A voiceprint recognition network lightweight method based on CKA feature cooperative distillation

Abstract

The application discloses a kind of voiceprint recognition network lightening method based on CKA feature cooperation distillation, belong to voiceprint recognition and depth learning model lightening technical field. Including: acquisition construction equipment sound signal, after being framed and windowed, frequency domain transformation and mel filter group extraction obtain log mel spectrum diagram as unified input feature;With pre-training high-precision voiceprint recognition network as teacher model, the intermediate layer features of it and student model are respectively subjected to layer-by-layer centralization kernel transformation, to obtain interlayer similarity matrix;According to the interlayer similarity matrix of teacher and student, the structural consistency coefficient of each layer feature is calculated, the feature layer needing to be aligned is screened and the feature structure constraint is applied, and the lightening student model training is completed.The scheme effectively maintains the discriminability of voiceprint feature structure while compressing the model size, achieving the balance between computational overhead and recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of voiceprint recognition and lightweight deep learning model technology, specifically a lightweight method for voiceprint recognition networks based on CKA feature co-distillation. Background Technology

[0002] Monitoring the operational status of construction machinery and equipment relies on voiceprint recognition technology to analyze the equipment's sound signals. Current high-precision voiceprint recognition networks are typically based on deep convolutional or Transformer architectures, resulting in a massive parameter scale that is difficult to deploy directly on edge monitoring nodes with limited computing resources. To meet the real-time inference requirements at the edge, the network needs to be lightweighted without significantly reducing recognition accuracy.

[0003] Existing lightweight network solutions mainly include channel pruning, low-rank decomposition, parameter quantization, and knowledge distillation. Knowledge distillation, which transfers teacher knowledge to a lightweight student network by having the student model fit the output distribution of the teacher model, is a common method for maintaining accuracy. However, traditional knowledge distillation methods primarily constrain the consistency between student output and teacher soft labels or intermediate layer pointwise features. This pointwise alignment strategy ignores the structural relationships within the feature space, making it difficult for the student model to reproduce the teacher's semantic organization of voiceprint features. Voiceprint signals are non-stationary and exhibit large intra-class variability; the correlation patterns between feature channels are crucial for distinguishing different device states. The lack of feature structure information blurs the class discrimination boundaries of the lightweight model. Furthermore, consistently selecting all intermediate layers for distillation introduces unnecessary alignment constraints. Some feature layers already have high similarity; forcing consistency on them may actually produce negative transfer, causing additional computational overhead.

[0004] Therefore, how to enable lightweight student models to effectively inherit the structured information of the intermediate layer features of the teacher network, and how to adaptively determine the feature layers that need to be aligned rather than full-layer distillation, have become urgent problems to be solved. Summary of the Invention

[0005] This paper proposes a method to achieve network lightweighting while ensuring the accuracy of voiceprint recognition. The method extracts the structural similarity of intermediate layer features between teacher and student models through a centered kernel transformation, and adaptively selects distillation layers based on the structural consistency coefficient to complete the training of a channel-adjustable student model.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a lightweight method for voiceprint recognition networks based on CKA feature co-distillation. For the voiceprint recognition task of construction equipment operation sound, a co-distillation strategy guided by structural consistency is used to significantly reduce the network computation while maintaining high-precision recognition capability.

[0007] This method performs frame-by-frame windowing processing on the collected sound signals from construction equipment to obtain a sound frame sequence. A frequency domain transformation is then performed on the sound frame sequence to obtain the frequency domain amplitude spectrum. The frequency band energy of the frequency domain amplitude spectrum is extracted using a Mel filter bank, and the logarithm is taken to generate a log-Mel spectrogram as a unified input feature. A pre-trained high-precision voiceprint recognition network is used as the teacher model. Using the log-Mel spectrogram as input, a layer-by-layer centering kernel transformation is performed on the selected intermediate layer output features of the teacher model. This includes flattening the feature map spatially and normalizing the column mean to obtain a centered feature matrix. Then, the intra-layer Gram matrix is ​​calculated and normalized to obtain the teacher inter-layer similarity matrix reflecting the structural relationships within the layer. Simultaneously, a student model with adjustable channel configuration parameters is constructed. The same centering kernel transformation is performed on the candidate intermediate layer features in the student model corresponding to the teacher's selected intermediate layer to obtain the student inter-layer similarity matrix.

[0008] Based on the similarity matrices between teacher and student layers, a structural consistency coefficient is calculated for each layer's features. Preferably, this coefficient is obtained by calculating the ratio of the inner product to the norm product of the corresponding submatrices, used to quantitatively measure the degree to which the student model preserves the feature structure of the current layer of the teacher model. Based on the comparison of each layer's structural consistency coefficient with a preset threshold, feature layers with insufficient structural consistency that require alignment are automatically selected as layers to be aligned. For each layer to be aligned, the difference matrix between the teacher and student submatrices after removing autocorrelation effects is calculated, and the feature structure difference of that layer is obtained through Frobenius norm operations. The feature structure differences of each layer to be aligned are weighted and summed to construct the total difference loss.

[0009] As a technical solution of this invention, the channel configuration parameters of the student model are iteratively updated with the goal of minimizing the total difference loss. During the update process, a set of candidate channel scaling ratios is set for each adjustable residual block in the student model. Under the constraint that the total computational cost does not exceed a preset budget threshold, the channel scaling ratio of each adjustable residual block is adjusted using gradient descent until the change in the total difference loss meets a preset convergence condition, thus obtaining the final channel configuration parameters. In this way, the student model can actively align with the intrinsic feature representation of the teacher model at the structural level during training, rather than merely imitating the final output. This allows it to maintain high discriminative power for construction equipment acoustic signatures even under conditions of significant channel compression, completing the training of a lightweight student model.

[0010] The technical effects and advantages provided by the present invention in the above technical solution are as follows: By processing the intermediate layer features of both the teacher and student models through a centralized kernel transformation, an inter-layer similarity matrix is ​​generated. This matrix describes the relationships between feature channels, rather than simply aligning individual feature values. This approach elevates the distillation objective from point-by-point matching to the level of internal feature structure distribution. Even with a significant reduction in the number of channels in the student model, the retained channels can still reproduce the activation patterns between channels in the teacher model. Addressing the complex spectral structure of construction equipment acoustic signals and the significant differences in energy distribution under different operating conditions, feature structure consistency constraints ensure that the lightweight student model maintains a manifold structure similar to the teacher model in the feature space, avoiding the loss of inter-category discriminative relationships by merely fitting numerical values. During the iteration process, the embedding vector output by the student model more closely resembles the teacher model's distribution under acoustic similarity measurement, maintaining a clear recognition boundary. This achieves recognition performance close to that of the teacher model while significantly reducing model size and computational cost.

[0011] The structural consistency coefficient of each layer's features is calculated based on the similarity matrices between teacher and student layers. This coefficient reflects the degree of matching between teachers and students in the feature structure of that layer. Layers with significant feature structure deviations are automatically identified as layers to be aligned, using a consistency coefficient below a preset threshold as a screening criterion. Feature structure difference loss is applied only to these layers, avoiding redundant constraints on layers with already high structural similarity. The difference matrix of the layers to be aligned is calculated and Frobenius norm operations are performed to obtain the difference quantity, which is then integrated into the total difference loss. With the goal of minimizing the total difference loss, the channel scaling ratio of each residual block in the student model is updated under computational budget constraints, completing the search for channel configurations. This adaptive selection of alignment layers reduces unnecessary distillation constraints, lowers the computational burden during training, and suppresses potential semantic negative transfer caused by forced alignment of high-level similar layers. The final lightweight model does not require manual specification of layer widths and can automatically obtain appropriate channel configurations under limited computational resources, enabling the model to balance inference speed and voiceprint recognition accuracy on edge devices. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0013] Figure 1 This is a flowchart of a lightweight method for voiceprint recognition networks based on CKA feature co-distillation; Figure 2 This is a flowchart of the preprocessing and log-Mel spectrum feature extraction of sound signals from construction equipment; Figure 3This is a flowchart of the feature structure difference loss calculation and model training process; Figure 4 It is a curve comparing the consistency coefficient of each layer of structure with the preset threshold value; Figure 5 This is the curve showing how the total difference loss changes with the number of training iterations; Figure 6 This is a curve showing the change in computational cost of the student model with training iterations, and a schematic diagram of the budget threshold. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] See Figure 1 This invention provides a lightweight method for voiceprint recognition networks based on CKA feature co-distillation, comprising: acquiring sound signals from construction equipment; performing frame-by-frame windowing processing on the sound signals to obtain a sound frame sequence; performing frequency domain transformation on the sound frame sequence to obtain a frequency domain amplitude spectrum; extracting the frequency band energy from the frequency domain amplitude spectrum using a Mel filter bank to obtain a log-Mel spectrogram, which serves as a unified input feature; using the log-Mel spectrogram as input and a pre-trained high-precision voiceprint recognition network as a teacher model, performing layer-by-layer centering kernel transformation on the intermediate layer features of the teacher model to obtain a teacher inter-layer similarity matrix; constructing a channel-adjustable student model, performing the same centering kernel transformation on the corresponding intermediate layer features of the student model to obtain a student inter-layer similarity matrix; calculating the structural consistency coefficient of each layer's features based on the teacher inter-layer similarity matrix and the student inter-layer similarity matrix; selecting feature layers that need alignment based on the structural consistency coefficient, and applying feature structure constraints to the selected feature layers to complete the training of the lightweight student model.

[0016] Example 1:

[0017] In specific implementation, please refer to Figure 2After acquiring the sound signal from the construction equipment, the sound signal is processed by framing and windowing. The frame length and frame shift are set; the frame length can be set to 25 milliseconds, and the frame shift to 10 milliseconds. The duration of the frame length and frame shift is determined based on the time-varying characteristics of the construction equipment sound signal. The construction equipment sound signal is segmented according to the frame length. Starting from the beginning of the sound signal, a segment is extracted at intervals of the frame length, resulting in multiple initial sound frames. Adjacent initial sound frames are overlapped according to the frame shift, so that the end of the previous initial sound frame overlaps with the beginning of the next initial sound frame. The length of the overlap region is equal to the frame length minus the frame shift. A Hanning window is applied to each overlapping sound frame. The Hanning window function expression is:

[0018] in, This indicates the Hanning window at the index position. Window function value at that point, This indicates the window length, which is the number of sampling points corresponding to the frame length. Represents pi (π). This represents the cosine function. Each sample point in the overlapping sound frames is multiplied by the corresponding window function value to obtain a windowed sound frame. All windowed sound frames are then arranged sequentially in chronological order to form a sound frame sequence.

[0019] A frequency domain transformation is performed on the audio frame sequence. A Discrete Fourier Transform (DFT) is applied to each audio frame in the sequence, converting the time-domain audio frame into a complex frequency spectrum. The DFT is calculated by performing point-by-point complex operations on the sampled points of each audio frame, generating a spectrum sequence containing real and imaginary components. The real and imaginary parts of the complex spectrum are extracted, and the sum of the squares of the real and imaginary parts is calculated. The square root of the sum of squares is then performed to obtain the amplitude value of each audio frame at each frequency point. All amplitude values ​​are arranged according to the chronological order of the frames and the frequency points, forming a two-dimensional frequency domain amplitude spectrum, where the rows correspond to the frame numbers and the columns correspond to the frequency point numbers.

[0020] Frequency band energy is extracted from the frequency domain amplitude spectrum using a Mel filter bank. A triangular Mel filter bank is constructed, consisting of multiple triangular filters, each with a center frequency. The center frequencies of adjacent triangular filters are evenly spaced on the Mel scale. The center frequency sequence of the triangular Mel filter bank is determined as follows: First, a Mel frequency range is defined. Based on the analysis requirements of the construction equipment's sound signals, the lowest Mel frequency corresponds to 0 Hz, and the highest Mel frequency corresponds to half the sampling rate in Hertz. The Mel frequency range is then linearly divided into several equal parts to obtain Mel domain division points. These Mel domain division points are then converted back to the Hertz domain using the following formula:

[0021] in, Represents frequency in the Hertz domain, with the unit being Hertz. The values ​​represent the Mel-domain frequencies in Mel units, with 700 and 2595 being the constant coefficients of the Mel scaling transform. The resulting Hertz-domain frequency points form the center frequency sequence of the triangular Mel filter bank.

[0022] Using the center frequency sequence as a reference, the frequency domain amplitude spectrum is divided into frequency bands. For each center frequency, the midpoint between the current center frequency and the previous center frequency is used as the starting frequency of the rising edge of the triangular filter, the current center frequency is used as the peak frequency of the triangular filter, and the midpoint between the current center frequency and the next center frequency is used as the ending frequency of the falling edge of the triangular filter, forming a triangular weighting coefficient. For the amplitude value sequences falling within each frequency band in the frequency domain amplitude spectrum, the weighting coefficients of the corresponding triangular filter are used for weighted summation to obtain the energy value of each frequency band. The natural logarithm of the energy value of each frequency band is taken to obtain the logarithmic energy value. The logarithmic energy values ​​of all frequency bands in all frames are arranged by frame and frequency band to obtain the log-Mel spectrogram. The log-Mel spectrogram serves as a unified input feature for subsequent processing of the teacher and student models.

[0023] Example 2:

[0024] In practice, the teacher model employs a pre-trained high-precision voiceprint recognition network. The core architecture of the teacher model comprises multiple cascaded convolutional blocks, each consisting of a convolutional layer, a batch normalization layer, and an activation function layer connected sequentially. The input to the teacher model is a log-Melogram, and the output is a voiceprint feature vector. Several intermediate layers are selected from all the convolutional blocks of the teacher model. These intermediate layers are distributed across the shallow, middle, and deep regions of the teacher model. The number of selected intermediate layers can be proportional to the total number of convolutional blocks in the teacher model; for example, half of all convolutional blocks can be selected as intermediate layers.

[0025] For each selected intermediate layer, the feature map output by that layer is extracted from the teacher model and used as the original teacher feature map. The original teacher feature map has three dimensions: channel dimension, height dimension, and width dimension. The original teacher feature map is then spatially flattened, merging the height and width dimensions into a single spatial dimension while keeping the channel dimension unchanged, resulting in a two-dimensional teacher feature matrix. The number of rows in the two-dimensional teacher feature matrix equals the number of channels in the original teacher feature map, and the number of columns equals the product of the height and width pixels in the original teacher feature map.

[0026] The two-dimensional feature matrix of teachers is mean-zeroed column-wise. The average of all elements in each column of the teacher's two-dimensional feature matrix is ​​calculated to obtain the column mean vector. The column mean of each element in the teacher's two-dimensional feature matrix is ​​subtracted from the column mean to obtain the teacher-centered feature matrix. The product of the teacher-centered feature matrix and its transpose is calculated; the result is a square matrix with the same number of rows and columns as the teacher-centered feature matrix. This product is used as the teacher-level Negram matrix. The [missing information - likely a specific matrix or structure] in the teacher-level Negram matrix... Line number The column elements represent the th element in the teacher-centered feature matrix. row and number Unnormalized similarity between rows, where and The index is from 1 to the channel number. Norm normalization is performed on the teacher layer Negram matrix using the Frobenius norm. The normalization method is as follows: take the square root of the sum of the squares of all elements in the teacher layer Negram matrix to obtain the normalized denominator value. Divide each element of the teacher layer Negram matrix by the teacher normalized denominator value; the resulting matrix is ​​the teacher layer similarity matrix.

[0027] The student model is a channel-adjustable model, with a structure corresponding to the teacher model. The student model contains the same number of adjustable residual blocks as the teacher model. Each adjustable residual block contains one standard convolutional path and one bottleneck convolutional path with an adjustable number of channels. The outputs of the two paths are concatenated by channel and then passed through an activation function. The input to the student model is the same as that of the teacher model, a log-Melogram. The output of the student model is a voiceprint feature vector. Candidate intermediate layers are set in the student model, corresponding to the selected intermediate layers. The positions of the candidate intermediate layers correspond one-to-one with the positions of the selected intermediate layers in the teacher model. For each candidate intermediate layer, the feature map output by the candidate intermediate layer is extracted as the student's original feature map. The student's original feature map has channel, height, and width dimensions. The student's original feature map is spatially flattened, merging the height and width dimensions into one spatial dimension, resulting in a two-dimensional feature matrix. The number of rows in the student's two-dimensional feature matrix equals the number of channels in the student's original feature map, and the number of columns equals the product of the height and width pixels in the student's original feature map.

[0028] The student two-dimensional feature matrix is ​​mean-zeroed column-wise. The average of all elements in each column of the student two-dimensional feature matrix is ​​calculated to obtain the student column mean vector. The student-centered feature matrix is ​​obtained by subtracting the student column mean from each element of the student two-dimensional feature matrix. The product of the student-centered feature matrix and its transpose is calculated; the product is a square matrix with the same number of rows and columns as the student-centered feature matrix. This product is used as the student-layer Negram matrix. Line number The column elements represent the student-centered feature matrix. row and number Unnormalized similarity between rows, where and Let be the index from 1 to the row number of the student-centered feature matrix. Norm normalization is performed on the student-layer Negram matrix using the Frobenius norm corresponding to the student-layer Negram matrix. The normalization method is as follows: take the square root of the sum of the squares of all elements in the student-layer Negram matrix to obtain the normalized denominator value. Divide each element of the student-layer Negram matrix by the student normalized denominator value; the resulting matrix is ​​the student-layer inter-similarity matrix.

[0029] Example 3:

[0030] In practice, the teacher inter-layer similarity matrix is ​​a three-dimensional data structure composed of the similarity matrices of all selected intermediate layers in the teacher model. The three dimensions of this three-dimensional data structure are denoted as row dimension, column dimension, and layer dimension. The row and column dimensions represent the similarity relationships between feature channels of the selected intermediate layers, while the layer dimension corresponds to different selected intermediate layers. The depth of the layer dimension in the teacher inter-layer similarity matrix is ​​equal to the total number of selected intermediate layers.

[0031] For a certain age index, it is marked as To obtain the teacher submatrix corresponding to the current layer in the teacher inter-layer similarity matrix, the method is as follows: extract the index along the layer dimension of the teacher inter-layer similarity matrix. The corresponding two-dimensional slice yields a size of OK A two-dimensional matrix of columns, serving as the teacher submatrix. Wherein, Let represent the order of the teacher submatrix, where the number of rows and columns of the teacher submatrix are equal. , The value is equal to the first in the teacher model. The number of output channels for a selected intermediate layer.

[0032] The student inter-layer similarity matrix is ​​a three-dimensional data structure composed of the similarity matrices of all candidate intermediate layers in the student model. The dimensional organization of this three-dimensional data structure is the same as that of the teacher inter-layer similarity matrix, containing row, column, and layer dimensions. The method for obtaining the student submatrix corresponding to the current layer in the student inter-layer similarity matrix is ​​as follows: extract the index along the layer dimension of the student inter-layer similarity matrix. The corresponding two-dimensional slice yields a size of OK A two-dimensional matrix of columns, used as the student submatrix. Wherein, Let represent the order of the student submatrix, where the number of rows and columns of the student submatrix are equal. , The value is equal to the first in the student model. The student model employs a channel-adjustable mechanism to ensure that the number of output channels for each candidate intermediate layer is equal to the number of output channels for the corresponding intermediate layer selected by the teacher model. and They have the same value.

[0033] After obtaining the teacher and student submatrices, calculate the inner product of the teacher and student submatrices. The specific calculation method for the inner product is as follows: Extract the row indices of the teacher submatrices one by one... Column index is element value and the index of the same row in the student submatrix Indexes on the same column element value ,Will and Multiply to obtain a single-point product. Index all rows. From 1 to And all column indexes From 1 to The sum of the products of the individual points is the inner product value.

[0034] Calculate the Frobenius norm of the teacher submatrix. The Frobenius norm of the teacher submatrix is ​​calculated as follows: for each element in the teacher submatrix... Perform squaring operations to obtain all squared values, then sum all the squared values, and finally take the square root of the sum. The result is the Frobenius norm of the teacher submatrix.

[0035] Calculate the Frobenius norm of the student submatrix. The Frobenius norm of the student submatrix is ​​calculated as follows: for each element in the student submatrix... Perform squaring operations to obtain all squared values, then sum all the squared values, and finally take the square root of the sum. The result is the Frobenius norm of the student submatrix.

[0036] The product of the Frobenius norm of the teacher submatrix and the Frobenius norm of the student submatrix is ​​the norm product.

[0037] Divide the inner product value by the norm product, and the quotient is used as the current level. The structural consistency coefficient of a layer. The formula for calculating the structural consistency coefficient is as follows:

[0038] in, Indicates the first The structural consistency coefficient of layer features The index is the sequence number of the current layer. The value range is from 1 to the total number of layers in the selected intermediate layer; The row number in the teacher submatrix is... , column number is The element value; The row number in the student submatrix is , column number is The element value; Represents the row index variable. The value ranges from 1 to ; Indicates column index variable, The value ranges from 1 to ; Let represent the order of the teacher submatrix, where the number of rows and columns of the teacher submatrix is ​​1. Molecular part Represents the inner product value; the first square root factor in the denominator. The Frobenius norm of the teacher submatrix and its second square root factor. Let Frobenius norm denote the student submatrix, and let their product denote the norm product.

[0039] See Figure 4 The horizontal axis in the figure represents the layer index. The value ranges from 1 to 500, and the vertical axis represents the structural consistency coefficient, ranging from 0 to 1. The blue solid line curve reflects the changing trend of the structural consistency coefficient of the teacher model and the student model at each layer, while the red dashed line represents the preset threshold value of 0.5. Overall, the structural consistency coefficient shows a fluctuating state, with the coefficient values ​​of most layers concentrated between 0.4 and 0.9, indicating that the feature structure of the teacher and student models has a high degree of consistency at most layers.

[0040] Specifically, the curve shows peaks in several intervals, with structural consistency coefficients exceeding 0.8, indicating a high degree of similarity in the characteristic structures of teachers and students in the corresponding layers. Conversely, significant troughs appear in other intervals, with the lowest value approaching 0.2, indicating substantial differences in the characteristic structures of teachers and students in these layers. The structural consistency coefficient repeatedly crosses the preset threshold of 0.5, reflecting that in approximately several hundred layers, the characteristic structure consistency of multiple layers is below the threshold.

[0041] Example 4:

[0042] In specific implementation, please refer to Figure 3 The structural consistency coefficient measures the similarity of the feature structures of the teacher model and the student model at a certain intermediate layer. The value of the structural consistency coefficient ranges from 0 to 1, with a value closer to 1 indicating a more consistent feature structure. The preset threshold can be set to 0.5. The reason for setting the preset threshold to 0.5 is that when the structural consistency coefficient is lower than 0.5, it indicates that the difference in feature structures between the teacher model and the student model at the corresponding layer has exceeded the similarity level, and that layer needs to be included in the alignment constraint.

[0043] The structural consistency coefficient of each layer is compared with a preset threshold value one by one, starting from the first layer and proceeding sequentially to the last. For a given layer, if its structural consistency coefficient is less than the preset threshold value, the layer is selected as a layer to be aligned; otherwise, it is not selected. All selected layers to be aligned form a set of layers to be aligned, and the layer numbers in this set are a subset of the original set of layer numbers.

[0044] For each layer in the set to be aligned, obtain the corresponding teacher submatrix. The teacher submatrix is ​​a two-dimensional slice matrix of the inter-layer similarity matrix of the teacher model, where the number of rows and columns equals the number of output channels of the layer to be aligned in the teacher model. The diagonal elements of the teacher submatrix represent the self-similarity of features within the same channel of the teacher model, while the off-diagonal elements represent the mutual similarity between features from different channels. The teacher submatrix of the layer to be aligned is processed by setting the diagonal elements to zero. This is done by replacing all elements in the teacher submatrix whose row index equals their column index with 0, while keeping the off-diagonal elements unchanged. The resulting matrix is ​​the teacher de-autocorrelation submatrix.

[0045] Simultaneously, obtain the student submatrix corresponding to the layer to be aligned. The student submatrix is ​​a two-dimensional slice matrix corresponding to the layer to be aligned in the student inter-layer similarity matrix. The number of rows and columns of the student submatrix is ​​equal to the number of output channels of the layer to be aligned in the student model. The main diagonal elements of the student submatrix of the layer to be aligned are set to zero. The processing method is as follows: replace the values ​​of all elements in the student submatrix whose row index equals the column index with 0, keeping the values ​​of the non-main diagonal elements unchanged. The resulting matrix is ​​the student de-autocorrelation submatrix.

[0046] Subtract each element in the teacher's deautocorrelation submatrix from the corresponding element in the student's deautocorrelation submatrix, which has the same row and column indices, to obtain the difference matrix elements. The values ​​of the difference matrix elements can be positive, negative, or zero. Arrange all the difference matrix elements according to their original row and column indices to form the difference matrix, which has the same number of rows and columns as both the teacher's and student's deautocorrelation submatrixes.

[0047] The difference matrix is ​​subjected to Frobenius norm operation. The Frobenius norm is defined as the square root of the sum of the squares of all elements in the matrix. The calculation method is as follows: squaring each element in the difference matrix, summing all the squares, and then taking the square root of the sum. The resulting value is the feature structure difference of the layer to be aligned.

[0048] The feature structure difference reflects the degree of difference in the inter-channel relationship structure between the teacher model and the student model at the layer to be aligned, after ignoring autocorrelation components. The feature structure difference of each layer in the set of layers to be aligned is weighted and summed, with the weighting coefficients set according to the depth of the layer in the network. The weighting coefficients for shallower layers are greater than those for deeper layers, emphasizing the alignment of shallower feature structures. The weighted sum is the total difference loss. The formula for calculating the total difference loss is as follows:

[0049] in, Indicates the total difference loss; This indicates the total number of layers to be aligned. Integers greater than or equal to 0; Indicates the index number of the layer to be aligned. The value ranges from 1 to ; Indicates the first The weighting coefficients corresponding to each layer to be aligned The value is determined based on the position of the layer to be aligned in the model. For shallow feature layers, The value can be set to a number between 1.0 and 2.0. For deep feature layers, The value can be set to a number between 0.5 and 1.0; Indicates the first The autocorrelation submatrix of the teachers in the layer to be aligned; Indicates the first The autocorrelation submatrix of the students in the layer to be aligned; Represents the Frobenius norm operation. Indicates the first The feature structure difference of each layer to be aligned.

[0050] With the goal of minimizing the total difference loss, in each training iteration, the gradient of the total difference loss with respect to each adjustable parameter in the student model's channel configuration parameters is calculated, and the gradient is propagated to the student model's channel configuration parameters via backpropagation. The channel configuration parameters include the channel scaling ratio of the bottleneck convolutional path within each adjustable residual block in the student model. Based on the gradient descent method, the channel scaling ratio is updated along the negative gradient direction, with the update step size controlled by a pre-set learning rate, which can be set to 0.001. As the total difference loss gradually decreases, the feature structure of the student model gradually approximates the feature structure of the teacher model.

[0051] See Figure 5 In the graph, the horizontal axis represents the number of training iterations, ranging from 0 to 500, and the vertical axis represents the total difference loss, using a logarithmic scale with values ​​ranging from approximately 1 to 1 × 10⁻⁶. -4 The blue solid line represents the trend of total difference loss as the number of training iterations increases; the red dashed line represents the preset convergence threshold, which has a value of 1×10. -4 .

[0052] As shown in the figure, the total difference loss generally decreases with training iterations, with a faster decrease in the initial stage (approximately the first 180 iterations), where the total difference loss rapidly decreases from close to 1 to approximately 10. -2 The order of magnitude. Afterward, as training progresses, the total difference loss continues to decrease slowly, but in the mid-to-late stages (around 180 to 300 iterations), fluctuations increase, and the value gradually approaches the convergence threshold. After more than 300 training iterations, the total difference loss tends to stabilize, showing a convergence trend.

[0053] As described in Example 4, when the change in total difference loss is less than a preset convergence threshold of 1×10⁻⁶ -4 At this point, the training process stops. The red dashed line in the figure clearly marks this convergence threshold, indicating that the training process has reached this threshold within approximately 300 to 500 iterations, meeting the preset stopping condition.

[0054] Example 5:

[0055] In practice, the student model consists of multiple sequentially connected adjustable residual blocks. Each adjustable residual block contains a standard convolutional path and a bottleneck convolutional path. The standard convolutional path contains a 3×3 standard convolutional layer, while the bottleneck convolutional path consists of a 1×1 point convolutional layer, a 3×3 depthwise separable convolutional layer, and another 1×1 point convolutional layer stacked sequentially. The output feature maps of the standard and bottleneck convolutional paths are concatenated along the channel dimension. The concatenated feature map is then processed by a batch normalization layer and a modified linear unit activation function layer to obtain the final output of the adjustable residual block. The input to the student model is a log-Melogram, and the output is a voiceprint feature vector processed by a global average pooling layer and a fully connected layer.

[0056] For each adjustable residual block in the student model, a set of candidate channel scaling ratios is defined, and the candidate channel scaling ratio set is: The rationale for setting the candidate channel scaling set to these discrete values ​​is that discretization at intervals of 0.2 can cover a uniform span from low to high computational cost, facilitating the selection of channel configurations under different computational budget conditions.

[0057] Each adjustable residual block has an independent channel scaling parameter, which represents the scaling factor of the number of channels in the bottleneck convolutional path relative to the baseline number of channels. Before model training, the channel scaling parameter of all adjustable residual blocks is initialized to the maximum value of 1.0 from the candidate channel scaling parameter set. The value of the channel scaling parameter is a continuous real number between 0 and 1.

[0058] Based on the channel scaling parameter of each adjustable residual block, the computational increment for that adjustable residual block is calculated. The computational increment represents the increase in the number of floating-point operations performed by the adjustable residual block when using the current channel scaling ratio, compared to the number of floating-point operations without using the bottleneck convolution path. The formula for calculating the computational increment of the adjustable residual block is expressed as follows:

[0059] in, Indicates the first The computational increment of an adjustable residual block, in units of floating-point operations, i.e., FLOPs; This represents the index of the adjustable residual block in the student model. The value ranges from 1 to , This represents the total number of adjustable residual blocks in the student model; Indicates the first The channel scaling parameter for each adjustable residual block. The initial value is 1.0, and it is updated in a continuous interval between 0 and 1 during training by gradient descent; Indicates the first The baseline computational cost of the bottleneck convolutional path in an adjustable residual block when the channel scaling parameter is 1.0, in FLOPs. It can be obtained in advance by calculating the sum of the floating-point operations of each convolution operation in the bottleneck convolution path.

[0060] The total computational cost of the student model consists of the incremental computational cost of all adjustable residual blocks and the base computational cost of the fixed part of the model. The fixed part of the model includes standard convolutional paths, global average pooling layers, and fully connected layers, and the base computational cost of the fixed part of the model is a fixed value.

[0061] The preset budget threshold represents the upper limit of the total computational load allowed for the student model. This threshold is set to 0.5 times the total computational load of the teacher model, designed to ensure that the computational load of the lightweight student model does not exceed half that of the teacher model, thus meeting deployment requirements in environments with limited computing resources. The total computational load of the teacher model is a known fixed value, obtained by counting the number of floating-point operations at each layer of the teacher model.

[0062] In a single training iteration, with the objective of minimizing the total difference loss, and under the condition that the total computational cost does not exceed a preset budget threshold, gradient descent is used to update the channel scaling parameters of each adjustable residual block in the student model. The gradient descent update employs a stochastic gradient descent optimizer with a learning rate set to 0.01. The total difference loss is calculated for each channel scaling parameter. The partial derivatives are obtained. The corresponding gradient value. Along the negative gradient direction. The update is performed, and the update magnitude is determined by the product of the learning rate and the gradient value.

[0063] After each gradient update of the channel scaling parameter, check whether the total computation of the current student model exceeds the preset budget threshold. If the total computation exceeds the preset budget threshold, reduce the channel scaling parameter of all adjustable residual blocks proportionally so that the reduced total computation equals the preset budget threshold.

[0064] The process of gradient descent updates and total computation constraint checks is performed iteratively. After each iteration, the absolute value of the difference between the total difference loss of the current iteration and the total difference loss of the previous iteration is calculated as the change in total difference loss. Iteration stops when the change in total difference loss is less than a preset convergence threshold. The preset convergence threshold is set to... The setting is based on the fact that when the change in total variance loss is less than When the iteration stops, it indicates that the improvement in feature structure alignment from updating the channel scaling parameters has reached saturation, and the benefit of continuing the iteration is minimal. After the iteration stops, the channel scaling parameters of each adjustable residual block obtained are the final channel configuration parameters.

[0065] See Figure 6 In the graph, the horizontal axis represents the number of training iterations, ranging from 0 to 500, and the vertical axis represents the total computational cost of the model, measured in floating-point operations (FLOPs). The solid curve represents the total computational cost of the student model, while the dashed red horizontal line represents the preset budget threshold, which is 50% of the total computational cost of the teacher model. The curve's trend can be divided into three distinct stages: During iterations 0 to 40, the total computational cost of the student model decreased rapidly, gradually dropping from approximately 150 FLOPs to close to 100 FLOPs. This indicates that the channel scaling parameter was quickly adjusted via gradient descent, significantly reducing the number of channels in the bottleneck convolutional pathway and thus lowering computational cost. From iterations 40 to 230, the total computational cost of the student model surged to approximately 160 FLOPs, exceeding the preset budget threshold, indicating that budget constraints were not considered during the update process, resulting in a peak in computational cost. From iterations 240 to 500, the total computational cost of the student model remained stable near the preset budget threshold of 100 FLOPs, with minimal fluctuations. This suggests that the channel scaling parameter was adjusted proportionally, ensuring that the total computational cost of the student model was strictly controlled within the budget threshold.

[0066] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A lightweight method for voiceprint recognition networks based on CKA feature co-distillation, characterized in that, The method includes: The sound signals of the construction equipment are collected, and the sound signals of the construction equipment are processed by frame segmentation and windowing to obtain a sound frame sequence; The frequency domain transform of the sound frame sequence is performed to obtain the frequency domain amplitude spectrum; The frequency domain amplitude spectrum is subjected to band energy extraction by a Mel filter bank to obtain a log-Mel spectrum, which is used as a unified input feature. Using the log-Mel spectrogram as input and a pre-trained high-precision voiceprint recognition network as the teacher model, the intermediate layer features of the teacher model are subjected to layer-by-layer centering kernel transformation to obtain the teacher inter-layer similarity matrix. Construct a student model with adjustable channels, and perform the same centering kernel transformation on the corresponding intermediate layer features of the student model to obtain the student inter-layer similarity matrix; Based on the teacher inter-layer similarity matrix and the student inter-layer similarity matrix, calculate the structural consistency coefficient of each layer's features; Based on the structural consistency coefficient, the feature layers that need to be aligned are selected, and feature structure constraints are applied to the selected feature layers to complete the training of the lightweight student model.

2. The lightweight method for voiceprint recognition networks based on CKA feature co-distillation according to claim 1, characterized in that, The step of performing frame-segmentation and windowing processing on the sound signal of the construction equipment to obtain a sound frame sequence specifically includes: Set the frame length and frame shift, and segment the sound signal of the construction equipment according to the frame length to obtain the initial sound frame; The adjacent initial sound frames are overlapped according to the frame shift to obtain overlapping sound frames; Apply a Hanning window to the overlapping audio frames to obtain windowed audio frames; All the windowed audio frames are arranged in chronological order to obtain the audio frame sequence.

3. The lightweight method for voiceprint recognition networks based on CKA feature co-distillation according to claim 2, characterized in that, The step of performing a frequency domain transformation on the sound frame sequence to obtain the frequency domain amplitude spectrum specifically includes: Perform a discrete Fourier transform on each sound frame in the sound frame sequence to obtain the complex spectrum corresponding to each sound frame; Extract the real and imaginary parts of the complex number's spectrum, and calculate the sum of the squares of the real and imaginary parts; The square root of the sum of the squares is taken to obtain the amplitude value of each audio frame; All the amplitude values ​​are arranged in frame order to form the frequency domain amplitude spectrum.

4. The lightweight method for voiceprint recognition networks based on CKA feature co-distillation according to claim 3, characterized in that, The step of extracting the bandgap energy from the frequency domain amplitude spectrum using a Mel filter bank to obtain a log-Melb spectrum, which serves as a unified input feature, specifically includes: Set up a triangular Mel filter bank and determine the center frequency sequence of the triangular Mel filter bank; Using the center frequency sequence as a reference, the frequency domain amplitude spectrum is divided into frequency bands to obtain the amplitude value sequence corresponding to each frequency band; The energy value of each frequency band is obtained by summing the amplitude value sequences corresponding to each frequency band. The logarithm of the energy values ​​of each frequency band is taken to obtain the logarithmic Mel spectrum.

5. A lightweight method for voiceprint recognition networks based on CKA feature co-distillation according to claim 1, characterized in that, The process involves using the log-Mel spectrum as input and a pre-trained high-precision voiceprint recognition network as the teacher model. The intermediate layer features of the teacher model undergo a layer-by-layer centering kernel transformation to obtain the teacher inter-layer similarity matrix. Specifically, this includes: Extract the output feature map of the selected intermediate layer from the teacher model, and use it as the original feature map of the teacher. The original teacher feature map is flattened in terms of spatial dimensions to obtain a two-dimensional teacher feature matrix. The mean of the teacher's two-dimensional feature matrix is ​​zeroed out column by column to obtain the teacher-centered feature matrix. Calculate the product of the teacher-centered feature matrix and its transpose to obtain the teacher-layer innergram matrix; The teacher layer inner Gram matrix is ​​normalized to obtain the teacher layer inter-layer similarity matrix.

6. A lightweight method for voiceprint recognition networks based on CKA feature co-distillation according to claim 5, characterized in that, The construction of the channel-adjustable student model involves performing a common centering kernel transformation on the corresponding intermediate layer features of the student model to obtain the student inter-layer similarity matrix, specifically including: In the student model, a candidate intermediate layer corresponding to the selected intermediate layer is set, and the output feature map of the candidate intermediate layer is extracted as the original feature map of the student. The original student feature map is flattened in terms of spatial dimensions to obtain a two-dimensional feature matrix of the student. The mean of the student two-dimensional feature matrix is ​​zeroed out column by column to obtain the student centered feature matrix; Calculate the product of the student-centered feature matrix and its transpose to obtain the student-layer innergram matrix; The student layer inner gram matrix is ​​normalized by norm to obtain the student layer inter-layer similarity matrix.

7. A lightweight method for voiceprint recognition networks based on CKA feature co-distillation according to claim 6, characterized in that, The step of calculating the structural consistency coefficient of each layer's features based on the teacher inter-layer similarity matrix and the student inter-layer similarity matrix specifically includes: Obtain the teacher submatrix corresponding to the current layer from the teacher inter-layer similarity matrix; Obtain the student submatrix corresponding to the current layer from the student inter-layer similarity matrix; Calculate the inner product of the teacher submatrix and the student submatrix to obtain the inner product value; Calculate the product of the norm of the teacher submatrix and the norm of the student submatrix to obtain the norm product; Divide the inner product value by the norm product to obtain the structural consistency coefficient of the current layer.

8. A lightweight method for voiceprint recognition networks based on CKA feature co-distillation according to claim 7, characterized in that, The step of selecting feature layers that need alignment based on the structural consistency coefficient and applying feature structure constraints to the selected feature layers to complete the training of the lightweight student model specifically includes: The structural consistency coefficient of each layer is compared with a preset threshold value, and the layer whose structural consistency coefficient is less than the preset threshold value is selected as the layer to be aligned. For each layer to be aligned, calculate the difference matrix between the teacher submatrix and the student submatrix of that layer; Perform Frobenius norm operation on the difference matrix to obtain the feature structure difference of the layer; The total difference loss is obtained by weighted summing of the feature structure differences of each of the layers to be aligned. With the goal of minimizing the total difference loss, the channel configuration parameters of the student model are updated.

9. A lightweight method for voiceprint recognition networks based on CKA feature co-distillation according to claim 8, characterized in that, For each layer to be aligned, calculating the difference matrix between the teacher submatrix and the student submatrix of that layer specifically includes: The teacher submatrix of the layer to be aligned is processed by setting the main diagonal elements to zero to obtain the teacher deautocorrelation submatrix; The student submatrix of the layer to be aligned is processed by setting the main diagonal elements to zero to obtain the student deautocorrelation submatrix; Subtract each element in the teacher's deautocorrelation submatrix from the corresponding element in the student's deautocorrelation submatrix to obtain the difference matrix elements; Arrange all the elements of the difference matrix in order of position to form the difference matrix.

10. A lightweight method for voiceprint recognition networks based on CKA feature co-distillation according to claim 8, characterized in that, The step of updating the channel configuration parameters of the student model with the objective of minimizing the total difference loss specifically includes: For each adjustable residual block in the student model, a set of candidate channel scaling ratios is set; The computational increment of the adjustable residual block is calculated based on each candidate scaling factor in the candidate channel scaling factor set. Under the condition that the total computational cost does not exceed the preset budget threshold, gradient descent is applied to the total difference loss to update the channel scaling ratio of each adjustable residual block; The update process is executed iteratively until the change in the total difference loss is less than a preset convergence threshold, and the final channel configuration parameters are obtained.

Images