Masking audio Transform model based on MATT-SED for multi-cabin transformer substation

Abstract

The invention discloses a masking audio Transform model based on MATT-SED (Matrix Asynchronous Transfer-Surface Event Description) and used for a multi-cabin transformer substation. The MATT-SED-based masking audio Transform model for the multi-cabin transformer substation comprises the following steps that an initial transformer model is constructed, and the initial transformer model comprises an encoder network and a context network; pre-training the initial transformer model based on masking reconstruction to generate a reconstruction sequence; performing fine tuning on the reconstruction sequence based on a preset fine tuning strategy to obtain a masking audio transformer model based on MATT-SED; and carrying out quality evaluation on the masking audio transformer model based on the MATT-SED. Through the MATT-SED-based masking audio Transform model for the multi-cabin transformer substation, the detection accuracy of the health condition of the multi-cabin equipment based on the voiceprint signal is improved, and the problem that the detection accuracy is low when the health condition of the multi-cabin equipment is detected in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of cabin-type power transformation technology, and in particular to a masked audio transformer model based on MAT-SED for a multi-cabin substation. Background Art

[0002] The internal equipment layout of the existing multi-cabin substation system is compact and the equipment is in a closed space, which causes the operating voiceprint signals between different types of equipment to interfere with each other. In addition, the voiceprint signals of the equipment are reflected between the cabin and the equipment, making it difficult to accurately detect the voiceprint information. As a result, when the health status of the multi-cabin equipment is detected based on the voiceprint signal, the detection accuracy is low.

[0003] Based on the above, in order to improve the detection accuracy of the health status of multi-cabin equipment based on voiceprint signals, the present application proposes a MAT-SED-based masked audio Transformer model for a multi-cabin substation. Summary of the invention

[0004] The embodiments of the present application provide a MAT-SED-based masked audio Transformer model for a multi-cabin substation, thereby solving the problem in the prior art that the detection accuracy rate is low when detecting the health status of multi-cabin equipment based on voiceprint signals.

[0005] An embodiment of the present application provides a masked audio Transformer model based on MAT-SED for a multi-cabin substation, comprising the following steps: constructing an initial transformer model, wherein the initial transformer model includes an encoder network and a context network; pre-training the initial transformer model based on masked reconstruction to generate a reconstruction sequence; fine-tuning the reconstruction sequence based on a preset fine-tuning strategy to obtain a masked audio transformer model based on MAT-SED; and performing quality evaluation on the masked audio transformer model based on MAT-SED based on a mean square error loss function.

[0006] Furthermore, the encoder network specifically includes: obtaining a mel spectrogram and dividing the mel spectrogram into a plurality of patch blocks of preset sizes; projecting the patch blocks into an embedding sequence in a linear relationship, and compressing the size of the spectrogram according to a PaSST mechanism to obtain a potential feature sequence.

[0007] Furthermore, the context network specifically includes: indicating the relationship between different position sequences in the potential feature sequence according to the multi-head attention mechanism, thereby obtaining the position information of the learnable potential feature sequence; and simultaneously detecting the position of the same feature at different times, thereby capturing the time correlation of the potential feature sequence.

[0008] Furthermore, the pre-training of the initial transformer model based on masked reconstruction specifically includes: freezing the weights of the initial transformer model, and randomly extracting a part of the frames in the potential feature sequence, hiding and covering the extracted frames, and marking them as mask frames to obtain a mask frame sequence; using a block masking strategy to divide the obtained mask frame sequence into a number of sequence blocks of preset sizes, and randomly masking a preset number of sequence blocks; reconstructing the sequence blocks after randomly masking the preset number to generate a reconstructed sequence.

[0009] Furthermore, the fine-tuning strategy includes a global-local feature fusion strategy, specifically including: the global-local feature fusion strategy uses two branches to extract different features from the spectrogram; the global branch inputs the reconstructed sequence into the encoder network to generate a global feature sequence; the local branch divides the reconstructed sequence into multiple overlapping blocks in the time dimension through a sliding window, and then independently inputs each block into the encoder network to generate a local feature sequence; the global feature sequence and the local feature sequence are linearly fused to obtain a fused feature sequence.

[0010] Furthermore, the masked audio transformer model based on MAT-SED is specifically as follows: a linear softmax is introduced to perform pooling processing on the detected latent feature sequence in the time dimension to obtain a clip-level prediction result; and a semi-supervised learning is performed on the clip-level prediction result through the average teacher algorithm to obtain a masked audio transformer model based on MAT-SED.

[0011] Furthermore, the method for obtaining the fused feature sequence is as follows: dividing the spectrogram into several overlapping blocks through a sliding time window; independently inputting each block into the encoder network for feature extraction; aggregating features of different blocks to obtain a local feature sequence; for blocks that overlap in time, averaging the features extracted within the overlapping duration to obtain a fused feature sequence.

[0012] Furthermore, the size of the spectrogram is compressed according to the PaSST mechanism to obtain a potential feature sequence, specifically: the expected output size of the spectrogram is obtained, and the feature points of the spectrogram are screened out by weighted averaging at non-uniform intervals in combination with a preset jump step size for compression to obtain a potential feature sequence.

[0013] Furthermore, a device using a masked audio Transformer model based on MAT-SED in a multi-cabin substation as described in any one of claims 1 to 9 is characterized in that it includes a transformer module, a training module, a task perception module and an evaluation module, and there is a connection between the modules; the transformer module: used to construct an initial transformer model, the initial transformer model includes an encoder network and a context network; the training module: pre-trains the initial transformer model based on masked reconstruction to generate a reconstruction sequence; the task perception module: fine-tunes the reconstruction sequence based on a preset fine-tuning strategy to obtain a masked audio transformer model based on MAT-SED; the evaluation module: performs quality evaluation on the masked audio transformer model based on MAT-SED based on the mean square error loss function.

[0014] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. Through the MAT-SED-based masked audio Transformer model used in multi-cabin substations, the detection accuracy of the health status of multi-cabin equipment based on voiceprint signals is improved, which effectively solves the problem of low detection accuracy when detecting the health status of multi-cabin equipment based on voiceprint signals in the existing technology.

[0015] 2. By constructing the context network in the MAT-SED structure based on the Transformer model with relative position encoding, the model is updated in a semi-supervised manner, thereby effectively improving the detection performance of sound events and better eliminating the impact of noise.

[0016] 3. Through relative position encoding, it can adapt to the changes in the temporal relationship between different frames, thereby ensuring that the model can accurately detect the same type of sound events at different time points.

[0017] 4. Through the global-local feature fusion strategy, the advantages of global and local features can be comprehensively utilized, thereby improving the ability to model the sound effects of events of different durations. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart during pre-training of a MAT-SED-based masked audio Transformer model for a multi-cabin substation provided in an embodiment of the present application; Figure 2 A schematic diagram of a model of a fine-tuning stage of a MAT-SED-based masked audio Transformer model for a multi-cabin substation provided in an embodiment of the present application; Figure 3A schematic diagram of a global-local feature fusion strategy of a masked audio Transformer model system based on MAT-SED for a multi-cabin substation provided in an embodiment of the present application; Figure 4 A comparison chart of the performance of MAT-SED of the MAT-SED-based masked audio transformer model system for a multi-cabin substation provided in an embodiment of the present application and other SED systems on the DCASE2023 dataset; Figure 5 A contextual network ablation study diagram of a MAT-SED-based masked audio Transformer model system for a multi-cabin substation provided in an embodiment of the present application; Figure 6 A comparison chart of the convergence curve of the masked audio Transformer model system based on MAT-SED for a multi-cabin substation provided in an embodiment of the present application, the curve of training MAT-SED from scratch and the end-to-end fine-tuning after masked reconstruction pre-training; Figure 7 A comparison diagram of masked reconstruction pre-training of a MAT-SED-based masked audio Transformer model system for a multi-cabin substation provided in an embodiment of the present application; Figure 8 The PSDS1 score of the masked audio Transformer model system based on MAT-SED for a multi-cabin substation provided in an embodiment of the present application when λ is set to 0, 0.5 and 1; DETAILED DESCRIPTION The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0020] The masked audio transformer model based on MAT-SED for multi-cabin substation is characterized by comprising the following steps: Constructing an initial transformer model, wherein the initial transformer model includes an encoder network and a context network; Pre-train the initial transformer model based on masked reconstruction to generate a reconstruction sequence; The reconstructed sequence is fine-tuned based on a preset fine-tuning strategy to obtain a masked audio transformer model based on MAT-SED; Quality evaluation of the MAT-SED based masked audio transformer model is performed based on the mean squared error loss function.

[0021] Model construction: The overall structure of MAT-SED includes an encoder network and a context network; The encoder network is used to extract features and output a latent feature sequence from the mel spectrogram, and the context network is responsible for capturing the temporal correlation between latent features; The encoder network of MAT-SED is based on PaSST, a large pre-trained Transformer model for audio tagging; each mel spectrogram is divided into several 16×16 patches, which are then linearly projected into an embedding sequence; this sequence passes through 10 layers of PaSST block transformers; according to PaSST, the frequency scale is compressed by average pooling and linearly upsampled by 10 times to recover the success in the patching process; A 3-layer Transformer block is used to form the context network, given the key requirement of positioning in the SED task; relative position encoding is used, where the learnable position encoding is determined by the relative position between frames, that is, for a given sound event, the translation is equivariant in the time dimension, that is, when the sound event changes in the temporal audio signal, the same features will be detected at the new time.

[0022] It should be noted that in the Transformer model, since it cannot process the absolute position information of the input sequence, it is necessary to introduce position encoding. The Transformer is a neural network architecture based on the self-attention mechanism and is widely used in sequence modeling tasks.

[0023] In the speech event detection task, a context network is constructed to encode and process the input speech features to capture the contextual information of the speech events. Each block includes a self-attention layer and a feedforward neural network layer. The use of a three-layer Transformer block can balance the model complexity and computational efficiency, while being able to effectively learn the complex features and relationships in the input sequence.

[0024] Relative position encoding takes into account the relative position relationship between frames, that is, it not only tells the model the position of a feature in the sequence, but also tells it the relative distance from other features. This encoding method is particularly useful for speech event detection tasks, because the temporal characteristics of sound events change in the sequence, and relative position encoding can help the model understand speech features at different time points.

[0025] The encoder network is based on PaSST and specifically includes: Divide the mel spectrogram into several patches of preset sizes; The obtained plaque patches are projected into the embedding sequence in a linear relationship, and the size of the spectrogram is compressed according to the PaSST mechanism to obtain the latent feature sequence.

[0026] The size of the spectrogram is compressed according to the PaSST mechanism to obtain a potential feature sequence, specifically: The expected output size of the spectrogram is obtained, and the characteristic points of the spectrogram are screened out by weighted averaging at non-uniform intervals in combination with the preset jump step size for compression to obtain a potential feature sequence.

[0027] Pre-training based on masked reconstruction, including: First, the encoder network is initialized using the PaSST model pre-trained on AudioSet and its weights are frozen to focus on the pre-trained context network. A mask reconstruction task is further designed as a mask task, and a mask language model is trained. A part of the frames in the latent feature sequence are hidden and covered and marked as mask frames to obtain a mask frame sequence.

[0028] It should be noted that the weights are frozen, which means that the encoder network will not update these pre-trained weights during training. This is done to focus on training a context network to enhance its modeling ability for time series data rather than relearning the representation of the underlying features.

[0029] We further adopt a block masking strategy, i.e., BERT image transformer pre-training, to divide the sequence into several blocks of size 10, and randomly mask the entire block; The mask sequence traverses through the context network and the reconstruction head consisting of two fully connected layers, producing a reconstructed sequence 576Z=[593z1, 593z2, …, 593zT]∈RC×T. The mean squared error loss function is used to evaluate the quality of the reconstruction:

[0030] Where: l m is the mean square error loss function, D represents the pre-training dataset, Mx represents the set corresponding to the mask frame, and Z t To reconstruct the sequence.

[0031] First, during fine-tuning, the reconstruction head is repositioned with a SED head consisting of fully connected layers, which outputs frame-level predictions; frame-level predictions are pooled in the temporal dimension through linear softmax to obtain clip-level prediction results. According to the task-aware module, AThead is set to focus on the audio tagging task; semi-supervised learning is performed through the average teacher algorithm with a consistency weight of 40; Secondly, a new fine-tuning strategy is set, namely the global-local feature fusion strategy, which uses two branches to extract different features from the reconstructed sequence; The global branch inputs the reconstructed sequence into the encoder network to generate a global feature sequence. In the local branch, the reconstructed sequence is split into several overlapping blocks along the time dimension by a sliding window; each block is then fed independently into the encoder network for feature extraction, and the output features from different blocks are aggregated to form a local feature sequence; since there is temporal overlap between nearby blocks, the features extracted from different blocks are averaged for the duration of the overlap, and the global features are linearly fused with the local features to obtain the fused features:

[0032] Where: Z fused is the fusion feature sequence, Z global is the global feature sequence, Z local is the local feature sequence, and ℷ is the number of independent input blocks.

[0033] In the model, λ is set to 0.5, so the fused features combine both local and global features, therefore, it works well for sound events with different durations; further, since the overlap between blocks significantly increases the memory consumption, this feature fusion strategy is only used for the teacher model of the mean teacher algorithm and no back-propagation is required.

[0034] The context network is responsible for capturing the temporal correlations between latent features, and different types of head layers follow the context network to handle specific tasks, such as reconstruction, audio tagging, and SED.

[0035] Preferably, the output of the encoder network is represented as Z = [z1, z2, ..., zT] ∈ RC×T, where C is the size of the embedding vector and T is the length of the encoder output in the time dimension.

[0036] Preferably, in the process of evaluating the quality of reconstruction using the mean square error loss, only the mask frame is used to calculate the reconstruction loss.

[0037] The device of the masked audio transformer model based on MAT-SED for the multi-cabin substation is characterized in that it includes a transformer module, a training module, a task perception module and an evaluation module, and there are connections between the modules; Transformer module: used to construct an initial transformer model, wherein the initial transformer model includes an encoder network and a context network; Training module: pre-train the initial transformer model based on masked reconstruction to generate a reconstruction sequence; Task-aware module: fine-tunes the reconstruction sequence based on a preset fine-tuning strategy to obtain a masked audio transformer model based on MAT-SED; Evaluation module: Quality evaluation of the MAT-SED based masked audio transformer model is performed based on the mean square error loss function.

[0038] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0039] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0040] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0041] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0042] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0043] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A MAT-SED-based masked audio Transformer model for multi-cabin substations, characterized in that: The following steps are involved: Constructing an initial transformer model, wherein the initial transformer model includes an encoder network and a context network; Pre-train the initial transformer model based on masked reconstruction to generate a reconstruction sequence; The reconstructed sequence is fine-tuned based on a preset fine-tuning strategy to obtain a masked audio transformer model based on MAT-SED; Quality evaluation of the MAT-SED based masked audio transformer model is performed based on the mean squared error loss function.

2. The MAT-SED-based masked audio Transformer model for a multi-cabin substation according to claim 1, characterized in that: The encoder network specifically includes: Get the mel spectrogram and divide it into several patches of preset sizes; The plaque patches are projected into the embedding sequence in a linear relationship, and the size of the spectrogram is compressed according to the PaSST mechanism to obtain a latent feature sequence.

3. The MAT-SED-based masked audio transformer model for a multi-cabin substation as claimed in claim 2, characterized in that: The context network specifically includes: According to the multi-head attention mechanism, the relationship between different position sequences in the potential feature sequence is indicated, and the position information of the learnable potential feature sequence is obtained; The positions of the same feature at different times are detected simultaneously, thereby capturing the temporal correlation of the underlying feature sequence.

4. The MAT-SED-based masked audio transformer model for a multi-cabin substation as claimed in claim 3, characterized in that: The pre-training of the initial transformer model based on masking reconstruction specifically includes: Freeze the weights of the initial transformer model, randomly extract a portion of the frames in the potential feature sequence, hide and cover the extracted frames, and mark them as mask frames to obtain a mask frame sequence; Dividing the obtained mask frame sequence into a number of sequence blocks of preset sizes by using a block masking strategy, and randomly masking a preset number of sequence blocks; The sequence blocks after the preset number of random masking are reconstructed to generate a reconstructed sequence.

5. The MAT-SED-based masked audio Transformer model for a multi-cabin substation as claimed in claim 4, characterized in that: The fine-tuning strategy includes a global-local feature fusion strategy, specifically including: The global-local feature fusion strategy uses two branches to extract different features from the spectrogram; The global branch inputs the reconstructed sequence into the encoder network to generate a global feature sequence; The local branch divides the reconstructed sequence into multiple overlapping blocks in the time dimension through a sliding window, and then inputs each block into the encoder network independently to generate a local feature sequence; The global feature sequence and the local feature sequence are linearly fused to obtain a fused feature sequence.

6. The MAT-SED-based masked audio transformer model for a multi-cabin substation as claimed in claim 5, characterized in that: The masked audio transformer model based on MAT-SED is specifically: Linear softmax is introduced to pool the detected latent feature sequence in the time dimension to obtain the clip-level prediction result; The clip-level prediction results are semi-supervised learned by the average teacher algorithm, resulting in a masked audio transformer model based on MAT-SED.

7. The MAT-SED-based masked audio transformer model for a multi-cabin substation according to claim 6, characterized in that: The quality assessment is specifically: The mean square error loss function is used to evaluate the quality of reconstruction. The calculation formula of the mean square error loss function is as follows: Where: is the mean square error loss function, D represents the pre-training data set, Denotes the set corresponding to the mask frame, Z t To reconstruct the sequence, x is the data in the data set.

8. The MAT-SED-based masked audio transformer model for a multi-cabin substation as claimed in claim 5, characterized in that: The method for obtaining the fusion feature sequence is as follows: Split the spectrogram into several overlapping blocks using a slidable time window; Independently input each block into the encoder network for feature extraction; Aggregate the features of different blocks to obtain a local feature sequence; For time-overlapping blocks, the features extracted within the overlapping duration are averaged to obtain a fused feature sequence.

9. The MAT-SED-based masked audio transformer model for a multi-cabin substation as claimed in claim 8, characterized in that: The fusion feature sequence is as follows: Where: Z fused is the fusion feature sequence, Z global is the global feature sequence, Z local is the local feature sequence, and ℷ is the number of independent input blocks.

10. A device using the MAT-SED-based masked audio Transformer model of a multi-cabin substation according to any one of claims 1 to 9, characterized in that: It includes transformer module, training module, task perception module and evaluation module, and there are connections between modules; Transformer module: used to construct an initial transformer model, wherein the initial transformer model includes an encoder network and a context network; Training module: pre-train the initial transformer model based on masked reconstruction to generate a reconstruction sequence; Task-aware module: fine-tunes the reconstruction sequence based on a preset fine-tuning strategy to obtain a masked audio transformer model based on MAT-SED; Evaluation module: Quality evaluation of the MAT-SED based masked audio transformer model is performed based on the mean square error loss function.

Images