Acoustic-semantic guided and hierarchical learning based anomaly sound detection system and method

Abstract

This invention discloses an abnormal sound detection system and method, belonging to the field of audio signal processing. The system includes: an audio encoder, a hierarchical structure learning module, and a detection result output module. The audio encoder, after being trained under the constraints of a text-audio cross-modal alignment module, can extract audio features. The hierarchical structure learning module, after being trained under the constraints of a latent acoustic feature learning module, can perform domain classification and attribute classification on the extracted audio features, and output a unified representation. The detection result output module can find training data of the same machine type that is closest in Euclidean distance to the preprocessed machine sound to be identified (used as test data) in the unified feature space formed by splicing the output embeddings of the domain and attribute branches. The Euclidean distance between the two embeddings is used as the anomaly score for output. If the anomaly score exceeds the corresponding anomaly threshold, it is considered an abnormal sound. This system can improve detection accuracy.

Description

Technical Field

[0001] This invention relates to the fields of audio signal processing and machine learning, and in particular to an abnormal sound detection system and method based on acoustic-semantic guidance and hierarchical representation learning. Background Technology

[0002] Anomalous Sound Detection (ASD) aims to identify unusual sounds emitted by machines, a technology crucial for monitoring the condition of industrial equipment. Early identification of anomalous sounds can prevent unexpected malfunctions, reduce maintenance costs, and improve operational safety. In real-world industrial environments, machines typically operate continuously under diverse conditions, making manual inspection for anomalous sounds impractical and costly. Therefore, developing intelligent ASD systems to replace manual monitoring and improve system reliability is essential. Because anomalous sounds are scarce in real-world scenarios, ASD systems are typically trained using only normal data. This allows the problem to be structured as a self-supervised learning (SSL) task, rather than a simple binary classification problem (i.e., normal or anomalous). During inference, the anomalous score of a given sample (reflecting the degree of anomalousness in the audio) is measured by its deviation from the distribution space of normal data.

[0003] In recent years, metadata-driven approaches have achieved state-of-the-art performance, becoming the mainstream paradigm for ASD systems. These strategies model the normal audio distribution space using metadata tags as classification targets, outperforming simple binary classification due to the lack of anomalous training data. Specifically, metadata (i.e., machine attributes) corresponds to various operating conditions (e.g., operating speed, operating voltage), providing a detailed description of the machine's operating state. By training models to classify different combinations of machine types and attributes, the models can capture fine-grained acoustic representations of normal machine sounds. Based on this technical approach, researchers have explored various methods to improve the performance of ASD systems. For example, they have explored different data augmentation techniques to expand the distribution space of training data; designed various deep learning network architectures to improve the accuracy of feature extraction and classification; proposed various improved classification loss functions to optimize the model's training objective; and introduced different anomaly score regularization techniques to improve detection results.

[0004] However, existing ASD systems typically treat metadata tags as flat, self-contained categories, ignoring their inherent hierarchical structure. Specifically, machine attributes are not shared across different domains; domain-level information defines coarse-grained acoustic features, while attribute-level information captures finer variations within each domain. Ignoring this hierarchy leads to a confusion between domain-specific and attribute-specific factors in the learned representation space.

[0005] Furthermore, even if attribute annotations are available, manually defined attribute labels cannot fully capture all acoustically relevant variations in machine sound. Attribute labels are typically designed from the perspective of system configuration or device parameters, which may overlook potential acoustic factors such as harmonic structure, resonance modes, or subtle operational changes.

[0006] Furthermore, existing ASD systems rely entirely on acoustic supervision signals, often neglecting the semantic properties inherent in the attribute information itself. Attributes such as speed, voltage, or operating mode are essentially semantic concepts that can be described using natural language. If this semantic information cannot be explicitly utilized during the modeling process, attribute supervision will remain at the level of pure classification labels.

[0007] In summary, despite significant progress in the field of ASD (Abnormal Sound Detection), many current anomalous sound detection methods fail to effectively capture the inherent hierarchical relationships of metadata tags and are insufficient in capturing data-driven latent acoustic features, resulting in incomplete modeling of normal sound distribution. Therefore, providing an anomalous sound detection system and method that can fully exploit the hierarchical structure and semantic information of metadata and capture the latent acoustic features in machine sounds is a problem that needs to be solved.

[0008] In view of this, the present invention is hereby proposed. Summary of the Invention

[0009] The purpose of this invention is to provide an acoustic-semantic guided and hierarchical learning-based abnormal sound detection system and method. This system utilizes metadata tags to perform hierarchical representation modeling, thereby explicitly modeling the hierarchical structure between domain-level and attribute-level information. It captures both coarse-grained and fine-grained machine acoustic features, enhancing the ability to detect abnormal sounds. Furthermore, it captures data-driven latent acoustic features through an acoustic pseudo-label contrastive learning mechanism, while aligning audio embedding with text attribute representation. This effectively handles abnormal sound detection tasks in real-world industrial scenarios, thus solving the aforementioned technical problems in existing technologies.

[0010] The objective of this invention is achieved through the following technical solution:

[0011] An abnormal sound detection system includes:

[0012] The system comprises an audio encoder, a hierarchical structure learning module, and a detection result output module; among which,

[0013] After the audio encoder is trained by the text-audio cross-modal alignment module under patch-level embedding constraints, it can take the pre-processed audio corresponding to the machine voice to be recognized as input, extract features from the pre-processed audio, and output the extracted audio features.

[0014] The hierarchical structure learning module is communicatively connected to the audio encoder. After being trained by the latent acoustic feature learning module under the constraint of acoustic pseudo-labels on the unified representation output by the hierarchical structure learning module, it can take the audio features extracted by the audio encoder as input and perform domain classification and attribute classification on the input audio features through parallel domain classification submodule and attribute classification submodule respectively. The domain classification submodule captures coarse-grained domain features, and the attribute classification submodule focuses on fine-grained operational changes. The outputs of the domain branch and the attribute branch are embedded and concatenated into a unified representation that retains both global domain information and local attribute semantics in a unified feature space before outputting.

[0015] The detection result output module is communicatively connected to the hierarchical structure learning module. It can find training data of the same machine type that is closest in Euclidean distance to the preprocessed machine sound to be identified as the test data in the unified feature space formed by splicing the output embeddings of the domain branch and attribute branch. The Euclidean distance between the two embeddings is used as the anomaly score for output. If the anomaly score exceeds the corresponding anomaly threshold, it indicates that the preprocessed machine sound to be identified is an abnormal sound.

[0016] An abnormal sound detection method using the abnormal sound detection system of the present invention includes:

[0017] After the audio encoder of the system is trained by the text-audio cross-modal alignment module under patch-level embedding constraints, it receives the preprocessed audio corresponding to the machine voice to be identified, performs feature extraction on the preprocessed audio, and outputs the extracted audio features.

[0018] After the hierarchical structure learning module of the system is trained under the constraint of acoustic pseudo-labels on the unified representation output by the latent acoustic feature learning module, the audio features extracted by the audio encoder are used as input. The parallel domain classification submodule and attribute classification submodule perform domain classification and attribute classification on the input audio features respectively. The domain classification submodule captures coarse-grained domain features, and the attribute classification submodule focuses on fine-grained operational changes. The outputs of the domain branch and the attribute branch are embedded and concatenated into a unified representation that retains both global domain information and local attribute semantics in a unified feature space before being output.

[0019] The detection result output module of the system finds training data of the same machine type that is closest to the Euclidean distance of the preprocessed machine sound to be identified in the unified feature space formed by splicing the output embeddings of the domain branch and attribute branch of the hierarchical structure learning module. The Euclidean distance between the two embeddings is used as the anomaly score for output. If the anomaly score exceeds the corresponding anomaly threshold, it indicates that the preprocessed machine sound to be identified is an abnormal sound.

[0020] Compared with existing technologies, the acoustic-semantic guided and hierarchical learning-based abnormal sound detection system and method provided by this invention have the following advantages:

[0021] By setting up a hierarchical structure learning module to represent the modeling strategy hierarchically, the hierarchical structure between domain-level and attribute-level information is explicitly modeled, thereby capturing coarse-grained and fine-grained machine acoustic features and enhancing the ability to detect abnormal sounds. During the training phase, a latent acoustic feature learning module utilizes an acoustic pseudo-label contrastive learning mechanism to capture data-driven latent acoustic features. Simultaneously, a semantically aware text-audio alignment module aligns audio embeddings with text attribute representations to effectively handle abnormal sound detection tasks in real-world industrial scenarios. This invention achieves comprehensive modeling of normal machine sounds, thus enabling accurate abnormal sound detection. This invention effectively solves the problem that many current abnormal sound detection methods fail to effectively capture the inherent hierarchical relationships when utilizing metadata tags, and are insufficient in capturing data-driven latent acoustic features, resulting in incomplete modeling of normal sound distribution. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A block diagram of an abnormal sound detection system is provided for embodiments of the present invention.

[0024] Figure 2 A flowchart is provided for an abnormal sound detection method according to an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them, and do not constitute a limitation on the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0026] First, the following explanations are provided for the terms that may be used in this article:

[0027] The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0028] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0029] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0030] Unless otherwise explicitly specified or limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this document according to the specific circumstances.

[0031] The terms “center,” “longitudinal,” “lateral,” “length,” “width,” “thickness,” “up,” “down,” “front,” “back,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” “outer,” “clockwise,” and “counterclockwise” indicate the current orientation or positional relationship, and are only for the convenience and simplification of description, and do not explicitly or implicitly suggest that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this document.

[0032] The technical solution provided by this invention will be described in detail below. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they shall be performed according to conventional conditions in the art or conditions recommended by the manufacturer. Reagents or instruments used in the embodiments of this invention whose manufacturers are not specified are all conventional products that can be purchased commercially.

[0033] like Figure 1As shown, this invention provides an abnormal sound detection system based on acoustic-semantic guidance and hierarchical learning, comprising:

[0034] The system comprises an audio encoder, a hierarchical structure learning module, and a detection result output module; among which,

[0035] After the audio encoder is trained by the text-audio cross-modal alignment module under patch-level embedding constraints, it can take the pre-processed audio corresponding to the machine voice to be recognized as input, extract features from the pre-processed audio, and output the extracted audio features.

[0036] The hierarchical structure learning module is communicatively connected to the audio encoder. After being trained by the latent acoustic feature learning module under the constraint of acoustic pseudo-labels on the unified representation output by the hierarchical structure learning module, it can take the audio features extracted by the audio encoder as input and perform domain classification and attribute classification on the input audio features through parallel domain classification submodule and attribute classification submodule respectively. The domain classification submodule captures coarse-grained domain features, and the attribute classification submodule focuses on fine-grained operational changes. The outputs of the domain branch and the attribute branch are embedded and concatenated into a unified representation that retains both global domain information and local attribute semantics in a unified feature space before outputting.

[0037] The detection result output module, communicatively connected to the hierarchical structure learning module, can find training data of the same machine type that is closest in Euclidean distance to the preprocessed machine voice to be identified in the unified feature space formed by splicing the output embeddings of the domain branch and attribute branch. The Euclidean distance between the two embeddings is used as the anomaly score for output. If the anomaly score exceeds the corresponding anomaly threshold, it indicates that the preprocessed machine voice to be identified is abnormal. Since the training uses normal machine voices, a higher anomaly score indicates a greater degree of deviation from normal data in the preprocessed machine voice to be identified, and a higher degree of anomaly. It can be seen that the corresponding anomaly threshold is not fixed, but dynamically adjusted for different machine types.

[0038] Preferably, in the above system, the text-audio cross-modal alignment module is communicatively connected to the audio encoder and can perform cross-modal alignment of audio embeddings with text features of metadata through contrastive learning during the training phase, enabling the audio encoder to capture semantic information beyond pure classification supervision and perform patch-level embedding constraints on the audio encoder.

[0039] The latent acoustic feature learning module, communicatively connected to the hierarchical structure learning module, can extract audio embeddings from preprocessed machine sounds (including sounds of different machine types in normal states) used as training data by an unsupervised pre-trained domain adaptive encoder during the training phase. Then, it uses a Ward linker to adaptively aggregate hierarchical clustering the extracted audio embeddings, assigning each machine type's cluster as a pseudo-label. The unified representation output by the hierarchical structure learning module is used as input and optimized through a supervised contrastive loss function. This allows samples sharing the same pseudo-label to form compact clusters in the embedding space, promoting the separation of acoustically dissimilar samples and directly constraining the unified representation output by the hierarchical structure learning module on the acoustic pseudo-labels.

[0040] Preferably, in the above system, the text-audio cross-modal alignment module performs a contrastive learning loss function to align the audio embedding with the text features of the metadata across modalities. for:

[0041] (1);

[0042] Where N represents the batch size; and They represent the first For each sample, audio and text embedding pairs are used, and so on. and They represent the first Audio and text embedding pairs for each sample; This is the second temperature hyperparameter; This represents the text embedding corresponding to the metadata, extracted by the text encoder. This indicates the audio embedding extracted by the audio encoder.

[0043] Preferably, in the above system, the latent acoustic feature learning module uses a supervised contrastive loss function. for:

[0044] (2);

[0045] in, This represents the index of all samples in a batch; For the first Regularized embedding of each sample ; This is the first temperature hyperparameter; Indicates that samples are not included. The self and the sample Sample sets with the same pseudo-labels Represents the sample set Embedding of samples; Indicates that in the same batch, excluding samples The set of all samples outside of, Represents a set Embedding of samples in the middle.

[0046] Preferably, in the above system, the system minimizes the total loss during training. To optimize the model parameters, the Adam optimizer is used for parameter updates. The model is trained on the DCASE ASD dataset and iterates multiple times until convergence.

[0047] The total loss mentioned for:

[0048] (3);

[0049] in, The attribute classification loss is used for the hierarchical representation learning module; For the domain classification loss of the hierarchical representation learning module, The weights for the domain classification loss of the hierarchical representation learning module; The supervised contrastive learning loss is used for the latent acoustic feature learning module. The weights are used for the supervised contrastive learning loss of the latent acoustic feature learning module; For the contrast loss of the text-audio cross-modal alignment module, Weights for the contrast loss of the text-audio cross-modal alignment module.

[0050] Preferably, in the above system, the audio encoder is composed of a ViT architecture, which includes a 12-layer Transformer encoder network. In each layer of the Transformer encoder network, there is a multi-head attention layer, followed by a batch normalization function, followed by an FFN network layer. The dimensionality is first increased, then activated by the GELU function, and finally reduced and batch normalized.

[0051] This audio encoder takes P patches of the log-Mel spectrum of the audio as input, and each layer of the Transformer encoder network outputs... The tensor of each layer of the Transformer encoder network is used to process each audio sample. Input features.

[0052] Preferably, in the above system, the hierarchical structure learning module includes: a domain classification submodule and an attribute classification submodule arranged in parallel; wherein,

[0053] The domain classification submodule includes an MLP layer that pools P patch-level embeddings into a sentence-level embedding, and an ArcFace classifier connected to the MLP layer.

[0054] The attribute classification submodule includes an MLP layer that pools P patch-level embeddings into a sentence-level embedding, and an ArcFace classifier connected to the MLP layer.

[0055] Preferably, in the above system, the processing procedure of the domain classification submodule is represented as follows:

[0056] (4);

[0057] (5);

[0058] (6);

[0059] in, Sentence-level embedding output for the MLP layer of the domain classification submodule; This is the output of the audio encoder; () represents the ArcFace attribute classifier in the domain classification submodule, which can predict sentence-level embeddings as corresponding predicted labels. CE() is the cross-entropy function, which can predict labels. With real labels Calculate the loss used for classification optimization ;

[0060] The processing procedure of the attribute classification submodule is as follows:

[0061] (7);

[0062] (8);

[0063] (9);

[0064] in, Sentence-level embedding output for the MLP layer of the attribute classification submodule; This is the output of the audio encoder; () represents the ArcFace attribute classifier in the attribute classification submodule, which can predict sentence-level embeddings as corresponding predicted labels. CE() is the cross-entropy function, which can predict labels. With real labels Calculate the loss used for classification optimization .

[0065] like Figure 2As shown, an abnormal sound detection method using the above-described abnormal sound detection system according to an embodiment of the present invention is characterized by comprising:

[0066] After the audio encoder of the system is trained by the text-audio cross-modal alignment module under patch-level embedding constraints, it receives the preprocessed audio corresponding to the machine voice to be identified, performs feature extraction on the preprocessed audio, and outputs the extracted audio features.

[0067] After the hierarchical structure learning module of the system is trained under the constraint of acoustic pseudo-labels on the unified representation output by the latent acoustic feature learning module, the audio features extracted by the audio encoder are used as input. The parallel domain classification submodule and attribute classification submodule perform domain classification and attribute classification on the input audio features respectively. The domain classification submodule captures coarse-grained domain features, and the attribute classification submodule focuses on fine-grained operational changes. The outputs of the domain branch and the attribute branch are embedded and concatenated into a unified representation that retains both global domain information and local attribute semantics in a unified feature space before being output.

[0068] The detection result output module of the system finds training data of the same machine type that is closest to the Euclidean distance of the preprocessed machine sound to be identified in the unified feature space formed by splicing the output embeddings of the domain branch and attribute branch of the hierarchical structure learning module. The Euclidean distance between the two embeddings is used as the anomaly score for output. If the anomaly score exceeds the corresponding anomaly threshold, it indicates that the preprocessed machine sound to be identified is an abnormal sound.

[0069] Preferably, in the above method, the system is pre-trained in the following manner: the model parameters are optimized by minimizing the total loss, the parameters are updated using the Adam optimizer, the model is trained on the DCASE ASD dataset, and the process is repeated multiple times until convergence.

[0070] During the training phase, the system's text-audio cross-modal alignment module utilizes contrastive learning to align audio embeddings with the text features of metadata across modalities. This enables the audio encoder to capture semantic information beyond pure classification supervision, and applies patch-level embedding constraints to the audio encoder.

[0071] During the training phase, the preprocessed machine sound, which serves as training data, is input into the unsupervised pre-trained domain adaptive encoder through the latent acoustic feature learning module of the system. The corresponding audio embeddings are extracted, and then Ward links are used to perform adaptive aggregation hierarchical clustering on the extracted audio embeddings to obtain cluster assignments for each machine type as pseudo-labels. The unified representation output by the hierarchical structure learning module is used as input and optimized through a supervised contrastive loss function. This allows samples sharing the same pseudo-labels to form compact clusters in the embedding space and promotes the separation of acoustically dissimilar samples. The unified representation output by the hierarchical structure learning module is directly constrained on the acoustic pseudo-labels.

[0072] To more clearly demonstrate the technical solution and its effects provided by the present invention, the following detailed description of the solution provided by the embodiments of the present invention is provided with reference to specific examples.

[0073] Example 1

[0074] like Figure 1 As shown, this embodiment provides an abnormal sound detection system, which is a framework for abnormal sound detection. Through three core modules—hierarchical structure learning, latent acoustic feature learning, and text-audio cross-modal alignment—it achieves comprehensive modeling of normal machine sounds, thereby enabling accurate abnormal sound detection. The system includes:

[0075] The system includes an audio encoder, a hierarchical structure learning module, a latent acoustic feature learning module, a text-audio cross-modal alignment module, and a detection result output module.

[0076] These modules work together to achieve accurate detection of abnormal sounds.

[0077] The audio encoder employs ViT (Vision Transformer) as its backbone network, providing the system with powerful audio feature extraction capabilities. The hierarchical structure learning module explicitly learns the hierarchical structure within the metadata to capture both coarse-grained and fine-grained machine features. The latent acoustic feature learning module introduces data-driven pseudo-labels as learning objectives to capture the machine's latent acoustic features. The text-audio cross-modal alignment module effectively fuses text and audio features to fully leverage the complementarity of multimodal information. These different modules work collaboratively with their respective optimization objectives, ensuring the model can comprehensively and deeply mine metadata information in a complementary manner to improve anomaly detection performance.

[0078] The hierarchical structure learning module introduces a hierarchical representation modeling strategy. Through a dual-branch architecture of domain-aware supervision and attribute-level supervision, the system can explicitly model the hierarchical relationships between metadata tags. Specifically, the audio spectrogram is first patched and input into a pre-trained ViT backbone network serving as the audio encoder to extract patch-level audio embeddings. Then, two parallel classification branches are used for domain classification and attribute classification, respectively. The domain branch captures coarse-grained domain features, while the attribute branch focuses on fine-grained operational variations. The output embeddings of these branches are concatenated to form a unified representation, thereby preserving both global domain information and local attribute semantics in a unified feature space, promoting more structured and discriminative audio representation learning.

[0079] The latent acoustic feature learning module, used during the training phase, guides supervised contrastive learning through cluster-induced pseudo-labels, enabling the system to capture latent acoustic features that cannot be represented by manually defined attributes. Specifically, preprocessed machine audio, used as training data, is first fed into an unsupervised pre-trained domain adaptive encoder to extract corresponding audio embeddings. These embeddings are then adaptively aggregated hierarchically using Ward links to obtain cluster assignments for each machine type as pseudo-labels. Next, the concatenated embeddings of domain-aware and attribute-aware features are used as input and optimized using a supervised contrastive loss function. This encourages samples sharing the same pseudo-labels to form compact clusters in the embedding space, while simultaneously promoting the separation of acoustically dissimilar samples. This mechanism allows the model to capture latent acoustic patterns and variations that are overlooked by individual attribute-level supervision.

[0080] A text-audio cross-modal alignment module, used during the training phase, introduces semantically aware text-audio alignment. Through a contrastive learning objective, it aligns audio embeddings with textual features of metadata across modalities, enabling the audio encoder to capture semantic information beyond pure classification supervision. Specifically, first, text descriptions are generated for each metadata label using a predefined natural language template function. Then, these text descriptions are encoded into text embeddings (i.e., semantic prototypes) using a pre-trained text encoder. The audio embeddings are then projected into a shared embedding space, maximizing the correspondence between audio samples and their associated text embeddings using a contrastive loss function. This alignment module further complements attribute and pseudo-label supervision, thereby facilitating the generation of a more discriminative and semantically structured embedding space.

[0081] To help those skilled in the art better understand the solutions of the embodiments of the present invention, the following detailed description, in conjunction with the accompanying drawings, describes the structure, training criteria, and practical applications of an abnormal sound detection system that includes multiple optimization targets utilizing metadata information.

[0082] (1) System structure:

[0083] The multi-optimization objective abnormal sound detection system of the present invention consists of an audio encoder (ViT architecture), a hierarchical structure learning module, a latent acoustic feature learning module, and a text-audio cross-modal alignment module. Different modules are implemented by different loss optimization objectives.

[0084] (11) Audio encoder:

[0085] The audio encoder is constructed using the ViT architecture, comprising a 12-layer Transformer encoder network. Each Transformer encoder layer includes a multi-head attention layer, followed by a batch normalization function, then an FFN network layer. The network first performs dimensionality increase, then passes through the GELU activation function, and finally reduces the dimensionality and performs batch normalization. The audio encoder takes P patches of the log-Melogram spectrogram of the audio as input, and each encoder layer outputs... The tensor. After passing through the audio encoder, the final result is the tensor of each audio sample. Input features.

[0086] (12) Hierarchical structure learning module:

[0087] The hierarchical structure learning module primarily uses classification modules targeting labels of varying fineness. Each module includes an MLP layer that pools P patch-level embeddings into a sentence-level embedding, and an ArcFace classifier that uses the MLP layer output for classification. The fine-grained labels include domain labels and attribute labels. The audio encoder output is represented as... Therefore, the processing procedure for the domain classification submodule can be represented as follows:

[0088] (4);

[0089] (5);

[0090] (6);

[0091] Among them, among them, Sentence-level embedding output for the MLP layer of the domain classification submodule; This is the output of the audio encoder; () represents the ArcFace attribute classifier in the domain classification submodule, which can predict sentence-level embeddings as corresponding predicted labels. CE() is the cross-entropy function, which can predict labels. With real labels Calculate the loss used for classification optimization .

[0092] The processing procedure for the attribute classification submodule can be represented as follows:

[0093] (7);

[0094] (8);

[0095] (9);

[0096] in, For the sentence-level embedding output of the MLP layer of the attribute classification submodule, () represents the ArcFace attribute classifier in the attribute classification submodule, used to predict sentence-level embeddings as corresponding tags. CE() is the cross-entropy function, which predicts the label. With real labels Calculate the loss used for classification optimization .

[0097] (13) Text-audio cross-modal alignment module:

[0098] This module uses a contrastive loss function to perform cross-modal alignment between the audio embedding and the text representation of the metadata, thereby capturing semantic information in the metadata and enhancing the discriminativeness and semantic consistency of the audio embedding. The text embedding extracted by the text encoder corresponding to the metadata is... And the audio embedded by the audio encoder The contrastive loss function can be expressed as:

[0099] (1);

[0100] Where N represents the batch size. and The audio and text embedding pair representing the i-th sample, and similarly, and They represent the first Audio and text embedding pairs for each sample. This refers to temperature hyperparameters.

[0101] (14) Latent acoustic feature learning module:

[0102] This module primarily uses data-driven clustering pseudo-labels as the learning objective. Unlike manually defined, system-configured true attribute labels, pseudo-labels focus on the actual acoustic patterns of the audio. By using supervised contrastive learning (SCL), with acoustic pseudo-labels as the guiding objective, the model can further capture latent acoustic patterns in the audio based on its hierarchical structural representation. This supervised contrastive learning process can be represented as:

[0103] (2);

[0104] in, This represents the index of all samples in a batch. For the first Regularized embedding of each sample , This refers to temperature hyperparameters. Representatives and Samples Sample sets with the same pseudo-labels, but excluding itself, Used to refer to the sample set Embedding of samples in the middle. This means that in the same batch, besides the sample The set of all samples outside of, Used to refer to the set Embedding of samples in the middle.

[0105] (2) Training principles:

[0106] (21) Loss function:

[0107] The total loss function used in this invention consists of four parts: attribute classification loss and domain classification loss in the hierarchical representation learning module; supervised contrastive learning loss in the latent acoustic feature learning module; and contrastive loss in the text-audio cross-modal alignment module. Total Loss It can be represented as:

[0108] (3);

[0109] in, , and These are the weights used to balance the losses of each module.

[0110] (22) Training process:

[0111] During training, by minimizing the total loss To optimize the model parameters, the Adam optimizer is used for parameter updates, and the model is trained on the DCASE ASD dataset, iterating multiple times until convergence.

[0112] (23) Practical Applications:

[0113] Here, taking an industrial equipment condition monitoring system as an example, this system can improve the accuracy and reliability of equipment anomaly monitoring. By identifying abnormal machine sound characteristics at an early stage, it helps maintenance personnel to conduct timely preventative maintenance interventions, avoiding unplanned downtime and production losses caused by unexpected failures. For example, in a smart manufacturing plant, the system can monitor various rotating mechanical equipment (such as bearings, gearboxes, valves, etc.) on the production line in real time, promptly capturing abnormal sound patterns (such as harmonic distortion caused by bearing wear, impact sounds caused by gear breakage, periodic vibration sounds caused by rotor imbalance, etc.), and automatically notifying maintenance personnel to conduct inspections and repairs.

[0114] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0115] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. An abnormal sound detection system characterized by, include: The system comprises an audio encoder, a hierarchical structure learning module, and a detection result output module; among which, After the audio encoder is trained by the text-audio cross-modal alignment module under patch-level embedding constraints, it can take the pre-processed audio corresponding to the machine voice to be recognized as input, extract features from the pre-processed audio, and output the extracted audio features. The hierarchical structure learning module is communicatively connected to the audio encoder. After being trained by the latent acoustic feature learning module under the constraint of acoustic pseudo-labels on the unified representation output by the hierarchical structure learning module, it can take the audio features extracted by the audio encoder as input and perform domain classification and attribute classification on the input audio features through parallel domain classification submodule and attribute classification submodule respectively. The domain classification submodule captures coarse-grained domain features, and the attribute classification submodule focuses on fine-grained operational changes. The outputs of the domain branch and the attribute branch are embedded and concatenated into a unified representation that retains both global domain information and local attribute semantics in a unified feature space before outputting. The detection result output module is communicatively connected to the hierarchical structure learning module. It can find training data of the same machine type that is closest in Euclidean distance to the preprocessed machine sound to be identified as the test data in the unified feature space formed by splicing the output embeddings of the domain branch and attribute branch. The Euclidean distance between the two embeddings is used as the anomaly score for output. If the anomaly score exceeds the corresponding anomaly threshold, it indicates that the preprocessed machine sound to be identified is an abnormal sound.

2. The abnormal sound detection system according to claim 1, characterized by, The text-audio cross-modal alignment module is communicatively connected to the audio encoder. During the training phase, it can perform cross-modal alignment between the audio embedding and the text features of the metadata through contrastive learning, enabling the audio encoder to capture semantic information beyond pure classification supervision and to perform patch-level embedding constraints on the audio encoder. The latent acoustic feature learning module, communicatively connected to the hierarchical structure learning module, can extract audio embeddings from the preprocessed machine sound input as training data using an unsupervised pre-trained domain adaptive encoder during the training phase. Then, it uses a Ward linker to adaptively aggregate and hierarchically cluster the extracted audio embeddings, obtaining cluster assignments for each machine type as pseudo-labels. The unified representation output by the hierarchical structure learning module is used as input and optimized through a supervised contrastive loss function. This allows samples sharing the same pseudo-labels to form compact clusters in the embedding space, promoting the separation of acoustically dissimilar samples and directly constraining the unified representation output by the hierarchical structure learning module on the acoustic pseudo-labels.

3. The abnormal sound detection system according to claim 2, characterized in that, The text-audio cross-modal alignment module is a cross-modal alignment contrastive learning loss function of aligning the audio embedding with the text feature of the metadata Is: (1)； Where N represents the batch size; and They represent the first For each sample, audio and text embedding pairs are used, and so on. and They represent the first Audio and text embedding pairs for each sample; This is the second temperature hyperparameter; This represents the text embedding corresponding to the metadata, extracted by the text encoder. This indicates the audio embedding extracted by the audio encoder.

4. The abnormal sound detection system according to claim 3, characterized in that, The supervised contrastive loss function used in the latent acoustic feature learning module for: (2)； in, This represents the index of all samples in a batch; For the first Regularized embedding of each sample ; This is the first temperature hyperparameter; Indicates that samples are not included. The self and the sample Sample sets with the same pseudo-labels Represents the sample set Embedding of samples; Indicates that in the same batch, excluding samples The set of all samples outside of, Represents a set Embedding of samples in the middle.

5. The abnormal sound detection system according to claim 4, characterized in that, The system minimizes the total loss during training. To optimize the model parameters, the Adam optimizer is used for parameter updates. The model is trained on the DCASE ASD dataset and iterates multiple times until convergence. The total loss mentioned for: (3)； in, The attribute classification loss is used for the hierarchical representation learning module; For the domain classification loss of the hierarchical representation learning module, The weights for the domain classification loss of the hierarchical representation learning module; The supervised contrastive learning loss is used for the latent acoustic feature learning module. The weights are used for the supervised contrastive learning loss of the latent acoustic feature learning module; For the contrast loss of the text-audio cross-modal alignment module, Weights for the contrast loss of the text-audio cross-modal alignment module.

6. The abnormal sound detection system according to any one of claims 1-5, characterized in that, The audio encoder is constructed using the ViT architecture, which includes a 12-layer Transformer encoder network. Each layer of the Transformer encoder network contains a multi-head attention layer, followed by a batch normalization function, and then an FFN network layer. The network first performs dimensionality increase, then passes through the GELU activation function, and finally performs dimensionality reduction and batch normalization. This audio encoder takes P patches of the log-Mel spectrum of the audio as input, and each layer of the Transformer encoder network outputs... The tensor of each layer of the Transformer encoder network is used to process each audio sample. Input features.

7. The abnormal sound detection system according to any one of claims 1-5, characterized in that, The hierarchical learning module includes: a domain classification submodule and an attribute classification submodule arranged in parallel; wherein... The domain classification submodule includes an MLP layer that pools P patch-level embeddings into a sentence-level embedding, and an ArcFace classifier connected to the MLP layer. The attribute classification submodule includes an MLP layer that pools P patch-level embeddings into a sentence-level embedding, and an ArcFace classifier connected to the MLP layer.

8. The abnormal sound detection system according to claim 7, characterized in that, The processing procedure of the domain classification submodule is represented as follows: (1)； (2)； (3)； in, Sentence-level embedding output for the MLP layer of the domain classification submodule; This is the output of the audio encoder; () represents the ArcFace attribute classifier in the domain classification submodule, which can predict sentence-level embeddings as corresponding predicted labels. CE() is the cross-entropy function, which can predict labels. With real labels Calculate the loss used for classification optimization ; The processing procedure of the attribute classification submodule is as follows: (4)； (5)； (6)； in, Sentence-level embedding output for the MLP layer of the attribute classification submodule; This is the output of the audio encoder; () represents the ArcFace attribute classifier in the attribute classification submodule, which can predict sentence-level embeddings as corresponding predicted labels. CE() is the cross-entropy function, which can predict labels. With real labels Calculate the loss used for classification optimization .

9. An abnormal sound detection method using the abnormal sound detection system according to any one of claims 1-8, characterized in that, include: After the audio encoder of the system is trained by the text-audio cross-modal alignment module under patch-level embedding constraints, it receives the preprocessed audio corresponding to the machine voice to be identified, performs feature extraction on the preprocessed audio, and outputs the extracted audio features. After the hierarchical structure learning module of the system is trained under the constraint of acoustic pseudo-labels on the unified representation output by the latent acoustic feature learning module, the audio features extracted by the audio encoder are used as input. The parallel domain classification submodule and attribute classification submodule perform domain classification and attribute classification on the input audio features respectively. The domain classification submodule captures coarse-grained domain features, and the attribute classification submodule focuses on fine-grained operational changes. The outputs of the domain branch and the attribute branch are embedded and concatenated into a unified representation that retains both global domain information and local attribute semantics in a unified feature space before being output. The detection result output module of the system finds training data of the same type that is closest in Euclidean distance to the preprocessed machine sound to be identified as the test data in the unified feature space formed by splicing the output embeddings of the domain branch and attribute branch of the hierarchical structure learning module. The Euclidean distance between the two embeddings is used as the anomaly score for output. If the anomaly score exceeds the corresponding anomaly threshold, it indicates that the preprocessed machine sound to be identified is an abnormal sound.

10. The abnormal sound detection method according to claim 9, characterized in that, In the method, the system is pre-trained in the following manner: the model parameters are optimized by minimizing the total loss, the parameters are updated using the Adam optimizer, the model is trained on the DCASE ASD dataset, and the process is repeated multiple times until convergence. During the training phase, the system's text-audio cross-modal alignment module utilizes contrastive learning to align the audio embedding with the text features of the metadata across modalities, enabling the audio encoder to capture semantic information beyond pure classification supervision and applying patch-level embedding constraints to the audio encoder. During the training phase, the preprocessed machine sound, which serves as training data, is input into the unsupervised pre-trained domain adaptive encoder through the latent acoustic feature learning module of the system. The corresponding audio embeddings are extracted, and then Ward links are used to perform adaptive aggregation hierarchical clustering on the extracted audio embeddings to obtain cluster assignments for each machine type as pseudo-labels. The unified representation output by the hierarchical structure learning module is used as input and optimized through a supervised contrastive loss function. This allows samples sharing the same pseudo-labels to form compact clusters in the embedding space and promotes the separation of acoustically dissimilar samples. The unified representation output by the hierarchical structure learning module is directly constrained on the acoustic pseudo-labels.

Images