Industrial noise identification method, system and device and storage medium

Abstract

The invention relates to an industrial noise identification method, system and device and a storage medium, and relates to the field of industrial Internet of Things and intelligent acoustic analysis. The method comprises the steps that a cross-modal shared dictionary and a projection matrix are generated at a client according to a local sensor for three modalities of acoustics, vibration and thermal imaging; constructing an initial prototype by using the cross-modal shared dictionary, the projection matrix and the prior knowledge base; receiving multi-modal data, and extracting and projecting original features of the multi-modal data; weighting the original features to obtain weighted original features, and training the initial prototype according to the multi-modal data and the weighted original features to obtain a local cross-modal prototype; performing sparse coding on the local cross-modal prototype to obtain a coefficient code, and uploading the coefficient code to a server side; and back-projecting the coefficient codes to construct a global prototype, wherein the global prototype is used for industrial noise identification. The technical effect of the invention is that the industrial noise identification accuracy and identification efficiency are improved.

Description

Technical Field

[0001] This application relates to the fields of industrial Internet of Things and intelligent acoustic analysis, and in particular to an industrial noise recognition method, system, device, and storage medium. Background Art

[0002] In the process of industrial intelligent transformation, federated learning technology provides new possibilities for cross-vendor collaboration in the field of equipment noise diagnosis.

[0003] Federated Learning technology (FL) is a distributed machine learning framework aimed at solving data privacy protection and data silo problems, enabling multiple participants to collaboratively train machine learning models without sharing raw data. Through a distributed parameter interaction mechanism, this technology allows manufacturers to jointly build a high-precision noise recognition model without exposing local data outside the domain, and it has been gradually applied to industrial scenarios such as equipment fault prediction and production line energy efficiency optimization.

[0004] However, the unique complexity of industrial noise recognition scenarios leads to multi-dimensional technical bottlenecks in federated learning:

[0005] Firstly, equipment noise data is highly fragmented due to production process confidentiality, forming data silos that are difficult to break. Although traditional centralized modeling schemes (such as CNN-LSTM multimodal fusion models) can improve recognition accuracy, they violate data sovereignty requirements such as GDPR due to forced data aggregation. Secondly, existing federated single-modal methods (such as FedAudio based on acoustic spectrum analysis) only focus on a single signal dimension, insufficiently utilize the complementary nature of multimodal features, and ignore the collaborative diagnostic value of mechanical state features in vibration signals and energy dissipation features in thermal imaging. Thirdly, the dynamic noise coupling effect caused by the parallel operation of multiple devices on the production line leads to non-linear superposition of acoustic signals in the time-frequency domain. Traditional signal processing methods (such as independent component analysis) are difficult to achieve noise source decoupling, and the time synchronization error of multimodal sensor data further significantly reduces the feature fusion efficiency. Fourthly, the problem of model heterogeneity is prominent in industrial fields. Sensing terminals deployed by different manufacturers have hardware generation gaps (such as the coexistence of 64-channel microphone arrays and 8-channel simple pickups), algorithm architecture differences (mixed deployment of ResNet and MobileNet), and uneven computing power resources, resulting in misalignment of feature spaces during the federated aggregation process.

[0006] To address the above challenges, the current mainstream solutions have significant technical limitations: The centralized training solution sacrifices data privacy leakage to improve model performance, which fundamentally conflicts with the industrial data compliance requirements; Although the federated unimodal method meets the privacy protection requirements, its single-dimensional modeling mode results in a 18.7% decrease in the noise source identification accuracy compared to the centralized method (measured in the automotive welding production line scenario); The static weight allocation strategy ignores the dynamic fluctuations of the sensor signal-to-noise ratio (±12dB) and the feature distribution drift caused by the working condition changes during multimodal federated aggregation, resulting in a 23.4% attenuation rate of the recognition accuracy of the model in continuous production scenarios.

[0007] These defects expose three systematic deficiencies of traditional methods: (1) Failure to construct a cross-modal dynamic calibration mechanism, making it difficult to eliminate the spatio-temporal asynchronous errors of multi-source sensing data; (2) Lack of an elastic feature mapping framework for device heterogeneity, leading to a continuous widening of the semantic gap between heterogeneous clients; (3) The fixed knowledge distillation strategy cannot adapt to the time-varying characteristics of complex noise coupling in industrial sites, resulting in a high false negative rate for long-tail category fault modes. In summary, these deficiencies lead to low industrial noise recognition accuracy and recognition efficiency. Summary of the Invention

[0008] To improve the industrial noise recognition accuracy and recognition efficiency, this application provides an industrial noise recognition method, system, device, and storage medium

[0009] In the first aspect, this application provides an industrial noise recognition method. The method is based on an industrial noise recognition system including a client and a server, a prior knowledge base, and local sensors. The method adopts the following technical solutions:

[0010] At the client, generate a cross-modal shared dictionary and a projection matrix for the three modalities of acoustics, vibration, and thermal imaging according to the local sensors;

[0011] Construct an initial prototype using the cross-modal shared dictionary, the projection matrix, and the prior knowledge base;

[0012] Receive multimodal data, extract and project the original features of the multimodal data, where the multimodal data includes acoustic data, vibration data, and thermal imaging data;

[0013] Introduce a gated attention mechanism to dynamically weight the original features to obtain weighted original features, and train the initial prototype according to the multimodal data and the weighted original features to obtain a local cross-modal prototype;

[0014] Perform sparse coding on the local cross-modal prototype to obtain coefficient codes, and upload the coefficient codes to the server side;

[0015] On the server side, back-project the coefficient encoding to construct a global prototype for industrial noise recognition.

[0016] Through the above technical solution, a hybrid federated learning framework based on prototype alignment and adaptive distillation is provided, focusing on three modalities of acoustics, vibration, and thermal imaging, constructing cross-modal prototypes to identify industrial noise. At the same time, to reduce communication overhead, sparse coding is performed on the cross-modal prototypes, improving the accuracy and efficiency of industrial noise recognition.

[0017] In a specific feasible implementation, the generating of the cross-modal shared dictionary and the projection matrix according to the local sensors for the three modalities of acoustics, vibration, and thermal imaging on the client side includes:

[0018] On the client side, generate an orthogonal basis vector set for the three modalities of acoustics, vibration, and thermal imaging respectively, and semantically associate the orthogonal basis vector sets to form a cross-modal shared dictionary;

[0019] On the client side, initialize the projection matrix according to the local sensors for the three modalities of acoustics, vibration, and thermal imaging. The projection matrix includes an acoustic projection matrix, a vibration projection matrix, and a thermal imaging projection matrix.

[0020] Through the above technical solution, generating the cross-modal shared dictionary and initializing the projection matrix are important steps in multi-modal federated initialization. The present invention adds steps for modal semantic association, making the alignment constraint objective between modalities enforce the consistency of semantic representations of different modalities in the shared space through mathematical or logical rules, breaking through modal barriers, achieving semantic fusion, and thus the industrial noise recognition method based on multi-modal federated learning has higher accuracy.

[0021] In a specific feasible implementation, the constructing of the initial prototype using the cross-modal shared dictionary, the projection matrix, and the prior knowledge base includes:

[0022] Generate a virtual prototype set based on the prior knowledge base:

[0023]

[0024] Among them, is the virtual prototype set of class c, is the simulated data set of class c, k c is the base vector index associated with class c, is the feature extraction of the simulated data x under modality m;

[0025] Construct the initial prototype using the cross-modal shared dictionary, the projection matrix, and the virtual prototype set;

[0026]

[0027] Among them, is the initial prototype, is the exclusive projection matrix for modality m of client i, is the feature extraction function of client i under modality m, which is defined by the local model parameter .

[0028] Through the above technical solution, local data can better reflect the user's specific scenario. Constructing the initial prototype can capture personalized features and improve the accuracy of the model in local scenarios. By aggregating the local prototypes of multiple clients, the global model can learn more general feature representations, avoid overfitting to a single data distribution, and balance personalized adaptation and global optimization.

[0029] In a specific feasible implementation, accepting the multi-modal data and extracting and projecting the original features of the multi-modal data include:

[0030] Accepting multi-modal data, which includes acoustic data, vibration data, and thermal imaging data;

[0031] Using a one-dimensional convolutional network to extract and project the original features of the acoustic data;

[0032] Using wavelet packet decomposition to extract and project the original features of the vibration data;

[0033] Using a three-dimensional convolutional network to extract and project the original features of the thermal imaging data.

[0034] Through the above technical solution, through heterogeneous feature projection, that is, extracting the original features and mapping them to the globally shared space, using a one-dimensional convolutional network, wavelet packet decomposition, and three-dimensional convolutional network to extract the original features of the acoustic data, vibration data, and thermal imaging data respectively, heterogeneous feature projection can break data silos, achieve cross-modal fusion, and improve the generalization and robustness of the model.

[0035] In a specific feasible implementation, after accepting the multi-modal data, it further includes:

[0036] Using dynamic time warping and adaptive weights to constrain the multi-modal data:

[0037]

[0038] Among them, S a (t), S v (t), S t (t) respectively represent the time series of acoustic, vibration, and thermal imaging signals, w t is the time shift variable, and λ controls the smoothness of the time series deformation.

[0039] Through the above technical solution, a cross-modal dynamic calibration mechanism is constructed to reduce the spatio-temporal asynchronous error of multi-source sensing data, making the constructed model for noise recognition more accurate.

[0040] In a specific feasible implementation, the method further includes a signal-to-noise ratio. The training of the initial prototype based on the multi-modal data and the weighted original features to obtain a local cross-modal prototype includes:

[0041] Training the initial prototype based on the multi-modal data and the weighted original features to obtain a local cross-modal prototype:

[0042] Designing a cross-modal loss function according to the signal-to-noise ratio and introducing the cross-modal loss function into the local cross-modal prototype.

[0043] Through the above technical solution, loss enhancement feature discrimination is introduced to improve the accuracy of the model in discriminating noise features, making industrial noise recognition more accurate.

[0044] In a specific feasible implementation, the back-projection of the coefficient encoding at the server side to construct a global prototype includes:

[0045] Obtaining a dynamic aggregation weight according to the signal-to-noise ratio by using a working condition similarity measurement method at the server side;

[0046] Using the dynamic aggregation weight to perform back-projection on the coefficient encoding to construct a global prototype:

[0047]

[0048] is the global prototype for class c, is the dynamic aggregation weight of client i in modality m, In the prototype of class c of client i in modality m, the k-th basis vector is the sparse coefficient.

[0049] Through the above technical solution, dynamic weight aggregation is performed when constructing the global prototype, making the model for noise recognition more accurate.

[0050] In a second aspect, the present application provides an industrial noise recognition system, adopting the following technical solution: The system includes:

[0051] A multi-modal federated initialization module, configured to generate a cross-modal shared dictionary and a projection matrix according to the local sensors for three modalities of acoustics, vibration, and thermal imaging at the client side;

[0052] An initial prototype construction module, configured to construct an initial prototype by using the cross-modal shared dictionary, the projection matrix, and the prior knowledge base;

[0053] A multi-modal data receiving and processing module for receiving multi-modal data, extracting and projecting the original features of the multi-modal data, where the multi-modal data includes acoustic data, vibration data, and thermal imaging data;

[0054] A local cross-modal prototype construction module for introducing a gated attention mechanism to dynamically weight the original features to obtain weighted original features, and training the initial prototype according to the multi-modal data and the weighted original features to obtain a local cross-modal prototype;

[0055] A sparse coding module for performing sparse coding on the local cross-modal prototype to obtain coefficient codes, and uploading the coefficient codes to the server side;

[0056] A global prototype construction module for back-projecting the coefficient coding on the server side to construct a global prototype, where the global prototype is used for industrial noise recognition.

[0057] In a third aspect, the present application provides a computer device, adopting the following technical solution: including a memory and a processor, and a computer program as described above is stored on the memory and can be loaded and executed by the processor.

[0058] In a fourth aspect, the present application provides a computer-readable storage medium, adopting the following technical solution: storing a computer program that can be loaded and executed by the processor as described above.

[0059] In summary, the present application has the following beneficial technical effects:

[0060] (1) The present invention provides a hybrid federated learning framework based on prototype alignment and adaptive distillation, focusing on three modalities of acoustics, vibration, and thermal imaging, constructing cross-modal prototypes to identify industrial noise. At the same time, in order to reduce communication overhead, sparse coding is performed on the cross-modal prototypes, which improves the accuracy and efficiency of industrial noise recognition.

[0061] (2) The present invention adds steps for modal semantic association, so that the alignment constraint objective between modalities is to force the semantic representations of different modalities to be consistent in the shared space through mathematical or logical rules, breaking through modal barriers and achieving semantic fusion, thereby making the accuracy of the industrial noise recognition method based on multi-modal federated learning higher.

[0062] (3) Through heterogeneous feature projection, that is, extracting the original features and mapping them to the global shared space, using a one-dimensional convolutional network, wavelet packet decomposition, and a three-dimensional convolutional network to extract the original features of the acoustic data, vibration data, and thermal imaging data respectively, heterogeneous feature projection can break data silos, achieve cross-modal fusion, and improve the generalization and robustness of the model.

[0063] (4) Introduce loss enhancement feature discrimination, perform dynamic weight aggregation when constructing the global prototype, improve the accuracy of the model's discrimination of noise features, and make industrial noise recognition more accurate. Description of the Drawings

[0064] Figure 1 This application proposes an industrial noise source recognition system based on multi-modal federated learning.

[0065] Figure 2 It is the flowchart in the embodiment of this application.

[0066] Figure 3 It is the comparison table of the accuracy rates of the method of this application and other advanced federated learning algorithms.

[0067] Figure 4 It is the structural block diagram in the embodiment of this application.

[0068] Reference Numerals: 401, multi-modal federated initialization module; 402, initial prototype construction module; 403, multi-modal data receiving and processing module; 404, local cross-modal prototype construction module; 405, sparse coding module; 406, global prototype construction module. Detailed Embodiments

[0069] The following further elaborates on this application in conjunction with the attached Figures 1 - 4 drawings.

[0070] The embodiment of this application discloses an industrial noise recognition method, which is used to improve the accuracy and efficiency of industrial noise recognition.

[0071] In the process of industrial intelligent transformation, federated learning technology provides new possibilities for cross-vendor collaboration in the field of equipment noise diagnosis. Through the distributed parameter interaction mechanism, this technology enables manufacturers to jointly build a high-precision noise recognition model on the premise that local data does not leave the domain, and has been gradually applied to industrial scenarios such as equipment fault prediction and production line energy efficiency optimization.

[0072] However, the unique complexity of the industrial noise recognition scenario leads to multi-dimensional technical bottlenecks for federated learning:

[0073] Firstly, due to the high degree of fragmentation of equipment noise data involving production process secrets, data islands that are difficult to break are formed. Although traditional centralized modeling schemes (such as the CNN-LSTM multi-modal fusion model) can improve the recognition accuracy, they violate data sovereignty requirements such as GDPR due to forced data aggregation.

[0074] Second, existing federal unimodal methods (such as FedAudio based on acoustic spectrum analysis) only focus on a single signal dimension, insufficiently utilize the complementary nature of multimodal features, and neglect the collaborative diagnostic value of mechanical state features in vibration signals and energy dissipation features in thermal imaging;

[0075] Third, the dynamic noise coupling effect caused by the parallel operation of multiple devices on the production line results in the non-linear superposition of acoustic signals in the time-frequency domain. Traditional signal processing methods (such as independent component analysis) are difficult to achieve noise source decoupling, and the time synchronization error of multimodal sensor data further significantly reduces the feature fusion efficiency;

[0076] Fourth, the problem of model heterogeneity is prominent in industrial sites. There are hardware generation gaps (such as the coexistence of 64-channel microphone arrays and 8-channel simple pickups), algorithm architecture differentiations (mixed deployment of ResNet and MobileNet), and uneven computing power resources among sensing terminals deployed by different manufacturers, leading to misalignment in feature space alignment during the federal aggregation process.

[0077] In response to the above challenges, current mainstream solutions have significant technical limitations: The centralized training solution sacrifices data privacy leakage to improve model performance, which fundamentally conflicts with industrial data compliance requirements; Although the federal unimodal method meets the privacy protection requirements, its single-dimensional modeling mode results in a 18.7% decrease in the noise source identification accuracy compared to the centralized method (measured in the automotive welding production line scenario); The static weight allocation strategy ignores the dynamic fluctuations of sensor signal-to-noise ratios (±12dB) and feature distribution drifts caused by working condition changes during multimodal federal aggregation, resulting in a 23.4% attenuation rate of the recognition accuracy of the model in continuous production scenarios.

[0078] These defects expose three systematic deficiencies of traditional methods: (1) The lack of a cross-modal dynamic calibration mechanism makes it difficult to eliminate the spatio-temporal asynchronous errors of multi-source sensing data; (2) The lack of an elastic feature mapping framework for device heterogeneity leads to the continuous expansion of the semantic gap between heterogeneous clients; (3) The fixed knowledge distillation strategy cannot adapt to the time-varying characteristics of complex noise coupling in industrial sites, resulting in a high missed detection rate for long-tail category fault modes. In summary, these deficiencies lead to low industrial noise recognition accuracy and recognition efficiency.

[0079] Therefore, this application proposes an industrial noise recognition method based on an industrial noise recognition system, a priori knowledge base, and local sensors, which includes a client and a server. The industrial noise source recognition system is as Figure 1 shown, and this method is used to improve the industrial noise recognition accuracy and recognition efficiency.

[0080] As Figure 2As shown, the method is based on an industrial noise recognition system including a client and a server, a prior knowledge base, and local sensors. For the client and the server, the method includes:

[0081] S10. On the client side, for the three modalities of acoustics, vibration, and thermal imaging, generate a cross-modal shared dictionary and a projection matrix according to local sensors.

[0082] Specifically, for the three modalities related to noise recognition, namely acoustics, vibration, and thermal imaging, respectively generate a set of orthogonal basis vectors to form a cross-modal shared dictionary:

[0083] And

[0084] where Φ is the cross-modal shared dictionary, d g is the dimension of the global feature space, is the unit vector of dimension d g K is the codebook capacity of each modality, represents the k-th vector of modality m. Ensure the orthogonality of the basis vectors within the modality through Gram-Schmidt orthogonalization. m is three different modalities.

[0085] The client initializes a modality-specific learnable projection matrix according to the local sensor configuration, and generates an acoustic projection matrix, a vibration projection matrix, and a thermal imaging projection matrix for the three modalities of acoustics, vibration, and thermal imaging respectively.

[0086] S20. Use the cross-modal shared dictionary, projection matrix, and prior knowledge base to construct an initial prototype.

[0087] Specifically, generate a virtual prototype set based on the prior knowledge base, and then use the cross-modal shared dictionary, projection matrix, and prior knowledge base to construct an initial prototype.

[0088] S30. Receive multi-modal data, extract and project the original features of the multi-modal data. The multi-modal data includes acoustic data, vibration data, and thermal imaging data.

[0089] Specifically, the multi-modal data (acoustics, vibration, thermal imaging) collected by the client, which includes acoustic data, vibration data, and thermal imaging data, perform heterogeneous feature projection on the multi-modal data, that is, extract the original features through local sensors and map them to the global shared space.

[0090] S40. Introduce a gated attention mechanism to dynamically weight the original features to obtain weighted original features, and train the initial prototype according to the multi-modal data and the weighted original features to obtain a local cross-modal prototype.

[0091] Specifically, a gated attention mechanism is designed to dynamically weight multi-modal features to complete cross-modal feature fusion.

[0092]

[0093] Among them, the reference matrix is a learnable parameter, α m is the weight corresponding to the modality, z m is the global feature vector of modality m, c is the bias term, and then the final joint representation is: z fusion = ∑ m α m ·z m ;

[0094] The initial prototype is trained based on multi-modal data and weighted original features to obtain a local cross-modal prototype.

[0095] S50, sparse coding is performed on the local cross-modal prototype to obtain coefficient codes, and the coefficient codes are uploaded to the server side.

[0096] Specifically, the local cross-modal prototype is sparsely coded by the orthogonal matching pursuit algorithm, and only the first s non-zero sparsities are retained and uploaded to the server side. The traditional method requires 3×c×d g communication volume. After sparse coding, only 3×C×(s + 1) communication volume needs to be uploaded. The formula is as follows:

[0097]

[0098] Among them is the original feature, is the pre-learned basis vector, is the combined weight, which controls the contribution of different basis vectors k of different modalities.

[0099] S60, the coefficient coding is back-projected on the server side to construct a global prototype, and the global prototype is used for industrial noise recognition.

[0100] Specifically, the server side dynamically adjusts the weights according to the signal-to-noise ratio of each modality, introduces a working condition similarity metric to adjust the aggregation weights, the server side back-projects the coefficient coding, constructs a global prototype, and the global prototype is used for industrial noise recognition.

[0101] In one embodiment, in order to improve the accuracy and efficiency of industrial noise recognition, the step of generating a cross-modal shared dictionary and a projection matrix according to local sensors for the three modalities of acoustics, vibration, and thermal imaging on the client side can be specifically implemented as:

[0102] First, on the client side, orthogonal basis vector sets are generated for the three modalities of acoustics, vibration, and thermal imaging respectively. The semantic association orthogonal basis vector sets constitute a cross-modal shared dictionary. Specifically, for the three modalities of acoustics (Acoustic), vibration (Vibration), and thermal imaging (Thermal) related to noise recognition, orthogonal basis vector sets are generated respectively to form a cross-modal shared dictionary:

[0103] And

[0104] Among them, Φ is the cross-modal shared dictionary, d g is the global feature space dimension, is the unit vector of dimension d g K is the codebook capacity of each modality, represents the k-th vector of modality m. The orthogonality of the basis vectors within the modality is ensured through Gram-Schmidt orthogonalization, and m is three different modalities.

[0105] Design a cross-modal similarity matrix to force the basis vectors of different modalities to satisfy semantic association in a unified space:

[0106]

[0107] Among them, among them is the predefined cross-modal alignment category.

[0108] Then, on the client side, projection matrices are initialized for the three modalities of acoustics, vibration, and thermal imaging according to local sensors. The projection matrices include an acoustic projection matrix, a vibration projection matrix, and a thermal imaging projection matrix. Specifically, the client initializes a modality-specific learnable projection matrix according to the local sensor configuration; Acoustic projection matrix: Adapts to different numbers of microphone array channels (such as 16 / 64 channels), Vibration projection matrix: Is compatible with the feature dimensions of three-axis / single-axis acceleration sensors Thermal imaging projection matrix: Supports different infrared resolution inputs (such as 640×480 vs 320×240).

[0109] Generating a cross-modal shared dictionary and initializing a projection matrix are important steps in multi-modal federated initialization. The present invention adds steps for modal semantic association, enabling the alignment constraint objective between modalities to force the semantic representations of different modalities to be consistent in a shared space through mathematical or logical rules, breaking through modal barriers and achieving semantic fusion, thereby making the industrial noise recognition method based on multi-modal federated learning more accurate.

[0110] In one embodiment, to improve the accuracy and efficiency of industrial noise recognition, the step of constructing an initial prototype using a cross-modal shared dictionary, a projection matrix, and a prior knowledge base can be specifically executed as follows:

[0111] First, generate a virtual prototype set based on the prior knowledge base:

[0112] where, is the virtual prototype set of class c, is the simulated data set of class c, k c is the index of the basis vector associated with class c, is the feature extraction of the simulated data x in modality m;

[0113] Specifically, generate a virtual prototype set based on the prior knowledge base (such as the ISO 13373 mechanical fault standard). In the calculation formula of the virtual prototype set, the modalities selected are the three modalities of Acoustic, Vibration, and Thermal in the prior knowledge base.

[0114] Then, construct an initial prototype using the cross-modal shared dictionary, the projection matrix, and the virtual prototype set;

[0115]

[0116] where, is the initial prototype, is the modality m exclusive projection matrix of client i, is the feature extraction function of client i in modality m, which is defined by the local model parameter Defined.

[0117] Local data can better reflect the user's specific scenario. Constructing the initial prototype can capture personalized features and improve the accuracy of the model in the local scenario. By aggregating the local prototypes of multiple clients, the global model can learn more general feature representations, avoid overfitting to a single data distribution, and balance personalized adaptation and global optimization.

[0118] In one embodiment, to improve the accuracy and efficiency of industrial noise recognition, the step of accepting multi-modal data and extracting and projecting the original features of the multi-modal data can be specifically executed as follows:

[0119] First, accept multi-modal data. The multi-modal data includes acoustic data, vibration data, and thermal imaging data. Specifically, the multi-modal data collected by the client (acoustic, vibration, thermal imaging), and the multi-modal data includes acoustic data, vibration data, and thermal imaging data.

[0120] Next, use a one-dimensional convolutional network to extract and project the original features of acoustic data, use wavelet packet decomposition to extract and project the original features of vibration data, and use a three-dimensional convolutional network to extract and project the original features of thermal imaging data. Specifically, map the original features extracted by local sensors to the global shared space. For acoustic feature projection, use one-dimensional residual convolution for processing: For vibration feature projection, multiply the initialized matrix by the wavelet packet energy spectrum input: z v = ReLU(W v ·WPD(x v ) + b v ). For thermal imaging projection, multiply the initialized matrix by a three-dimensional convolution: z t = ReLU(W t ·f t (x t ) + b t ), where W a , W v , W t are initialized projection matrices, WPD() is wavelet packet decomposition, f a () is a one-dimensional convolutional network, f t () is a three-dimensional convolutional network, and b a , b v , b t are the biases of acoustic, vibration, and thermal imaging features respectively.

[0121] Through heterogeneous feature projection, that is, extract the original features and map them to the global shared space, and use a one-dimensional convolutional network, wavelet packet decomposition, and three-dimensional convolutional network to extract the original features of acoustic data, vibration data, and thermal imaging data respectively. Heterogeneous feature projection can break data silos, achieve cross-modal fusion, and improve the generalization and robustness of the model.

[0122] In one embodiment, to improve the accuracy and efficiency of industrial noise recognition, the following steps can also be performed after receiving multi-modal data:

[0123] Adopt dynamic time warping and adaptive weight to constrain multi-modal data:

[0124]

[0125] Among them, S a (t), S v (t), S t (t) respectively represent the time series of acoustic, vibration, and thermal imaging signals, w t is the time shift variable, and λ controls the smoothness of the time series deformation;

[0126] Specifically, dynamic time warping (DTW) and adaptive weight constraints are used to achieve cross-modal time synchronization.

[0127] A cross-modal dynamic calibration mechanism is constructed to reduce the spatio-temporal asynchronous error of multi-source sensing data, making the constructed model for noise recognition more accurate.

[0128] In one embodiment, in order to improve the accuracy and efficiency of industrial noise recognition, the step of training an initial prototype based on multi-modal data and weighted original features to obtain a local cross-modal prototype can be specifically executed as follows:

[0129] First, train an initial prototype based on multi-modal data and weighted original features to obtain a local cross-modal prototype. Specifically, calculate the modality-specific prototype:

[0130]

[0131] At the same time, calculate the fused feature prototype using the following formula:

[0132] Among them, is the modality-specific prototype, is the modality feature fusion prototype, is the dataset belonging to class c, z n (x) is the global feature vector of sample x under modality m, z fusion (x) is the fused vector.

[0133] Then, design a cross-modal loss function according to the signal-to-noise ratio and introduce the cross-modal loss function into the local cross-modal prototype. Specifically, for each different sample, introduce cross-modal contrast loss to enhance feature discrimination; for different modal features (z a ,z v ),(z v ,z t ),(z a ,z t ) of the same sample, we use them to represent positive sample pairs, and random modal features of different samples are used to represent negative sample pairs. Therefore, the multi-modal contrast loss function is expressed as:

[0134]

[0135] Among them, z m represents the feature vectors of different modalities, represents the positive sample feature from the same device as z m , z - represents the positive sample feature from the same device as z mNegative sample features from different devices, sin(·) represents the cosine similarity function to measure the similarity between two vectors, τ is the temperature coefficient to control the sharpness of the probability distribution;

[0136] Adjust the weights of each modality in the loss function according to the real-time signal-to-noise ratio (SNR): where, ∈ m is the noise estimator. Therefore, combining the classification loss, feature alignment loss, and contrast loss, the total loss function is:

[0137]

[0138] λ1 and λ2 are balance coefficients, and the global prototype is synchronized from the server regularly, is the cross-entropy classification loss, is the class probability distribution predicted by the model, and SNR m is the signal-to-noise ratio of modality m.

[0139] Introduce loss enhancement feature discrimination to improve the accuracy of the model in discriminating noise features and make industrial noise recognition more accurate.

[0140] In one embodiment, in order to improve the accuracy and efficiency of industrial noise recognition, the step of back-projecting the coefficient coding on the server side to construct the global prototype can be specifically executed as:

[0141] First, obtain the dynamic aggregation weight according to the signal-to-noise ratio using the working condition similarity measurement method on the server side. Specifically, the server side dynamically adjusts the weight according to the signal-to-noise ratio of the modality:

[0142]

[0143] where, η is the sharpening factor, and T m is the modality feature dimension normalization term. Then, the low-quality modality automatically reduces the weight, and the high-quality modality has a higher proportion. is the signal-to-noise ratio of modality m of the i-th client;

[0144] Introduce the working condition similarity measurement to adjust the aggregation weight:

[0145]

[0146] Then the final weight is γ i,centroid is the centroid of the current global prototype. Among them, γ ij represents the working condition similarity measurement between client i and client j in the acoustic modality (a), is the local prototype vector of class c of client i in the acoustic modality (a), Initial weights for calculating the signal-to-noise ratio (SNR) in the i-th client modality.

[0147] Then, back-project the coefficient encoding using the dynamic aggregation weights to construct the global prototype:

[0148]

[0149] is the global prototype for class c, is the dynamic aggregation weight of the i-th client in modality m, In the prototype of class c for the i-th client in modality m, the k-th basis vector of the sparse coefficient;

[0150] Specifically, in the multi-modal prototype decoupling aggregation, the server back-projects the coefficient encoding to construct the global prototype. At the same time, to prevent cross-modal semantic drift, a regularization term is added to the loss function for constraint:

[0151]

[0152] where, is the feature prototype of class c in another modality m′, is the feature prototype of class c in modality m.

[0153] Performing dynamic weight aggregation when constructing the global prototype makes the model for noise recognition more accurate.

[0154] In addition, this solution also involves optimizing the feature space dimension. Specifically, the spectrum analysis of the Hessian matrix is used to dynamically adjust the global dimension d g : where, represents the global feature space dimension during the t-th round of federated training, and the initial value is set by the pre-trained model; H represents the Hessian matrix, and λ max (H) is the maximum eigenvalue of the Hessian matrix, and λ avg (H) is the average value of the Hessian matrix;

[0155] Update the codebook every R rounds to generate a new codebook:

[0156]

[0157] where, φ k is the k-th basis vector in the global codebook, is the regularization constraint function, C is the total number of sample classes, and η is the learning rate.

[0158] Regenerate the orthogonal basis vectors {φ k} And broadcast it to the client to maintain the stability of sparse coding.

[0159] To comprehensively evaluate the superior performance of the method innovatively proposed in the present invention in practical applications, the present invention selects special experimental data sets including the CWRU bearing fault data set, the NASA turbine engine degradation data set, and the FLIR Thermal thermal imaging data set, and according to the evaluation criteria detailed in this section, compares and analyzes the proposed method with traditional federated learning algorithms. The experimental environment was built on the Ubuntu 18.04.3 LTS platform, using an Intel COREI7 13700KF CPU and an NVIDIA GeForce RTX 3090TI GPU (24GB RAM). In terms of the selection of programming languages and frameworks, the deep learning models in this article were built using the Python language and the Pytorch framework.

[0160] In the specific implementation, the data set is divided into a training set and a test set, where the training set accounts for 75% and the test set accounts for 25%.

[0161] The specific implementation parameters in the present invention are set as follows:

[0162] (1) Dirichlet parameter: The Dirichlet parameter α is set to 0.1 to simulate a relatively high degree of client data heterogeneity.

[0163] (2) Number of clients: The total number of clients is set to 20, and the number of online clients participating in training in each round is 10. The online clients are selected by random sampling.

[0164] (3) Training of client model parameters: The batch size is set to 64, the learning rate is 0.001, the number of local training rounds of the client is set to 1, and the total number of training rounds is 200.

[0165] To verify the effectiveness of the method proposed in the present invention, the most conventional accuracy metric is used to measure the effectiveness of the present invention. The accuracy calculation formula is as follows:

[0166]

[0167] In the specific implementation method, the method of the present invention is experimentally compared with advanced models such as the most classic federated learning algorithm FedAvg. As shown in the figure, there is a certain degree of optimization and improvement in the accuracy evaluation. Compared with existing federated learning methods, the method proposed in the present invention has significant advantages over the comparison models in terms of performance indicators on the above data sets, and optimizes the performance of federated learning in the field of industrial noise.

[0168] Based on the above method, embodiments of the present application also disclose. As Figure 4 , the system includes the following modules:

[0169] The multimodal federated initialization module 401 is used to generate a cross-modal shared dictionary and a projection matrix for acoustic, vibration, and thermal imaging modalities according to local sensors at the client side;

[0170] The initial prototype construction module 402 is used to construct an initial prototype by using the cross-modal shared dictionary, the projection matrix, and the prior knowledge base;

[0171] The multimodal data receiving and processing module 403 is used to receive multimodal data, extract and project the original features of the multimodal data, and the multimodal data includes acoustic data, vibration data, and thermal imaging data;

[0172] The local cross-modal prototype construction module 404 is used to introduce a gated attention mechanism to dynamically weight the original features to obtain weighted original features, and train the initial prototype according to the multimodal data and the weighted original features to obtain a local cross-modal prototype;

[0173] The sparse coding module 405 is used to perform sparse coding on the local cross-modal prototype to obtain coefficient codes, and upload the coefficient codes to the server side;

[0174] The global prototype construction module 406 is used to perform back-projection on the coefficient coding at the server side to construct a global prototype, and the global prototype is used for industrial noise recognition.

[0175] In one embodiment, the multimodal federated initialization module 401 is specifically configured to generate an orthogonal basis vector set for acoustic, vibration, and thermal imaging modalities respectively at the client side, and the semantic association orthogonal basis vector set constitutes a cross-modal shared dictionary; at the client side, the projection matrix is initialized according to local sensors for acoustic, vibration, and thermal imaging modalities, and the projection matrix includes an acoustic projection matrix, a vibration projection matrix, and a thermal imaging projection matrix.

[0176] In one embodiment, the initial prototype construction module 402 is specifically configured to generate a virtual prototype set based on the prior knowledge base:

[0177] Wherein, is the virtual prototype set of category c, is the simulated data set of category c, k c is the basis vector index associated with class c, is the feature extraction of the simulated data x under modality m;

[0178] Construct an initial prototype by using the cross-modal shared dictionary, the projection matrix, and the virtual prototype set;

[0179]

[0180] Among them, is the initial prototype, is the exclusive projection matrix for modality m of client i, is the feature extraction function of client i in modality m, which is defined by the local model parameters Define.

[0181] In one embodiment, the multi-modal data receiving and processing module 403 is specifically configured to receive multi-modal data, where the multi-modal data includes acoustic data, vibration data, and thermal imaging data; extract and project the original features of the acoustic data using a one-dimensional convolutional network; extract and project the original features of the vibration data using wavelet packet decomposition; extract and project the original features of the thermal imaging data using a three-dimensional convolutional network.

[0182] In one embodiment, the local cross-modal prototype construction module 404 is specifically configured to train the initial prototype based on the multi-modal data and the weighted original features to obtain the local cross-modal prototype: design a cross-modal loss function according to the signal-to-noise ratio, and introduce the cross-modal loss function into the local cross-modal prototype.

[0183] In one embodiment, the global prototype construction module 406 is specifically configured to obtain the dynamic aggregation weight using the working condition similarity measurement method according to the signal-to-noise ratio on the server side; use the dynamic aggregation weight to perform back-projection on the coefficient coding to construct the global prototype:

[0184]

[0185] is the global prototype of class c, is the dynamic aggregation weight of client i in modality m, In the prototype of class c of client i in modality m, the k-th basis vector Sparse coefficient.

[0186] The embodiments of the present application also disclose a computer device.

[0187] Specifically, the computer device includes a memory and a processor, and a computer program capable of being loaded and executed by the processor for the above-mentioned industrial noise recognition method is stored on the memory.

[0188] The embodiments of the present application also disclose a computer-readable storage medium.

[0189] Specifically, the computer-readable storage medium stores a computer program that can be loaded and executed by a processor, such as an industrial noise recognition method as described above. The computer-readable storage medium includes, for example: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs.

[0190] This specific embodiment is only an explanation of the present invention and does not limit the present invention. After reading this specification, those skilled in the art can make modifications to this embodiment without creative contributions as needed, but as long as it is within the scope of the claims of the present invention, it is protected by the patent law.

Claims

1. An industrial noise recognition method, characterized in that, The method is based on an industrial noise recognition system including a client and a server, a prior knowledge base, and local sensors, and the method includes: At the client, generate a cross-modal shared dictionary and a projection matrix according to the local sensors for three modalities of acoustics, vibration, and thermal imaging; Construct an initial prototype using the cross-modal shared dictionary, the projection matrix, and the prior knowledge base; Receive multi-modal data, extract and project the original features of the multi-modal data, where the multi-modal data includes acoustic data, vibration data, and thermal imaging data; Introduce a gated attention mechanism to dynamically weight the original features to obtain weighted original features, and train the initial prototype according to the multi-modal data and the weighted original features to obtain a local cross-modal prototype; Perform sparse coding on the local cross-modal prototype to obtain coefficient codes, and upload the coefficient codes to the server; At the server, perform back-projection on the coefficient coding to construct a global prototype for industrial noise recognition.

2. The method according to claim 1, wherein The generating, at the client, a cross-modal shared dictionary and a projection matrix according to the local sensors for three modalities of acoustics, vibration, and thermal imaging includes: At the client, generate an orthogonal basis vector set for three modalities of acoustics, vibration, and thermal imaging respectively, and semantically associate the orthogonal basis vector sets to form a cross-modal shared dictionary; At the client, initialize a projection matrix according to the local sensors for three modalities of acoustics, vibration, and thermal imaging, where the projection matrix includes an acoustic projection matrix, a vibration projection matrix, and a thermal imaging projection matrix.

3. The method according to claim 1, wherein The constructing an initial prototype using the cross-modal shared dictionary, the projection matrix, and the prior knowledge base includes: Generate a virtual prototype set based on the prior knowledge base: Among them, is the virtual prototype set of category c, is the simulated data set of category c, k c is the base vector index associated with class c, is the feature extraction of the simulated data x in modality m; Construct an initial prototype using the cross-modal shared dictionary, the projection matrix, and the virtual prototype set; Among them, is the initial prototype, is the exclusive projection matrix for modality m of client i, is the feature extraction function of client i under modality m, which is defined by the local model parameters Define.

4. The method according to claim 1, wherein The receiving multi-modal data, extracting and projecting the original features of the multi-modal data includes: Receive multi-modal data, where the multi-modal data includes acoustic data, vibration data, and thermal imaging data; Use a one-dimensional convolutional network to extract and project the original features of the acoustic data; Use wavelet packet decomposition to extract and project the original features of the vibration data; Use a three-dimensional convolutional network to extract and project the original features of the thermal imaging data.

5. The method according to claim 4, wherein After receiving the multi-modal data, it further includes: Use dynamic time warping and adaptive weights to constrain the multi-modal data: Among them, S a (t), S v (t), S t (t) respectively represent the time series of acoustic, vibration, and thermal imaging signals. w t is the time shift variable, and λ controls the smoothness of the time series deformation.

6. The method according to claim 1, wherein The method further includes a signal-to-noise ratio, and the training the initial prototype according to the multi-modal data and the weighted original features to obtain a local cross-modal prototype includes: Train the initial prototype according to the multi-modal data and the weighted original features to obtain a local cross-modal prototype: Design a cross-modal loss function according to the signal-to-noise ratio, and introduce the cross-modal loss function into the local cross-modal prototype.

7. The method according to claim 6, wherein The performing, at the server, back-projection on the coefficient coding to construct a global prototype includes: At the server, obtain a dynamic aggregation weight according to the signal-to-noise ratio using a working condition similarity measurement method; Use the dynamic aggregation weight to perform back-projection on the coefficient coding to construct a global prototype: is the global prototype for class c, is the dynamic aggregation weight of client i in modality m, In the prototype of class c of client i in modality m, the k-th basis vector of the sparse coefficient.

8. An industrial noise recognition system, characterized in that, The system includes: A multimodal federated initialization module (401) for generating a cross-modal shared dictionary and a projection matrix for acoustic, vibration, and thermal imaging modalities based on the local sensors at the client side; An initial prototype construction module (402) for constructing an initial prototype using the cross-modal shared dictionary, the projection matrix, and the prior knowledge base; A multimodal data reception and processing module (403) for receiving multimodal data, extracting and projecting the original features of the multimodal data, where the multimodal data includes acoustic data, vibration data, and thermal imaging data; A local cross-modal prototype construction module (404) for introducing a gated attention mechanism to dynamically weight the original features to obtain weighted original features, and training the initial prototype according to the multimodal data and the weighted original features to obtain a local cross-modal prototype; A sparse coding module (405) for performing sparse coding on the local cross-modal prototype to obtain coefficient codes and uploading the coefficient codes to the server side; A global prototype construction module (406) for back-projecting the coefficient coding at the server side to construct a global prototype, where the global prototype is used for industrial noise recognition.

9. A computer device, characterized in that, It includes a memory and a processor, and a computer program capable of being loaded and executed by the processor, such as any one of the methods in claims 1 to 7, is stored on the memory.

10. A computer-readable storage medium, characterized in that, A computer program capable of being loaded and executed by the processor, such as any one of the methods in claims 1 to 7, is stored.

Images