Sound source localization model training method, sound source localization method and device

Abstract

This application discloses a sound source localization model training method, a sound source localization method, and an apparatus, belonging to the field of sound signal processing technology. This application is applied to sound source localization scenarios, such as predicting the spatial position of a loosely connected component, wherein the component is mounted on a mechanical connection structure. This application uses sound signals as the data processing object, trains a sound source localization model based on pre-built training samples, and achieves automatic sound source localization based on the trained model. Since this method requires no manual intervention, compared to manual inspection, it not only saves labor costs and avoids the risks of working at heights, but is also more efficient, enabling real-time monitoring of the fastening status of the connected component.

Description

Technical Field

[0001] This application relates to the field of acoustic signal processing technology, and in particular to a sound source localization model training method, a sound source localization method and device. Background Technology

[0002] In tall steel structure systems such as communication base station towers and power transmission towers, high-strength bolts are core connectors that ensure the stability of the tower structure and the safe operation of the facilities. However, due to long-term wind vibration, changes in environmental temperature, and material performance degradation, bolts are likely to loosen. If such hidden dangers are not detected and addressed in time, they will gradually lead to connection failure, structural instability, and may ultimately result in major safety accidents such as tower collapse. Therefore, it is essential to conduct bolt loosening detection on the tower structure.

[0003] Currently, bolt loosening detection still relies primarily on manual inspections. Specifically, technicians periodically climb the tower to visually inspect or tap the bolts to determine if they are loose. However, this method has many limitations: it is inefficient, involves high-altitude work, and cannot provide real-time monitoring of bolt conditions. Furthermore, because loose bolt detection heavily depends on the experience and subjective judgment of technicians, it is difficult to accurately locate loose bolts, easily creating safety hazards. Summary of the Invention

[0004] This application provides a method for training a sound source localization model, a sound source localization method, an apparatus, and a storage medium. The technical solution is shown below.

[0005] On the one hand, a method for training a sound source localization model is provided, the method comprising: Acquire training samples, which include sound signals of a loose connector and a reference spatial position of the connector, which is mounted on a mechanical connection structure; The time-frequency graph of the sound signal is input into the sound source localization model, and the sound source localization model extracts features from the time-frequency graph to obtain a first feature vector; the first feature vector is used to indicate the spatial propagation characteristics of the sound signal and the physical properties of the connector. The sound source localization model converts the first feature vector into attention-weighted features, and the spatial position of the connector is predicted based on the attention-weighted features. The attention-weighted features are used to reflect the key features in the first feature vector that are related to the state of the connector from multiple dimensions. The sound source localization model is trained based on the reference spatial location and the spatial location predicted by the sound source localization model.

[0006] In some embodiments, the sound source localization model includes a multi-head attention module, and the step of converting the first feature vector into attention-weighted features through the sound source localization model includes: For the i-th attention head, based on the linear projection matrix of the i-th attention head and the first feature vector, the query matrix, key matrix, and value matrix of the i-th attention head are generated; wherein, the parameters of the linear projection matrix are updated during model training; i is a positive integer; Based on the query matrix and key matrix of the i-th attention head, the attention weight map of the i-th attention head is obtained; wherein, the attention weight map marks the importance of each feature component in the first feature vector by the attention weight; The output features of the i-th attention head are obtained by weighted summation of the attention weight map and value matrix. The attention-weighted features are generated based on the output features of the multiple attention heads.

[0007] In other embodiments, generating the attention-weighted features based on the output features of the plurality of attention heads includes: The output features of the multiple attention heads are concatenated; The concatenated features are fused based on the output projection matrix to obtain the fused output features; wherein the parameters of the output projection matrix are updated during model training. An addition operation is performed on the first feature vector and the fused output feature, and a layer normalization operation is performed on the added feature to obtain the attention-weighted feature.

[0008] In other embodiments, generating the query matrix, key matrix, and value matrix of the i-th attention head based on the linear projection matrix of the i-th attention head and the first feature vector includes: Based on the first linear projection matrix of the i-th attention head, the first feature vector is projected onto the query space to obtain the query matrix of the i-th attention head; Based on the second linear projection matrix of the i-th attention head, the first feature vector is projected onto the key space to obtain the key matrix of the i-th attention head; Based on the third linear projection matrix of the i-th attention head, the first feature vector is projected onto the value space to obtain the value matrix of the i-th attention head.

[0009] In other embodiments, the step of extracting features from the time-frequency map using the sound source localization model to obtain the first feature vector includes: The spatial propagation characteristics of the sound signal are obtained by extracting features from the time-frequency diagram using the first encoder of the sound source localization model. The second encoder of the sound source localization model is used to extract features from the time-frequency map to obtain the physical property features of the connector. The spatial propagation features and the physical attribute features are fused to obtain the first feature vector.

[0010] In other embodiments, the step of extracting features from the time-frequency map using the first encoder of the sound source localization model to obtain the spatial propagation features of the sound signal includes: An acoustic feature map is generated based on the time-frequency diagram, and the acoustic feature map includes feature values ​​at any spatial location within each feature channel; Global average pooling is performed on the acoustic feature map to obtain channel feature vectors; wherein each component of the channel feature vector is the global pixel mean value within a single feature channel of the acoustic feature map; Based on the channel correlation matrix, the channel feature vectors are compressed to obtain compressed channel feature vectors; the channel correlation matrix is ​​used to learn the weights corresponding to different feature channel combinations. Based on the channel weight reconstruction matrix, the compressed channel feature vectors are dimension-restored to obtain the channel weight vectors; the channel weight reconstruction matrix is ​​used to learn the importance of each feature channel. Based on the channel weight vector, the acoustic feature map is weighted to obtain a new acoustic feature map, which is then converted into the spatial propagation feature.

[0011] In other embodiments, generating the acoustic feature map based on the time-frequency map includes: A non-dilated convolutional layer is used to perform a convolution operation on the time-frequency map to obtain time-frequency features; Multiple parallel dilated convolutional layers are used to perform convolution operations on the time-frequency features to obtain multiple output feature maps; wherein, different dilated convolutional layers have different dilation rates. The multiple output feature maps are concatenated to obtain a multi-scale feature set; The acoustic feature map is obtained by fusing channel-dimensional features of the multi-scale feature set using a multi-scale feature fusion processor.

[0012] In other embodiments, the sound signal is acquired by a microphone array, which includes multiple microphones deployed in different spatial locations; The method further includes: For any channel of the microphone array, the spatial propagation characteristics corresponding to the channel are mapped to three-dimensional spatial coordinates to obtain the first spatial position of the connector; based on the first spatial position and the reference spatial position of the connector relative to the channel, the positioning loss of the channel is determined. The average positioning loss of each channel in the microphone array is obtained to obtain the first positioning loss; The spatial propagation features corresponding to each channel are fused to obtain a second feature vector; the second feature vector is mapped to three-dimensional spatial coordinates to obtain the second spatial position of the connector; The second positioning loss is determined based on the second spatial position and the reference spatial position of the connector relative to the microphone array; The sound source localization model is trained based on the first localization loss and the second localization loss.

[0013] In other embodiments, the second encoder of the sound source localization model extracts features from the time-frequency map to obtain the physical property features of the connector, including: A non-dilated convolutional layer is used to perform a convolution operation on the time-frequency map to obtain time-frequency features; While maintaining frequency resolution, an asymmetric convolution kernel is used to perform a convolution operation on the time-frequency features along the time direction, and the time-frequency features are added to the time-frequency features after the convolution operation through residual connections to obtain new time-frequency features; wherein, the time dimension kernel size of the asymmetric convolution kernel is larger than the frequency dimension kernel size; An attention weight map is generated based on the new time-frequency features; wherein, the attention weight map marks the importance of each time-frequency region in the time-frequency map by attention weight; Based on the attention weight map and the new time-frequency features, the optimized time-frequency features are generated; The physical property features are generated based on the optimized time-frequency features.

[0014] In other embodiments, generating the attention weight map based on the new time-frequency features includes: Perform channel-dimensional average pooling on the new time-frequency features to obtain the average energy distribution of each time-frequency point across all feature channels; Max pooling is performed on the new time-frequency features to obtain the maximum energy distribution of each time-frequency point across all feature channels; The attention weight map is generated based on the average energy distribution and maximum energy distribution at each time frequency point.

[0015] In other embodiments, generating the physical attribute features based on the optimized time-frequency features includes: Global average pooling is performed on the optimized time-frequency features to obtain channel feature vectors; wherein each component of the channel feature vector is the global pixel mean value within a single feature channel of the optimized time-frequency features; Based on the channel feature vector, the physical properties of the connector are predicted to obtain multiple attribute probability distributions; The probability distributions of the multiple attributes are converted into the physical attribute features.

[0016] In other embodiments, converting the plurality of attribute probability distributions into the physical attribute features includes: Based on the semantic projection matrix, the probability distributions of the multiple attributes are mapped to the same continuous semantic subspace to obtain multiple attribute vectors; Obtain the dot product similarity between the query vector and each attribute vector to get the attention score of each physical attribute of the connector; The attention scores of each physical attribute are normalized to obtain the attention weights of each physical attribute. The attention weights of each physical attribute and the multiple attribute vectors are weighted and summed to obtain an attribute fusion vector, which is then converted into the physical attribute features. One of the attribute vectors is used to characterize a physical attribute of the connector; the parameters of the semantic projection matrix and the query vector are updated during model training.

[0017] In other embodiments, the step of mapping the multiple attribute probability distributions to the same continuous semantic subspace based on a semantic projection matrix to obtain multiple attribute vectors includes: Based on the first semantic projection matrix, the probability distribution of the first attribute indicating the connector model is mapped to the continuous semantic subspace to obtain the first attribute vector; the first attribute vector is a continuous semantic feature containing the connector model information. Based on the second semantic projection matrix, the probability distribution of the first attribute indicating the material of the connector is mapped to the continuous semantic subspace to obtain the second attribute vector; the second attribute vector is a continuous semantic feature containing the material information of the connector. Based on the third semantic projection matrix, the probability distribution of the first attribute indicating the fastening state of the connector is mapped to the continuous semantic subspace to obtain the third attribute vector; the third attribute vector is a continuous semantic feature containing the state information of the connector. Based on the fourth semantic projection matrix, the probability distribution of the first attribute indicating the installation position of the connector is mapped to the continuous semantic subspace to obtain the fourth attribute vector; the fourth attribute vector is a continuous semantic feature containing the installation position information of the connector.

[0018] In other embodiments, the step of performing a convolution operation on the time-frequency map using a non-dilated convolutional layer to obtain time-frequency features includes: The time-frequency graph is subjected to data augmentation processing to obtain the processed time-frequency graph; The non-dilated convolutional layer is used to perform a convolution operation on the processed time-frequency map to obtain the time-frequency features.

[0019] In other embodiments, the step of performing data augmentation processing on the time-frequency graph to obtain the processed time-frequency graph includes: Based on the stretching factor, the time axis of the time-frequency graph is scaled. Interpolation is performed on the scaled time axis to fill in the missing data points in the time dimension, resulting in the processed time-frequency graph.

[0020] In other embodiments, the step of performing data augmentation processing on the time-frequency graph to obtain the processed time-frequency graph includes: Select the mask start frequency within the effective frequency range of the time-frequency diagram; Based on the mask width and the mask start frequency, determine the frequency range to be masked; The energy values ​​in the frequency range to be masked in the time-frequency graph are replaced with preset values ​​to obtain the processed time-frequency graph; the preset values ​​are random noise values, the mean background noise of the time-frequency graph, or zero.

[0021] In other embodiments, the method further includes: For any training sample, a first similarity is obtained between the first time-frequency map and the second time-frequency map; the first time-frequency map corresponds to the sound signal included in the training sample, and the second time-frequency map corresponds to the sound signal included in the positive sample; Obtain a second similarity between the first time-frequency map and the third time-frequency map; the third time-frequency map corresponds to the sound signal included in the negative sample. The sound source localization model is trained based on the first similarity and the second similarity. The training samples include sound signals from the first connector in the first state and corresponding to the first sound propagation path; the positive samples include sound signals from the first connector in the first state and corresponding to the second sound propagation path; the negative samples include sound signals from the first connector in the second state, or from a second connector different from the first connector.

[0022] In other embodiments, the training samples also include multiple reference attributes of the connector; the second encoder includes multiple classifiers, different classifiers being used to perform different attribute classification tasks; the training process of the multiple classifiers includes: The physical property features are input into the multiple classifiers, and the multiple classifiers are used to predict the attributes of the connector to obtain multiple predicted attributes of the connector. For any attribute classification task, the classification loss of the attribute classification task is obtained based on the reference attribute and the predicted attribute corresponding to the connector. The multiple classifiers are trained based on the classification loss for each attribute classification task.

[0023] In other embodiments, the sound signal is acquired by a microphone array, which includes multiple microphones deployed in different spatial locations; The spatial location prediction of the connector based on the attention-weighted features includes: The attention-weighted features corresponding to each channel in the microphone array are fused to obtain global fused features; the global fused features are mapped to three-dimensional spatial coordinates to obtain the predicted spatial position of the connector.

[0024] In other embodiments, fusing the attention-weighted features corresponding to each channel in the microphone array to obtain global fused features includes: Multiple nonlinear transformations are performed on the attention weighted features corresponding to each channel to obtain nonlinear transformation results; wherein, the nonlinear transformation results are used to characterize the correlation between the attention weighted features corresponding to each channel; The nonlinear transformation result is dimensionally compressed, and a layer normalization operation is performed on the dimensionally compressed nonlinear transformation result to obtain the global fusion feature.

[0025] In other embodiments, the sound source localization model includes a feature fusion network; the step of performing multiple nonlinear transformations on the attention-weighted features corresponding to each channel to obtain the nonlinear transformation result includes: Based on the first layer weight matrix of the feature fusion network, a nonlinear transformation is performed on the attention-weighted features corresponding to each channel to obtain intermediate transformation results; the first layer weight matrix is ​​used to learn the basic correlation between the attention-weighted features corresponding to each channel. Based on the second-layer weight matrix of the feature fusion network, the intermediate transformation result is subjected to a nonlinear transformation to obtain the nonlinear transformation result; the second-layer weight matrix is ​​used to learn the higher-order correlation of the attention-weighted features corresponding to each channel after the first nonlinear transformation.

[0026] In other embodiments, the sound source localization model includes a coordinate regression network; the step of mapping the global fusion features to three-dimensional spatial coordinates to obtain the predicted spatial location of the connector includes: Based on the first layer weight matrix of the coordinate regression network, the global fusion features are mapped to a first intermediate representation; the first layer weight matrix is ​​used to learn to map the global fusion features to an intermediate physical quantity. Based on the second-layer weight matrix of the coordinate regression network, the first intermediate representation is mapped to the second intermediate representation; the second-layer weight matrix is ​​used to learn the mutual influence between the intermediate physical quantities. The second intermediate representation is mapped to the predicted spatial location of the connector.

[0027] In other embodiments, the training process of the coordinate regression network includes: Based on the reference spatial position and the predicted spatial position of the connector, the main loss is obtained; An auxiliary loss is obtained based on a height constraint term and a horizontal distance constraint term; wherein, the height constraint term is used to constrain the predicted height of the connector to not exceed the height of the mechanical connection structure; the horizontal distance constraint term is used to constrain the horizontal radius of the predicted position to not exceed the minimum horizontal radius of the mechanical connection structure; The coordinate regression network is trained based on the main loss and the auxiliary loss.

[0028] On the other hand, a sound source localization method is provided, the method comprising: Acquire a sound signal to be detected; wherein the sound signal to be detected is acquired by microphones deployed around the mechanical connection structure to be detected, and the mechanical connection structure is equipped with connectors; The time-frequency graph of the sound signal to be detected is input into the sound source localization model, and the spatial location of the loose connector is predicted by the sound source localization model; wherein, the sound source localization model is trained by the above-mentioned training method.

[0029] In some embodiments, the number of microphones is multiple, with different microphones deployed at different spatial locations around the mechanical structure; the method further includes: When the signal energy of any microphone channel exceeds the energy threshold, multiple microphones are triggered to collect a sound signal of a preset duration. The sound signal to be detected includes multiple sound signals with a duration of the preset duration.

[0030] In other embodiments, the step of inputting the time-frequency graph of the sound signal to be detected into a sound source localization model, and predicting the spatial location of the loose connector using the sound source localization model, includes: Perform time-frequency analysis on each segment of the audio signal with a preset duration to obtain multiple time-frequency diagrams; The multiple time-frequency maps are input into the sound source localization model, and the spatial location of the loose connector is predicted by the sound source localization model.

[0031] On the other hand, a sound source localization model training device is provided, the device comprising: The first acquisition module is configured to acquire training samples, the training samples including sound signals of a loose connector and a reference spatial position of the connector, the connector being mounted on a mechanical connection structure; The first processing module is configured to input the time-frequency diagram of the sound signal into the sound source localization model, and extract features from the time-frequency diagram through the sound source localization model to obtain a first feature vector; the first feature vector is used to indicate the spatial propagation characteristics of the sound signal and the physical properties of the connector. The second processing module is configured to convert the first feature vector into attention-weighted features using the sound source localization model; the attention-weighted features are used to reflect the key features in the first feature vector related to the state of the connector from multiple dimensions. The prediction module is configured to predict the spatial location of the connector based on the attention-weighted features and the sound source localization model. The training module is configured to train the sound source localization model based on the reference spatial location and the spatial location predicted by the sound source localization model.

[0032] In some embodiments, the sound source localization model includes a multi-head attention module, and the second processing module is configured to: For the i-th attention head, based on the linear projection matrix of the i-th attention head and the first feature vector, the query matrix, key matrix, and value matrix of the i-th attention head are generated; wherein, the parameters of the linear projection matrix are updated during model training; i is a positive integer; Based on the query matrix and key matrix of the i-th attention head, the attention weight map of the i-th attention head is obtained; wherein, the attention weight map marks the importance of each feature component in the first feature vector by the attention weight; The output features of the i-th attention head are obtained by weighted summation of the attention weight map and value matrix. The attention-weighted features are generated based on the output features of the multiple attention heads.

[0033] In other embodiments, the second processing module is configured to: The output features of the multiple attention heads are concatenated; The concatenated features are fused based on the output projection matrix to obtain the fused output features; wherein the parameters of the output projection matrix are updated during model training. An addition operation is performed on the first feature vector and the fused output feature, and a layer normalization operation is performed on the added feature to obtain the attention-weighted feature.

[0034] In other embodiments, the second processing module is configured to: Based on the first linear projection matrix of the i-th attention head, the first feature vector is projected onto the query space to obtain the query matrix of the i-th attention head; Based on the second linear projection matrix of the i-th attention head, the first feature vector is projected onto the key space to obtain the key matrix of the i-th attention head; Based on the third linear projection matrix of the i-th attention head, the first feature vector is projected onto the value space to obtain the value matrix of the i-th attention head.

[0035] In other embodiments, the first processing module includes: The first processing submodule is configured to extract features from the time-frequency diagram using the first encoder of the sound source localization model to obtain the spatial propagation features of the sound signal. The second processing submodule is configured to extract features from the time-frequency diagram using the second encoder of the sound source localization model to obtain the physical property features of the connector. The fusion submodule is configured to fuse the spatial propagation features and the physical attribute features to obtain the first feature vector.

[0036] In other embodiments, the first processing submodule is configured to: An acoustic feature map is generated based on the time-frequency diagram, and the acoustic feature map includes feature values ​​at any spatial location within each feature channel; Global average pooling is performed on the acoustic feature map to obtain channel feature vectors; wherein each component of the channel feature vector is the global pixel mean value within a single feature channel of the acoustic feature map; Based on the channel correlation matrix, the channel feature vectors are compressed to obtain compressed channel feature vectors; the channel correlation matrix is ​​used to learn the weights corresponding to different feature channel combinations. Based on the channel weight reconstruction matrix, the compressed channel feature vectors are dimension-restored to obtain the channel weight vectors; the channel weight reconstruction matrix is ​​used to learn the importance of each feature channel. Based on the channel weight vector, the acoustic feature map is weighted to obtain a new acoustic feature map, which is then converted into the spatial propagation feature.

[0037] In other embodiments, the first processing submodule is configured to: A non-dilated convolutional layer is used to perform a convolution operation on the time-frequency map to obtain time-frequency features; Multiple parallel dilated convolutional layers are used to perform convolution operations on the time-frequency features to obtain multiple output feature maps; wherein, different dilated convolutional layers have different dilation rates. The multiple output feature maps are concatenated to obtain a multi-scale feature set; The acoustic feature map is obtained by fusing channel-dimensional features of the multi-scale feature set using a multi-scale feature fusion processor.

[0038] In other embodiments, the sound signal is acquired by a microphone array, which includes multiple microphones deployed in different spatial locations; The prediction module is also configured to map the spatial propagation features corresponding to any channel of the microphone array into three-dimensional spatial coordinates to obtain the first spatial position of the connector. The training module is also configured to determine the positioning loss of the channel based on the first spatial position and the reference spatial position of the connector relative to the channel; The training module is also configured to obtain the average localization loss of each channel in the microphone array to obtain the first localization loss; The prediction module is further configured to fuse the spatial propagation features corresponding to each channel to obtain a second feature vector; and to map the second feature vector to three-dimensional spatial coordinates to obtain the second spatial position of the connector. The training module is further configured to determine a second localization loss based on the second spatial position and the reference spatial position of the connector relative to the microphone array; The training module is also configured to train the sound source localization model based on the first localization loss and the second localization loss.

[0039] In other embodiments, the second processing submodule is configured to: A non-dilated convolutional layer is used to perform a convolution operation on the time-frequency map to obtain time-frequency features; While maintaining frequency resolution, an asymmetric convolution kernel is used to perform a convolution operation on the time-frequency features along the time direction, and the time-frequency features are added to the time-frequency features after the convolution operation through residual connections to obtain new time-frequency features; wherein, the time dimension kernel size of the asymmetric convolution kernel is larger than the frequency dimension kernel size; An attention weight map is generated based on the new time-frequency features; wherein, the attention weight map marks the importance of each time-frequency region in the time-frequency map by attention weight; Based on the attention weight map and the new time-frequency features, the optimized time-frequency features are generated; The physical property features are generated based on the optimized time-frequency features.

[0040] In other embodiments, the second processing submodule is configured to: Perform channel-dimensional average pooling on the new time-frequency features to obtain the average energy distribution of each time-frequency point across all feature channels; Max pooling is performed on the new time-frequency features to obtain the maximum energy distribution of each time-frequency point across all feature channels; The attention weight map is generated based on the average energy distribution and maximum energy distribution at each time frequency point.

[0041] In other embodiments, the second processing submodule is configured to: Global average pooling is performed on the optimized time-frequency features to obtain channel feature vectors; wherein each component of the channel feature vector is the global pixel mean value within a single feature channel of the optimized time-frequency features; Based on the channel feature vector, the physical properties of the connector are predicted to obtain multiple attribute probability distributions; The probability distributions of the multiple attributes are converted into the physical attribute features.

[0042] In other embodiments, the second processing submodule is configured to: Based on the semantic projection matrix, the probability distributions of the multiple attributes are mapped to the same continuous semantic subspace to obtain multiple attribute vectors; Obtain the dot product similarity between the query vector and each attribute vector to get the attention score of each physical attribute of the connector; The attention scores of each physical attribute are normalized to obtain the attention weights of each physical attribute. The attention weights of each physical attribute and the multiple attribute vectors are weighted and summed to obtain an attribute fusion vector, which is then converted into the physical attribute features. One of the attribute vectors is used to characterize a physical attribute of the connector; the parameters of the semantic projection matrix and the query vector are updated during model training.

[0043] In other embodiments, the second processing submodule is configured to: Based on the first semantic projection matrix, the probability distribution of the first attribute indicating the connector model is mapped to the continuous semantic subspace to obtain the first attribute vector; the first attribute vector is a continuous semantic feature containing the connector model information. Based on the second semantic projection matrix, the probability distribution of the first attribute indicating the material of the connector is mapped to the continuous semantic subspace to obtain the second attribute vector; the second attribute vector is a continuous semantic feature containing the material information of the connector. Based on the third semantic projection matrix, the probability distribution of the first attribute indicating the fastening state of the connector is mapped to the continuous semantic subspace to obtain the third attribute vector; the third attribute vector is a continuous semantic feature containing the state information of the connector. Based on the fourth semantic projection matrix, the probability distribution of the first attribute indicating the installation position of the connector is mapped to the continuous semantic subspace to obtain the fourth attribute vector; the fourth attribute vector is a continuous semantic feature containing the installation position information of the connector.

[0044] In other embodiments, the second processing submodule is configured to: The time-frequency graph is subjected to data augmentation processing to obtain the processed time-frequency graph; The non-dilated convolutional layer is used to perform a convolution operation on the processed time-frequency map to obtain the time-frequency features.

[0045] In other embodiments, the second processing submodule is configured to: Based on the stretching factor, the time axis of the time-frequency graph is scaled. Interpolation is performed on the scaled time axis to fill in the missing data points in the time dimension, resulting in the processed time-frequency graph.

[0046] In other embodiments, the second processing submodule is configured to: Select the mask start frequency within the effective frequency range of the time-frequency diagram; Based on the mask width and the mask start frequency, determine the frequency range to be masked; The energy values ​​in the frequency range to be masked in the time-frequency graph are replaced with preset values ​​to obtain the processed time-frequency graph; the preset values ​​are random noise values, the mean background noise of the time-frequency graph, or zero.

[0047] In other embodiments, the training module is further configured to: For any training sample, a first similarity is obtained between the first time-frequency map and the second time-frequency map; the first time-frequency map corresponds to the sound signal included in the training sample, and the second time-frequency map corresponds to the sound signal included in the positive sample; Obtain a second similarity between the first time-frequency map and the third time-frequency map; the third time-frequency map corresponds to the sound signal included in the negative sample. The sound source localization model is trained based on the first similarity and the second similarity. The training samples include sound signals from the first connector in the first state and corresponding to the first sound propagation path; the positive samples include sound signals from the first connector in the first state and corresponding to the second sound propagation path; the negative samples include sound signals from the first connector in the second state, or from a second connector different from the first connector.

[0048] In other embodiments, the training samples also include multiple reference attributes of the connector; the second encoder includes multiple classifiers, different classifiers being used to perform different attribute classification tasks; the training process of the multiple classifiers includes: The physical property features are input into the multiple classifiers, and the multiple classifiers are used to predict the attributes of the connector to obtain multiple predicted attributes of the connector. For any attribute classification task, the classification loss of the attribute classification task is obtained based on the reference attribute and the predicted attribute corresponding to the connector. The multiple classifiers are trained based on the classification loss for each attribute classification task.

[0049] In other embodiments, the prediction module is configured to: The attention-weighted features corresponding to each channel in the microphone array are fused to obtain global fused features; the global fused features are mapped to three-dimensional spatial coordinates to obtain the predicted spatial position of the connector.

[0050] In other embodiments, the prediction module is configured to: Multiple nonlinear transformations are performed on the attention weighted features corresponding to each channel to obtain nonlinear transformation results; wherein, the nonlinear transformation results are used to characterize the correlation between the attention weighted features corresponding to each channel; The nonlinear transformation result is dimensionally compressed, and a layer normalization operation is performed on the dimensionally compressed nonlinear transformation result to obtain the global fusion feature.

[0051] In other embodiments, the sound source localization model includes a feature fusion network; the prediction module is configured to: Based on the first layer weight matrix of the feature fusion network, a nonlinear transformation is performed on the attention-weighted features corresponding to each channel to obtain intermediate transformation results; the first layer weight matrix is ​​used to learn the basic correlation between the attention-weighted features corresponding to each channel. Based on the second-layer weight matrix of the feature fusion network, the intermediate transformation result is subjected to a nonlinear transformation to obtain the nonlinear transformation result; the second-layer weight matrix is ​​used to learn the higher-order correlation of the attention-weighted features corresponding to each channel after the first nonlinear transformation.

[0052] In other embodiments, the sound source localization model includes a coordinate regression network; the prediction module is configured to: Based on the first layer weight matrix of the coordinate regression network, the global fusion features are mapped to a first intermediate representation; the first layer weight matrix is ​​used to learn to map the global fusion features to an intermediate physical quantity. Based on the second-layer weight matrix of the coordinate regression network, the first intermediate representation is mapped to the second intermediate representation; the second-layer weight matrix is ​​used to learn the mutual influence between the intermediate physical quantities. The second intermediate representation is mapped to the predicted spatial location of the connector.

[0053] In other embodiments, the training process of the coordinate regression network includes: Based on the reference spatial position and the predicted spatial position of the connector, the main loss is obtained; An auxiliary loss is obtained based on a height constraint term and a horizontal distance constraint term; wherein, the height constraint term is used to constrain the predicted height of the connector to not exceed the height of the mechanical connection structure; the horizontal distance constraint term is used to constrain the horizontal radius of the predicted position to not exceed the minimum horizontal radius of the mechanical connection structure; The coordinate regression network is trained based on the main loss and the auxiliary loss.

[0054] On the other hand, a sound source localization device is provided, the device comprising: The second acquisition module is configured to acquire a sound signal to be detected; wherein the sound signal to be detected is acquired by microphones deployed around the mechanical connection structure to be detected, and the mechanical connection structure is equipped with connectors; The sound source localization module is configured to input the time-frequency diagram of the sound signal to be detected into the sound source localization model, and predict the spatial position of the loose connector through the sound source localization model; wherein the sound source localization model is trained by the aforementioned training device.

[0055] In some embodiments, the number of microphones is multiple, with different microphones deployed at different spatial locations around the mechanical structure; the device further includes: The control module is configured to trigger multiple microphones to collect a sound signal of a preset duration when the signal energy of any microphone channel exceeds the energy threshold. The sound signal to be detected includes multiple sound signals with a duration of the preset duration.

[0056] In other embodiments, the sound source localization module is configured to: Perform time-frequency analysis on each segment of the audio signal with a preset duration to obtain multiple time-frequency diagrams; The multiple time-frequency maps are input into the sound source localization model, and the spatial location of the loose connector is predicted by the sound source localization model.

[0057] On the other hand, a computer device is provided, the device including a processor and a memory, the memory storing computer program code, the computer program code being loaded and executed by the processor to implement the above-described training method or sound source localization method for the sound source localization model.

[0058] On the other hand, a computer-readable storage medium is provided, wherein computer program code is stored in the storage medium, and the computer program code is loaded and executed by the processor of a computer device to implement the above-described training method or sound source localization method for the sound source localization model.

[0059] On the other hand, a computer program product is provided, the computer program product including computer program code stored in a computer-readable storage medium, a processor of a computer device reading the computer program code from the computer-readable storage medium, the processor executing the computer program code, causing the computer device to execute the above-described training method or sound source localization method for the sound source localization model.

[0060] This application embodiment is applied to sound source localization scenarios, such as predicting the spatial location of a loose connector, where the connector is mounted on a mechanical connection structure. Specifically, this application embodiment uses sound signals as the data processing object and trains a sound source localization model based on pre-built training samples, achieving automatic sound source localization based on the trained model. Since this method requires no manual intervention, compared to manual inspection, it not only saves labor costs and avoids the risks of working at heights, but is also more efficient, enabling real-time monitoring of the connector's tightness. Furthermore, during model training, this application embodiment extracts feature vectors containing sound spatial propagation characteristics and the connector's physical properties based on the time-frequency diagram of the sound signal. An attention mechanism is then introduced to automatically strengthen features in the feature vector that are crucial for sound source localization, while suppressing irrelevant noise or interference features. This allows the model to focus on core discriminative features, effectively reducing feature redundancy under complex conditions, resulting in a sound source localization model with higher localization accuracy during the inference phase. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a schematic diagram of an implementation environment for locating the sound source of loose bolts installed on a tower structure, provided in an embodiment of this application. Figure 2 This is a schematic diagram of the training process of a sound source localization model provided in an embodiment of this application; Figure 3 This is a flowchart of a training method for a sound source localization model provided in an embodiment of this application; Figure 4 This is a schematic diagram of a channel feature vector provided in an embodiment of this application; Figure 5 This is a flowchart of a sound source localization method provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a training device for a sound source localization model provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a sound source localization device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0064] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items that have essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or execution order. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms.

[0065] These terms are simply used to distinguish one element from another. For example, without departing from the various examples, the first element can be referred to as the second element, and similarly, the second element can be referred to as the first element. Both the first and second elements can be elements, and in some cases, they can be separate and distinct elements.

[0066] "At least one" refers to one or more elements. For example, at least one element can be one element, two elements, three elements, or any integer number of elements greater than or equal to one. "Multiple" refers to two or more elements. For example, multiple elements can be two elements, three elements, or any integer number of elements greater than or equal to two.

[0067] In this article, "and / or" indicates that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0068] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0069] In this embodiment, the sound source localization object is a loose connector, which is mounted on a mechanical connection structure. As an example, the mechanical connection structure is a tower structure such as a communication base station tower or a power transmission tower, and the connector is a bolt mounted on the tower that has become loose.

[0070] It should be noted that this article only uses a loose bolt as the sound source localization object and a tower structure as the mechanical connection structure as an example to introduce the training process of the sound source localization model and the process of sound source localization based on the trained sound source localization model. Besides this example, the sound source localization scheme provided in this application can also be applied to other sound source localization scenarios with similar needs, and this application does not limit it in this regard.

[0071] Besides the aforementioned method of manually inspecting for bolt loosening, a vibration sensor-based method has been proposed. This involves installing accelerometers at key locations on the tower structure and then analyzing the overall structural vibration characteristics to infer the bolt condition at each location. However, this method has the following drawbacks: (1) Ambiguous positioning: It is difficult to separate the weak loosening signal from the complex structural vibration background noise and accurately locate the specific bolt; (2) Insufficient detection sensitivity: For early and slight loosening of bolts, the vibration signal changes are not obvious, and it is easy to miss the detection; (3) High deployment and maintenance costs: This position detection method requires the installation of a large number of acceleration sensors and a lot of wiring.

[0072] It should be noted that, based on the aforementioned problems in the location detection of loose bolts using related technologies, this application proposes a scheme for sound source localization of loose bolts on a tower body based on an adaptive attention mechanism. This scheme constructs a deep learning network (which, based on an adaptive attention mechanism, can dynamically focus on the acoustic features most relevant to the currently input sound signal), adaptively learning and fusing acoustic features related to the spatial propagation characteristics of sound and the physical properties of the bolts, ultimately achieving accurate three-dimensional spatial localization of loose bolts in a strong noise background.

[0073] Typically, sound signal-based location detection of loose bolts faces significant challenges due to environmental interference and acoustic signature diversity. Environmental interference is a concern, as the tower's surroundings often include strong background noise such as wind, electromagnetic interference, and bird calls. Acoustic signature diversity is another challenge, as bolts at different locations (e.g., tower feet, crossbeams, or tower top) and of different types (e.g., M20, M24, M30) exhibit significantly different acoustic signature characteristics (frequency characteristics, attenuation properties, or propagation modes) when they become loose. However, the sound source localization scheme for loose bolts provided in this application not only overcomes the problem of susceptibility to environmental noise interference during sound source localization but also effectively addresses the acoustic signature diversity caused by differences in bolt location and type, achieving accurate and robust localization of loose bolts even in complex environments.

[0074] Figure 1This is a schematic diagram of an implementation environment for locating the sound source of loose bolts installed on a tower body, as provided in an embodiment of this application. See also... Figure 1 The implementation environment includes: A first computer device 110 for training a sound source localization model, a second computer device 120 for locating the sound source of loose bolts on the tower body, and a tower structure 130.

[0075] In this embodiment, non-contact sensors, such as a microphone array, are deployed around the tower structure 130. During the training phase, a first computer device 110 is used to train a sound source localization model based on a constructed sample dataset. During the inference phase, a second computer device 120 is used to perform a sound source localization task targeting loose bolts on the tower structure 130 based on the trained sound source localization model.

[0076] It should be noted that the first computer device 110 and the second computer device 120 can be terminal nodes, edge computing nodes, or servers (belonging to cloud computing centers), etc., and this application does not limit them in this regard. Furthermore, the first computer device 110 and the second computer device 120 can be the same device or different devices. When the first computer device 110 and the second computer device 120 are different devices, the first computer device 110 can be an edge computing node or a server, while the second computer device 120 can be a terminal node.

[0077] The following is based on Figure 2 The training process of the sound source localization model provided in the embodiments of this application is introduced.

[0078] Step 1: Data preparation and preprocessing. This step is used to provide standardized input for subsequent steps.

[0079] Step 2: Feature extraction and adaptive attention mechanism based on dual-path encoder.

[0080] This step is the core step, such as... Figure 2 As shown, the process includes steps 2.1 and 2.2. Step 2.1 is used to extract features from the input time-frequency map (output of step 1) based on a dual-path encoder. Further, step 2.1 includes extracting spatial propagation features of sound based on a time-space encoder and extracting physical attribute features of the bolt based on a acoustic signature encoder. Step 2.2 includes feature concatenation and an adaptive attention mechanism.

[0081] Step 3: Feature fusion, decoding and localization.

[0082] like Figure 2As shown, step 3 includes steps 3.1 and 3.2. Step 3.1 is used to perform cross-channel feature fusion, and step 3.2 is used to perform localization coordinate regression.

[0083] In summary, the entire model training process is an end-to-end deep learning workflow with data-driven, adaptive attention mechanisms. The training process of the sound source localization model and the process of sound source localization based on the trained model are described in detail below through the following implementation methods.

[0084] Figure 3 This is a flowchart illustrating a training method for a sound source localization model provided in an embodiment of this application. The method is executed by a computer device, such as... Figure 1 The first computer device 110 in the system. See also Figure 3 The method includes the following steps.

[0085] 301. The computer equipment acquires training samples, which include sound signals of bolts in a loose state and reference spatial positions of the bolts, wherein the bolts are installed on the tower body.

[0086] This step corresponds to Figure 2 Step 1 in the process is used for data preparation and preprocessing.

[0087] In this embodiment, the number of training samples used to train the sound source localization model is multiple. Each training sample contains a sound signal of a loose bolt and three-dimensional spatial coordinates labeled for this sound signal. These three-dimensional spatial coordinates are referred to herein as the reference spatial location, the true spatial location, or the true installation location.

[0088] For the data preparation stage, in an experimental or simulation environment, bolts of different types and materials can be used in different spatial locations to simulate different degrees of loosening, and the abnormal noises produced by the loose bolts can be collected. At the same time, the three-dimensional spatial coordinates of the loose bolts (sound source) can be accurately measured to construct a labeled sample dataset.

[0089] It should be noted that this sample dataset includes multiple training samples, which cover various environmental noise scenarios. Furthermore, this solution uses a microphone array to collect abnormal noises from loose bolts. The noise produced by the same loose bolt is simultaneously captured by each microphone in the array, generating multiple time-domain signals, each corresponding to a different channel of the microphone array.

[0090] As an example, preprocessing audio signals includes, but is not limited to, the following methods: 1. DC removal and filtering: Remove the DC component of the signal and use a bandpass filter (e.g., 300Hz–8kHz band) to preprocess each segment of the acquired sound signal to suppress low-frequency wind noise and high-frequency interference, while retaining the characteristic frequency band signal of bolt loosening noise.

[0091] 2. Framing and Windowing: Each acquired audio signal (time domain signal) is framed and windowed (e.g., Hamming window) to prepare for subsequent time-frequency analysis.

[0092] 3. Generate time-frequency diagram: Perform a short-time Fourier transform on each frame of signal and convert the time-domain signal of each microphone channel into a time-frequency diagram (Spectrogram) to serve as input for the sound source localization model.

[0093] At this point, each sound event (the noise caused by a loose bolt) will be represented as a feature tensor [N, F, T]. Here, N refers to the number of channels in the microphone array, F refers to the frequency dimension (e.g., 256 frequency points), and T refers to the number of time frames (e.g., 128 frames).

[0094] 302. The computer equipment inputs the time-frequency diagram of the sound signal into the sound source localization model, and extracts features from the input time-frequency diagram through the sound source localization model to obtain the first feature vector; wherein, the first feature vector is used to indicate the spatial propagation characteristics of the sound signal and the physical properties of the bolt.

[0095] This step corresponds to Figure 2 Step 2 in the process includes feature splicing in steps 2.1 and 2.2.

[0096] This application embodiment constructs a dual-path encoder to extract features from the input time-frequency map. Path one corresponds to a spatial-temporal encoder, which uses ordinary 2D convolution to process the time-frequency map of each microphone channel separately. In this application embodiment, the two-dimensional convolutional layer of this encoder (non-dilated convolutional layer, performing ordinary 2D convolution) is responsible for extracting the basic frequency domain features of the sound signal, such as harmonic features and formant features; the deep convolutional layer of this encoder (such as multi-scale dilated convolutional layer) captures high-level features strongly correlated with the spatial propagation characteristics of sound, including key information such as signal attenuation patterns and multipath effect features, through layer-by-layer feature dimensionality reduction and abstraction.

[0097] The goal of Path 1 is to focus on extracting spatial propagation information contained in sound signals, such as the implicit feature representation of Time Difference of Arrival (TDoA), signal attenuation patterns during sound wave propagation, and interference features caused by multipath propagation. These features directly point to the spatial location of the sound source. Accordingly, the output of Path 1 is used to encode spatial location information of "where the sound is coming from," providing a core basis for subsequent sound source localization.

[0098] In this embodiment of the application, the sound signals included in each training sample correspond to N time-frequency diagrams. The input to the space-time encoder (also known as the first encoder) is the time-frequency plot of a single microphone channel. Where i refers to the microphone channel index (i=1,2,...,N).

[0099] In this embodiment of the application, a first feature vector is obtained by extracting features from the input time-frequency map using a sound source localization model. This includes steps 3021 and 3022 as described below.

[0100] 3021. Using the first encoder of the sound source localization model, feature extraction is performed on the input time-frequency map to obtain the spatial propagation features of the sound. .

[0101] This process includes steps 3021-1 to 3021-4 below.

[0102] 3021-1. Generating Acoustic Feature Maps Based on Input Time-Frequency Maps Among them, acoustic feature map It includes the feature values ​​at any spatial location within each feature channel.

[0103] Shared layer As mentioned earlier, the first encoder uses two-dimensional convolutional layers to process the time-frequency map of each microphone channel individually. That is, a two-dimensional convolutional layer is first used to perform ordinary 2D convolution operations on the input time-frequency map. For example, small-sized convolutional kernels such as 3x3 are used in the first 2-3 layers to focus on basic edge and texture feature extraction in order to capture local feature patterns in the time-frequency map such as transient impulses and continuous harmonics.

[0104] Taking a 2D-CNN design with two layers of two-dimensional convolutional layers as an example, time-frequency features can be extracted using the following formulas 1 and 2.

[0105] Formula 1:

[0106] Formula 2:

[0107] in, This refers to the time-frequency diagram of the i-th microphone channel; The size refers to a 3×3 convolution kernel; stride=1 means that the convolution kernel moves 1 pixel at a time. This refers to batch normalization, which normalizes each channel of the convolution output separately, accelerating training convergence and improving stability. This refers to modifying the activation function of linear units, introducing nonlinearity to enable the network to learn complex patterns; This refers to the feature map of the i-th microphone channel after the first convolutional layer; stride=2 means that the convolutional kernel moves by 2 pixels each time. This stride configuration will perform downsampling on the feature map, which can effectively reduce the size of the feature map while retaining the core feature information. It refers to the feature map of the i-th microphone channel after the second convolutional layer.

[0108] It should be noted that this stage extracts basic frequency domain features, including impact features, harmonic features, and resonance peak features. Among them, impact features are used for time positioning to accurately identify the moment of loosening and provide a precise time reference for TDOA; harmonic features are used for bolt type identification, as different bolts have different fundamental frequencies and harmonic structures, which can help distinguish different bolt types such as M20 and M24; resonance peak features are used for location inference, as the resonant frequencies of bolts at different locations are affected by the structure differently, and the resonant characteristics of tower foot bolts and tower top bolts are different.

[0109] It should be noted that the spatial-temporal encoder of Path 1 and the voiceprint attribute encoder of Path 2 share a two-dimensional convolutional layer for extracting basic frequency domain features to improve feature extraction efficiency. Therefore, the two-dimensional convolutional layer is also called a shared layer. Path 1 and Path 2 begin to diverge after the shared layer; Path 1 uses network modules that place greater emphasis on temporal and inter-channel relationships.

[0110] Multi-scale void convolutional layer In obtaining the time-frequency characteristics H of the sound signal ( ¹ ) After that, the time-frequency feature H ( ¹ ) The input will be processed by a multi-scale dilated convolutional layer of a spatial-temporal encoder, employing multiple parallel dilated convolutional layers to process the time-frequency features H separately. ( ¹ ) Perform convolution operations to obtain multiple output feature maps.

[0111] Different dilated convolutional layers have different dilation rates (also known as expansion rates). As an example, this scheme uses three parallel dilated convolutional branches, that is, three 3×3 dilated convolutional layers (dilation rates of 1, 2, and 4, respectively) to process the time-frequency feature H in parallel.( ¹ ) .

[0112] A dilation rate of 1 corresponds to ordinary convolution (without dilation), resulting in a small receptive field that captures fine temporal and frequency structures. A dilation rate of 2 or 4 expands the receptive field, allowing for the extraction of local, mid-range, and long-range acoustic feature patterns in parallel. Increasing the dilation rate exponentially expands the receptive field, but the number of parameters remains unchanged.

[0113] The purpose of performing multi-scale dilated convolution is that a larger receptive field can effectively capture the long-range dependencies between spatially distant feature points in the time-frequency map. This is crucial for analyzing the multipath effects that occur as sound propagates from the sound source to the microphone. These effects typically manifest as repetition or tailing of specific feature patterns in the time-frequency map. These multipath propagation patterns are strongly correlated with the spatial layout of the reflector (i.e., the tower structure).

[0114] In this embodiment of the application, the time-frequency feature H ( ¹ ) The process of inputting into a multi-scale dilated convolutional layer and performing convolution operations in parallel can be expressed by the following formula 3-5.

[0115] Formula 3:

[0116] Formula 4:

[0117] Formula 5:

[0118] In this context, Formula 3 corresponds to a dilatation rate of 1, which is the standard convolution; Formula 4 corresponds to a dilatation rate of 2; and Formula 5 corresponds to a dilatation rate of 4. It refers to the feature map obtained after processing the time-frequency maps of N microphone channels through a two-dimensional convolutional layer. F refers to the frequency dimension (e.g., 128 frequency points); T refers to the time dimension (e.g., 64 frames); and C refers to the number of feature channels (e.g., 64).

[0119] in addition, , , These are multi-scale feature detectors, as shown in Table 1 below. Used for local time-frequency pattern detection (with a small receptive field, such as 3). 3) Accordingly, This is a local acoustic feature map, which contains a fine time-frequency structure; Used for medium-range time-frequency pattern detection (medium receptive field, e.g., 5). 5), Accordingly, This is a mid-range acoustic feature map, containing preliminary characteristics of the propagation effect; Used for long-range time-frequency pattern detection (with a large receptive field, for example), accordingly, It is a long-range acoustic feature map that can capture complex multipath and reverberation effects.

[0120] Table 1

[0121] Next, multiple output feature maps are concatenated using Equation 6 to obtain a multi-scale feature set M. Finally, the multi-scale feature set M is fused along the channel dimension using Equation 7 via a multi-scale feature fusion processor to obtain the acoustic feature map. .

[0122] Formula 6:

[0123] Since this scheme integrates feature information from three scales to form a multi-scale feature set M, it provides a rich source of information for subsequent feature fusion.

[0124] Formula 7:

[0125] In Formula 7, This refers to a multi-scale feature fusion processor, which learns how to fuse 96 multi-scale feature channels into 64 more meaningful feature channels, achieving cross-scale information integration. (Acoustic feature map) Multi-scale propagation characteristics, integrating local, mid-range, and long-range acoustic information, encompass various effects during sound propagation, providing high-quality input for subsequent attention mechanisms. Among these, , , This refers to the spatial position of the c-th feature channel. The eigenvalue at that location.

[0126] Channel attention module This module is used for acoustic feature maps. The channel attention mechanism is applied. As an example, first, global average pooling is performed to obtain a vector representing the global information of each feature channel; then, channel weights are learned through a fully connected layer; finally, the learned channel weights are used to process the acoustic feature map input to the channel attention module. Feature recalibration is then performed, which involves weighting the feature channels. These three steps are described below.

[0127] 3021-2, Global Information Extraction: Extraction of Acoustic Feature Map Perform global average pooling to obtain channel feature vectors. Among them, channel feature vectors Each component is an acoustic feature map The global pixel mean within a single feature channel.

[0128] In this embodiment, global average pooling is used to pool the two-dimensional features of each feature channel using the following formula 8. Compress it into a scalar.

[0129] Formula 8:

[0130] in, "C" refers to the global statistics of the c-th feature channel; "H" refers to the acoustic feature map. The height corresponds to the frequency dimension in the time-frequency diagram; W refers to the acoustic feature map. The width corresponds to the time dimension in a time-frequency graph.

[0131] By applying Formula 8 above to all feature channels to calculate the global statistics, we can obtain the following vector z.

[0132] Formula 9:

[0133] In Formula 9, C refers to the total number of feature channels; z refers to the global statistical vector of the channels, i.e. This represents the average activation intensity of the c-th feature channel across the entire time-frequency plane, reflecting the overall importance of the corresponding frequency range.

[0134] It should be noted that the specific significance of global information extraction in loose bolt localization is that each feature channel corresponds to a specific set of frequency filter responses. The higher the value, the more prominent the frequency pattern corresponding to the c-th feature channel is in the input audio signal. For example, suppose... The high value indicates that low-frequency components (such as the 0-500Hz frequency band) dominate the input sound signal.

[0135] 3021-3, Channel Weight Learning: Based on the channel correlation matrix W1, the channel feature vectors are processed... Dimensional compression is performed to obtain the compressed channel feature vectors. ; and, based on the channel weight reconstruction matrix W2, the compressed channel feature vectors Dimension recovery is performed to obtain the channel weight vector s.

[0136] In this embodiment, the channel correlation matrix W1 is used to learn the weights corresponding to different feature channel combinations; that is, this matrix is ​​used to discover feature channel combinations that play a key role in locating loose bolts. The channel weight reconstruction matrix W2 is used to learn the importance of each feature channel.

[0137] As an example, this scheme learns channel weights through two fully connected layers. Matrix W1 represents the weights of the first fully connected layer, as shown in Equation 10 below. This scheme uses matrix multiplication to recombine channel information, thereby identifying which feature channel combinations are more meaningful for the sound source localization task.

[0138] Formula 10:

[0139] Where δ refers to the ReLU activation function; the compressed channel feature vector It retains the original channel statistics The most crucial information is the interaction information between feature channels, and it has a lower dimension, reducing the amount of subsequent computation.

[0140] In addition, matrix W2 represents the weights of the second fully connected layer. This scheme calculates the unnormalized channel weight vector s' (the output of the second fully connected layer) using the following formula 11.

[0141] Formula 11:

[0142] Finally, using the Sigmoid activation function, the weights of each feature channel are normalized according to Equation 12 below, that is, the weights of each feature channel are mapped to... The interval is used to obtain the channel weight vector s as shown in Formula 13 below.

[0143] Formula 12:

[0144] Formula 13:

[0145] In formula 12, This refers to the unnormalized weights of the c-th feature channel. This refers to the normalized weight of the c-th feature channel; in Formula 13, C refers to the total number of feature channels; additionally, in Formulas 12 and 13, This refers to the Sigmoid activation function.

[0146] 3021-4. Feature Channel Weighting: Based on Channel Weight Vector Acoustic feature map The weighted data is then processed to obtain a new acoustic feature map, which is then converted into spatial propagation features. .

[0147] In this embodiment of the application, based on the learned channel weights, the acoustic feature map is processed using the following formula 14. eigenvalues ​​in Perform feature recalibration.

[0148] Formula 14:

[0149] in, It is an acoustic feature map The c-th feature channel, spatial location eigenvalues ​​at that location It is the weight (scalar) assigned to the c-th feature channel based on the channel attention mechanism, and These are the eigenvalues ​​after feature recalibration.

[0150] Similarly, applying the feature recalibration shown in Formula 14 above to the entire feature map yields a new acoustic feature map. This process can be represented by Formula 15 below.

[0151] Formula 15:

[0152] As an example, this scheme transforms the new acoustic feature map into a feature vector with fixed dimensions through a global average pooling layer and a fully connected layer. This application does not impose any limitations on this.

[0153] The channel attention mechanism will be illustrated with a specific example below.

[0154] This example assumes an acoustic feature map. It contains 64 feature channels, and the specific frequency correspondence of these 64 feature channels is shown in Table 2 below.

[0155] Table 2

[0156] Taking the channel correlation matrix W1 as shown in Formula 16 below as an example, the physical meaning of the row vectors of matrix W1 is as follows: Line 1: Learn how to extract the first channel combination pattern from 64 feature channels; Line 2: Learn how to extract the second combination pattern from 64 feature channels; Line 3: Learn how to extract the third channel combination pattern from 64 feature channels; Line 4: Learn how to extract the fourth combination pattern from 64 feature channels.

[0157] Furthermore, the physical meaning of the weights in matrix W1: A larger value indicates that the j-th feature channel contributes more to the i-th channel combination; A small value indicates that the j-th feature channel is not important to the i-th channel combination.

[0158] Formula 16:

[0159] Taking the abnormal noise caused by a loose bolt at a distance as an example, and the channel feature vector like Figure 4 As shown, the training matrix W1 might learn the following patterns: The weights in the first row are: [0.8, 0.7, 0.6, ..., 0.1, 0.05, 0.01].

[0160] Weight distribution: Low-frequency channels have high weights, and high-frequency channels have low weights.

[0161] Physical meaning: Long-distance sound feature extractor.

[0162] The weights in the second row are: [0.1, 0.2, 0.3, ..., 0.9, 0.8, 0.7].

[0163] Weight distribution: High-frequency channels have high weights.

[0164] Physical meaning: Transient feature extractor (activated at close range).

[0165] The weights in row 3 are: [0.5, 0.5, 0.4, ..., 0.3, 0.2, 0.1].

[0166] Weight distribution: balanced across the entire frequency band.

[0167] Physical meaning: General feature extractor.

[0168] The weights in row 4 are: [0.2, 0.8, 0.3, ..., 0.1, 0.05, 0.01].

[0169] Weight distribution: Certain intermediate frequency channels have high weights.

[0170] Physical meaning: Resonance feature extractor.

[0171] The calculation method for the u vector is shown in Formula 17 below.

[0172] Formula 17:

[0173] After the ReLU activation function, assuming we get u = [4.2, 0.0, 2.1, 3.8], then u1=4.2 represents strong activation of long-range feature patterns; u2=0.0 represents that transient feature patterns are suppressed (ReLU sets negative values ​​to 0); u3=2.1 represents moderate activation of general features; and u4=3.8 represents strong activation of resonant features.

[0174] In summary, the purpose of the channel attention mechanism is to enable the sound source localization model to automatically learn which frequency bands are more important for determining sound direction or distance. For example, low-frequency signals have strong diffraction capabilities and slow attenuation, making them more sensitive to distance; high-frequency signals have strong directionality but attenuate quickly. This scheme, based on the channel attention mechanism, can adaptively enhance or suppress information in specific frequency bands.

[0175] 3022. Using the second encoder of the sound source localization model, feature extraction is performed on the input time-frequency map to obtain the physical property features of the bolt. .

[0176] In this embodiment, the training objective of the voiceprint attribute encoder focuses on recognizing the physical properties of sound. For example, by designing an auxiliary classification task (requiring pre-labeled data), the model learns to distinguish typical voiceprint patterns of loose bolts in different locations, of different models, or made of different materials.

[0177] The goal of Path 2 is to focus on learning the physical properties of the sound source contained in the sound signal, i.e., the sound signature, such as resonant frequency, damping characteristics, and impact response spectrum. These characteristics are determined by the bolt type, material, tightening condition, and installation location. Accordingly, the output of Path 2 is used to encode the information of "what kind of bolt made this sound".

[0178] In addition, the voiceprint attribute encoder shares a two-dimensional convolutional layer with the spatial-temporal encoder (used to extract basic time-frequency features). Subsequently, path one and path two branch off, each trained with different network structures, loss functions, and supervision signals to force them to learn feature representations with different focuses.

[0179] 3022-1. Use non-dilated convolutional layers to perform convolution operations on the input time-frequency map to obtain time-frequency features.

[0180] This step can be referred to in the previous introduction to the shared layer, and will not be repeated here.

[0181] 3022-2, Refined Convolution Processing along the Frequency Axis and Residual Connections: While maintaining frequency resolution, asymmetric convolution kernels are used to refine time-frequency features along the time direction. Perform convolution operations and combine time-frequency features through residual connections. Time-frequency features after convolution operation Adding them together yields new time-frequency characteristics. .

[0182] The input for this step is the time-frequency feature. As mentioned earlier, this feature refers to the feature map obtained after processing the time-frequency maps of N microphone channels through a two-dimensional convolutional layer. From the time-frequency features... Initially, path two uses a dedicated network structure to extract the physical property features of the bolts.

[0183] In this scheme, the temporal kernel size of the asymmetric convolution kernel is larger than the frequency kernel size. The proposed scheme uses an asymmetric convolution kernel (e.g., 1x5 in size, with a height of 1 and a width of 5). This kernel has a receptive field of 1 in the frequency axis (vertical direction) and a receptive field of 5 in the temporal axis (horizontal direction). It emphasizes pattern capture along the temporal dimension while maintaining fine resolution in the frequency dimension.

[0184] To maintain frequency resolution and capture temporal patterns, path two uses an asymmetric convolution kernel to perform convolution operations. To extract the temporal dynamic features of the signal while preserving frequency resolution, this scheme employs an asymmetric convolution kernel, performing convolution operations along the time axis of the time-frequency plot. This design allows each frequency point to independently complete local feature extraction in the time dimension, accurately capturing the temporal evolution patterns and temporal dependencies of sound events, while effectively avoiding information aliasing between different frequency channels.

[0185] For the input time-frequency features Applying asymmetric convolution kernels (k is the length of the convolution kernel in the time direction), time-frequency features after convolution operation The calculation formula is shown below.

[0186] Formula 18:

[0187] In Formula 18, f refers to the frequency index (values ​​from 1 to F); t refers to the time index (values ​​from 1 to T); c refers to the input channel index (values ​​from 1 to C); and c' refers to the output channel index (values ​​from 1 to C'). 1 refers to the offset in the time direction; W refers to the kernel weight, whose first dimension is 1, indicating that no mixing occurs in the frequency direction, thus maintaining frequency resolution; b refers to the bias term.

[0188] Residual connections refer to adding the input and output of a convolutional layer to form a residual block. Its mathematical expression is shown in Equation 19 below.

[0189] Formula 19:

[0190] In Equation 19, F refers to operations such as convolution, batch normalization, and activation; additionally... , This scheme uses residual connections to preserve the features of the input residual blocks, avoiding information loss and reducing training difficulty.

[0191] 3022-3, Spatial Attention Mechanism: Based on Novel Time-Frequency Features Generate an attention weight map.

[0192] In this embodiment, the spatial attention mechanism allows the model to focus on regions of energy abrupt change in the time-frequency graph (such as the transient pulse corresponding to the clicking sound of a loose bolt) or specific resonant frequency bands, thereby extracting more accurate voiceprint features. Correspondingly, the attention weight map marks the importance of each time-frequency region in the time-frequency graph using attention weights.

[0193] In some embodiments, based on new time-frequency characteristics Generate an attention weight map, including: Step 1: Apply the following formula 20 to the new time-frequency characteristics. Perform average pooling along the channel dimension to obtain the average energy distribution of each time-frequency point across all feature channels; and, using the following formula 21, perform average pooling on the new time-frequency features. Max pooling is performed to obtain the maximum energy distribution of each time frequency point across all feature channels.

[0194] Formula 20:

[0195] Formula 21:

[0196] Formulas 20 and 21 are used to calculate each time-frequency point. Channel statistics.

[0197] For average pooling characteristics It is used to characterize the average energy distribution of each time-frequency point across all feature channels, reflecting the average activation intensity of each spatial location across all feature channels, and is suitable for identifying the main frequency regions in the time-frequency domain. Essentially, it reflects the overall response of a time-frequency point across all feature channels. In the task of locating loose bolts, average pooling features can effectively highlight key regions where strong responses exist in multiple feature channels. These regions typically correspond to common acoustic features of events directly related to bolt loosening, such as impacts and harmonics.

[0198] For max pooling features Max pooling is used to characterize the maximum energy distribution across all feature channels at each time-frequency point, highlighting the impact initiation point, i.e., the maximum activation intensity at each spatial location across all feature channels. It reflects the strongest response at that time-frequency point on a particular feature channel. Furthermore, max pooling can capture the most salient features at each location. For example, one feature channel might be particularly sensitive to impact initiation, while another might be particularly sensitive to resonance peaks; max pooling can preserve these most salient features.

[0199] Step 2: Generate an attention weight map based on the average energy distribution and maximum energy distribution at each time frequency point. .

[0200] This step concatenates the two channel statistics calculated using Equations 20 and 21, and then generates an attention map using a convolutional layer and a sigmoid activation function. The above process can be represented by the following formula 22.

[0201] Formula 22:

[0202] In Equation 22, σ refers to the Sigmoid activation function, W s This refers to the weight matrix of the convolution kernel. This indicates a convolution operation.

[0203] Finally, the attention weight map Applied to new time-frequency features For details, please refer to step 3022-4 below.

[0204] 3022-4. Feature Recalibration: Based on the attention weight map and the new time-frequency features, the optimized time-frequency features are generated using the following formula 23. .

[0205] Formula 23:

[0206] 3022-5. Based on the optimized time-frequency characteristics The physical properties of the generated bolts are determined.

[0207] This process includes steps 1 through 4 below.

[0208] Step 1: Global Information Extraction: Extracting the optimized time-frequency features Global average pooling is performed to obtain the channel feature vectors. Each component of the channel feature vector in this step represents the optimized time-frequency feature. The global pixel mean within a single feature channel.

[0209] In this embodiment of the application, the optimized time-frequency features are expressed by the following formula 24. Convert to channel feature vector .

[0210] Formula 24:

[0211] in, As a scalar, it refers to the first... The global statistics for each feature channel; F refers to the transformed time-frequency features. The height of corresponds to the frequency dimension in a time-frequency plot; T refers to the transformed time-frequency feature. The width corresponds to the time dimension in a time-frequency graph.

[0212] Applying Formula 24 above to all feature channels to calculate the global statistic yields the following vector. .

[0213] Formula 25:

[0214] In Formula 25, This refers to the total number of feature channels; This refers to the channel's global statistical vector, i.e. Represents the first The average activation intensity of each feature channel across the entire time-frequency plane reflects the overall importance of the corresponding frequency range.

[0215] Step 2, Multi-task classification: Based on the channel feature vector, predict the physical properties of the bolt to obtain the probability distribution of multiple attributes.

[0216] This step is used to extract the channel feature vector. Input multiple fully connected layers and predict the physical properties of bolts, such as bolt type, material, fastening condition and installation location, using the following formulas 26 to 29.

[0217] Formula 26:

[0218] Formula 27:

[0219] Formula 28:

[0220] Formula 29:

[0221] In formulas 26-29, W This refers to the weight matrix, used to weight the eigenvectors. Mapping to the decision space corresponding to a specific attribute, from the feature vector We select and weight the feature dimensions that are most effective in identifying this type of attribute. For example, W tightness W is used to screen out frequency resonance characteristics that are sensitive to changes in preload. type Spectral envelope features used to filter out reactive bolt geometry, W material W is used to filter out feature dimensions that are sensitive to material changes. location Used to filter feature dimensions related to the installation location. This refers to the bias vector, used to help the model adapt to the inherent distributional differences between different categories. For example, if a certain type of bolt is more common in the sample dataset, then the corresponding bias vector (b)... type The value for the corresponding category may be higher, and it will be incorporated into the model as prior knowledge. + b This is a scoring system for the original evidence that a bolt belongs to a certain category under a specific attribute. A higher score indicates a stronger feature vector. The closer the match is to the pattern of the corresponding category. This refers to the probability distribution, which is the output after normalization using the Softmax function. Its sum is 1, representing the final diagnostic conclusion and confidence level for a certain attribute of the bolt. For example, P... tightness = [0.02, 0.15, 0.83] indicates that the bolt is in a severely loose state with an 83% confidence level.

[0222] Step 3: Convert the probability distributions of multiple attributes into the physical property characteristics of the bolt. .

[0223] This step is used to calculate the probability distributions P of the four independent attributes. type P material P tightness and P location Merged into a fixed-dimensional feature vector , where the feature vector In this paper, these are also referred to as the physical property characteristics of bolts. As an example, the above property probability distribution is transformed into a feature vector. The process consists of three steps: subspace mapping, attention weighting, and feature encoding. These three steps will be described in detail below.

[0224] Subspace mapping This step maps the aforementioned attribute probability distributions to the same continuous semantic subspace based on the semantic projection matrix, thereby obtaining multiple attribute vectors. One attribute vector represents a physical property of the bolt. Furthermore, the parameters of the semantic projection matrix are updated during model training.

[0225] It should be noted that this step is performed to map the probability distribution of each attribute to a unified, comparable, continuous semantic subspace, so that the attribute vectors have consistent dimensions and are comparable.

[0226] In some embodiments, based on the semantic projection matrix, the above attribute probability distributions are mapped to the same continuous semantic subspace to obtain multiple attribute vectors, including: A. Based on the first semantic projection matrix, the probability distribution of the first attribute indicating the bolt type is mapped to the continuous semantic subspace using the following formula 30 to obtain the first attribute vector.

[0227] Formula 30:

[0228] in, It is the probability distribution of bolt type, also referred to as the first attribute probability distribution in this paper. Each element in the array represents the probability that the bolt belongs to a specific model. The output from the multi-task classification head reflects the characteristics of the sound signal related to the bolt geometry.

[0229] This is the semantic projection matrix of the model attribute, also referred to as the first semantic projection matrix in this paper. It represents the probability distribution of the attributes corresponding to the model. From C type A linear mapping of the category space of dimension to d k A continuous semantic subspace of dimension. The learnable parameters of this matrix enable the model to transform discrete model category information into continuous semantic features and interact with other attribute features.

[0230] This is the bias term corresponding to the model projection, used to adjust the mapped features. This scheme enhances the expressive power of the model by introducing a priori offset.

[0231] This refers to the first attribute vector, which is a continuous semantic feature containing bolt model information. In other words, it is the semantic embedding vector of the model attribute. It is an attribute probability distribution. The feature representation after linear mapping and ReLU activation function captures the semantic information of bolt type in a unified continuous semantic subspace. Furthermore, due to the presence of the ReLU activation function, therefore... It preserves nonnegativity and introduces sparsity, which helps with feature selection.

[0232] B. Based on the second semantic projection matrix, the probability distribution of the second attribute indicating the bolt material is mapped to the continuous semantic subspace using the following formula 31 to obtain the second attribute vector.

[0233] Formula 31:

[0234] in, It is the probability distribution of bolt material, also referred to as the second attribute probability distribution in this paper. Each element in the signal represents the probability that the bolt is made of different materials (such as carbon steel, stainless steel, alloy steel, etc.), which reflects the characteristics of the sound signal related to material properties (such as density and elastic modulus).

[0235] This is the semantic projection matrix of the material properties, also referred to as the second semantic projection matrix in this paper. It represents the probability distribution of the material's corresponding properties. From C material A linear mapping of the category space of dimension to dk A continuous semantic subspace of dimension.

[0236] It is the bias term corresponding to the material projection.

[0237] This refers to the second attribute vector, which is a continuous semantic feature containing bolt material information. In other words, it is a semantic embedding vector of material properties.

[0238] C. Based on the third semantic projection matrix, the probability distribution of the third attribute indicating the bolt fastening state is mapped to the continuous semantic subspace using the following formula 32 to obtain the third attribute vector.

[0239] Formula 32:

[0240] in, It is the probability distribution of the bolt tightening state, also referred to as the third attribute probability distribution in this paper. Each element in the array represents the probability that the bolt is in different tightness states (such as normal, slightly loose, severely loose, etc.). This is the most critical attribute in the loose bolt location task, as it directly reflects the bolt's tightness.

[0241] This is the semantic projection matrix of the fastened state attributes, also referred to as the third semantic projection matrix in this paper. It represents the probability distribution of the attributes corresponding to the fastened state. From C tightness A linear mapping of the category space of dimension to d k A continuous semantic subspace of dimension. Notably, in the task of locating loose bolts, this matrix may be trained to emphasize feature variations related to bolt loosening.

[0242] It is the bias term corresponding to the projection of the fastened state.

[0243] This is the third attribute vector, which is a continuous semantic feature containing bolt state information. That is, it is a semantic embedding vector of the fastening state attribute. It encodes semantic information related to the bolt fastening state and is a crucial component in subsequent feature fusion.

[0244] D. Based on the fourth semantic projection matrix, the probability distribution of the fourth attribute indicating the bolt installation position is mapped to the continuous semantic subspace using the following formula 33 to obtain the fourth attribute vector.

[0245] Formula 33:

[0246] in, It is the probability distribution of bolt installation positions, also referred to as the fourth attribute probability distribution in this paper. Each element in the array represents the probability of a bolt being located in a different position (such as the top of a tower, the foot of a tower, or a beam), reflecting the contextual information of the sound propagation path and the structural coupling.

[0247] This is the semantic projection matrix of the installation location attributes, also referred to as the fourth semantic projection matrix in this paper. It represents the probability distribution of attributes corresponding to the installation location. From C location A linear mapping of the category space of dimension to d k A continuous semantic subspace of dimension. This matrix may learn position-dependent acoustic modulation patterns.

[0248] It is the offset term corresponding to the position projection.

[0249] It is the fourth attribute vector, which is a continuous semantic feature containing bolt installation position information. That is, it is a semantic embedding vector of the installation position attribute.

[0250] It should be noted that in formulas 30-33, C type C material C tightness C location These are the number of categories for each physical attribute (model, material, fastening condition, and installation location), d k It is the unified dimension after mapping.

[0251] In summary, by using a learnable semantic projection matrix, this scheme maps discrete attribute probability distributions to a unified continuous semantic subspace, forming attribute feature vectors with clear physical meaning. This provides comparable input for subsequent adaptive weighted fusion, thereby realizing joint modeling and refined expression of multi-dimensional bolt attributes.

[0252] Adaptive attention weighting This step introduces a learnable query vector. It calculates the relative importance of each physical attribute in the current context and generates a weighted context vector. The parameters of the query vector are updated during model training. These parameters are updated during model training.

[0253] 1. Calculate the attention score: Obtain the query vector using the following formula 34. The attention score for each physical attribute of the bolt is obtained by calculating the dot product similarity between the bolt and each attribute vector. .

[0254] Formula 34:

[0255] 2. Normalization to obtain attention weights: Normalize the attention scores of each physical attribute to obtain the attention weights of each physical attribute.

[0256] For example, this scheme uses the Softmax function via the following formula 35 to calculate the attention score. Normalized to a probability distribution, the attention weights for each physical attribute are obtained. .

[0257] Formula 35:

[0258] in, Query vector This represents the focus of the current task. The model can learn the optimal query direction for different implicit contexts through training. Additionally, the current task could be: determining the tightness of bolts, where the model focuses more on features related to preload, contact stiffness, and nonlinear vibration; an asset inventory task, identifying bolt identities (e.g., model, material), where the model focuses more on inherent properties such as bolt geometry and material resonance; or determining the installation location of bolts, where the model focuses more on structural transfer functions and environmental noise characteristics.

[0259] It should be noted that attention weight It is not fixed, but dynamically updated. It simulates the emphasis human experts place on different attributes of evidence when making comprehensive judgments. For example, in a task involving locating loose bolts, the model might learn to assign different attributes to P. tightness Assign higher attention weights; while in asset inventory tasks, the model may learn to give P... type and P material Assign higher attention weights. That is, through training, the query vector It can automatically adjust its orientation in different task modes, thereby assigning higher attention weights to the corresponding physical attributes.

[0260] 3. Generate a weighted context vector: The attention weights of each physical attribute and multiple attribute vectors are weighted and summed to obtain the attribute fusion vector. That is, this scheme uses the calculated attention weights to perform a weighted summation of the semantic embedding vectors of each physical attribute using the following formula 36, ​​thereby obtaining the attribute fusion vector. .

[0261] Formula 36:

[0262] in, It is a context-aware attribute fusion vector. It is no longer a simple concatenation of attribute information, but rather a dynamically weighted fusion result based on task relevance. It focuses more on the combination of attribute information most relevant to the current context. Here, the current context can be understood as the current task.

[0263] Feature encoding This step is used to fuse the attribute vector using the following formula 37. Convert to fixed dimensions (number of dimensions is d) f Physical properties .

[0264] Formula 37:

[0265] In formula 37, W f and b f This refers to a feature compression encoder, used to compress and encode dynamically fused features of dimension d. k context vector Non-linear mapping to a dimension of d f Physical property characteristics (e.g., 128, 256) .in, It is a compact feature representation with strong semantics, fixed dimensions, and adaptability to downstream tasks.

[0266] It should be noted that during the training phase, this scheme also applies data augmentation processing to the input time-frequency graph to simulate spatial propagation effects, such as time stretching (simulating minute phase changes caused by different sound propagation distances) and frequency masking (used to simulate the situation where some frequency bands are blocked). Accordingly, the dual-path encoder uses the data-augmented time-frequency graph as the original input.

[0267] The purpose of data augmentation is to force the model to learn the intrinsic features of the sound source that are relatively invariant to changes in the sound propagation path, ensuring that the core acoustic features of the same type of bolt have path invariance, thereby improving the model's robustness to changes in spatial propagation. That is, regardless of whether the sound wave signal originates from the top of the tower, the base of the tower, or other spatial locations, the model can stably identify the target features.

[0268] Step 4: Data Augmentation In some embodiments, the data augmentation method includes at least one of time stretching or frequency masking. Time stretching and frequency masking are described below.

[0269] Time stretching This data augmentation method is used to simulate minute phase changes caused by different sound propagation distances. As shown in Equation 38 below, the time stretching method interpolates the input time-frequency graph on the time axis, where the stretching factor is randomly selected.

[0270] Formula 38:

[0271] In formula 38, This refers to the stretching factor, used to simulate the minute phase changes that occur when sound travels to a microphone at different distances. Typically, The value ranges from 0.9 to 1.1, indicating that the time axis is compressed or stretched by 10%. t refers to the original time coordinate, i.e., the time axis coordinate of the input time-frequency graph, representing the time point of the audio signal. t' refers to the stretched time coordinate, i.e., the time coordinate after time stretching, t' = t. When When the value is greater than 1, the time axis is stretched, indicating that the sound travels to a microphone further away (increased delay); when... When the value is less than 1, the time axis is compressed, meaning the sound travels to a closer microphone (delay is reduced). It refers to the amplitude (or energy) value of the input time-frequency graph at frequency f and time t. The amplitude (or energy) values ​​at frequency f and time t' of the time-frequency graph after time stretching. This refers to the interpolation function. Since the stretched time coordinate t' may not be an integer, interpolation is needed to calculate the amplitude (or energy) value at non-integer time points.

[0272] In summary, data augmentation processing is performed on the input time-frequency graph to obtain the processed time-frequency graph, including: based on a randomly selected stretching factor. The time axis of the input time-frequency graph is scaled; interpolation is then performed on the scaled time axis to fill in the missing data points in the time dimension, resulting in the processed time-frequency graph.

[0273] It's important to note that in the loose bolt localization task, the varying distances between the microphones in the microphone array and the sound source result in slight differences in sound arrival times. Time stretching can simulate these differences, enhancing the model's robustness to time delays. In other words, the physical meaning of time stretching in the loose bolt localization task is to simulate the variation in time delay as sound travels from the loose bolt to different microphones by randomly changing the scaling of the time axis. This data augmentation allows the model to adapt to the subtle time differences in sound signals received by different microphones during learning, thereby improving the model's robustness to sound source localization.

[0274] Frequency mask This data augmentation method randomly masks certain frequency bands in the input time-frequency graph, simulating the situation where some frequency bands are blocked. This forces the model to learn more robust feature representations, thereby improving the model's generalization ability in complex environments.

[0275] In some embodiments, data augmentation processing is performed on the input time-frequency graph to obtain a processed time-frequency graph, including: selecting a mask start frequency within the effective frequency range of the input time-frequency graph; determining the frequency range to be masked based on the mask width and the mask start frequency; and replacing the energy values ​​in the frequency range to be masked in the input time-frequency graph with preset values ​​to obtain the processed time-frequency graph.

[0276] It should be noted that the preset values ​​are random noise values, the mean background noise of the input time-frequency plot, or zero. Taking random zeroing as an example, the masking operation can be expressed as the following formula 39.

[0277] Formula 39:

[0278] In Equation 39, f represents the frequency index and t represents the time index.

[0279] f0 refers to the mask's starting frequency index, and f0 is usually randomly selected. During training, f0 is randomly sampled from a uniform distribution, typically ranging from [0, F]. Δf]. Here, F is the length of the frequency dimension, i.e., the effective frequency range. Physically, this simulates the situation where a certain frequency band is blocked or attenuated during sound propagation. For example, due to obstacles in the environment or microphone malfunction, the signal within a specific frequency range may be lost. Δf refers to the mask width (in frequency indices). Δf is usually set to a random value ranging from [1, F]. max ]. Among them, F max This refers to the maximum mask width (e.g., 10% or 20% of the frequency dimension of the input time-frequency plot). Physically, this simulates different widths of frequency occlusion. Wider masks generally correspond to larger obstructions or more severe frequency attenuation.

[0280] Time-frequency graph It represents the energy distribution of a sound signal on the frequency-time plane. For abnormal noises caused by loose bolts, the time-frequency diagram includes characteristics such as impact components, harmonics, and resonance peaks.

[0281] Masking (zeroing): Replaces the energy values ​​within the selected frequency range with 0. Physically, this simulates the complete loss of the corresponding frequency band signal. In real-world environments, frequency information loss may occur due to the following reasons: insufficient microphone sensitivity in a specific frequency band; strong ambient noise or interference overwhelming the information in that frequency band; frequency-selective attenuation in the sound propagation path (e.g., air absorption, obstacle reflection, etc.).

[0282] It should be noted that, sometimes, for a more natural effect, the energy values ​​within the selected frequency range can be replaced with random noise values ​​or the mean background noise of the input time-frequency graph, and this application does not limit this.

[0283] In addition, the purpose of performing frequency masking on the input time-frequency graph includes, but is not limited to, the following aspects.

[0284] Enhancing model robustness: This forces the model to learn features distributed across the entire frequency range, rather than relying on any single frequency feature. This way, even if some frequency bands are affected by noise or information loss, the model can still accurately identify the sound of loose bolts.

[0285] For example, suppose the input time-frequency plot contains a significant resonant peak near frequency index f=100, with a width of approximately 10 frequency units. If a random mask happens to cover this region (e.g., f0=95, Δf=10), the model will be unable to utilize this resonant peak information. Therefore, the model must learn to rely on other frequency features (such as the impulse initiation point, other harmonics, etc.) to perform loose bolt localization. In this way, the model will not become overly dependent on any particular frequency feature, thereby improving its robustness.

[0286] To prevent overfitting: Avoid making the model too dependent on specific frequency patterns that appear in the training samples, thereby improving generalization ability.

[0287] Simulating frequency occlusion in real-world environments: In real-world environments, sound propagation may be affected by various factors, resulting in the loss of certain frequency components. Through frequency masks, the model can learn to process this incomplete frequency information.

[0288] In summary, Path 2 utilizes asymmetric convolution, residual connections, and spatial attention mechanisms to extract voiceprint attribute features from sound signals that are independent of the propagation path. Furthermore, this scheme simulates sound variations under different propagation conditions through data augmentation, enabling the model to learn voiceprint attribute features insensitive to the propagation path. Additionally, invariant learning allows the model to ignore propagation differences and focus on the physical properties of the bolt itself. A detailed description of invariant learning can be found in step 305.

[0289] 3023. Characteristics of spatial propagation and physical properties The first feature vector is obtained by fusion. .

[0290] In this embodiment, through different network structure designs, path one and path two are guided to learn complementary information. It should contain the spatial propagation information of sound as purely as possible, and The physical properties information of the bolts should be included as purely as possible.

[0291] In addition, through the design of differentiated loss functions, path one and path two are also guided to learn complementary information. Please refer to step 305 for details.

[0292] In addition, spatial propagation characteristics and physical properties This laid the foundation for subsequent adaptive attention mechanisms. Specifically, the query matrix Q, key matrix K, and value matrix V of the multi-head attention module are based on spatial propagation features. and physical properties The features are generated through splicing or interaction. Since these two sets of features have been professionally refined, the adaptive attention mechanism can make the following judgment more clearly and specifically: when making the final localization decision, should it rely more on the spatial cues of the current sound or on its voiceprint attribute cues?

[0293] For example, suppose we collect a very typical and clear abnormal noise (physical property characteristic) caused by a loose bolt of a certain type. (Very strong), even if the collected sound signal is weak due to obstructions (spatial propagation characteristics) (Fuzzy), adaptive attention mechanisms also assign physical property features. With higher weighting, the model combines historical data or tower structure diagrams to ultimately deduce the location of the bolt most likely to cause the abnormal noise.

[0294] As an example, regarding spatial propagation characteristics and physical properties Fusion refers to: using the following formula 40 to integrate spatial propagation characteristics and physical properties By concatenating the features along the feature dimension, a fused feature vector containing "spatial-attribute" information is formed.

[0295] Formula 40:

[0296] in, This represents the physical property features output by path two, and the dimension of the physical property features output by path two is... ; The spatial propagation feature representing the output of path one has the following dimension: ; This represents the fused feature vector, and the dimension of the fused feature vector is... .

[0297] It should be noted that the feature vector fusion Also referred to as the first feature vector in this paper, it achieves information complementarity. This scheme forces the subsequent network structure of the model to learn a deep association between the two features through cross-modal correlation. For example, the propagation pattern of the loosening sound of a specific type of bolt in a microphone array at a specific location provides a complete "space-attribute" joint representation for the subsequent adaptive attention mechanism. In other words, It is an input vector that integrates physical property information and spatial propagation information of sound. In the task of locating loose bolts, it contains not only attribute information such as bolt type, material or condition, but also spatial propagation information of sound.

[0298] 303. The computer equipment uses a sound source localization model to convert the first feature vector into attention-weighted features; whereby the attention-weighted features are used to reflect the key features related to the bolt state in the first feature vector from multiple dimensions.

[0299] In this embodiment, the sound source localization model includes a multi-head attention module. The adaptive attention mechanism based on the multi-head attention module mainly includes the following steps: Linear projection: passing the first feature vector through a learnable weight matrix W Q W K and W V These are mapped to a query matrix Q, a key matrix K, and a value matrix V, respectively. The query matrix Q can be understood as the "question" about the part of the current sound signal that needs to be focused on, the key matrix K consists of "labels" in the features used to match the query matrix Q, and the value matrix V is the actual "content."

[0300] Calculate attention weights: First, calculate the dot product of the query matrix Q and all key matrices K. Then, divide the dot product result by a scaling factor to alleviate the numerical distortion caused by dimensionality increase. Finally, normalize the result using the Softmax function to generate an attention weight map that represents the importance of features.

[0301] In this process, the model automatically learns which part of the feature map should be focused on for the current input signal (for example, whether to focus on the features of a specific frequency range or the features of a microphone channel), and this part may correspond to the key voiceprint clues that best represent the loosening of a certain location and type of bolt.

[0302] Weighted summation: The attention weight map and the value matrix V are weighted and summed to generate the output of the current attention head.

[0303] In summary, the attention mechanism extracts key information by weighted summation of the value matrix based on the matching degree between the query matrix and the key matrix. Furthermore, multiple attention heads operate in parallel, each focusing on a different feature subspace or different feature interaction patterns. In the loose bolt localization task, different attention heads may focus on different aspects. For example, one attention head might focus on frequency features (to identify specific frequency components of bolt loosening); another might focus on the time delay features between microphones (for sound source localization); one might focus on the correlation between the bolt's physical properties and acoustic features; and another might focus on the impact of environmental noise, etc. This application does not limit the scope of these approaches.

[0304] Multi-head output fusion: The outputs of all attention heads are concatenated again and fused through a linear layer to obtain a feature representation modulated by the attention mechanism.

[0305] It should be noted that, based on the adaptive attention mechanism, this scheme achieves dynamic and adaptive focusing on the joint representation of "space-attribute".

[0306] 3031. For the i-th attention head, based on the linear projection matrix of the i-th attention head and the first eigenvector... Generate the query matrix, key matrix, and value matrix of the i-th attention head.

[0307] The parameters of the linear projection matrix are updated during model training. i is a positive integer.

[0308] In some embodiments, the query matrix, key matrix, and value matrix of the i-th attention head are generated based on the linear projection matrix of the i-th attention head and the first feature vector, including the following steps.

[0309] 1. Based on the first linear projection matrix of the i-th attention head, the first feature vector is projected using the following formula 41. Projecting onto the query space yields the query matrix Q for the i-th attention head.

[0310] Formula 41:

[0311] This refers to the query projection matrix, also known in this paper as the first linear projection matrix.

[0312] 2. Based on the second linear projection matrix of the i-th attention head, project the first feature vector onto the key space to obtain the key matrix of the i-th attention head.

[0313] Formula 42:

[0314] It refers to the key projection matrix, which is also called the second linear projection matrix in this paper.

[0315] 3. Based on the third linear projection matrix of the i-th attention head, project the first feature vector onto the value space to obtain the value matrix of the i-th attention head.

[0316] Formula 43:

[0317] It refers to the value projection matrix, which is also called the third linear projection matrix in this paper.

[0318] In formulas 42-43, and This refers to the projection dimension, which is usually... = .

[0319] In this embodiment, the query vector Q represents what needs to be considered in the current signal. For example, when the model detects a loose M24 bolt in the tower base area, the query vector Q will autonomously learn and focus on the combination features of low-frequency components and specific microphone channels. Furthermore, this solution generates different query vectors Q for different application scenarios. Assuming scenario A involves a loose bolt at the top of the tower, the query vector Q might focus more on high-frequency attenuation patterns (corresponding to air propagation characteristics). Assuming scenario B involves loose bolts at the tower base, the query vector Q might focus more on low-frequency structural waves (corresponding to solid propagation characteristics). Assuming scenario C involves a loose small-diameter bolt, the query vector Q might focus more on high-frequency impact components.

[0320] In conclusion, W Q This enables the model to intelligently ask questions, such as: for a specific bolt (whose model and material are known), in which characteristics is the loosening sound most likely to be present?

[0321] The key vector K represents the available information in the features, such as the energy distribution of each frequency band and the correlation patterns between microphone channels. W K The purpose is to create "index labels" for each part of the feature information, and to construct a feature index space that can be retrieved and matched by the query vector Q. When the query vector Q (such as the target feature of loose M24 bolts on the focusing tower foot) is highly matched with the key vector K (such as the low-frequency energy feature collected by microphone 3), it can be determined that the microphone channel corresponding to the key vector is very important in the current sound source localization task.

[0322] The value vector V represents the actual content; it is the information carrier after the original features have been transformed. W VThe original features are transformed into features more suitable for sound source localization tasks. By suppressing noise features that are irrelevant to sound source localization, the physical features that are key to sound source localization (such as the temporal resolution of time delay features and the frequency domain clarity of resonant frequencies) are enhanced, thereby improving the adaptability of the features to sound source localization tasks.

[0323] 3032. Based on the query matrix and key matrix of the i-th attention head, obtain the attention weight map of the i-th attention head; perform a weighted summation of the attention weight map and value matrix of the i-th attention head to obtain the output features of the i-th attention head; wherein, the attention weight map marks the importance of each feature component in the first feature vector through attention weight.

[0324] In this embodiment of the application, the attention weight map and value matrix of the i-th attention head are weighted and summed using the following formula 44 to obtain the output features of the i-th attention head.

[0325] Formula 44:

[0326] In formula 44, , , .

[0327] in, It is the first linear projection matrix of the i-th attention head. The second linear projection matrix of the i-th attention head and This is the third linear projection matrix of the i-th attention head. These three matrices (the parameters of which are updated during model training) are used to respectively... Projected onto the corresponding feature subspaces, namely the query space, key space, and value space. In the task of locating loose bolts, these three matrices can be understood as... Extract features related to the content focused on by the i-th attention head. For example, for an attention head with a certain frequency of attention, the W of that attention head... Q W K and W V It may be trained to extract frequency-related features.

[0328] , and This refers to the query matrix, key matrix, and value matrix of the i-th attention head. This can be understood as the content to be queried. In the task of locating loose bolts, the content to be queried may be "the frequency range corresponding to the bolt loosening characteristics" or "the microphone pair combination corresponding to the sound source direction information", etc. It can be viewed as a dedicated index label for the feature space, used for matching with the query matrix. Each key vector corresponds to an independent information dimension within the feature space, such as the energy characteristics of a specific frequency range or the correlation characteristics of a pair of microphones, thereby representing whether that dimension contains the target content required by the query vector. It is the actual content; when the query matrix matches a certain key matrix, the corresponding value matrix will be extracted.

[0329] It is the input for the adaptive multi-head attention mechanism; , where represents the i-th attention head, and h is the total number of attention heads. Let be the output feature of the i-th attention head, corresponding to the focused representation in its dedicated feature subspace; it represents the focus of the input from a specific perspective (such as frequency perspective, microphone channel perspective). The reweighting and reorganization.

[0330] Additionally, in formula 44, It refers to the attention weight map of the i-th attention head.

[0331] 3033. Generate attention-weighted features based on the output features of multiple attention heads. .

[0332] In some embodiments, attention-weighted features are generated based on the output features of multiple attention heads, including the following steps.

[0333] 1. Multi-head output fusion: Concatenates the output features of multiple attention heads; based on the output projection matrix. The spliced ​​features are fused to obtain the fused output features.

[0334] Wherein, the output projection matrix The parameters are updated during model training. This step can be represented by the following formula 45.

[0335] Formula 45:

[0336] In step 303, This refers to the dimension of the query matrix Q and the key matrix K. This dimension determines the representational capability of the adaptive attention mechanism when calculating similarity. In the task of locating loose bolts, The size of the value affects the precision with which the model calculates the similarity between features. This refers to the dimension of the value matrix. Specifically, it's the dimension of the output features of each attention head. It determines the amount of information in the feature representation after attention weighting. Used to map the concatenated features back to the original data. Same dimensions Or another suitable dimension. In the task of locating loose bolts, this matrix can integrate information from different attention heads and reduce feature dimensions for subsequent processing. It is the feature after splicing, which is both The result of fusion after parallel processing by multiple attention heads is also a context-aware feature that has undergone intelligent focusing and fusion, representing the model's weighted integration of information from different dimensions.

[0337] 2. Residual Connections and Layer Normalization: Using the following formula 46, the first feature vector is... and the fused output features Perform an addition operation, and then perform a layer normalization operation on the added features to obtain attention-weighted features. .

[0338] Formula 46:

[0339] in, This is the result after residual connection; This is a layer normalization operation used to standardize the results after residual connections. It acts as a feature stabilizer, ensuring that features have a consistent distribution across different samples and environments, thus improving training stability and generalization ability. It is the final output feature vector of the adaptive attention mechanism. It is an enhanced feature after attention modulation and stabilization processing, and serves as the input to the subsequent localization module, possessing better robustness and discriminativeness.

[0340] 304. The computer equipment uses attention-weighted features to predict the spatial location of bolts through a sound source localization model.

[0341] This step corresponds to Figure 2 Step 3 in the process.

[0342] 3041. Cross-channel feature fusion: The attention-weighted features corresponding to each channel in the microphone array are fused to obtain the global fused features. .

[0343] This step involves inputting the output of step 303, namely the channel features after attention modulation and stabilization, into a lightweight feature fusion network (e.g., including several fully connected layers). The feature fusion network learns the correlation between features collected by microphones deployed at different locations, comprehensively determining the direction and distance of the sound source.

[0344] First, multiple nonlinear transformations are performed on the attention-weighted features corresponding to each channel to obtain the nonlinear transformation results. These results are used to characterize the correlation between the attention-weighted features corresponding to each channel. Taking a feature fusion network consisting of three fully connected layers as an example, the cross-channel feature fusion is described in detail below.

[0345] Step 1: Based on the first-layer weight matrix of the feature fusion network, perform a nonlinear transformation on the attention-weighted features corresponding to each channel using the following formula 47 to obtain the intermediate transformation result. The first-layer weight matrix is ​​used to learn the basic correlations between the attention-weighted features corresponding to each channel.

[0346] Formula 47:

[0347] Step 2: Based on the weight matrix of the second layer of the feature fusion network, the intermediate transformation results are processed using the following formula 48. Perform a nonlinear transformation to obtain the nonlinear transformation result. The second-layer weight matrix is ​​used to learn the higher-order correlation of the attention-weighted features corresponding to each channel after the first nonlinear transformation.

[0348] Formula 48:

[0349] Step 3: Process the nonlinear transformation results Dimensional compression is performed, and layer normalization is applied to the nonlinear transformation result after dimensional compression to obtain the global fusion feature. .

[0350] Formula 49:

[0351] In summary, step 3041 is used to gradually fuse attention-weighted local features into global features. Its core capabilities are reflected in two main dimensions: First, collaborative modeling of the microphone array. By learning the nonlinear correlations between features corresponding to microphones deployed at different locations, it mines the spatial information implicit in the features. For example, when multiple microphones simultaneously detect high-frequency impact components, it may indicate the presence of a near-field sound source. Furthermore, the consistent pattern of phase differences between microphones can accurately characterize the propagation direction of sound waves. Second, environmental adaptive adjustment. The model can dynamically adjust the contribution weights of the features collected by each microphone according to changes in different environmental noise backgrounds, ensuring stable extraction of effective features even under complex noise interference.

[0352] For formulas 47-49, This refers to the first-layer weight matrix. It will... From d f dimensional mapping to d mDimension (intermediate layer dimension). This matrix learns how to... This allows for the combination of more advanced abstract features. In tasks such as locating loose bolts, it may learn to combine the focusing features and attribute features of different microphones into intermediate features related to spatial location.

[0353] This refers to the intermediate layer dimension (e.g., 512). It determines the capacity of feature fusion. If If it's too small, it may not be able to fully integrate multi-channel information; if Too many dimensions can lead to overfitting. In loose bolt localization tasks, the number of microphones may be small (e.g., 8), but the features corresponding to each microphone can be very rich, thus requiring sufficient dimensions to accommodate the information. Typically... Setting it to 512 is an empirical value, large enough to capture complex relationships.

[0354] It is the bias of the first fully connected layer, providing an offset for each intermediate layer neuron, enabling the model to adapt to different baseline conditions. For example, different ambient noise levels, different default bolt tightening states, etc.

[0355] H1 represents the features after activation of the first fully connected layer. This approach introduces nonlinearity through the ReLU activation function, enabling the model to learn more complex feature interactions. In the loose bolt localization task, H1 might represent the features after the first nonlinear transformation, fusing information from different channels.

[0356] This refers to the second-level weight matrix. It is in the same dimension. The inner layer performs transformations to further deepen the feature representation. This layer can learn more complex feature interactions, such as higher-order correlations between features corresponding to different microphones.

[0357] It is the bias of the second fully connected layer.

[0358] H2 is the feature after activation of the second fully connected layer. After two layers of nonlinear transformation, H2 is a highly abstract feature representation that integrates information from all channels and encodes complex patterns related to bolt position.

[0359] This refers to the third-layer weight matrix. It shifts the features from the middle layer dimensions. Mapping to global feature dimension The purpose of this fully connected layer is to compress the high-dimensional abstract features learned earlier into a more compact global feature representation, which facilitates subsequent coordinate regression. This refers to the global feature dimension (e.g., 256). The dimension needs to be large enough to represent the bolt's position information, but it cannot be too large to avoid making the coordinate regression network overly complex. Typically... Setting it to 256 is a value that balances expressive power and complexity.

[0360] This refers to the bias of the third fully connected layer.

[0361] This refers to the final output of the globally fused features. It has undergone layer normalization to stabilize the feature distribution, which is beneficial for training subsequent regression networks. In the task of locating loose bolts, It is a feature vector that contains all channel information, attribute information, and spatial information, and has been highly abstracted and compressed. It can be directly used to regress the three-dimensional spatial coordinates of the bolt.

[0362] In summary, in the loose bolt location task, multiple microphones are deployed at different locations on the tower. The sound signals collected by each microphone are processed through feature extraction to form initial features. These features are then processed through path one and path two, and further focused using an adaptive attention mechanism to ultimately form... .but, It may still be in a high-dimensional and complex hybrid state, so further feature fusion is needed to extract higher-level abstract features.

[0363] In other words, the purpose of cross-channel feature fusion is to deeply fuse these mixed features and extract global features directly related to location. For example, it may learn spatial location correlations; assuming microphones 1 and 3 simultaneously exhibit specific frequency peaks, and microphone 2 has a shorter signal delay, it determines that the bolt loosening location is on the northwest side of the tower. Another example is that it may learn material property correlations; when the bolt material is carbon steel, its sound attenuates faster, thus requiring an increase in the contribution weight of the near-field microphone signal.

[0364] Taking a feature fusion network consisting of three fully connected layers as an example, the model can learn these complex cross-channel interactions through the first two fully connected layers. Then, a dimensionality reduction operation is performed through the third fully connected layer to obtain compact global features.

[0365] In addition, layer normalization standardizes features, making training more stable. In the task of locating loose bolts, changes in the environment (such as temperature and humidity) may cause changes in feature distribution, and layer normalization helps improve the robustness of the model.

[0366] 3042. Localization Coordinate Regression: Integrating Global Features The location of the bolt is predicted by mapping it to three-dimensional spatial coordinates.

[0367] In this embodiment, the sound source localization model further includes a coordinate regression network. The input to the coordinate regression network is global fusion features. The output is the predicted 3D spatial coordinates of the bolt by the model. Assuming the coordinate regression network is a multilayer perceptron, its output layer has 3 nodes, corresponding to the 3D spatial coordinates. As mentioned earlier, this feature has removed redundancy but retains sufficient information to regress the 3D spatial coordinates. It is a global comprehensive decision feature regarding the bolt position, incorporating information from all microphones and undergoing intelligent arbitration and compression, containing all the key information needed to determine the bolt position. This includes encoding the sound source direction, estimating the sound source distance, compensating for environmental interference, and positioning confidence.

[0368] Taking a coordinate regression network as an example, which is a multilayer perceptron with two fully connected layers in its hidden layers, the coordinate regression network will be introduced in detail below through the following steps.

[0369] Step 1: Based on the first layer weight matrix of the coordinate regression network, the global fusion features are obtained using the following formula 50. This is mapped to a first intermediate representation. The first-layer weight matrix is ​​used to learn how to fuse global features. It is mapped to an intermediate physical quantity.

[0370] Formula 50:

[0371] Step 2: Based on the second-layer weight matrix of the coordinate regression network, the first intermediate representation is mapped to the second intermediate representation using the following formula 51. The second-layer weight matrix is ​​used to learn the interactions between intermediate physical quantities.

[0372] Formula 51:

[0373] Step 3: Map the second intermediate representation to the predicted spatial location of the bolt using the following formula 52.

[0374] Formula 52:

[0375] The parameters of the coordinate regression network are introduced below.

[0376] The first fully connected layer: This refers to the first-layer weight matrix, which will weigh d g Global fusion features in 256 dimensions (e.g., 256 dimensions) Mapped to the first intermediate representation Z1 (dimension d) r (e.g., 128-dimensional). In bolt positioning tasks, Learn how to decompose comprehensive features into more fundamental physical quantities. For example, converting sound source direction encoding into direction cosine, converting sound source distance encoding into a preliminary estimate of the actual distance, and extracting feature components that are related to height. It is the bias of the first fully connected layer, used to compensate for system errors, such as microphone array installation deviations and changes in sound speed with temperature. In addition, nonlinearity is introduced by combining it with the ReLU activation function, while ensuring that certain intermediate physical quantities (such as distance or intensity) are non-negative.

[0377] The second fully connected layer is used for higher-order coupling and correction between intermediate physical quantities. This refers to the second-layer weight matrix, which further processes intermediate features and learns the interactions between different physical quantities. In the bolt positioning task, it may learn: the coupling between direction and distance (e.g., the attenuation rate of sound wave signals in a specific propagation direction is significantly higher than in other directions); the geometric constraints of height and distance (e.g., when the height of the sound source increases, its horizontal propagation distance is often limited by the tower structure, forming a fixed geometric relationship); and the influence of environmental factors, such as the modulation effect of wind speed changes on the sound wave propagation path and attenuation. It is the bias of the second fully connected layer, used to adjust the reference for coupling.

[0378] The third fully connected layer (output layer) is used to perform linear transformations from intermediate physical quantities to three-dimensional spatial coordinates. This refers to the third-layer weight matrix, which will weigh d r The second intermediate feature of a dimension (e.g., 128-dimensional) represents the Z2 mapping to three-dimensional spatial coordinates. In bolt positioning tasks, Each column corresponds to a coordinate axis, and they learn how to combine intermediate physical quantities to predict the three-dimensional spatial coordinates of the bolt. This refers to the offset of the coordinate system. Assuming the origin of the coordinate system is the center of the tower, the bolt positions are usually not at the origin, hence the need for offset. Of course, It can also be used to compensate for measurement deviations.

[0379] This refers to the intermediate layer dimension, which determines the expressive power of the coordinate regression network. It is usually set to 128 or 256, a value sufficient to represent the features fused from the global fusion. Nonlinear mapping to three-dimensional spatial coordinates. Additionally, if... If the value is too small, underfitting will occur, and complex relationships will be unable to be captured; if... Too large an amount may lead to overfitting.

[0380] It should be noted that, This refers to the predicted position of the loose bolt in the tower coordinate system. In other words, the coordinate regression network predicts the spatial position of the loose bolt. and These are horizontal coordinates, usually with the center of the tower as the origin. These are coordinates in the vertical direction, typically calculated from the ground or the base of the tower.

[0381] As an example, the coordinate system of the tower is defined as follows: Origin point: Usually located at the center of the tower.

[0382] x-axis: East-west direction (or prevailing wind direction).

[0383] y-axis: North-South direction (or perpendicular to the prevailing wind direction).

[0384] z-axis: Vertical upward direction.

[0385] 305. The computer equipment trains the sound source localization model based on the reference spatial position of the bolt and the spatial position predicted by the sound source localization model.

[0386] In some embodiments, this scheme trains the entire model in an end-to-end manner based on training samples included in the sample dataset. The optimization objective is to minimize the error between the predicted spatial position and the reference spatial position of the bolt, i.e., the positioning error loss.

[0387] It should be noted that the above describes the extraction of corresponding features through path one and path two respectively, and after feature fusion, the features are fused globally. Three-dimensional spatial coordinates are predicted, and then loss is calculated based on the predicted and actual three-dimensional spatial coordinates. In this case, the actual three-dimensional spatial coordinates are the spatial position of the bolt relative to the entire microphone array.

[0388] In addition, the loss can be calculated using the loss function designed for path one, and the parameters of the entire model can be updated accordingly. This process will be described in detail below.

[0389] For path one, the training objective is to force all parameter optimizations along this path to serve a single goal: to infer the sound source location as accurately as possible from the sound signal corresponding to a single microphone channel. This motivates the model to extract all spatially relevant clues and ignore irrelevant details such as the physical properties of bolts.

[0390] First, a regression head is designed for each microphone channel. The regression head can be a simple fully connected network, such as a multilayer perceptron; this application does not limit this to a specific type.

[0391] As an example, assuming the regression head is a multilayer perceptron, the spatial propagation features corresponding to the i-th microphone channel can be mapped to three-dimensional spatial coordinates using the following formula 53. .

[0392] Formula 53:

[0393] in, It means The spatial propagation characteristics corresponding to the i-th microphone channel.

[0394] Next, the auxiliary regression loss for each microphone channel is calculated. For example, the predicted position for each microphone channel is calculated using the following formula 54. With reference spatial position The mean square error loss between them.

[0395] Formula 54:

[0396] in, This refers to the mean square error loss of the i-th microphone channel.

[0397] Next, the average loss of all microphone channels is calculated using the following formula 55. .

[0398] Formula 55:

[0399] The above process can be summarized as follows: For any channel of the microphone array, map the spatial propagation characteristics corresponding to that channel into three-dimensional spatial coordinates to obtain the first spatial position of the bolt; based on the first spatial position and the reference spatial position of the bolt relative to that channel, determine the positioning loss of that channel. ; Obtain the average positioning loss of each channel in the microphone array to obtain the first positioning loss. .

[0400] In addition, this solution fuses the spatial propagation features corresponding to each microphone channel to obtain fused features; then, a regression head maps the fused features into three-dimensional spatial coordinates. .

[0401] As an example, this scheme fuses the spatial propagation features corresponding to each microphone channel through a cross-channel attention mechanism. The core of cross-channel attention fusion is to model the feature correlation between microphone channels. First, the weights corresponding to each microphone channel are calculated through the attention mechanism. Then, the spatial propagation features corresponding to each microphone channel are weighted and summed accordingly to obtain the fused features.

[0402] Next, calculate the three-dimensional spatial coordinates. The second positioning loss is obtained by considering the mean square error between the bolt's position relative to the microphone array's reference spatial position. .

[0403] Finally, the total loss is calculated using the following formula 56.

[0404] Formula 56:

[0405] in, These are hyperparameters used to balance the initial localization loss. Second positioning loss .

[0406] For example, this scheme updates all parameters of the model by minimizing the total loss using gradient descent, including parameters of path one, path two, and the parameters of the multi-head attention module.

[0407] The above process can be summarized as follows: fuse the spatial propagation features corresponding to each channel to obtain the second feature vector; map the second feature vector to three-dimensional spatial coordinates to obtain the second spatial position of the bolt. Based on the second spatial location The second positioning loss is determined by the reference spatial position of the bolt relative to the microphone array. Ultimately, based on the first positioning loss... Second positioning loss Train the sound source localization model.

[0408] It should be noted that this scheme uses attribute classification tasks for multi-task learning supervision. That is, in addition to optimizing the model to reduce positioning error loss, the model can also be guided to learn features related to the physical properties of the bolts through attribute classification tasks.

[0409] Accordingly, the second encoder includes multiple classifiers, with different classifiers used to perform different attribute classification tasks. Additionally, each training sample includes multiple reference attributes of the bolt.

[0410] As an example, the training process for these classifiers includes: physical property characteristics Input each classifier and use each classifier to predict the attributes of the bolt to obtain multiple predicted attributes of the bolt; for any attribute classification task, obtain the classification loss of the attribute classification task based on the reference attribute and the predicted attribute of the bolt; train multiple classifiers based on the classification loss of each attribute classification task.

[0411] For example, suppose the attribute classification task includes a model classification task and an installation location classification task. Then, the model classification task corresponds to a classifier used to predict the specific model of the bolt. Similarly, the installation location classification task also corresponds to a classifier used to predict the installation location of the bolt on the tower body.

[0412] For model classification tasks, physical attribute features Input the corresponding classifier to obtain the model number predicted by the classifier; then, the cross-entropy loss between the predicted model number (predicted label) and the reference model number (true label) can be calculated using the cross-entropy loss function.

[0413] For the installation location classification task, physical attribute features Input the corresponding classifier to obtain the installation location predicted by the classifier; then, the cross-entropy loss between the predicted installation location and the actual installation location can be calculated using the cross-entropy loss function.

[0414] Next, calculate the total loss, which is the weighted sum of the two losses.

[0415] The training objective of path two is to force the parameter optimization under path two to accurately identify the physical properties of the sound source. The model must learn to extract features from the sound signal that can distinguish the physical properties of the bolts. For example, small bolts may have a crisper sound (rich in high frequencies), while the sound of tower bolts may contain more low-frequency components reflected from the ground.

[0416] In summary, this approach, through multi-task learning, can simultaneously predict multiple bolt attributes, providing rich feature information for locating loose bolts. In other words, multi-task learning enables the model to learn multiple physical attributes of the bolt simultaneously. These physical attributes collectively constitute the bolt's acoustic signature features, and there may be correlations between them. Joint learning can improve the performance of each attribute classification task.

[0417] It should be noted that this solution also supports training the coordinate regression network separately.

[0418] In this embodiment of the application, the training process of the coordinate regression network includes the following steps.

[0419] First, based on the reference spatial position and the predicted spatial position of the bolt, the main loss is obtained using the following formula 57.

[0420] Formula 57:

[0421] The principal loss minimizes the Euclidean distance between the predicted 3D spatial coordinates and the true 3D spatial coordinates. The principal loss forces the network to learn accurate coordinate mappings, directly optimizing positioning accuracy. In Equation 57, N refers to the number of samples in the current batch. These are the three-dimensional spatial coordinates included in the i-th training sample, which are usually obtained through prior measurement or simulation.

[0422] Next, based on the height constraint and the horizontal distance constraint, the auxiliary loss is obtained using the following formula 58. The height constraint is used to ensure that the predicted height of the bolt does not exceed the height of the tower; the horizontal distance constraint is used to ensure that the horizontal radius of the predicted location does not exceed the minimum horizontal radius of the tower.

[0423] Formula 58:

[0424] The auxiliary loss, also known as the geometric constraint loss, utilizes the physical structural constraints of the tower to improve the rationality of the network predictions. Specifically, the loss function shown in Equation 58 leverages prior knowledge of the tower's structure.

[0425] First item, height constraint .

[0426] Among them, H tower This refers to the tower height. Since bolts cannot be located above the tower height, if the predicted height... Greater than H tower If a violation occurs, a penalty is imposed. This prevents the network from predicting unreasonable high-altitude locations (e.g., errors due to multipath reflections or noise).

[0427] Second item: Horizontal distance constraint .

[0428] in, This refers to the horizontal radius of the predicted spatial location (i.e. ); R min It is the minimum horizontal radius of the tower (e.g., the radius of the tower foundation).

[0429] Since bolts cannot be inside the tower (because the tower is a solid structure and bolts are on the surface), if the predicted horizontal distance is less than R... min This indicates that the prediction reached the interior of the tower, which is unreasonable, hence the penalty. It should be noted that this could also refer to a penalty when the horizontal distance is less than a certain minimum value, because in reality, towers typically have a central axis, and bolts are on the tower surface, so the horizontal distance should be approximately equal to the radius of the tower at that height.

[0430] Finally, the total loss is calculated based on the main loss and auxiliary loss using the following formula 59, and the coordinate regression network is trained based on the calculated total loss.

[0431] Formula 59:

[0432] It should be noted that when calculating the total loss, the geometric constraint loss is usually multiplied by the weights. (Under normal circumstances, The value is relatively small, for example, 0.1, because it is only an auxiliary constraint, and the parameter updates of the coordinate regression network mainly rely on the data-driven main loss.

[0433] The following example will provide a detailed explanation of the process of a coordinate regression network.

[0434] Suppose we need to detect a loose bolt located on the northwest side of a tower at a height of 20 meters, and the input of the coordinate regression network is... The indicated sound source is located northwest, approximately 10 meters horizontally and 20 meters above the ground. The coordinate regression network processing would then look like this: The first-layer weight matrix U1 is used to fuse global features. The values ​​are converted into intermediate physical quantities. The second-layer weight matrix U2 is used for coupling correction. For example, due to the limitations of the tower structure, sound propagation in the northwest direction may be blocked, so the distance needs to be corrected. Also, there is a geometric relationship between height and horizontal distance (the tower is conical). The third-layer weight matrix U3 is used to output the predicted coordinates, assumed to be (-5.2, 3.8, 20.1), in meters.

[0435] Main loss calculation: Calculate the Euclidean distance between the predicted coordinates (-5.2, 3.8, 20.1) and the actual coordinates (-5.0, 4.0, 20.0).

[0436] Geometric constraint loss calculation: Check if the predicted height of 20.1 exceeds the tower height (assumed to be 50 meters); since it does not exceed it, the height constraint term is 0; then check the horizontal distance. Is ≈6.45 less than the minimum radius (assumed to be 1 meter)? Since it is not less than, the horizontal distance constraint term is also 0. Accordingly, total loss = main loss.

[0437] Network optimization: Update parameters through backpropagation to make predictions more accurate.

[0438] It should be noted that, in order to improve training stability and make the numerical range more suitable for neural network processing, this scheme will also normalize the three-dimensional spatial coordinates included in each training sample to [ ] during the training phase. The interval is [1, 1]. The coordinate normalization operation can be expressed as formulas 60-62 below.

[0439] Formula 60:

[0440] Formula 61:

[0441] Formula 62:

[0442] In formulas 60-62, x, y, and z refer to the actual three-dimensional spatial coordinates of the bolt in the tower coordinate system. norm ,y norm , z norm It refers to the normalized three-dimensional spatial coordinates used for neural network training.

[0443] μ x It refers to the average x-coordinate contained in each training sample in the sample dataset. μ y This refers to the average y-coordinate of each training sample in the dataset. The physical meaning of these two values ​​is: the "average center" position of the tower on the horizontal plane.

[0444] Since the bolts are distributed throughout the tower, this average center can be considered the geometric center of the tower's horizontal projection. In reality, the distribution of bolts on the tower may not be perfectly symmetrical, so this center may deviate slightly from the center of the tower base (the origin of the coordinate system). By subtracting the mean, the center of the data can be shifted to zero, so that the normalized coordinates are distributed with zero as the center, which helps the neural network learn.

[0445] σ x σ refers to the standard deviation of the x-coordinates contained in each training sample in the dataset. y This refers to the standard deviation of the y-coordinate in each training sample within the dataset. The physical meaning of these two values ​​is that they reflect the extent to which bolts are distributed horizontally. A large standard deviation indicates a wide distribution range of bolts in the x or y direction (large lateral dimension of the tower); a small standard deviation indicates a concentrated distribution of bolts in the x or y direction (small lateral dimension of the tower). Dividing by the standard deviation allows for data scaling, ensuring that the coordinate data of towers of different sizes have a similar numerical range, facilitating neural network processing.

[0446] In the task of locating loose bolts, the z-coordinate represents the height of the bolt from the ground or the tower base, so the upper limit of the z-coordinate value is H. tower Divide the z-coordinate by H tower This normalizes the height to the [0,1] interval. The physical meaning of doing so is to convert the absolute height into a relative height (the proportion relative to the total height of the tower), so that the z-coordinates of towers at different heights have the same scale.

[0447] In summary, due to the different dimensions of different tower bodies, the coordinate range of the bolts also varies. By performing coordinate normalization, this solution can apply the same neural network to tower bodies of different sizes, requiring only the use of the corresponding normalization parameter (μ) during input. x , σ x , μ y , σ y H tower Coordinate normalization can be performed. Furthermore, coordinate normalization helps accelerate the training process of neural networks, preventing certain coordinate axes from dominating training due to excessively large numerical ranges, while also improving the convergence speed and stability of the network.

[0448] During the training phase, training samples are first acquired, and then the corresponding normalization parameters are calculated. The normalized three-dimensional spatial coordinates are then used as the reference spatial position of the bolt. Correspondingly, during the inference phase, the model output is normalized coordinates, and the actual physical coordinates of the bolt are obtained through the inverse normalization operation shown in formulas 63-65 below.

[0449] Formula 63:

[0450] Formula 64:

[0451] Formula 65:

[0452] The following section introduces invariant learning.

[0453] For any training sample (original sample), this scheme further filters the corresponding positive and negative samples in the sample dataset. Assuming the sound signal in the training sample originates from the first bolt in the first state and corresponds to the first sound propagation path, then the sound signal in the positive sample originates from the first bolt in the first state and corresponds to the second sound propagation path; the sound signal in the negative sample originates from the first bolt in the second state, or from a second bolt different from the first bolt. Here, the first and second sound propagation paths are different, and the first and second states are different.

[0454] Based on this, this scheme obtains a first similarity between the first time-frequency map and the second time-frequency map. The first time-frequency map corresponds to the sound signals included in the training samples, and the second time-frequency map corresponds to the sound signals included in the positive samples. In addition, this scheme also obtains a second similarity between the first time-frequency map and the third time-frequency map. The third time-frequency map corresponds to the sound signals included in the negative samples.

[0455] Next, based on the first similarity and the second similarity, the contrastive learning loss is calculated using the contrastive learning loss function shown in Formula 66 below.

[0456] Formula 66:

[0457] in, This refers to the time-frequency diagram of the original sample, also known as the first time-frequency diagram. It represents the baseline acoustic signature characteristics of a specific bolt collected at a specific location under a specific condition. It contains the intrinsic information of that condition, but also includes information such as the specific propagation path and environmental noise at the time of collection. This refers to the time-frequency diagram of the positive sample, i.e., the second time-frequency diagram. It represents the acoustic signature characteristics of the same bolt under the same conditions but with different propagation paths, which is consistent with... The corresponding bolts are in the same state, but are affected by different path interferences. It refers to the time-frequency diagram of the negative sample, that is, the third time-frequency diagram, which represents the acoustic characteristics of the same bolt in different states or different bolts. This represents a sound source that is fundamentally different from the target that needs to be identified.

[0458] This refers to the first similarity. This refers to the second similarity; τ is a scalar hyperparameter used to adjust the similarity distribution.

[0459] It's important to note that the core of the contrastive learning loss function is to maximize the similarity between the time-frequency features of the original sample and the time-frequency features of the positive sample, while minimizing the similarity between the time-frequency features of the original sample and the time-frequency features of the negative sample. Furthermore, minimizing this loss function directly achieves two objectives: 1. Maximize invariance: Maximize the numerator This forces the model to ignore the differences in propagation paths simulated by time stretching and frequency masking, and extracts... and The inherent characteristics of the bolt state shared behind it.

[0460] 2. Maximize discriminative power: Minimize the number of negative samples in the denominator. The sum of similarities forces the model to enhance the feature differences between different bolt states. The feature space learned in this way allows samples of the same state but different paths to cluster together, while samples of different states are far apart, which meets the requirements of this scheme for "path robust and state sensitive" voiceprint recognition and feature extraction.

[0461] In some embodiments, the contrastive learning loss can be weighted and fused with the localization error loss described in step 305 to construct a multi-objective joint training loss function, thereby achieving end-to-end optimization of the sound source localization model. This application does not limit this.

[0462] In other embodiments, It can also be by After data augmentation, the following approach can be used to calculate the contrastive learning loss: For any training sample, using it as the original sample, obtain the third similarity between the time-frequency map of the sound signal included in the original sample and the time-frequency map after data augmentation; and obtain the fourth similarity between the time-frequency map of the sound signal included in the original sample and the time-frequency map of the sound signal included in the negative sample; then, based on the third and fourth similarities, calculate the contrastive learning loss using the contrastive learning loss function.

[0463] In summary, the embodiments of this application achieve at least the following aspects.

[0464] I. This solution achieves high-precision and robust sound source localization.

[0465] This solution employs a deep neural architecture, enabling it to directly learn complex acoustic feature mapping relationships from the original time-frequency plot. Its powerful nonlinear modeling capabilities allow it to better separate effective signals related to bolt loosening from complex background noise. More importantly, the adaptive attention mechanism embedded in the model acts as an intelligent feature filter, dynamically enhancing features crucial for sound source localization based on the current input sound signal, while suppressing irrelevant noise or interference features. This data-driven adaptive capability allows the model to maintain high localization accuracy even when faced with changes in bolt position or type.

[0466] Second, this solution can effectively overcome the problem of voiceprint diversity.

[0467] This solution employs a dual-path encoder structure to achieve decoupled explicit learning of spatial propagation characteristics and physical property characteristics. One path focuses on modeling the propagation patterns of sound waves at different spatial locations, while the other path focuses on extracting the physical property characteristics of the bolt itself. Building upon this, the model introduces a multi-head attention mechanism, enabling the network to adaptively combine and weight the two types of features for different working conditions. For example, in the scenario of loose bolts on tower tops, the model emphasizes the feature patterns of multipath reflections; in the scenario of large-diameter bolts, the model prioritizes capturing their low-frequency energy characteristics. Furthermore, during the inference phase, since the attention weights are dynamically calculated and generated from the real-time input sound signal, the model possesses excellent generalization ability and adaptive characteristics, eliminating the need for custom-designed models for different bolt types or installation locations.

[0468] It should be noted that after the sound source localization model is trained, the sound source can be located for the loose bolt based on the acoustic signal to be detected during the inference phase. That is, the embodiments of this application support online localization. The trained model is deployed on an edge computing device or server, and a microphone array located on site is responsible for collecting sound signals. After preprocessing, the collected sound signals can be input into the trained sound source localization model, and the model can directly output the spatial location of the loose bolt.

[0469] Figure 5 This is a flowchart illustrating a sound source localization method provided in an embodiment of this application. The method is executed by a computer device. See also... Figure 5 The method includes the following steps.

[0470] 501. The computer equipment acquires the sound signal to be detected and performs time-frequency analysis on the sound signal to be detected to obtain the time-frequency diagram of the sound signal to be detected; wherein, the sound signal to be detected is collected by microphones deployed around the tower body to be detected, and bolts are installed on the tower body.

[0471] The system employs multiple microphones, deployed at various spatial locations around the tower to be monitored. Furthermore, this solution supports the deployment of sparse microphone arrays (e.g., 4-8 nodes) on-site. In other words, the microphone array design considers a balance between spatial coverage and cost.

[0472] Furthermore, when the signal energy of any microphone channel exceeds the energy threshold, this solution triggers multiple microphones (usually all microphones) to collect a sound signal of a preset duration. The sound signal to be detected includes multiple sound signals of the preset duration. For example, the preset duration can be 1 second, and this application does not limit this value.

[0473] 502. The computer equipment inputs the time-frequency diagram of the sound signal to be detected into the trained sound source localization model, and predicts the spatial position of the loose bolt on the tower body through the sound source localization model.

[0474] In some embodiments, the computer device performs time-frequency analysis on each audio signal with a preset duration to obtain multiple time-frequency maps; correspondingly, the computer device inputs these time-frequency maps into a sound source localization model, and then uses the sound source localization model to predict the spatial location of the bolt in a loose state.

[0475] In summary, the embodiments of this application are applied to sound source localization scenarios, such as predicting the spatial position of a loose connector, where the connector is mounted on a mechanical connection structure. Specifically, these embodiments use sound signals as the data processing object and train a sound source localization model based on pre-built training samples, achieving automatic sound source localization based on the trained model. Since this method requires no manual intervention, it saves labor costs and avoids the risks of working at heights compared to manual inspection methods, while also being more efficient and enabling real-time monitoring of the connector's tightness. Furthermore, during model training, these embodiments extract feature vectors containing sound spatial propagation characteristics and the connector's physical properties based on the time-frequency diagram of the sound signal. An attention mechanism is then introduced to automatically strengthen features in the feature vector that are crucial for sound source localization, while suppressing irrelevant noise or interference features. This allows the model to focus on core discriminative features, effectively reducing feature redundancy under complex conditions, resulting in a sound source localization model with higher localization accuracy during the inference phase.

[0476] Furthermore, this solution enhances the model's practicality in complex environments. On one hand, the attention mechanism indirectly improves the model's anti-interference capability by focusing on the signal itself. On the other hand, the solution employs a sparse microphone array, eliminating the need for numerous microphones on the tower, thus reducing deployment costs and maintenance complexity. The entire solution is based on end-to-end deep learning, automating the "sound input - coordinate output" process, significantly improving detection efficiency and automation levels, and providing a feasible technical path for intelligent and routine monitoring of tower structure safety.

[0477] Figure 6 This is a schematic diagram of the structure of a sound source localization model training device provided in an embodiment of this application. See also... Figure 6 The device includes: The first acquisition module 601 is configured to acquire training samples, the training samples including sound signals of a loose connector and a reference spatial position of the connector, the connector being mounted on a mechanical connection structure; The first processing module 602 is configured to input the time-frequency diagram of the sound signal into the sound source localization model, and extract features from the time-frequency diagram through the sound source localization model to obtain a first feature vector; the first feature vector is used to indicate the spatial propagation characteristics of the sound signal and the physical properties of the connector. The second processing module 603 is configured to convert the first feature vector into attention-weighted features through the sound source localization model; the attention-weighted features are used to reflect the key features in the first feature vector related to the state of the connector from multiple dimensions. Prediction module 604 is configured to predict the spatial position of the connector based on the attention-weighted features and the sound source localization model. Training module 605 is configured to train the sound source localization model based on the reference spatial location and the spatial location predicted by the sound source localization model.

[0478] This application embodiment is applied to sound source localization scenarios, such as predicting the spatial location of a loose connector, where the connector is mounted on a mechanical connection structure. Specifically, this application embodiment uses sound signals as the data processing object and trains a sound source localization model based on pre-built training samples, achieving automatic sound source localization based on the trained model. Since this method requires no manual intervention, compared to manual inspection, it not only saves labor costs and avoids the risks of working at heights, but is also more efficient, enabling real-time monitoring of the connector's tightness. Furthermore, during model training, this application embodiment extracts feature vectors containing sound spatial propagation characteristics and the connector's physical properties based on the time-frequency diagram of the sound signal. An attention mechanism is then introduced to automatically strengthen features in the feature vector that are crucial for sound source localization, while suppressing irrelevant noise or interference features. This allows the model to focus on core discriminative features, effectively reducing feature redundancy under complex conditions, resulting in a sound source localization model with higher localization accuracy during the inference phase.

[0479] In some embodiments, the sound source localization model includes a multi-head attention module, and the second processing module is configured to: For the i-th attention head, based on the linear projection matrix of the i-th attention head and the first feature vector, the query matrix, key matrix, and value matrix of the i-th attention head are generated; wherein, the parameters of the linear projection matrix are updated during model training; i is a positive integer; Based on the query matrix and key matrix of the i-th attention head, the attention weight map of the i-th attention head is obtained; wherein, the attention weight map marks the importance of each feature component in the first feature vector by the attention weight; The output features of the i-th attention head are obtained by weighted summation of the attention weight map and value matrix. The attention-weighted features are generated based on the output features of the multiple attention heads.

[0480] In other embodiments, the second processing module is configured to: The output features of the multiple attention heads are concatenated; The concatenated features are fused based on the output projection matrix to obtain the fused output features; wherein the parameters of the output projection matrix are updated during model training. An addition operation is performed on the first feature vector and the fused output feature, and a layer normalization operation is performed on the added feature to obtain the attention-weighted feature.

[0481] In other embodiments, the second processing module is configured to: Based on the first linear projection matrix of the i-th attention head, the first feature vector is projected onto the query space to obtain the query matrix of the i-th attention head; Based on the second linear projection matrix of the i-th attention head, the first feature vector is projected onto the key space to obtain the key matrix of the i-th attention head; Based on the third linear projection matrix of the i-th attention head, the first feature vector is projected onto the value space to obtain the value matrix of the i-th attention head.

[0482] In other embodiments, the first processing module includes: The first processing submodule is configured to extract features from the time-frequency diagram using the first encoder of the sound source localization model to obtain the spatial propagation features of the sound signal. The second processing submodule is configured to extract features from the time-frequency diagram using the second encoder of the sound source localization model to obtain the physical property features of the connector. The fusion submodule is configured to fuse the spatial propagation features and the physical attribute features to obtain the first feature vector.

[0483] In other embodiments, the first processing submodule is configured to: An acoustic feature map is generated based on the time-frequency diagram, and the acoustic feature map includes feature values ​​at any spatial location within each feature channel; Global average pooling is performed on the acoustic feature map to obtain channel feature vectors; wherein each component of the channel feature vector is the global pixel mean value within a single feature channel of the acoustic feature map; Based on the channel correlation matrix, the channel feature vectors are compressed to obtain compressed channel feature vectors; the channel correlation matrix is ​​used to learn the weights corresponding to different feature channel combinations. Based on the channel weight reconstruction matrix, the compressed channel feature vectors are dimension-restored to obtain the channel weight vectors; the channel weight reconstruction matrix is ​​used to learn the importance of each feature channel. Based on the channel weight vector, the acoustic feature map is weighted to obtain a new acoustic feature map, which is then converted into the spatial propagation feature.

[0484] In other embodiments, the first processing submodule is configured to: A non-dilated convolutional layer is used to perform a convolution operation on the time-frequency map to obtain time-frequency features; Multiple parallel dilated convolutional layers are used to perform convolution operations on the time-frequency features to obtain multiple output feature maps; wherein, different dilated convolutional layers have different dilation rates. The multiple output feature maps are concatenated to obtain a multi-scale feature set; The acoustic feature map is obtained by fusing channel-dimensional features of the multi-scale feature set using a multi-scale feature fusion processor.

[0485] In other embodiments, the sound signal is acquired by a microphone array, which includes multiple microphones deployed in different spatial locations; The prediction module is also configured to map the spatial propagation features corresponding to any channel of the microphone array into three-dimensional spatial coordinates to obtain the first spatial position of the connector. The training module is also configured to determine the positioning loss of the channel based on the first spatial position and the reference spatial position of the connector relative to the channel; The training module is also configured to obtain the average localization loss of each channel in the microphone array to obtain the first localization loss; The prediction module is further configured to fuse the spatial propagation features corresponding to each channel to obtain a second feature vector; and to map the second feature vector to three-dimensional spatial coordinates to obtain the second spatial position of the connector. The training module is further configured to determine a second localization loss based on the second spatial position and the reference spatial position of the connector relative to the microphone array; The training module is also configured to train the sound source localization model based on the first localization loss and the second localization loss.

[0486] In other embodiments, the second processing submodule is configured to: A non-dilated convolutional layer is used to perform a convolution operation on the time-frequency map to obtain time-frequency features; While maintaining frequency resolution, an asymmetric convolution kernel is used to perform a convolution operation on the time-frequency features along the time direction, and the time-frequency features are added to the time-frequency features after the convolution operation through residual connections to obtain new time-frequency features; wherein, the time dimension kernel size of the asymmetric convolution kernel is larger than the frequency dimension kernel size; An attention weight map is generated based on the new time-frequency features; wherein, the attention weight map marks the importance of each time-frequency region in the time-frequency map by attention weight; Based on the attention weight map and the new time-frequency features, the optimized time-frequency features are generated; The physical property features are generated based on the optimized time-frequency features.

[0487] In other embodiments, the second processing submodule is configured to: Perform channel-dimensional average pooling on the new time-frequency features to obtain the average energy distribution of each time-frequency point across all feature channels; Max pooling is performed on the new time-frequency features to obtain the maximum energy distribution of each time-frequency point across all feature channels; The attention weight map is generated based on the average energy distribution and maximum energy distribution at each time frequency point.

[0488] In other embodiments, the second processing submodule is configured to: Global average pooling is performed on the optimized time-frequency features to obtain channel feature vectors; wherein each component of the channel feature vector is the global pixel mean value within a single feature channel of the optimized time-frequency features; Based on the channel feature vector, the physical properties of the connector are predicted to obtain multiple attribute probability distributions; The probability distributions of the multiple attributes are converted into the physical attribute features.

[0489] In other embodiments, the second processing submodule is configured to: Based on the semantic projection matrix, the probability distributions of the multiple attributes are mapped to the same continuous semantic subspace to obtain multiple attribute vectors; Obtain the dot product similarity between the query vector and each attribute vector to get the attention score of each physical attribute of the connector; The attention scores of each physical attribute are normalized to obtain the attention weights of each physical attribute. The attention weights of each physical attribute and the multiple attribute vectors are weighted and summed to obtain an attribute fusion vector, which is then converted into the physical attribute features. One of the attribute vectors is used to characterize a physical attribute of the connector; the parameters of the semantic projection matrix and the query vector are updated during model training.

[0490] In other embodiments, the second processing submodule is configured to: Based on the first semantic projection matrix, the probability distribution of the first attribute indicating the connector model is mapped to the continuous semantic subspace to obtain the first attribute vector; the first attribute vector is a continuous semantic feature containing the connector model information. Based on the second semantic projection matrix, the probability distribution of the first attribute indicating the material of the connector is mapped to the continuous semantic subspace to obtain the second attribute vector; the second attribute vector is a continuous semantic feature containing the material information of the connector. Based on the third semantic projection matrix, the probability distribution of the first attribute indicating the fastening state of the connector is mapped to the continuous semantic subspace to obtain the third attribute vector; the third attribute vector is a continuous semantic feature containing the state information of the connector. Based on the fourth semantic projection matrix, the probability distribution of the first attribute indicating the installation position of the connector is mapped to the continuous semantic subspace to obtain the fourth attribute vector; the fourth attribute vector is a continuous semantic feature containing the installation position information of the connector.

[0491] In other embodiments, the second processing submodule is configured to: The time-frequency graph is subjected to data augmentation processing to obtain the processed time-frequency graph; The non-dilated convolutional layer is used to perform a convolution operation on the processed time-frequency map to obtain the time-frequency features.

[0492] In other embodiments, the second processing submodule is configured to: Based on the stretching factor, the time axis of the time-frequency graph is scaled. Interpolation is performed on the scaled time axis to fill in the missing data points in the time dimension, resulting in the processed time-frequency graph.

[0493] In other embodiments, the second processing submodule is configured to: Select the mask start frequency within the effective frequency range of the time-frequency diagram; Based on the mask width and the mask start frequency, determine the frequency range to be masked; The energy values ​​in the frequency range to be masked in the time-frequency graph are replaced with preset values ​​to obtain the processed time-frequency graph; the preset values ​​are random noise values, the mean background noise of the time-frequency graph, or zero.

[0494] In other embodiments, the training module is further configured to: For any training sample, a first similarity is obtained between the first time-frequency map and the second time-frequency map; the first time-frequency map corresponds to the sound signal included in the training sample, and the second time-frequency map corresponds to the sound signal included in the positive sample; Obtain a second similarity between the first time-frequency map and the third time-frequency map; the third time-frequency map corresponds to the sound signal included in the negative sample. The sound source localization model is trained based on the first similarity and the second similarity. The training samples include sound signals from the first connector in the first state and corresponding to the first sound propagation path; the positive samples include sound signals from the first connector in the first state and corresponding to the second sound propagation path; the negative samples include sound signals from the first connector in the second state, or from a second connector different from the first connector.

[0495] In other embodiments, the training samples also include multiple reference attributes of the connector; the second encoder includes multiple classifiers, different classifiers being used to perform different attribute classification tasks; the training process of the multiple classifiers includes: The physical property features are input into the multiple classifiers, and the multiple classifiers are used to predict the attributes of the connector to obtain multiple predicted attributes of the connector. For any attribute classification task, the classification loss of the attribute classification task is obtained based on the reference attribute and the predicted attribute corresponding to the connector. The multiple classifiers are trained based on the classification loss for each attribute classification task.

[0496] In other embodiments, the prediction module is configured to: The attention-weighted features corresponding to each channel in the microphone array are fused to obtain global fused features; the global fused features are mapped to three-dimensional spatial coordinates to obtain the predicted spatial position of the connector.

[0497] In other embodiments, the prediction module is configured to: Multiple nonlinear transformations are performed on the attention weighted features corresponding to each channel to obtain nonlinear transformation results; wherein, the nonlinear transformation results are used to characterize the correlation between the attention weighted features corresponding to each channel; The nonlinear transformation result is dimensionally compressed, and a layer normalization operation is performed on the dimensionally compressed nonlinear transformation result to obtain the global fusion feature.

[0498] In other embodiments, the sound source localization model includes a feature fusion network; the prediction module is configured to: Based on the first layer weight matrix of the feature fusion network, a nonlinear transformation is performed on the attention-weighted features corresponding to each channel to obtain intermediate transformation results; the first layer weight matrix is ​​used to learn the basic correlation between the attention-weighted features corresponding to each channel. Based on the second-layer weight matrix of the feature fusion network, the intermediate transformation result is subjected to a nonlinear transformation to obtain the nonlinear transformation result; the second-layer weight matrix is ​​used to learn the higher-order correlation of the attention-weighted features corresponding to each channel after the first nonlinear transformation.

[0499] In other embodiments, the sound source localization model includes a coordinate regression network; the prediction module is configured to: Based on the first layer weight matrix of the coordinate regression network, the global fusion features are mapped to a first intermediate representation; the first layer weight matrix is ​​used to learn to map the global fusion features to an intermediate physical quantity. Based on the second-layer weight matrix of the coordinate regression network, the first intermediate representation is mapped to the second intermediate representation; the second-layer weight matrix is ​​used to learn the mutual influence between the intermediate physical quantities. The second intermediate representation is mapped to the predicted spatial location of the connector.

[0500] In other embodiments, the training process of the coordinate regression network includes: Based on the reference spatial position and the predicted spatial position of the connector, the main loss is obtained; An auxiliary loss is obtained based on a height constraint term and a horizontal distance constraint term; wherein, the height constraint term is used to constrain the predicted height of the connector to not exceed the height of the mechanical connection structure; the horizontal distance constraint term is used to constrain the horizontal radius of the predicted position to not exceed the minimum horizontal radius of the mechanical connection structure; The coordinate regression network is trained based on the main loss and the auxiliary loss.

[0501] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0502] Figure 7 This is a schematic diagram of the structure of a sound source localization device provided in an embodiment of this application. See also... Figure 7 The device includes: The second acquisition module 701 is configured to acquire a sound signal to be detected; wherein the sound signal to be detected is acquired by microphones deployed around the mechanical connection structure to be detected, and the mechanical connection structure is equipped with connectors; The sound source localization module 702 is configured to input the time-frequency diagram of the sound signal to be detected into the sound source localization model, and predict the spatial position of the loose connector through the sound source localization model; wherein, the sound source localization model is trained by the aforementioned sound source localization model training device.

[0503] This application embodiment is applied to sound source localization scenarios, such as predicting the spatial location of a loose connector, where the connector is mounted on a mechanical connection structure. Specifically, this application embodiment uses sound signals as the data processing object and trains a sound source localization model based on pre-built training samples, achieving automatic sound source localization based on the trained model. Since this method requires no manual intervention, compared to manual inspection, it not only saves labor costs and avoids the risks of working at heights, but is also more efficient, enabling real-time monitoring of the connector's tightness. Furthermore, during model training, this application embodiment extracts feature vectors containing sound spatial propagation characteristics and the connector's physical properties based on the time-frequency diagram of the sound signal. An attention mechanism is then introduced to automatically strengthen features in the feature vector that are crucial for sound source localization, while suppressing irrelevant noise or interference features. This allows the model to focus on core discriminative features, effectively reducing feature redundancy under complex conditions, resulting in a sound source localization model with higher localization accuracy during the inference phase.

[0504] In some embodiments, the number of microphones is multiple, with different microphones deployed at different spatial locations around the mechanical structure; the device further includes: The control module is configured to trigger multiple microphones to collect a sound signal of a preset duration when the signal energy of any microphone channel exceeds the energy threshold. The sound signal to be detected includes multiple sound signals with a duration of the preset duration.

[0505] In other embodiments, the sound source localization module is configured to: Perform time-frequency analysis on each segment of the audio signal with a preset duration to obtain multiple time-frequency diagrams; The multiple time-frequency maps are input into the sound source localization model, and the spatial location of the loose connector is predicted by the sound source localization model.

[0506] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0507] It should be noted that the training method for the sound source localization model provided in the above embodiments is only illustrated by the division of the above functional modules when training the sound source localization model or when the sound source localization method performs sound source localization on the connector. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the sound source localization model training device and the sound source localization model training method embodiments provided in the above embodiments belong to the same concept, and the sound source localization device and the sound source localization method embodiments provided in the above embodiments belong to the same concept. For details of their specific implementation process, please refer to the method embodiments, which will not be repeated here.

[0508] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application.

[0509] The computer device 800 can be a server. The computer device 800 can vary considerably depending on its configuration or performance, and includes one or more Central Processing Units (CPUs) 801 and one or more memories 802. The memories 802 store computer program code, which is loaded and executed by the processors 801 to implement the aforementioned sound source localization model training method or sound source localization method. Of course, the computer device 800 also has wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The computer device 800 also includes other components for implementing the device's functions, which will not be elaborated upon here.

[0510] In some embodiments, this application also provides a computer-readable storage medium, such as a memory including computer program code, which can be executed by a processor in a computer device to complete the above-described sound source localization model training method or sound source localization method. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0511] In some embodiments, this application also provides a computer program product, the computer program product including computer program code, the computer program code being stored in a computer-readable storage medium, a processor of a computer device reading the computer program code from the computer-readable storage medium, the processor executing the computer program code, causing the computer device to execute the above-described sound source localization model training method or sound source localization method.

[0512] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0513] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A training method for a sound source localization model, characterized in that, The method includes: Acquire training samples, which include sound signals of a loose connector and a reference spatial position of the connector, which is mounted on a mechanical connection structure; The time-frequency graph of the sound signal is input into the sound source localization model, and the sound source localization model extracts features from the time-frequency graph to obtain a first feature vector; the first feature vector is used to indicate the spatial propagation characteristics of the sound signal and the physical properties of the connector. The sound source localization model converts the first feature vector into attention-weighted features, and the spatial position of the connector is predicted based on the attention-weighted features. The attention-weighted features are used to reflect the key features in the first feature vector that are related to the state of the connector from multiple dimensions. The sound source localization model is trained based on the reference spatial location and the spatial location predicted by the sound source localization model.

2. The method according to claim 1, characterized in that, The sound source localization model includes a multi-head attention module. The step of converting the first feature vector into attention-weighted features using the sound source localization model includes: For the i-th attention head, based on the linear projection matrix of the i-th attention head and the first feature vector, the query matrix, key matrix, and value matrix of the i-th attention head are generated; wherein, the parameters of the linear projection matrix are updated during model training; i is a positive integer; Based on the query matrix and key matrix of the i-th attention head, the attention weight map of the i-th attention head is obtained; wherein, the attention weight map marks the importance of each feature component in the first feature vector by the attention weight; The output features of the i-th attention head are obtained by weighted summation of the attention weight map and value matrix. The attention-weighted features are generated based on the output features of the multiple attention heads.

3. The method according to claim 2, characterized in that, The generation of the attention-weighted features based on the output features of the multiple attention heads includes: The output features of the multiple attention heads are concatenated; The concatenated features are fused based on the output projection matrix to obtain the fused output features; wherein the parameters of the output projection matrix are updated during model training. An addition operation is performed on the first feature vector and the fused output feature, and a layer normalization operation is performed on the added feature to obtain the attention-weighted feature.

4. The method according to claim 2, characterized in that, The step of generating the query matrix, key matrix, and value matrix of the i-th attention head based on the linear projection matrix of the i-th attention head and the first feature vector includes: Based on the first linear projection matrix of the i-th attention head, the first feature vector is projected onto the query space to obtain the query matrix of the i-th attention head; Based on the second linear projection matrix of the i-th attention head, the first feature vector is projected onto the key space to obtain the key matrix of the i-th attention head; Based on the third linear projection matrix of the i-th attention head, the first feature vector is projected onto the value space to obtain the value matrix of the i-th attention head.

5. The method according to claim 1, characterized in that, The step of extracting features from the time-frequency map using the sound source localization model to obtain the first feature vector includes: The spatial propagation characteristics of the sound signal are obtained by extracting features from the time-frequency diagram using the first encoder of the sound source localization model. The second encoder of the sound source localization model is used to extract features from the time-frequency map to obtain the physical property features of the connector. The spatial propagation features and the physical attribute features are fused to obtain the first feature vector.

6. The method according to claim 5, characterized in that, The first encoder of the sound source localization model extracts features from the time-frequency map to obtain the spatial propagation features of the sound signal, including: An acoustic feature map is generated based on the time-frequency diagram, and the acoustic feature map includes feature values ​​at any spatial location within each feature channel; Global average pooling is performed on the acoustic feature map to obtain channel feature vectors; wherein each component of the channel feature vector is the global pixel mean value within a single feature channel of the acoustic feature map; Based on the channel correlation matrix, the channel feature vectors are compressed to obtain compressed channel feature vectors; the channel correlation matrix is ​​used to learn the weights corresponding to different feature channel combinations. Based on the channel weight reconstruction matrix, the compressed channel feature vectors are dimension-restored to obtain the channel weight vectors; the channel weight reconstruction matrix is ​​used to learn the importance of each feature channel. Based on the channel weight vector, the acoustic feature map is weighted to obtain a new acoustic feature map, which is then converted into the spatial propagation feature.

7. The method according to claim 6, characterized in that, The generation of the acoustic feature map based on the time-frequency map includes: A non-dilated convolutional layer is used to perform a convolution operation on the time-frequency map to obtain time-frequency features; Multiple parallel dilated convolutional layers are used to perform convolution operations on the time-frequency features to obtain multiple output feature maps; wherein, different dilated convolutional layers have different dilation rates. The multiple output feature maps are concatenated to obtain a multi-scale feature set; The acoustic feature map is obtained by fusing channel-dimensional features of the multi-scale feature set using a multi-scale feature fusion processor.

8. The method according to claim 6 or 7, characterized in that, The sound signal is acquired through a microphone array, which includes multiple microphones deployed in different spatial locations; The method further includes: For any channel of the microphone array, the spatial propagation characteristics corresponding to the channel are mapped to three-dimensional spatial coordinates to obtain the first spatial position of the connector; based on the first spatial position and the reference spatial position of the connector relative to the channel, the positioning loss of the channel is determined. The average positioning loss of each channel in the microphone array is obtained to obtain the first positioning loss; The spatial propagation features corresponding to each channel are fused to obtain a second feature vector; the second feature vector is mapped to three-dimensional spatial coordinates to obtain the second spatial position of the connector; The second positioning loss is determined based on the second spatial position and the reference spatial position of the connector relative to the microphone array; The sound source localization model is trained based on the first localization loss and the second localization loss.

9. The method according to claim 5, characterized in that, The second encoder, using the sound source localization model, extracts features from the time-frequency map to obtain the physical attribute features of the connector, including: A non-dilated convolutional layer is used to perform a convolution operation on the time-frequency map to obtain time-frequency features; While maintaining frequency resolution, an asymmetric convolution kernel is used to perform a convolution operation on the time-frequency features along the time direction, and the time-frequency features are added to the time-frequency features after the convolution operation through residual connections to obtain new time-frequency features; wherein, the time dimension kernel size of the asymmetric convolution kernel is larger than the frequency dimension kernel size; An attention weight map is generated based on the new time-frequency features; wherein, the attention weight map marks the importance of each time-frequency region in the time-frequency map by attention weight; Based on the attention weight map and the new time-frequency features, the optimized time-frequency features are generated; The physical property features are generated based on the optimized time-frequency features.

10. The method according to claim 9, characterized in that, The generation of the attention weight map based on the new time-frequency features includes: Perform channel-dimensional average pooling on the new time-frequency features to obtain the average energy distribution of each time-frequency point across all feature channels; Max pooling is performed on the new time-frequency features to obtain the maximum energy distribution of each time-frequency point across all feature channels; The attention weight map is generated based on the average energy distribution and maximum energy distribution at each time frequency point.

11. The method according to claim 9, characterized in that, The step of generating the physical attribute features based on the optimized time-frequency features includes: Global average pooling is performed on the optimized time-frequency features to obtain channel feature vectors; wherein each component of the channel feature vector is the global pixel mean value within a single feature channel of the optimized time-frequency features; Based on the channel feature vector, the physical properties of the connector are predicted to obtain multiple attribute probability distributions; The probability distributions of the multiple attributes are converted into the physical attribute features.

12. The method according to claim 11, characterized in that, The step of converting the probability distributions of the multiple attributes into the physical attribute features includes: Based on the semantic projection matrix, the probability distributions of the multiple attributes are mapped to the same continuous semantic subspace to obtain multiple attribute vectors; Obtain the dot product similarity between the query vector and each attribute vector to get the attention score of each physical attribute of the connector; The attention scores of each physical attribute are normalized to obtain the attention weights of each physical attribute. The attention weights of each physical attribute and the multiple attribute vectors are weighted and summed to obtain an attribute fusion vector, which is then converted into the physical attribute features. One of the attribute vectors is used to characterize a physical attribute of the connector; the parameters of the semantic projection matrix and the query vector are updated during model training.

13. The method according to claim 12, characterized in that, The method of mapping the probability distributions of multiple attributes to the same continuous semantic subspace based on the semantic projection matrix yields multiple attribute vectors, including: Based on the first semantic projection matrix, the probability distribution of the first attribute indicating the connector model is mapped to the continuous semantic subspace to obtain the first attribute vector; the first attribute vector is a continuous semantic feature containing the connector model information. Based on the second semantic projection matrix, the probability distribution of the first attribute indicating the material of the connector is mapped to the continuous semantic subspace to obtain the second attribute vector; the second attribute vector is a continuous semantic feature containing the material information of the connector. Based on the third semantic projection matrix, the probability distribution of the first attribute indicating the fastening state of the connector is mapped to the continuous semantic subspace to obtain the third attribute vector; the third attribute vector is a continuous semantic feature containing the state information of the connector. Based on the fourth semantic projection matrix, the probability distribution of the first attribute indicating the installation position of the connector is mapped to the continuous semantic subspace to obtain the fourth attribute vector; the fourth attribute vector is a continuous semantic feature containing the installation position information of the connector.

14. The method according to claim 7 or 9, characterized in that, The step of performing a convolution operation on the time-frequency map using a non-dilated convolutional layer to obtain time-frequency features includes: The time-frequency graph is subjected to data augmentation processing to obtain the processed time-frequency graph; The non-dilated convolutional layer is used to perform a convolution operation on the processed time-frequency map to obtain the time-frequency features.

15. The method according to claim 14, characterized in that, The step of performing data augmentation processing on the time-frequency graph to obtain the processed time-frequency graph includes: Based on the stretching factor, the time axis of the time-frequency graph is scaled. Interpolation is performed on the scaled time axis to fill in the missing data points in the time dimension, resulting in the processed time-frequency graph.

16. The method according to claim 14, characterized in that, The step of performing data augmentation processing on the time-frequency graph to obtain the processed time-frequency graph includes: Select the mask start frequency within the effective frequency range of the time-frequency diagram; Based on the mask width and the mask start frequency, determine the frequency range to be masked; The energy values ​​in the frequency range to be masked in the time-frequency graph are replaced with preset values ​​to obtain the processed time-frequency graph; the preset values ​​are random noise values, the mean background noise of the time-frequency graph, or zero.

17. The method according to claim 5, characterized in that, The method further includes: For any training sample, a first similarity is obtained between the first time-frequency map and the second time-frequency map; the first time-frequency map corresponds to the sound signal included in the training sample, and the second time-frequency map corresponds to the sound signal included in the positive sample; Obtain a second similarity between the first time-frequency map and the third time-frequency map; the third time-frequency map corresponds to the sound signal included in the negative sample. The sound source localization model is trained based on the first similarity and the second similarity. The training samples include sound signals from the first connector in the first state and corresponding to the first sound propagation path; the positive samples include sound signals from the first connector in the first state and corresponding to the second sound propagation path; the negative samples include sound signals from the first connector in the second state, or from a second connector different from the first connector.

18. The method according to claim 5, characterized in that, The training samples also include various reference attributes of the connector; the second encoder includes multiple classifiers, with different classifiers used to perform different attribute classification tasks; the training process of the multiple classifiers includes: The physical property features are input into the multiple classifiers, and the multiple classifiers are used to predict the attributes of the connector to obtain multiple predicted attributes of the connector. For any attribute classification task, the classification loss of the attribute classification task is obtained based on the reference attribute and the predicted attribute corresponding to the connector. The multiple classifiers are trained based on the classification loss for each attribute classification task.

19. The method according to claim 1, characterized in that, The sound signal is acquired through a microphone array, which includes multiple microphones deployed in different spatial locations; The spatial location prediction of the connector based on the attention-weighted features includes: The attention-weighted features corresponding to each channel in the microphone array are fused to obtain global fused features; The global fusion features are mapped to three-dimensional spatial coordinates to obtain the predicted spatial position of the connector.

20. The method according to claim 19, characterized in that, The process of fusing the attention-weighted features corresponding to each channel in the microphone array to obtain global fused features includes: Multiple nonlinear transformations are performed on the attention weighted features corresponding to each channel to obtain nonlinear transformation results; wherein, the nonlinear transformation results are used to characterize the correlation between the attention weighted features corresponding to each channel; The nonlinear transformation result is dimensionally compressed, and a layer normalization operation is performed on the dimensionally compressed nonlinear transformation result to obtain the global fusion feature.

21. The method according to claim 20, characterized in that, The sound source localization model includes a feature fusion network; the process of performing multiple nonlinear transformations on the attention-weighted features corresponding to each channel to obtain the nonlinear transformation results includes: Based on the first layer weight matrix of the feature fusion network, a nonlinear transformation is performed on the attention-weighted features corresponding to each channel to obtain intermediate transformation results; the first layer weight matrix is ​​used to learn the basic correlation between the attention-weighted features corresponding to each channel. Based on the second-layer weight matrix of the feature fusion network, the intermediate transformation result is subjected to a nonlinear transformation to obtain the nonlinear transformation result; the second-layer weight matrix is ​​used to learn the higher-order correlation of the attention-weighted features corresponding to each channel after the first nonlinear transformation.

22. The method according to claim 20, characterized in that, The sound source localization model includes a coordinate regression network; the step of mapping the global fusion features to three-dimensional spatial coordinates to obtain the predicted spatial location of the connector includes: Based on the first layer weight matrix of the coordinate regression network, the global fusion features are mapped to a first intermediate representation; the first layer weight matrix is ​​used to learn to map the global fusion features to an intermediate physical quantity. Based on the second-layer weight matrix of the coordinate regression network, the first intermediate representation is mapped to the second intermediate representation; the second-layer weight matrix is ​​used to learn the mutual influence between the intermediate physical quantities. The second intermediate representation is mapped to the predicted spatial location of the connector.

23. The method according to claim 22, characterized in that, The training process of the coordinate regression network includes: Based on the reference spatial position and the predicted spatial position of the connector, the main loss is obtained; An auxiliary loss is obtained based on a height constraint term and a horizontal distance constraint term; wherein, the height constraint term is used to constrain the predicted height of the connector to not exceed the height of the mechanical connection structure; the horizontal distance constraint term is used to constrain the horizontal radius of the predicted position to not exceed the minimum horizontal radius of the mechanical connection structure; The coordinate regression network is trained based on the main loss and the auxiliary loss.

24. A method for locating a sound source, characterized in that, The method includes: Acquire a sound signal to be detected; wherein the sound signal to be detected is acquired by microphones deployed around the mechanical connection structure to be detected, and the mechanical connection structure is equipped with connectors; The time-frequency graph of the sound signal to be detected is input into the sound source localization model, and the spatial position of the loose connector is predicted by the sound source localization model; wherein, the sound source localization model is trained by the training method shown in claims 1 to 23.

25. The method according to claim 24, characterized in that, The number of microphones is multiple, and different microphones are deployed in different spatial locations around the mechanical structure; The method further includes: When the signal energy of any microphone channel exceeds the energy threshold, multiple microphones are triggered to collect a sound signal of a preset duration. The sound signal to be detected includes multiple sound signals with a duration of the preset duration.

26. The method according to claim 25, characterized in that, The step of inputting the time-frequency graph of the sound signal to be detected into the sound source localization model, and predicting the spatial location of the loose connector using the sound source localization model, includes: Perform time-frequency analysis on each segment of the audio signal with a preset duration to obtain multiple time-frequency diagrams; The multiple time-frequency maps are input into the sound source localization model, and the spatial location of the loose connector is predicted by the sound source localization model.

27. A training device for a sound source localization model, characterized in that, The device includes: The first acquisition module is configured to acquire training samples, the training samples including sound signals of a loose connector and a reference spatial position of the connector, the connector being mounted on a mechanical connection structure; The first processing module is configured to input the time-frequency diagram of the sound signal into the sound source localization model, and extract features from the time-frequency diagram through the sound source localization model to obtain a first feature vector; the first feature vector is used to indicate the spatial propagation characteristics of the sound signal and the physical properties of the connector. The second processing module is configured to convert the first feature vector into attention-weighted features using the sound source localization model; the attention-weighted features are used to reflect the key features in the first feature vector related to the state of the connector from multiple dimensions. The prediction module is configured to predict the spatial location of the connector based on the attention-weighted features and the sound source localization model. The training module is configured to train the sound source localization model based on the reference spatial location and the spatial location predicted by the sound source localization model.

28. A sound source localization device, characterized in that, The device includes: The second acquisition module is configured to acquire a sound signal to be detected; wherein the sound signal to be detected is acquired by microphones deployed around the mechanical connection structure to be detected, and the mechanical connection structure is equipped with connectors; The sound source localization module is configured to input the time-frequency diagram of the sound signal to be detected into the sound source localization model, and predict the spatial position of the loose connector through the sound source localization model; wherein the sound source localization model is trained by the training device shown in claims 1 to 23.

29. A computer device, characterized in that, The device includes a processor and a memory, the memory storing computer program code, the computer program code being loaded and executed by the processor to implement the training method of the sound source localization model as described in any one of claims 1 to 23, or the sound source localization method as described in any one of claims 24 to 26.

30. A computer-readable storage medium, characterized in that, The storage medium stores computer program code, which is loaded and executed by a processor to implement the training method for the sound source localization model as described in any one of claims 1 to 23, or the sound source localization method as described in any one of claims 24 to 26.

31. A computer program product, characterized in that, The computer program product includes computer program code stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the computer device to perform a training method for a sound source localization model as described in any one of claims 1 to 23, or a sound source localization method as described in any one of claims 24 to 26.

Images