A method for environmental noise monitoring based on multimodal data

Abstract

The present invention provides an environmental noise monitoring method based on multimodal data, which relates to the technical field of environmental monitoring and comprises the following steps: S1: synchronously collecting multimodal data in an environment, the multimodal data including original acoustic signals, meteorological data and video image data; S2: data processing and feature extraction; S3: dynamically adjusting the number of heads and weight calculation method of a self-attention mechanism based on the signal-to-noise ratio and time-domain variance of a spectrum graph, and then extracting a voiceprint feature vector based on the adjusted self-attention mechanism; S4: splicing the voiceprint feature vector, the meteorological feature vector and the visual feature vector into a multimodal feature matrix along the feature dimension, and performing weighted fusion on the multimodal feature matrix to generate a fused feature vector; S5: noise classification and early warning; the present invention solves the problem of limitations of single sensor data by fusing three heterogeneous modal information of acoustic signals, meteorological data and video images and combining a cross-modal self-attention mechanism to achieve feature weighted fusion.

Description

Technical Field

[0001] The present invention relates to the technical field of environmental noise monitoring, and in particular to an environmental noise monitoring method based on multimodal data. Background Art

[0002] Currently, traditional environmental noise monitoring technologies rely on a single sensor to collect acoustic signals and then use fixed algorithms for feature extraction and classification. However, this single-modal data-based monitoring approach has significant limitations. First, the generation and propagation of environmental noise are significantly affected by meteorological conditions (such as temperature, humidity, and air pressure). For example, changes in humidity alter the attenuation characteristics of sound, and temperature differences cause changes in the speed of sound. A single acoustic signal cannot effectively reflect the interference of these environmental factors on noise. Second, traditional static self-attention models cannot dynamically adjust their model structure and computational strategies based on real-time noise environment characteristics (such as the signal-to-noise ratio (SNR) of the spectrogram and the temporal variance (Var). This leads to performance flaws such as insufficient feature extraction and reduced classification accuracy in highly dynamic noise scenarios, making it difficult to meet the needs of high-precision noise monitoring in complex environments. Summary of the Invention

[0003] In order to solve the above technical problems, the present invention provides an environmental noise monitoring method based on multimodal data.

[0004] The technical solution adopted in the present invention is as follows:

[0005] A method for monitoring environmental noise based on multimodal data, comprising the following steps:

[0006] S1: synchronously collecting multimodal data in the environment, wherein the multimodal data includes original acoustic signals, meteorological data, and video image data, wherein the meteorological data includes temperature, humidity, and air pressure;

[0007] S2: performing short-time Fourier transform on the original acoustic signal to generate a spectrogram; normalizing the meteorological data and splicing them into meteorological feature vectors; extracting the video image data through a convolutional neural network to obtain a visual feature vector;

[0008] S3: Dynamically adjust the number of heads and weight calculation method of the self-attention mechanism based on the signal-to-noise ratio and time-domain variance of the spectrogram, and then extract the voiceprint feature vector based on the adjusted self-attention mechanism;

[0009] S4: splicing the voiceprint feature vector, the meteorological feature vector, and the visual feature vector along the feature dimension into a multimodal feature matrix, calculating the cross-modal correlation weight by fusing the self-attention layer, and performing weighted fusion on the multimodal feature matrix to generate a fused feature vector;

[0010] S5: Input the fused feature vector into the classification model and output the noise category probability distribution. If the decibel value of the category corresponding to the maximum probability value exceeds the preset threshold, an early warning is triggered and the noise source is located.

[0011] Furthermore, the dimension of the meteorological feature vector is 3, which is composed of the standardized temperature, humidity and air pressure values ​​​​concatenated in sequence; the visual feature vector is extracted through the ResNet-50 model, and the output dimension is 2048.

[0012] Furthermore, in step S3, when the signal-to-noise ratio of the spectrogram is lower than 10dB or the time domain variance is higher than 5dB, the number of self-attention heads is increased; when the signal-to-noise ratio of the spectrogram is not lower than 10dB and the time domain variance is not higher than 5dB, the number of self-attention heads is reduced; when the signal-to-noise ratio is lower than 10dB, the additive attention mechanism is used to calculate the voiceprint feature vector; when the signal-to-noise ratio is not lower than 10dB, the dot product attention mechanism is used to calculate the voiceprint feature vector.

[0013] Furthermore, the dimension of the multimodal feature matrix is ​​3×d, where d is the maximum dimension of the voiceprint feature vector, meteorological feature vector and visual feature vector; the fused self-attention layer calculates the query matrix, key matrix and value matrix through the trainable parameter matrix, and generates cross-modal correlation weights based on the Softmax function.

[0014] Furthermore, in step S5, the classification model is a fully connected neural network, and the output dimension is equal to the total number of noise categories.

[0015] Furthermore, the additive attention mechanism calculates attention weights through a trainable weight matrix and a bias vector.

[0016] Furthermore, the output dimension of the fused feature vector is 1024.

[0017] Furthermore, the noise categories include traffic noise, industrial noise, living noise and natural noise.

[0018] Furthermore, the preset threshold is dynamically set according to the regional environmental noise standard.

[0019] Furthermore, in step S2, the meteorological data is standardized using a Z-score normalization method.

[0020] Beneficial effects of the present invention:

[0021] (1) This paper solves the limitation of single sensor data by fusing three heterogeneous modal information: acoustic signals, meteorological data (temperature, humidity, air pressure) and video images, and combining them with a cross-modal self-attention mechanism to achieve feature weighted fusion.

[0022] (2) Based on the real-time feedback of the signal-to-noise ratio (SNR) and the time domain variance (Var) of the spectrum graph, the number of self-attention heads and the weight calculation strategy are adaptively adjusted to solve the performance defects of the static model in high dynamic noise scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Flowchart of an environmental noise monitoring method based on multimodal data according to an embodiment of the present invention;

[0024] Figure 2 Flowchart of data processing and feature extraction according to an embodiment of the present invention;

[0025] Figure 3 This is a flow chart of a dynamic self-attention adjustment mechanism according to an embodiment of the present invention;

[0026] Figure 4 This is a flowchart of multimodal feature fusion according to an embodiment of the present invention;

[0027] Figure 5 Flowchart of noise classification and warning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0029] like Figure 1-Figure 5 As shown, an environmental noise monitoring method based on multimodal data according to an embodiment of the present invention includes the following steps:

[0030] S1: Synchronously collect multimodal data in the environment, including original acoustic signals , meteorological data and video image data , among which meteorological data is a three-dimensional vector, including temperature, humidity and air pressure, where , where Indicates temperature, Indicates humidity, Indicates air pressure.

[0031] It should be understood that in the environmental noise monitoring scenario, the data of a single mode often cannot fully reflect the actual situation of noise generation. While it carries characteristic information about noise, the generation and propagation of noise can be affected by meteorological conditions, such as wind speed and humidity, which can alter the speed and attenuation of sound. Video image data, on the other hand, can intuitively reveal the physical form and surroundings of noise sources. For example, image recognition can be used to identify noise generated by construction machinery at a construction site or the noise caused by vehicle horns on congested roads. Therefore, the present invention collaboratively collects raw acoustic signals, meteorological data, and video image data to create multimodal data, improving the accuracy and reliability of noise monitoring.

[0032] Specifically, in a specific embodiment of the present invention, the original acoustic signal The acquisition frequency can be set to 44.1kHz or 48kHz, meeting the sampling requirements of the human audible sound frequency band (20Hz-20kHz), ensuring that the original acoustic signal collected can fully retain the noise characteristics.

[0033] For meteorological data , which includes three key indicators: temperature, humidity and air pressure. It can be deployed in the target area through small weather station equipment, and data collection is carried out at intervals of one minute or every five minutes to ensure real-time and stable meteorological data.

[0034] For video image data This can be achieved by using a high-definition camera with infrared night vision to ensure that image data can be collected normally even at night. During installation, the camera's installation height and angle should be reasonably planned to avoid obstruction and ensure effective coverage of the target area.

[0035] S2: original acoustic signal Perform short-time Fourier transform to generate spectrum ; Weather data Perform standardization and splicing into meteorological feature vectors; extract video image data through convolutional neural network Get the visual feature vector.

[0036] Among them, the original acoustic signal It is an analog signal that changes continuously in the time domain. It is difficult to analyze it directly, so it needs to be converted to the frequency domain for processing. Specifically, the present invention uses short-time Fourier transform to convert the original acoustic signal Convert to spectrogram , where the short-time Fourier transform formula is:

[0037]

[0038] Where, Represents the original acoustic signal at time The sampling value of Indicates the number of sampling points, which determines the signal length of each analysis; represents the window function; represents the number of Fourier transform points, where The larger it is, the smaller the frequency interval is, and the more detailed the analysis of frequency components is; represents the frequency index; Represents a time index.

[0039] Calculated by this formula , the horizontal axis is time, the vertical axis is frequency, and the amplitude of each point reflects the intensity of the sound signal at the corresponding time and frequency, ultimately forming a spectrum diagram , the changes of different frequency components in the sound signal over time are intuitively presented for subsequent extraction of voiceprint features.

[0040] Weather data Temperature, humidity, and air pressure all have different numerical ranges and physical meanings. For example, the temperature range is typically around -40°C to 50°C, the humidity range is 0% to 100%, and the air pressure is typically between 950hPa and 1050hPa. This is done to eliminate the impact of differences in dimensions and numerical ranges between data, making different features comparable.

[0041] Specifically, the present invention uses the Z-score normalization method to normalize meteorological data. The standardization process is carried out using the formula:

[0042]

[0043] Where, Represents the standardized meteorological data; Indicates the original data features, among which Corresponding to temperature, humidity, and air pressure respectively; Indicates the The mean of a feature, which represents the average level of the feature; Indicates the The standard deviation of a feature reflects the degree of dispersion of the data.

[0044] Through this formula, each original eigenvalue is subtracted from its mean and then divided by the standard deviation, so that the mean of the standardized data is 0 and the standard deviation is 1, mapping meteorological data of different magnitudes to the same scale.

[0045] The standardized meteorological data is spliced ​​into a meteorological feature vector with a dimension of 3 ,in, .

[0046] In a specific embodiment of the present invention, the visual feature vector is extracted by the ResNet-50 model, and the output dimension is 2048. The ResNet-50 model is a classic CNN model, which consists of 50 convolutional layers, pooling layers, and fully connected layers.

[0047] Processing video image data When the image is first input into the first convolutional layer of the ResNet-50 model, the convolutional layer performs convolution operations with the image through multiple convolution kernels with different parameters to extract low-level features such as edges and textures in the image. As the number of network layers increases, subsequent convolutional layers will gradually extract more advanced and abstract semantic features, such as the shape and category of the object, based on the features extracted by the previous layer. The pooling layer downsamples the feature map output by the convolutional layer to reduce the data dimension and the amount of calculation while retaining the main features. After multiple layers of convolution and pooling operations, the feature map is finally flattened and mapped to a 2048-dimensional visual feature vector through the fully connected layer. ,in, .

[0048] These feature vectors condense the key information in the image, can effectively characterize the scene and object features related to noise in the image, and provide information support in the visual dimension for multimodal feature fusion.

[0049] S3: Based on spectrogram The signal-to-noise ratio (SNR) and time domain variance (Var) are used to dynamically adjust the number of heads and weight calculation method of the self-attention mechanism, and then the voiceprint feature vector is extracted based on the adjusted self-attention mechanism. .

[0050] Among them, the signal-to-noise ratio SNR is a measure of the spectrum An important indicator of the relative size of signal strength and noise strength in the medium. Its calculation formula is:

[0051]

[0052] Where, Indicates that the signal part of the spectrum is at all frequencies and time The total energy on . It represents the amplitude of the signal component in the spectrum graph that has been manually labeled or identified based on a specific algorithm (such as a deep learning algorithm (convolutional neural network CNN), a machine learning algorithm (support vector machine SVM algorithm), etc.), which reflects the intensity distribution of the effective sound information of interest. It represents the sum of the energy of the noise part in the entire spectrum, which represents the amplitude of the noise component at the corresponding frequency and time point.

[0053] The present invention converts the signal-to-noise energy ratio into a signal-to-noise ratio (SNR) value in decibels (dB) by taking the base-10 logarithm of the signal-to-noise energy ratio and multiplying it by 10. A higher SNR value indicates better signal quality and less noise interference. For example, an SNR of 10dB means the signal energy is 10 times greater than the noise energy; while an SNR of 20dB means the signal energy is 100 times greater than the noise energy.

[0054] The time domain variance Var is used to measure the degree of fluctuation of the original acoustic signal in the time dimension. Its calculation formula is:

[0055]

[0056] Where, Represents the original acoustic signal at time The sampling value of , which records the instantaneous change of sound intensity over time; Indicates the number of sampling points, which is the total number of sampling points of the signal segment used to calculate the variance; Represents the mean value of the signal segment, which reflects the average intensity level of the signal in this time period, where Calculated by the following formula:

[0057]

[0058] Used to calculate the square of the deviation of each sampling point from the mean, and then divide it by the number of sampling points after accumulation. The resulting time-domain variance Var can quantify the degree of signal fluctuation. A larger variance indicates a more dramatic temporal change in the signal, potentially containing more sudden noise components. A smaller variance indicates a relatively stable signal with less noise interference.

[0059] For example, the variance of acoustic signals collected in a quiet indoor environment is usually small; however, the variance of signals collected at a busy intersection is often large due to the superposition of various noises such as vehicle horns and engine sounds.

[0060] According to the calculation results of the signal-to-noise ratio (SNR) and the time domain variance (Var) of the spectrum graph, the number of heads and the weight calculation method of the self-attention mechanism are dynamically adjusted to adapt to the feature extraction requirements in different noise environments.

[0061] Specifically, the self-attention mechanism adjustment strategy is as follows:

[0062] When the signal-to-noise ratio (SNR) of the spectrogram is lower than 10dB or the time domain variance (Var) is higher than 5dB, it indicates that the current environmental noise is strong and the signal quality is poor. In this case, the number of self-attention heads is increased. For example, the number of self-attention heads is increased to 12. By increasing the number of heads, the model can capture voiceprint features from multiple different angles. By processing different subspace information in parallel, the model's ability to extract complex noise features is enhanced, avoiding the loss of key features due to noise interference.

[0063] When the signal-to-noise ratio (SNR) of the spectrogram is not less than 10 dB and the time domain variance (Var) is not higher than 5 dB, it indicates that the ambient noise is relatively weak and the signal is relatively clear. In this case, the number of self-attention heads is reduced. For example, the number of self-attention heads is reduced to 4. Reducing the number of heads can reduce the computational complexity of the model, improve computational efficiency, and avoid overfitting problems that may be caused by too many heads in a simple noise environment.

[0064] When the signal-to-noise ratio (SNR) is lower than 10dB, the additive attention mechanism is used to calculate the voiceprint feature vector A; the additive attention mechanism uses a trainable weight matrix and the bias vector Calculating attention weights .

[0065] The specific calculation process is as follows:

[0066] First, calculate the attention score , the calculation formula is:

[0067]

[0068] Where, 、 They are query matrix and key matrix respectively, which are obtained by linear transformation of input data and are used for information retrieval and matching in feature space; 、 Represents a trainable weight matrix, which continuously adjusts parameters during model training to optimize the accuracy of feature matching; Represents the hyperbolic tangent activation function, which maps the input data to the range of -1 to 1 to increase the nonlinear expression ability of the model. and Processing, mining the correlation between different features, and thus obtaining a preliminary attention score .

[0069] Then, the attention score Perform normalization to obtain attention weight :

[0070]

[0071] Attention weight Indicates the importance of each feature in the current calculation. The larger the value, the more critical the corresponding feature is in voiceprint feature extraction. The normalization operation makes the sum of the attention weights of all features equal to 1, which facilitates subsequent weighted calculations.

[0072] Finally, through the formula , based on attention weight Pair Matrix Perform weighted summation to obtain the voiceprint feature vector A. Value matrix It is also obtained by linear transformation of input data, which contains the feature information that needs to be paid attention to. Pair Matrix Weighting can highlight important features and suppress minor features, thereby effectively mining voiceprint features masked by noise.

[0073] When the signal-to-noise ratio (SNR) is not less than 10dB, the dot product attention mechanism is used to calculate the voiceprint feature vector A. The expression of the dot product attention mechanism is:

[0074]

[0075]

[0076] Where, Represents the bond matrix By introducing the dimension Scaling is performed to prevent the dot product result from being too large, resulting in The function gradient disappears. By calculating the query matrix With the key matrix The dot product of the transpose and scaling is used to get the attention weight , and then go through The function is normalized, and finally the value matrix is ​​calculated based on the normalized weight pair Perform weighted summation to obtain the voiceprint feature vector A. Among them, the dot product attention mechanism has higher computational efficiency in a high signal-to-noise ratio environment and can extract voiceprint features quickly and accurately.

[0077] Through the above dynamic adjustment strategy, the self-attention mechanism can adaptively extract the voiceprint feature vector A according to different noise environments, providing high-quality acoustic feature input for subsequent multimodal feature fusion and noise classification, thereby improving the accuracy and robustness of the entire environmental noise monitoring method.

[0078] S4: The voiceprint feature vector A and the meteorological feature vector and visual feature vector The multimodal feature matrix M is spliced ​​along the feature dimension, the cross-modal correlation weight is calculated by fusing the self-attention layer, and the multimodal feature matrix is ​​weighted fused to generate a fused feature vector, where the output dimension of the fused feature vector is 1024.

[0079] Since the dimensions of the voiceprint feature vector, meteorological feature vector and visual feature vector may be different, let d

[0080] is the maximum dimension among the three. For feature vectors with a dimension less than d, zero padding and other methods are used to align the dimensions to ensure the feasibility of splicing.

[0081] For example, if the dimension of the voiceprint feature vector A is , meteorological characteristic vector The dimension is 3, the visual feature vector The dimension is 2048, then the voiceprint feature vector A and the meteorological feature vector Perform zero padding operation to make the dimensions reach 2048, and then splice them to finally obtain a multimodal feature matrix M with a dimension of 3×d, where each row corresponds to the feature vector of a modality, and d is the maximum dimension of the voiceprint feature vector, meteorological feature vector, and visual feature vector.

[0082] The fused self-attention layer calculates the query matrix, key matrix, and value matrix through the trainable parameter matrix, and generates cross-modal correlation weights based on the Softmax function.

[0083] Specifically, the fusion self-attention layer is trained through the parameter matrix 、 、 Perform linear transformation on the multimodal feature matrix M and calculate the query matrix Q, key matrix K and value matrix V respectively. Specifically, the calculation formula is:

[0084]

[0085]

[0086]

[0087] Where, 、 、 This matrix, whose parameters are continuously adjusted during model training via the backpropagation algorithm, is used to map the multimodal feature matrix M to different feature spaces to explore potential relationships between data from different modalities. Specifically, the query matrix Q is used to initiate queries in the feature space to find relevant information; the key matrix K is used to store queryable information as a reference for queries; and the value matrix V contains the feature information that actually needs to be focused on and integrated.

[0088] Based on the calculated query matrix Q and key matrix K, the cross-modal relevance weight is calculated by the following formula :

[0089]

[0090] Where, is the dimension of the key matrix K, and Scaling is performed to prevent the dot product result from being too large, causing the Softmax function gradient to disappear and affecting the accuracy of weight calculation. The calculation of is essentially to measure the similarity between each element in the query matrix Q and the elements in the key matrix K. The higher the similarity, the larger the corresponding weight value. After the Softmax function is processed, the calculation result is normalized to between 0 and 1, and the sum of all weights is 1. The cross-modal correlation weight is obtained It represents the relative importance of different modal features in the fusion process. For example, the weight corresponding to the meteorological feature vector is larger, indicating that in the current environment, the contribution of meteorological data to noise analysis is higher.

[0091] Generation-based cross-modal relevance weights , perform weighted summation on the value matrix V to generate the fused feature vector F. The generation formula of the fused feature vector F is:

[0092]

[0093] This step weights the different modal features in the value matrix V using weights β, highlighting important feature information and suppressing relatively less important information, thereby achieving effective fusion of multimodal features. The resulting fused feature vector F has an output dimension of 1024. The fused feature vector F integrates key information from the original acoustic signal, meteorological data, and video image data, and can comprehensively reflect the characteristics of environmental noise.

[0094] The present invention organically combines data from different sources and of different natures through multimodal feature fusion in step S4, utilizes the powerful computing power of the fused self-attention layer to explore the deep correlation between data, significantly improves the feature expression ability, and provides strong support for achieving accurate environmental noise monitoring and early warning.

[0095] S5: Input the fused feature vector F into the classification model and output the noise category probability distribution P. If the decibel value of the category corresponding to the maximum probability exceeds the preset threshold, an early warning is triggered and the noise source is located, providing effective support for environmental noise control.

[0096] Specifically, the classification model uses a fully connected neural network, and its output dimension is equal to the total number of noise categories. Among them, the noise categories include traffic noise, industrial noise, life noise and natural noise, with a total of C = 4 categories. The output of the classification model is a 4-dimensional vector ,in, ( =1, 2, 3, 4) indicates that the input fusion feature vector belongs to the The probability of noise-like .

[0097] It should be understood that a fully connected neural network consists of multiple neuron layers, and the neurons in each layer are connected to all neurons in the next layer. For the input 1024-dimensional fusion feature vector F, it first enters the hidden layer of the fully connected neural network. Each neuron in the hidden layer processes the input through weighted summation and activation function. If the hidden layer The input of a neuron is ,in, Represents the connection input layer nodes and hidden layer The weight of the node, Represents the hidden layer The bias of each node is output after the activation function, where the activation function can use the ReLU function: .

[0098] Through multiple hidden layers, features are extracted and transformed in sequence, and the original fusion features are gradually mapped to the feature space related to the noise category. Finally, the output is converted into a probability distribution P through the linear transformation and Softmax function of the output layer, thereby realizing the prediction of the noise category; among them, the Softmax function expression is:

[0099]

[0100] Where, represents the input vector of the output layer, Represents the category index.

[0101] In the model training phase, a large number of fused feature vectors F with labeled noise categories are used as training data, and the weights in the network are continuously adjusted through the back propagation algorithm. and bias , using the cross entropy loss function as the optimization objective, minimizing the difference between the predicted probability distribution and the true label, so that the model can continuously learn the mapping relationship between noise features and categories during the training process to improve the accuracy of classification. Among them, the expression of the cross entropy loss function is:

[0102]

[0103] Where, Indicates the number of training samples; Representation sample Belong to category The true label of Represents the model prediction sample Belong to category probability.

[0104] After obtaining the noise category probability distribution P, the model compares the probability values ​​to find the maximum value, max(P). The corresponding category is the noise category predicted by the model. Each noise category has a pre-defined decibel range; for example, traffic noise corresponds to 60-90dB, industrial noise corresponds to 70-110dB, and so on. The decibel value corresponding to the predicted category is compared with a preset threshold. If it exceeds the threshold, an alert is triggered.

[0105] In addition, the preset threshold is dynamically set according to the regional environmental noise standard. For example, taking residential areas as an example, according to national standards, the noise limit during the day is 55dB and at night is 45dB. In actual applications, the attribute information of the current monitoring area (such as whether it is a residential area) is obtained in real time, and the corresponding preset threshold is automatically loaded according to the current time period (daytime or nighttime). For example, when monitoring a residential area at night, if the model predicts that the noise category is traffic noise, and the decibel value corresponding to this category is greater than 45dB after calculation or query of relevant data, an early warning will be triggered immediately, notifying relevant personnel that there is excessive noise pollution in the area.

[0106] Noise sources are located by matching GPS coordinates with video image data. During the data acquisition phase, the video image acquisition device records the GPS coordinates corresponding to each frame. These coordinates precisely identify the geographic location where the image was captured. When an alert is triggered, the system first searches the video image data for frames at or near the same time, based on the timestamp corresponding to the alert. Computer vision technology is then used to analyze the image. First, object detection algorithms (such as YOLO and Faster R-CNN) are used to identify objects related to noise sources in the image. For example, in traffic noise alerts, these algorithms detect vehicles in the image; in industrial noise alerts, these algorithms detect factory equipment. Second, by combining geographic reference objects in the image (such as buildings and road signs) with the geographic location information provided by GPS coordinates, a geographic information system (GIS) is used to perform spatial matching and coordinate transformation. This maps the location of noise source objects in the image to actual geographic space, thereby accurately determining the specific location of the noise source. For example, on city streets, by identifying a honking vehicle in an image and combining it with GPS coordinates and street maps, the specific location of the vehicle on the street can be accurately pointed out, providing precise guidance for noise control personnel to quickly locate and deal with noise problems.

[0107] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of such features. "Multiple" means two or more, unless otherwise specifically defined.

[0108] In the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," "connect," "fixed," etc. should be understood broadly. For example, they may refer to fixed connection, detachable connection, or integration; mechanical connection or electrical connection; direct connection or indirect connection through an intermediate medium; internal communication between two components or interaction between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0109] In the present invention, unless otherwise expressly specified or limited, when a first feature is "above" or "below" a second feature, it may mean that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediary. Furthermore, when a first feature is "above," "above," or "above" a second feature, it may mean that the first feature is directly above or diagonally above the second feature, or simply means that the first feature is at a higher level than the second feature. When a first feature is "below," "below," or "below" a second feature, it may mean that the first feature is directly below or diagonally below the second feature, or simply means that the first feature is at a lower level than the second feature.

[0110] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0111] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0112] The logic and / or steps represented in a flowchart or otherwise described herein, for example, can be considered a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (not exhaustive) of computer-readable media include: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0113] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0114] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0115] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0116] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for monitoring environmental noise based on multimodal data, characterized in that: The following steps are involved: S1: synchronously collecting multimodal data in the environment, wherein the multimodal data includes original acoustic signals, meteorological data, and video image data, wherein the meteorological data includes temperature, humidity, and air pressure; S2: performing short-time Fourier transform on the original acoustic signal to generate a spectrogram; normalizing the meteorological data and splicing them into meteorological feature vectors; extracting the video image data through a convolutional neural network to obtain a visual feature vector; S3: Dynamically adjust the number of heads and weight calculation method of the self-attention mechanism based on the signal-to-noise ratio and time-domain variance of the spectrogram, and then extract the voiceprint feature vector based on the adjusted self-attention mechanism; wherein, when the signal-to-noise ratio of the spectrogram is lower than 10dB or the time-domain variance is higher than 5dB, increase the number of self-attention heads; when the signal-to-noise ratio of the spectrogram is not lower than 10dB and the time-domain variance is not higher than 5dB, reduce the number of self-attention heads; when the signal-to-noise ratio is lower than 10dB, use the additive attention mechanism to calculate the voiceprint feature vector; when the signal-to-noise ratio is not lower than 10dB, use the dot-product attention mechanism to calculate the voiceprint feature vector; S4: splicing the voiceprint feature vector, the meteorological feature vector, and the visual feature vector along the feature dimension into a multimodal feature matrix, calculating the cross-modal correlation weight by fusing the self-attention layer, and performing weighted fusion on the multimodal feature matrix to generate a fused feature vector; S5: Input the fused feature vector into the classification model and output the noise category probability distribution. If the decibel value of the category corresponding to the maximum probability value exceeds the preset threshold, an early warning is triggered and the noise source is located.

2. The environmental noise monitoring method based on multimodal data according to claim 1, characterized in that: The dimension of the meteorological feature vector is 3, and it is composed of the standardized temperature, humidity and air pressure values ​​in sequence; The visual feature vector is extracted through the ResNet-50 model, and the output dimension is 2048.

3. The environmental noise monitoring method based on multimodal data according to claim 2, characterized in that: The dimension of the multimodal feature matrix is ​​3×d, where d is the maximum dimension of the voiceprint feature vector, meteorological feature vector, and visual feature vector; The fused self-attention layer calculates the query matrix, key matrix and value matrix through the trainable parameter matrix, and generates cross-modal correlation weights based on the Softmax function.

4. The environmental noise monitoring method based on multimodal data according to claim 3, characterized in that: In the step S5, The classification model is a fully connected neural network with an output dimension equal to the total number of noise categories.

5. The environmental noise monitoring method based on multimodal data according to claim 4, characterized in that: The additive attention mechanism calculates the attention weights through a trainable weight matrix and a bias vector.

6. The environmental noise monitoring method based on multimodal data according to claim 5, characterized in that: The output dimension of the fused feature vector is 1024.

7. The environmental noise monitoring method based on multimodal data according to claim 6, characterized in that: The noise categories include traffic noise, industrial noise, living noise and natural noise.

8. The environmental noise monitoring method based on multimodal data according to claim 7, characterized in that: The preset threshold is dynamically set according to the regional environmental noise standard.

9. The environmental noise monitoring method based on multimodal data according to claim 8, characterized in that: In step S2, the meteorological data is standardized using a Z-score normalization method.

Images