Urban sound event marking and identifying method based on saliency judgment

Abstract

The invention discloses a city sound event labeling and recognition method based on significance judgment, and belongs to the technical field of city sound environment monitoring and intelligent recognition. The method comprises the following steps: continuously collecting real environment audio; segmenting the audio data and extracting samples, and performing de-identification preprocessing to reduce privacy risks; screening reliable annotators through classification capability averaging and annotation consistency, and achieving a significance judgment consensus; performing significant sound event labeling based on a significance judgment consensus, and constructing a data set; the judgment rule of human on acoustic significance is analyzed and concluded through labeling consistency, and then the model is guided to learn significance standards with cognitive fitness; a deep learning model integrated with a channel attention mechanism is adopted to extract spectrum features, and intelligent judgment of a significant sound event is realized under supervised classification guidance of a significant sound event data set. According to the method, redundant data can be remarkably compressed while sound classification accuracy is kept, and the method is suitable for identification and analysis of multi-source and key sound events in a real urban environment.

Description

Technical Field

[0001] This invention covers the field of urban acoustic environment monitoring and intelligent recognition technology, specifically a method for intelligently processing acoustic events that integrates human auditory saliency perception with deep learning. This method is suitable for extracting and accurately identifying key events from mixed sound sources such as traffic sounds, human activity sounds, and natural sounds in complex urban environments. The method encompasses technical areas such as audio signal processing, deep learning model construction, and annotator training and selection. Background Art

[0002] Traditional acoustic event detection (SED) technology faces significant challenges in current urban sound environment monitoring. Traditional SED is often trained on "clean" datasets, achieving multi-label classification based on the clear spectral and energy characteristics of a single sound source. However, the high degree of intermingling between traffic noise, conversations, and natural sounds in urban environments leads to blurred boundaries between acoustic events and a surge in information redundancy, making it difficult to efficiently extract key events. Psychological research shows that humans prioritize attention to prominent sounds with prominent volume, frequency, or rhythm in complex soundscapes (which play a decisive role in soundscape perception). However, traditional methods fail to model this mechanism, resulting in insufficient scene understanding capabilities. For example, patent CN202010184468.6 utilizes a deep neural network architecture for sound event detection. However, this method ignores the importance of significant acoustic events to human perception in urban multi-source mixed scenes and lacks effective focus on key sounds. Patent CN201910153124.6 focuses on anomaly detection, but does not comprehensively extract key information from complex soundscapes or model a mechanism for distinguishing salience between acoustic events, making it difficult to adapt to the diverse use cases in public spaces. Furthermore, given the significant differences in distribution between real urban environments and training datasets, many methods experience a sharp decline in generalization performance in complex scenarios, making them unable to meet the needs of smart governance, such as noise source tracing and spatial usage analysis. Therefore, a method for acoustic event labeling and recognition that incorporates human perception and is adaptable to complex scenarios is urgently needed. Summary of the Invention

[0003] To address the problems of "complex sound source mixing and insufficient extraction of salient information" in urban sound environments, this paper introduces a human group perception consensus mechanism to construct a dataset and deep learning model that integrates salient features, achieving efficient labeling and accurate identification of key events in urban soundscapes.

[0004] The technical solution of the present invention:

[0005] The urban sound event labeling and identification method based on significance judgment includes the following steps:

[0006] S1: Data acquisition and preprocessing;

[0007] S2: saliency annotation and dataset construction;

[0008] S3: Deep learning model construction and training;

[0009] S4: generalization test and duration restoration;

[0010] In the above technical solution, step S1 includes the following steps.

[0011] S101 Audio Data Collection: Select typical public spaces in four functional areas: transportation facilities, commercial centers, residential areas, and comprehensive public buildings. Record the audio continuously at a sampling frequency to obtain the original audio.

[0012] S102 Sample Segmentation: Extract multiple segments from continuous recordings hour by hour and divide them into fixed-length audio samples to ensure data scale and event integrity.

[0013] S103 De-identification (Privacy Protection): Through manual review or algorithmic processing of clips containing private information (such as clear human voices and specific environmental identifiers), the privacy is removed while retaining the time domain and frequency domain characteristics of the acoustic event;

[0014] S104 Label Sample Generation: Integrate the collected, segmented, and de-identified audio to generate a structured label sample set for subsequent saliency annotation, deep learning model training, and verification.

[0015] In the above technical solution, step S2 includes the following steps.

[0016] S201 Pre-labeling: Organize several annotators trained on the sound event classification standard for open source datasets to perform initial labeling of the pure sound dataset (Pre-WSD, single sound source samples) based on unified semantic rules, clarifying the framework of the five first-level categories of "natural, quiet, human, traffic, and mechanical" and their corresponding second-level categories; collect mixed sound data, and annotate the mixed sound data to obtain a mixed sound dataset (Pre-SPD, containing overlapping sound events) through annotators trained on the sound event classification standard for open source datasets;

[0017] S202 Annotator Capability Assessment: Crowdsourced annotators annotate Pre-WSD, and calculate the crowdsourced annotator classification accuracy (SCA) to measure the accuracy of single sound source recognition. Crowdsourced annotators annotate Pre-SPD, and use the Cohens Kappa coefficient (CK) to assess the group consistency of annotators' annotation results in mixed scenarios and quantify the consensus on significance judgments.

[0018] S203 Accuracy and Consistency Verification Analysis: Conduct nonlinear trend analysis of SCA and CK to analyze their correlation patterns. Determine SCA and CK screening thresholds, retaining annotators with good classification ability and high consistency. S204 Data Sample Redundant Annotation: Assign candidate samples (including single sound source and mixed multi-source types) to multiple screened annotators for redundant annotation (≥5 annotators per sample). Each annotator's tasks must cover both "single sound salient events" and "mixed sound salient events."

[0019] S205 Label opinion aggregation: The results are aggregated using the "majority voting" rule. Samples with large disagreements (label difference > 30%) are manually reviewed or discarded to generate acoustic event labels that are both representative and reliable.

[0020] S206 Significant Acoustic Event Dataset Generation: Integrate all acoustic event labels, sort out the classification distribution of single and mixed acoustic events, and construct a structured significant acoustic event dataset (USSD) for deep learning model training and verification.

[0021] In the above technical solution, step S3 includes the following steps.

[0022] S301 Deep Learning Model Architecture Design: Based on the multi-source mixed and complex characteristics of urban acoustic environments, we built an ECAPA-TDNN model that integrates multi-scale residual blocks and channel attention mechanism (SE) to enhance feature expression capabilities.

[0023] S302 Feature Extraction: The audio samples are sequentially processed through audio framing (preferably with a window length of 1200 points and a frame shift of 480 points). The framed audio is then subjected to FFT transformation to calculate the power spectrum. This is then transformed into dB and normalized with the frame-by-frame mean-variance. Finally, the result is compressed using Mel spectrum (preferably converted to an 80-dimensional Mel spectrum) to obtain voiceprint features coupled between the time and frequency domains.

[0024] S303 Deep Learning Model Training and Validation: Using a ten-fold cross-validation strategy, the dataset is randomly split into n parts, with m parts used for training and nm parts used for validation. The model's robustness in identifying significant acoustic events is comprehensively evaluated using classification accuracy, confusion matrix, and time-reduction error metrics.

[0025] S304 Deep Learning Model Optimization Output: After multiple rounds of training and parameter optimization, a mature model with the ability to classify significant acoustic event types and restore their duration is output, providing algorithmic support for intelligent analysis of urban soundscapes.

[0026] In the above technical solution, step S4 includes the following steps.

[0027] S401 Test Set Verification: Using the trained model, significant acoustic event detection is performed on the Pre-SPD dataset (including overlapping acoustic events). The audio sample is input and the detection results of event type and duration are output.

[0028] S402 Strong Label Comparison Verification: Compare the model detection results with the strong labels (real results) pre-labeled by experts to verify the accuracy of event type recognition and duration restoration;

[0029] S403 Generalization Ability Evaluation: Based on the comparison results, the model's generalization and adaptation capabilities to unseen data are evaluated from the dimensions of event type recognition accuracy (Precision>0.95), duration restoration Pearson correlation coefficient (PCC≥0.97), and mean absolute percentage error (MAPE≤6.94%).

[0030] Compared with the prior art, the urban sound event labeling and identification method based on significance judgment of the present invention has the following beneficial effects:

[0031] Through the "unified classification standard + multi-annotator crowdsourcing + opinion aggregation" mechanism, the consistency of annotators' judgments on significant acoustic events reached 78.79%, solving the problem of "large individual differences and unreliable results" in traditional labeling and providing high-quality labels for the model; the model achieved an accuracy rate of 91.3% in ten-fold cross-validation, significantly outperforming the traditional SED model that relies on "clean data" training; in the generalization test, the main event classification accuracy was >0.95, and the duration restoration MAPE was as low as 6.94%, accurately adapting to the complex urban sound environment; it can efficiently extract "significant events carrying key information" in urban soundscapes (such as the strong correlation between the proportion of human voices in parks and the number of users\(r = 0.974\)), providing a non-visual data source for smart governance scenarios such as noise control, emergency warning, and space use analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1: Overall flow chart of the significant acoustic event detection method;

[0033] Figure 2: Schematic diagram of the distribution of 15 sampling points;

[0034] Figure 3 : Quadratic regression curve of annotator SCA and CK;

[0035] Figure 4 : USSD data set composition;

[0036] Figure 5 : ECAPA-TDNN model architecture diagram;

[0037] Figure 6: Comparison of the time distribution of model detection and real acoustic events;

[0038] Figure 7 : Duration verification diagram of the main events, where (a) is the comparison of the cumulative duration of exercise sounds, (b) is the comparison of the cumulative duration of children's voices, and (c) is the comparison of the cumulative duration of conversation sounds. DETAILED DESCRIPTION

[0039] The following is a combination of specific implementation cases and attached Figure 1-7 The present invention is further described below, but the present invention is not limited to these embodiments.

[0040] The present invention is developed around four core processes ( Figure 1 The project involves "data collection and preprocessing," "data labeling," "model training and validation," and "generalization testing." First, audio recordings are collected from urban environments and preprocessed. Second, expert pre-labeling and crowdsourcing are performed to create a dataset for training and validation. Third, a deep learning model is constructed for training and validation. Finally, the model's generalization ability is tested using the pre-labeled, strongly labeled dataset.

[0041] Considering the typicality of public spaces, the study selected representative public spaces around four urban functional areas: transportation facilities, commercial centers, residential areas, and comprehensive public buildings, and arranged sampling points ( Figure 2 ). A high-fidelity recorder is placed at each point, and the parameters are set to single channel, 16 bit, sampling frequency 48kHz, and wav format. During the recording period, the average temperature must be kept comfortable and normal (recommended between 15-30 degrees Celsius, and 18.5 degrees Celsius in the research case), and the weather must be sunny. Each sampling point has been continuously recorded for at least one week. Multiple fragments are extracted from the continuous recording hour by hour and uniformly divided into 10-second audio samples to ensure the data scale and event integrity; fragments containing privacy information (such as clear human voices, specific environmental signs) are processed through manual review or algorithm to remove privacy while retaining the time domain characteristics and frequency domain characteristics of the sound event; the collected, segmented, and de-identified audio are integrated to generate a structured label sample set for subsequent saliency labeling, deep learning model training and verification.

[0042] To evaluate the classification capabilities of different crowdsourced annotators and obtain data for validation testing, annotators trained on open-source datasets pre-annotated candidate labels for clean sounds. This pre-annotation process referenced ISO classification standards and relevant research to develop a classification scheme, defining a framework for five primary categories: "natural, quiet, human, traffic, and mechanical," and their associated secondary categories. This annotation process also resulted in a single sound dataset (Pre-WSD, 180 samples). A space with a rich variety of event types was selected, and 30 minutes of audio and video data were simultaneously recorded. Annotators trained on open-source datasets annotated this 30 minutes of audio and video data to create a strongly labeled dataset (Pre-SPD).

[0043] 80 crowdsourced annotation volunteers were trained on the two datasets. The classification accuracy (SCA) of each crowdsourced annotator was obtained by annotating the Pre-WSD. The consistency evaluation (CK) of the salient sound judgment results of each crowdsourced annotator was obtained by annotating the mixed sound samples in the Pre-SPD. The relationship between SCA and CK was analyzed ( Figure 3 ) and screened out crowdsourced annotators with strong classification capabilities. These 60 annotators achieved scores above the optimal grouping thresholds of SCA = 0.6456 and CK = 0.6659. This group of annotators demonstrated excellent classification proficiency and consistency (mean SCA = 0.8417; mean CK = 0.7919). Subsequent annotation of the significant acoustic event dataset was completed by these 60 annotators with strong classification capabilities. This approach reduced the impact of the crowdsourced annotators' classification capabilities on the perceived annotation errors of significant acoustic events in mixed sounds, thereby ensuring the reliability of the annotation results.

[0044] Finally, the urban significant sound dataset USSD was generated. Figure 4 As shown in Figure 2, the dataset contains 3094 labels, including 1749 single-sound event labels and 1345 mixed-sound salient event labels. The first-level classifications are ranked by sample size as follows: traffic sound (1076 samples), human (1083), natural (547), and mechanical (276).

[0045] The deep learning model based on ECAPA-TDNN is used for acoustic event classification. ECAPA-TDNN is a high-performance model designed for voiceprint recognition tasks. The model structure is as follows Figure 5As shown in the figure, the sound data is first converted into a Mel spectrum as input feature for the model to capture the voiceprint characteristics of the sound signal in the time and frequency domains. Specifically, an 80-dimensional Mel spectrum is calculated, using an FFT window length of 1200 points and a frame shift of 480 points. The power spectrum is then decibel-transformed and frame-by-frame mean-variance normalization is performed. Secondly, the main model uses a multi-scale residual block module with channel attention (SE-Res2Block), which constructs the basic computational unit through one-dimensional convolution (Conv1D), activation function (ReLU), and batch normalization (BN). This module combines the grouped convolution strategy of the multi-scale residual block (Res2Block) (with 8 groups and dilation rates of 2, 3, and 4, respectively) to effectively capture local and global dependencies in the sound signal. Furthermore, the module integrates the Squeeze-and-Excitation (SE) channel attention mechanism, which achieves inter-channel interaction and weight distribution through "squeeze" global information aggregation and "excitation" weight calibration. Overall, the multi-scale residual block module with channel attention significantly improves the model's classification robustness and generalization capabilities, effectively adapting to complex and changing acoustic environments. Finally, the Attentive Stats Pooling module dynamically weights frame features in the temporal dimension to achieve global feature aggregation, ultimately outputting a 192-dimensional embedding vector. This vector is then passed through a fully connected layer to complete acoustic event classification. Training utilizes the Adam optimizer with an initial learning rate of 0.001 and a cosine annealing learning rate schedule to improve training efficiency and convergence stability. To strike a balance between prediction quality and time cost, the number of iterations was ultimately fixed at 50. Training encompassed a total of 3094 training sets. Experiments were conducted on an Intel(R) i5-14600KF 14-core 20-thread processor running the 64-bit Windows 10 operating system, 32GB of RAM, and an NVIDIA RTX 4070TiSuper GPU.

[0046] In order to evaluate the error of the significant acoustic event detection results relative to the true distribution of acoustic events, the study further compared the model's detection results of the actual duration of the Pre-SPD acoustic event type. The results are as follows: Figure 6As shown. In the figure, row T (blue) shows the true distribution of acoustic event types and durations in Pre-SPD. The dataset contains 180 10-second audio samples, with a total of 610 overlapping acoustic events recorded. On average, each sample contains about 3.4 acoustic events, and the number of acoustic events ranges from 1 to . Row P (orange) shows the detection results of the best model (ACC=92.2%) obtained by ten-fold cross-validation training on the same dataset. Each audio sample corresponds to an identified significant acoustic event, totaling 180. The model detection results successfully identified 12 event types in Pre-SPD (12 / 14, recognition rate 85.7%). Among the identified event types, the model prediction results have a high accuracy (accuracy>0.95), that is, most of the significant acoustic event types predicted by the model actually exist.

[0047] In order to evaluate the model's ability to restore the real acoustic environment characteristics, the differences between the cumulative results of duration detection on the main acoustic event types and the cumulative results of the real duration are compared. Figure 7 The model detection results restored the duration of exercise sounds, children's voices, and conversation sounds at rates of 90.64%, 87.72%, and 80.51%, respectively. The PCC correlation coefficients between the model-predicted cumulative duration and the actual cumulative duration were 0.97, 0.98, and 0.99, respectively, indicating that all three types of sound events were well restored, especially for exercise sounds and children's voices, which had longer durations. This means that the model can stably capture the duration and scale of major sound events, reflecting the model's overall ability to estimate the duration and scale distribution of sound events.

Claims

1. A method for labeling and identifying urban sound events based on saliency judgment, characterized by: The steps include: S1: Data acquisition and preprocessing; S2: saliency annotation and dataset construction; S3: Deep learning model construction and training; S4: Generalization test and duration restoration.

2. The urban sound event labeling and identification method based on significance judgment according to claim 1 is characterized in that: Step S1 includes the following steps: S101 Audio Data Collection: Select typical public spaces in four functional areas: transportation facilities, commercial centers, residential areas, and comprehensive public buildings. Record the audio continuously at a sampling frequency to obtain the original audio. S102 Sample Segmentation: Extract multiple segments from continuous recordings hour by hour and divide them into fixed-length audio samples to ensure data scale and event integrity. S103 De-identification: Through manual review or algorithmic processing of fragments containing private information, privacy is removed while retaining the time domain and frequency domain characteristics of the acoustic event; S104 Label Sample Generation: Integrate the collected, segmented, and de-identified audio to generate a structured label sample set for subsequent saliency annotation, deep learning model training, and verification.

3. The urban sound event labeling and identification method based on significance judgment according to claim 1 is characterized in that: In step S2, the following steps are included: S201 Pre-annotation: Organize a number of annotators trained in the open-source sound event classification standard to perform initial annotation of the pure sound dataset Pre-WSD based on unified semantic rules, clarifying the framework of the five first-level categories of "nature, quiet, human, traffic, and machinery" and their corresponding second-level categories; collect mixed sound data, and annotate the mixed sound data with annotators trained in the open-source sound event classification standard to obtain the mixed sound dataset Pre-SPD; S202 Annotator Capability Assessment: Crowdsourced annotators annotate Pre-WSD, and calculate the crowdsourced annotator classification accuracy (SCA) to measure the accuracy of single sound source recognition. Crowdsourced annotators annotate Pre-SPD, and use the Cohens Kappa coefficient (CK) to assess the group consistency of annotators' annotation results in mixed scenarios and quantify the consensus on significance judgments. S203 Accuracy and Consistency Verification Analysis: Conduct nonlinear trend analysis of SCA and CK to analyze the correlation between the two; determine SCA and CK screening thresholds, and retain annotators with good classification ability and high consistency; S204 Data Sample Redundant Annotation: Assign candidate samples to multiple screened annotators for redundant annotation. Each annotator's tasks must cover both "single sound salient events" and "mixed sound salient events"; S205 Labeling Opinion Aggregation: Aggregate the results using the "majority voting" rule, manually review or discard samples with significant disagreements, and generate acoustic event labels that are both representative and reliable. S206 Significant Acoustic Event Dataset Generation: Integrate all acoustic event labels, sort out the classification distribution of single and mixed acoustic events, and construct a structured significant acoustic event dataset for deep learning model training and verification.

4. The urban sound event labeling and identification method based on significance judgment according to claim 1 is characterized in that: In step S3, the following steps are included: S301 Deep Learning Model Architecture Design: Based on the multi-source mixed and complex characteristics of urban acoustic environments, we built an ECAPA-TDNN model that integrates multi-scale residual blocks and a channel attention mechanism to enhance feature expression capabilities. S302 Feature Extraction: The audio samples are sequentially framed. The framed audio is transformed by FFT to calculate the power spectrum. The power spectrum is then normalized by dB transformation and frame-by-frame mean-variance normalization. Finally, the Mel spectrum is compressed to obtain the voiceprint features coupled in the time and frequency domains. S303 Deep Learning Model Training and Validation: Using a ten-fold cross-validation strategy, the dataset is randomly split into n parts, with m parts used for training and nm parts used for validation. The model's robustness in identifying significant acoustic events is comprehensively evaluated using classification accuracy, confusion matrix, and time-reduction error metrics. S304 Deep Learning Model Optimization Output: After multiple rounds of training and parameter optimization, a mature model with the ability to classify significant acoustic event types and restore their duration is output, providing algorithmic support for intelligent analysis of urban soundscapes.

5. The urban sound event labeling and identification method based on significance judgment according to claim 1 is characterized in that: In step S4, the following steps are included: S401 Test Set Verification: Use the trained model to detect significant acoustic events on the Pre-SPD dataset. Input audio samples and output detection results of event type and duration. S402 Strong Label Comparison Verification: Compare the model detection results with the strong labels pre-labeled by experts to verify the accuracy of event type recognition and duration restoration; S403 Generalization Ability Evaluation: Based on the comparison results, the model's generalization and adaptation ability to unseen data is evaluated from the dimensions of event type recognition accuracy, duration restoration Pearson correlation coefficient, and mean absolute percentage error.

Images