A method and system for classifying and marking sounds of multiple hidden dangers in power transmission channels

Abstract

The present invention proposes a method and system for classifying and labeling multiple hidden danger sounds in a power transmission channel, which belongs to the technical field of classifying multiple hidden danger sounds in a power transmission channel, including: continuously collecting sound signals around the power transmission channel; processing the sound signals and extracting Mel-spectrogram features; inputting the extracted Mel-spectrogram features into a trained classification labeling model for prediction, and outputting the sound type contained in the sound signal. The classification labeling of single or multiple hidden danger sound types contained in the sound signal is realized, the monitoring capability when multiple hidden dangers occur at the same time is improved, and the safety of power facilities is further guaranteed.

Description

Technical Field

[0001] The present invention belongs to the field of sound classification of multiple hidden dangers in power transmission channels, and in particular relates to a method and system for classifying and marking sounds of multiple hidden dangers in power transmission channels. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] The transmission channel scenario refers to the outdoor power transmission line scenario. The transmission lines are installed in the air through poles and towers to achieve power transmission. There are many hidden dangers when the transmission lines are exposed to the outdoor environment. For example, large birds can easily cause short circuits in the transmission lines, and construction around the transmission lines may cause damage to the transmission lines. Therefore, it is necessary to use some technical means to monitor the hidden dangers around the transmission lines, issue timely warnings when hidden dangers occur, and take other preventive measures later.

[0004] Currently, there are many hidden dangers in power transmission channel scenarios. The sound classification technology in this scenario mainly uses acoustic features combined with deep learning to perform multi-category single-label analysis. Each time an audio signal is input, only the category with the highest score is output. In other words, only one category is output for an input signal each time.

[0005] There are many current studies on sound classification, including traditional machine learning methods and deep learning methods. For example, the traditional machine learning method of support vector machine is suitable for small samples and high-dimensional data, and can handle multi-label classification tasks after audio feature extraction; deep learning methods include convolutional neural networks, which are mainly used to process audio feature maps, can capture local features, and are suitable for multi-label classification tasks. In recent years, the Transformer architecture based on the self-attention mechanism has also been applied to audio classification, which can model long-range dependencies; there are also many mainstream features for sound classification, such as: Mel spectrum, Mel cepstral coefficients, and Fbank.

[0006] The technologies existing in the existing literature mainly include: Application No.: CN202110679471.X A method for environmental sound classification based on convolutional neural network relationship modeling; Application No.: CN202410093149.2, A method, system, device and storage medium for underwater sound classification model training; Application No.: CN202410313134.2, A method and system for sound environment recognition based on artificial intelligence; Pu Zijun, Zhang Shouming. Research on sound event localization and detection algorithm based on feature fusion and Transformer model [J]. Computer Engineering&Science / Jisuanji Gongcheng yu Kexue, 2023, 45(6).

[0007] After analysis, it was found that the existing technology for environmental sound classification mostly uses the category with the largest prediction score in the output sound signal, which is suitable for situations with a single type. When the sound signal contains multiple types of sounds at the same time, it is easy to miss detection or make a false detection.

[0008] In addition, the existing literature also discloses CN114464215B_A method and device for monitoring hidden dangers in power transmission line channels. Through the sound classification algorithm in the audio-visual acquisition device, the hidden danger audio data is determined, and the audio data without hidden dangers can be filtered out, thereby reducing the audio data for hidden danger processing and improving the speed of hidden danger analysis. Secondly, the embodiment of the present application can identify the category of hidden danger data and the sound start time through the preset audio feature extractor and the preset sound classifier, so as to obtain detailed transmission line hidden danger information. In addition, the embodiment of the present application combines the first transmission line hidden danger information identified by sound with the second transmission line hidden danger information identified by image to obtain the transmission line hidden danger level, which can monitor the transmission line in real time due to weather, environmental changes, etc., and timely and accurately determine the hidden dangers of the transmission line, thereby ensuring the stable operation of the transmission line.

[0009] The above scheme is effective for the alternating appearance of hidden danger sounds and non-hidden danger sounds, but it is easy to cause misjudgment for the overlapping appearance of hidden danger and non-hidden danger sounds. Moreover, from the above text description, the scheme only distinguishes between hidden dangers and non-hidden dangers, and does not identify the specific type of hidden dangers. It is difficult to take corresponding effective preventive measures against the hidden dangers later. Summary of the invention

[0010] In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides a method for classifying and marking multiple hidden danger sounds in a power transmission channel, thereby realizing the classification of hidden danger sounds containing multiple types at the same time.

[0011] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0012] In a first aspect, a method for classifying and marking multiple hidden danger sounds in a power transmission channel is disclosed, comprising:

[0013] Continuously collect sound signals around the power transmission channel;

[0014] Process the sound signal and extract the Mel-spectrogram features;

[0015] The extracted Mel-spectrogram features are input into the trained classification labeling model for prediction, and the sound type contained in the sound signal is output;

[0016] The classification labeling model includes a convolutional layer, a depth-separable convolutional layer, a bottleneck layer, an SE module, and a global average pooling and classification layer;

[0017] The convolution layer is used to extract the features of the input Mel spectrum graph;

[0018] The depth-wise separable convolutional layer processes the extracted features again to reduce computational complexity.

[0019] The bottleneck layer mainly uses 1x1 convolution for channel compression;

[0020] The SE module is used to recalibrate the features in the channel dimension. By learning the importance weight of each channel, the channel weight of the feature map is adaptively adjusted so that the network can better focus on important features.

[0021] Global average pooling is used to reduce feature dimensions and obtain a more compact feature representation;

[0022] The classification layer is used to provide a probability for each target sound type.

[0023] As a further technical solution, the classification labeling model is trained using a data set of multi-hazardous sound in power transmission channels during the training process.

[0024] As a further technical solution, the construction process of the transmission channel multi-hazard sound data set is as follows:

[0025] Collect sound data around the power transmission channel;

[0026] Analyze the collected sound data, determine the annotated sound type based on statistical analysis and the sounds of interest in the power transmission scenario, and construct the corresponding annotation files according to the divided sound types. Subsequently, annotate the specific sound categories contained in each sound file under the corresponding annotation file category;

[0027] Finally, all the annotated audios are placed in a folder, and the annotation labels corresponding to all the audios are organized into a csv table file. The annotation file is split into training label data csv file, test label data csv file, and verification label data csv file according to the proportion, plus a data set containing all sound type files, to complete the construction of the transmission channel multi-hazard sound data set.

[0028] As a further technical solution, each sound file is annotated under a corresponding annotation file category according to the specific sound category contained in the sound file. The specific annotation process is as follows:

[0029] Generate a total annotation file for all the sound files to be annotated, which contains the file names of all the sound files and the corresponding type tags;

[0030] All audios are annotated, and each audio is checked to confirm whether the sound type in the annotated sound type exists, and scores are given in order according to whether the sound exists and the clarity of the sound contained in the audio;

[0031] Then, after all the sound data are labeled, there will be three labeled files. A screening mechanism is used to determine the final data labels for the three labeled files.

[0032] As a further technical solution, the process of using a screening mechanism to determine the final data label is as follows:

[0033] Check whether the number and names of the sound files in the three annotation files are consistent to ensure that the three annotation files mark the same audio data;

[0034] After confirming the consistency, the labels of all categories in the annotation file are binarized to 0 or 1 according to the annotation scores;

[0035] Finally, the three annotated categories corresponding to the same audio are intersected to obtain the final audio data annotation. The annotation label corresponding to the audio may be one category or multiple categories, depending on the actual sound it contains.

[0036] As a further technical solution, the labels are binarized to 0 or 1. Specifically, the scores of each category of each audio are processed with a threshold of 0.8. If the score is greater than or equal to 0.8, it is assigned a value of 1, otherwise it is assigned a value of 0.

[0037] As a further technical solution, the sound signal is processed and the Mel-spectrogram features are extracted. The specific process is as follows:

[0038] Use a high-pass filter to pre-emphasize the sound signal to equalize the intensity difference between high and low frequencies;

[0039] Perform frame-by-frame windowing on the input pre-emphasized audio;

[0040] Perform Fourier transform on each frame of the sound signal to transform the signal from the time domain to the frequency domain;

[0041] The obtained Fourier transformed data is processed by a Mel filter bank;

[0042] Then all the data processed by the Mel filter bank are concatenated and integrated to obtain the Mel spectrum graph.

[0043] In a second aspect, a system for classifying and marking multiple hidden danger sounds in power transmission channels is disclosed, comprising:

[0044] The sound signal collection module is configured to: collect sound signals around the power transmission channel;

[0045] The feature extraction module is configured to: process the sound signal and extract the Mel-spectrogram features;

[0046] The prediction module is configured to: input the extracted Mel-spectrogram features into the trained classification labeling model for prediction, and output the sound type contained in the sound signal;

[0047] The classification labeling model includes a convolutional layer, a depth-separable convolutional layer, a bottleneck layer, an SE module, and a global average pooling and classification layer;

[0048] The convolution layer is used to extract the features of the input Mel spectrum graph;

[0049] The depth-wise separable convolutional layer processes the extracted features again to reduce computational complexity.

[0050] The bottleneck layer mainly uses 1x1 convolution for channel compression;

[0051] The SE module is used to recalibrate the features in the channel dimension. By learning the importance weight of each channel, the channel weight of the feature map is adaptively adjusted so that the network can better focus on important features.

[0052] Global average pooling is used to reduce feature dimensions and obtain a more compact feature representation;

[0053] The classification layer is used to provide a probability for each target sound type.

[0054] One or more of the above technical solutions have the following beneficial effects:

[0055] The technical solution of the present invention analyzes the sound signals around the transmission channel based on the deep neural network model, and can classify and mark the situations containing one or more types of hidden danger sounds. The acoustic features of specific dimensions are extracted through multi-step processing; the obtained Mel spectrum features are input into the MobileNet model that can perform time convolution for iterative training. Finally, the trained model can classify and mark the single or multiple types of hidden danger sounds contained in the sound signal, avoiding the defect that the general classification algorithm only outputs the category with the highest score, improving the monitoring ability when multiple hidden dangers occur at the same time, and taking corresponding measures when hidden dangers are detected. For example, if bird activities are detected here, the bird repellent device here can be linked based on this information to prevent bird activities from causing damage to power facilities; if construction is detected nearby, the shouting device can be linked to remind the surrounding construction personnel and other measures to further ensure the safety of power facilities.

[0056] The specific data annotation and processing technology proposed in the technical solution of the present invention realizes the construction of a high-quality transmission channel multi-hazard sound data set and provides a basis for the development of a classification and labeling model.

[0057] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0059] Figure 1 Schematic diagram of feature extraction in an embodiment of the present invention;

[0060] Figure 2 This is a schematic diagram of model training in an embodiment of the present invention;

[0061] Figure 3 A schematic diagram of sound classification prediction according to an embodiment of the present invention;

[0062] Figure 4 Schematic diagram of the Mobilenetv3 classification labeling model structure according to an embodiment of the present invention. DETAILED DESCRIPTION

[0063] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0064] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0065] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0066] The overall idea proposed by the present invention is:

[0067] Firstly, a high-quality transmission channel multi-hidden danger sound database is constructed according to the proposed specific data annotation and processing method, and all data are divided into different parts according to the set ratio; then, the acoustic features of specific dimensions are extracted through multi-step processing; the obtained Mel-spectrogram features are input into the MobileNet model that can perform time convolution for iterative training. Finally, the trained model can classify and mark the single or multiple hidden danger sound types contained in the sound signal, avoiding the defect of general classification algorithms that only output the category with the highest score, improving the monitoring capability when multiple hidden dangers occur at the same time, and further ensuring the safety of power facilities.

[0068] Glossary: ​​sum: is the sum function.

[0069] Embodiment 1

[0070] See attached Figure 3 As shown, this embodiment discloses a method for classifying and marking multiple hidden danger sounds in a power transmission channel, comprising:

[0071] First, the acquisition device placed in the application environment continuously collects the environmental sound signals around the power transmission channel;

[0072] After being pre-processed by the device, the sound signal is uploaded to the algorithm server. After receiving the sound signal, the server starts to store the data, pre-processes the sound signal, and then performs feature extraction and extracts the above-mentioned Mel spectrum features;

[0073] After feature extraction, it is input into the trained model for prediction, and the prediction results are obtained, and then post-processing is performed, including: after the model analysis is completed, the prediction scores for all preset sound types will be output. The scores of each category are independent of each other. The final output score of each category can be judged by the preset threshold. If the category exceeds its own preset threshold, it is considered that the sound signal contains this type of sound. All categories that exceed their respective preset thresholds are output as the types contained in the sound signal;

[0074] The final results are uploaded to the platform for display or downstream tasks are processed based on the results.

[0075] Since the trained model is obtained by training on the constructed transmission channel multi-hazard sound dataset, the construction and processing process of the transmission channel multi-hazard sound dataset is introduced below:

[0076] When establishing a data set to obtain data, a device with an audio acquisition function is placed in a real scene to collect real-time sound data around the transmission channel. After a long period of collection and accumulation, the sound data gradually reaches a certain amount, and then is annotated to form a data set for model training and testing. All the collected sounds come from the tower-hanging equipment. There are many devices installed on different transmission towers, and all the collected sound data may not be from the same location.

[0077] Specifically, the collected sound data is analyzed, and the sound types are determined based on statistical analysis and the sounds of interest in the power transmission scenario. There are 9 categories, including bird calls, wind sounds, engines, thunder sounds, metal collisions, whistles, human voices, rain sounds, and others. The corresponding annotation files are also constructed according to the 9 categories. Subsequent annotation personnel will annotate the specific categories contained in each sound file under the corresponding annotation file category. The specific method is as follows:

[0078] First, a total annotation file is generated for all the sound files to be annotated, which contains the file names of all the sound files and the corresponding 9 types of labels. Three annotators are assigned to annotate all the audios. The annotators check each audio to confirm whether there is a sound type in the 9 categories. They score it according to whether it exists and the clarity of the sound contained in the audio. The sound of the type included is marked as 1 in the corresponding category. If there is an unclear situation, the corresponding score is given between 0 and 1 according to the similarity of the sound. If it does not exist, it is marked as 0.

[0079] Then, after all the sound data are labeled, there will be three labeled files. A screening mechanism is used to determine the final data labels for the three labeled files. The specific screening method is as follows:

[0080] a. Check whether the number and names of the sound files in the three annotation files are consistent to ensure that the three annotation files mark the same audio data.

[0081] b. After confirming the consistency, the labels of all categories in the annotation file are binarized to 0 or 1 according to the annotation scores. The specific operation is: the score of each audio corresponding to each category is processed with 0.8 as the threshold, and the score greater than or equal to 0.8 is assigned 1, otherwise it is assigned 0.

[0082] c. Finally, take the intersection of the three annotated categories corresponding to the same audio to obtain the final audio data annotation. The annotation label corresponding to the audio may be one category or multiple categories, depending on the actual sound it contains.

[0083] Finally, all the annotated audios are placed in a folder, and the annotation labels corresponding to all the audios are organized into a csv table file. The annotation file is split into training label data csv file, test label data csv file, and verification label data csv file according to the proportion, plus a data set containing all sound type files. At this point, the construction of the transmission channel multi-hazard sound data set is completed.

[0084] The model training process of the sub-technical solution of this embodiment includes: the data reading and processing method in the database requires that the audio and label data are constructed according to the above structure.

[0085] For the training process of the Mobilenetv3 classification labeling model, see the attached Figure 2 As shown, it includes obtaining a sound signal. After obtaining the sound signal, it is necessary to extract acoustic features and select a Mel spectrum graph;

[0086] The text labels in the transmission channel multi-hazard sound dataset are mapped to the data labels by ID, and then iterative training is performed to obtain the trained model.

[0087] The above ID mapping is performed because the model processes numbers. Therefore, when the model is iteratively trained, each category is represented by a number such as 0, 1, or 2. Similarly, when the model predicts the input sound, the output is also a number 0, 1, or 2. Then, after ID mapping, the numbers 0, 1, or 2 are converted into previously defined text labels, such as 0->bird, 1->engine, etc.

[0088] In this implementation example, the acoustic feature used is a Mel-spectrogram. All audios that need to extract acoustic features are pre-emphasized before extraction. The sound signal is passed through a high-pass filter to balance the intensity difference between high and low frequencies, thereby increasing the high-frequency resolution of the sound. The first-order filter used for pre-emphasis can be expressed as:

[0089]

[0090] H(z) is the transfer function of the digital filter, describing the relationship between the input signal X(z) and the output signal Y(z). µ is a filter coefficient, usually related to the cutoff frequency of the high-pass filter. −1 is the delay operator, which means that the signal is delayed by one sample in time. For example, if the input signal is x[n], then z −1 x[n] is x[n−1], which means that the signal has shifted backward in time by one sampling period.

[0091] Next, we will enter the Mel spectrum extraction process, see the attached Figure 1 As shown in the figure, firstly, the sound signal has a short-term stable characteristic. In order to facilitate the subsequent Fourier transform operation, the input pre-emphasized audio needs to be framed and windowed, that is, the sound signal is divided into several segments according to time, each segment becomes a frame, and the frame length is usually 20~40ms. After comprehensive analysis, the frame length is set to 25ms, the frame shift is set to 15ms, and there is an overlap of 10ms between the two frames to ensure a smooth transition between the two frames to prevent information leakage; the signal processing method requires that the signal is continuous, but it is disconnected in the middle during the frame processing. In order to ensure that the cut signal has global continuity and facilitate the subsequent frequency domain analysis, a specific window function is multiplied for each frame signal. This step is called windowing. Next, Fourier transform is performed on each frame of the sound signal to transform the signal from the time domain to the frequency domain. The obtained Fourier transform data is processed by the Mel filter bank, and then all the processed data are spliced ​​and integrated using the sum operation to obtain the Mel spectrum. The characteristics expressed by the Mel spectrum are consistent with the perception of the human ear, have a lower dimension, are noise robust, and contain relatively rich information.

[0092] See attached Figure 4As shown in the figure, Mobilenetv3 is a lightweight convolutional neural network model. Its main structure includes input layer, convolution layer, depth-separable convolution layer, bottleneck layer, SE module, global average pooling and classification layer.

[0093] The initial convolution layer is used for feature extraction, and the subsequent multiple stages of depth-separable convolution layers reduce the computational complexity; the bottleneck layer mainly uses 1x1 convolution for channel compression to enhance the model's feature extraction capability; the SE module recalibrates the features in the channel dimension, and by learning the importance weights of each channel, it can adaptively adjust the channel weights of the feature map so that the network can better focus on important features, thereby improving the network's representation and generalization capabilities; the introduction of global average pooling can further reduce the feature dimension and obtain a more compact feature representation.

[0094] Specifically, for the Mel-spectrogram feature, the feature is input into the model and processed by the convolution layer to obtain a high-level feature representation, which is presented in the form of a multi-dimensional array.

[0095] The above-mentioned multiple stages refer to the fact that the entire model structure contains many modules of depthwise separable convolution. Serial processing of multiple such modules can reduce the number of parameters and the computational complexity.

[0096] Depthwise separable convolution reduces the computational complexity and number of parameters of the model by splitting the convolution operation into depthwise convolution and pointwise convolution. Depthwise convolution uses an independent convolution kernel for each input channel for convolution operation. The convolution operation is only calculated with the features of the channel and does not fuse the information of other channels. Pointwise convolution uses a 1×1 convolution kernel to process the output of depthwise convolution to fuse the information of different channels.

[0097] Channel compression can effectively reduce the number of feature channels, thereby reducing computational complexity and model memory usage. At the same time, it can enable the model to skip some redundant information and retain the most important features.

[0098] The recalibration process is divided into two steps:

[0099] 1. Squeeze operation (compression): The SE module first compresses the spatial features of each channel into a single value through global average pooling.

[0100] 2. Excitation operation: The SE module processes the compressed features through a small neural network (usually composed of two fully connected layers) to generate channel weights. Finally, the generated channel weights are used to recalibrate the original feature map.

[0101] Adaptively adjust the channel weights of the feature map: This needs to be combined with the SE module processing process mentioned above. For adaptive adjustment, the channel weights obtained by the SE module are different according to the different input features, and then the weights are applied to the original features initially input to the SE module, that is, adaptive adjustment.

[0102] In order to make the network structure suitable for processing time series data such as sound, two adjustments are made. First, the sound time series data is divided into windows of fixed length, and then each window data is input as an image to the mobilenetv3 model for analysis; secondly, a time convolution layer is added to mobilenetv3, that is, the time series information is extracted by sliding the convolution kernel in the time dimension, and the multi-head attention mechanism pooling layer in the network is used to obtain the time series dependencies of the sound signal. Finally, the output layer uses a sigmoid unit to provide a probability for each target type.

[0103] The sound time series data segmentation is processed before the model input, and the specific segmentation is before the Mel spectrum is obtained.

[0104] This temporal convolution layer is placed at the very beginning of the model, which can effectively extract features in the time dimension, followed by the activation layer. The features are input to the convolution layer, and then the set temporal convolution kernel is used to slide on the time axis, the weighted sum of each position is calculated, and the new features obtained are used as the input of the subsequent network layer.

[0105] The temporal convolutional layer is calculated:

[0106]

[0107] in, H in and W in represents the height and width of the input feature, H out , W out Represents the height and width of the output feature, F is the size of the convolution kernel, P is the thickness of the zero padding at the edge of the feature map, and S represents the step size of the convolution kernel movement.

[0108] The multi-head attention mechanism pooling layer is located in the penultimate layer of the entire model network structure, that is, before the linear classification layer. It mainly performs targeted pooling processing on the input features. First, the input features will be transformed linearly to generate three weight matrices Q, K, and V, and then the weight-attention score will be calculated. Then, the corresponding weight score is multiplied by the input feature map to obtain the weighted features for subsequent classification processing.

[0109] In the above multi-head attention mechanism pooling layer, the mathematical expression for the generation of query, key, and value using linear transformation can be expressed as:

[0110] ;

[0111] W Q , W K , W V Represent the weight matrices of query, key, and value respectively, and X is the input data.

[0112] ;

[0113] in, d k Indicates the dimension of the key vector.

[0114] ;

[0115] Y is the output feature.

[0116] In the method of this embodiment, the model training input is 128xT-dimensional acoustic features. The final output is obtained after processing by different neural network layers. The model prediction output is compared with the label, and the loss value between the prediction and the true label is calculated. The weights and biases in the model are continuously updated and iterated through multiple rounds of back propagation, and finally a prediction model that best fits the data distribution of the data set is obtained.

[0117] Based on the Mel-spectrogram features extracted by fixed windows and the MobileNet network structure with temporal convolution, the classification of sounds containing multiple types of hidden dangers is achieved.

[0118] Embodiment 2

[0119] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0120] Embodiment 3

[0121] The purpose of this embodiment is to provide a computer-readable storage medium.

[0122] A computer-readable storage medium stores a computer program, which executes the steps of the above method when executed by a processor.

[0123] Embodiment 4

[0124] The purpose of this embodiment is to provide a system for classifying and marking multiple hidden danger sounds in power transmission channels, including:

[0125] The sound signal collection module is configured to: collect sound signals around the power transmission channel;

[0126] The feature extraction module is configured to: process the sound signal and extract the Mel-spectrogram features;

[0127] The prediction module is configured to: input the extracted Mel-spectrogram features into the trained classification labeling model for prediction, and output the sound type contained in the sound signal;

[0128] The classification labeling model includes a convolutional layer, a depth-separable convolutional layer, a bottleneck layer, an SE module, and a global average pooling and classification layer;

[0129] The convolution layer is used to extract the features of the input Mel spectrum graph;

[0130] The depth-wise separable convolutional layer processes the extracted features again to reduce computational complexity.

[0131] The bottleneck layer mainly uses 1x1 convolution for channel compression;

[0132] The SE module is used to recalibrate the features in the channel dimension. By learning the importance weight of each channel, the channel weight of the feature map is adaptively adjusted so that the network can better focus on important features.

[0133] Global average pooling is used to reduce feature dimensions and obtain a more compact feature representation;

[0134] The classification layer is used to provide a probability for each target sound type.

[0135] Embodiment 5

[0136] The purpose of this embodiment is to provide a computer program product containing instructions, which, when running on a computer, enables the computer to execute the methods and functions involved in any of the above embodiments.

[0137] The steps involved in the apparatus of the above embodiment correspond to the method embodiment 1, and the specific implementation method can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0138] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0139] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for classifying and marking multiple hidden danger sounds in a power transmission channel, characterized in that: include: Continuously collect sound signals around the power transmission channel; Process the sound signal and extract the Mel-spectrogram features; The extracted Mel-spectrogram features are input into the trained classification labeling model for prediction, and the sound type contained in the sound signal is output; The classification labeling model includes a convolutional layer, a depth-separable convolutional layer, a bottleneck layer, an SE module, and a global average pooling and classification layer; The convolution layer is used to extract the features of the input Mel spectrum graph; The depth-wise separable convolutional layer processes the extracted features again to reduce computational complexity. The bottleneck layer mainly uses convolution for channel compression; The SE module is used to recalibrate the features in the channel dimension. By learning the importance weight of each channel, the channel weight of the feature map is adaptively adjusted so that the network can better focus on important features. Global average pooling is used to reduce feature dimensions and obtain a more compact feature representation; The classification layer is used to provide a probability for each target sound type; The trained classification labeling model is obtained by training on the constructed transmission channel multi-hazard sound dataset. The construction and processing process of the transmission channel multi-hazard sound dataset is as follows: When acquiring data from the sound data set of multiple hidden dangers in power transmission channels, the equipment with audio collection function collects the sound data around the power transmission channels in real time; after a long period of collection and accumulation, the sound data gradually reaches the set amount, and then is annotated to form a data set for model training and testing; The sound data are all collected from tower-mounted equipment, and multiple devices are installed on different transmission towers; The annotated sound types are determined based on statistical analysis and sounds of interest in power transmission scenarios. They are divided into nine categories, including bird calls, wind sounds, engines, thunder sounds, metal collisions, whistles, human voices, rain sounds, and others. The corresponding annotation files are constructed according to the nine categories. The specific categories contained in each sound file are annotated under the corresponding annotation file category. The specific method is as follows: First, a total annotation file is generated for all the sound files to be annotated, which contains the file names of all the sound files and the corresponding 9 types of labels; all audios are annotated, and each audio is checked to confirm whether there is a sound type in the 9 categories, and scores are given in turn according to whether it exists and the clarity of the sound contained in the audio. The type of sound confirmed is marked as 1 in the corresponding category. If there is an unclear situation, a corresponding score between 0-1 is given according to the degree of sound similarity. If it does not exist, it is marked as 0; Then, after all the sound data are labeled, they include three labeled files. A screening mechanism is used to determine the final data labels for the three labeled files. The specific screening method is as follows: a. Check whether the number and names of the sound files in the three annotation files are consistent to ensure that the three annotation files mark the same audio data; b. After confirming the consistency, binarize the labels of all categories in the annotation file according to the annotation scores to 0 or 1. The specific operation is as follows: the score of each category of each audio is processed with 0.8 as the threshold, and the score greater than or equal to 0.8 is assigned 1, otherwise it is assigned 0; c. Finally, the three annotated categories corresponding to the same audio are intersected to obtain the final audio data annotation. The annotation labels corresponding to the audio are one category or multiple categories. All the annotated labels corresponding to the audio are sorted into a csv table file, and the annotated file is split into a training label data csv file, a test label data csv file, and a verification label data csv file according to the proportion. A data set containing all sound type files is added to complete the construction of the transmission channel multi-hazard sound data set; The labels in the data set of multi-hazardous sounds in the power transmission channel are mapped to IDs to obtain the target output, and iterative training is performed based on the target output and Mel-spectrogram features to obtain the trained classification labeling model; All audio sound data that need to extract acoustic features are first processed by pre-emphasis before extraction. The first-order filter used for pre-emphasis can be expressed as: H(z)=1-μz -1 H(z) is the transfer function of the digital filter, which describes the relationship between the input signal X(z) and the output signal Y(z). μ is a filter coefficient related to the cutoff frequency of the high-pass filter. -1 is a delay operator, which means that the signal is delayed by one sample in time; if the input signal is x[n], then z -1 x[n] is x[n-1], which means that the signal has moved backward in time by one sampling period; The input pre-emphasized audio is processed by frame division and windowing, that is, the sound signal is divided into several segments according to time, each segment becomes a frame, the frame length is usually 20 to 40ms, the frame length is set to 25ms, the frame shift is set to 15ms, and there is an overlap of 10ms between the previous and next frames; Multiply each frame of the signal by the window function, then perform Fourier transform on each frame of the sound signal to transform the signal from the time domain to the frequency domain, process the obtained Fourier transform data through the Mel filter bank, and then use the sum operation to splice and integrate all the processed data to obtain the Mel spectrum. When the network structure of the classification labeling model processes the sound time series data, it divides the sound time series data into windows of fixed length, and then inputs each window data into the mobilenetv3 model as an image; a time convolution layer is added to mobilenetv3, that is, the time series information is extracted by sliding the convolution kernel in the time dimension, and the multi-head attention mechanism pooling layer in the network is used to obtain the time series dependency of the sound signal. The output layer uses a sigmoid unit to provide a probability for each target type; The temporal convolution layer is placed at the very beginning of the model to extract features in the temporal dimension, followed by an activation layer. After feature extraction, it is input into the trained model for prediction, and the prediction results are obtained, and then post-processing is performed, including: after the model analysis is completed, the prediction scores for all preset sound types will be output. The previous scores of each category are independent of each other. The final output score of each category is judged by the preset threshold. If the category exceeds its own preset threshold, it is considered that the sound signal contains this type of sound. All categories that exceed their respective preset thresholds are output as the types contained in the sound signal; The obtained Mel-spectrogram features are input into the time convolution MobileNet model for iterative training. Finally, the trained model can classify and mark the single or multiple types of hidden danger sound types contained in the sound signal, and take corresponding measures when hidden dangers are detected. If bird activity is detected here, the bird-repellent device here will be activated based on this information; if construction is detected nearby, the shouting device can be activated to remind the surrounding construction workers to ensure the safety of power facilities.

2. A transmission channel multi-hazard sound classification and marking system, using the transmission channel multi-hazard sound classification and marking method according to claim 1, characterized in that: include: The sound signal collection module is configured to: collect sound signals around the power transmission channel; The feature extraction module is configured to: process the sound signal and extract the Mel-spectrogram features; The prediction module is configured to: input the extracted Mel-spectrogram features into the trained classification labeling model for prediction, and output the sound type contained in the sound signal; The classification labeling model includes a convolutional layer, a depth-separable convolutional layer, a bottleneck layer, an SE module, and a global average pooling and classification layer; The convolution layer is used to extract the features of the input Mel spectrum graph; The depth-wise separable convolutional layer processes the extracted features again to reduce computational complexity. The bottleneck layer mainly uses 1x1 convolution for channel compression; The SE module is used to recalibrate the features in the channel dimension. By learning the importance weight of each channel, the channel weight of the feature map is adaptively adjusted so that the network can better focus on important features. Global average pooling is used to reduce feature dimensions and obtain a more compact feature representation; The classification layer is used to provide a probability for each target sound type.

3. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to claim 1 is implemented.

4. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to claim 1 are implemented.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to claim 1 are performed.

Images