Deep learning method and system for wind speed measurement based on surveillance camera audio data
The wind sound characteristics are extracted from the audio data of the monitoring camera through deep learning methods and dual attention mechanisms, which solves the problem of wind speed measurement in complex terrain areas and urban areas, and realizes accurate measurement of wind speed, reduces costs and is suitable for a variety of monitoring scenarios.
Patent Information
- Application Number
- CN202410917282.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-10
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-07-10
AI Technical Summary
The prior art is difficult to accurately capture the spatial variation characteristics of wind speed in complex terrain areas and urban areas, and ambient sound interference in the audio data of the monitoring camera increases the difficulty of wind speed measurement.
The deep learning method is adopted to extract the time-frequency domain characteristics of wind sound from the audio data of the monitoring camera through a dual attention mechanism, and SA-CNN and CA-CNN models are constructed to suppress the external acoustic interference and achieve accurate measurement of wind speed.
It improves the accuracy and real-time performance of wind speed measurement, reduces dependence on special equipment, reduces costs, is suitable for a variety of monitoring scenarios, and has a wide range of application prospects.
Smart Images

Figure CN118465305B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the fields of meteorology, computer hearing, and deep learning technology, and in particular, relates to a deep learning method and system for wind speed measurement based on audio data of a surveillance camera. Background Art
[0002] Wind plays an important role in the turbulent exchange of mass, energy and momentum between the land / water surface and the atmosphere. Surface wind speed perception with high temporal and spatial resolution is crucial to understanding urban microclimate, weather and pollution transmission laws, improving wind energy utilization efficiency, and promoting low-altitude economic development. However, as a basic means of surface wind speed observation, meteorological stations can obtain point wind speed information with high temporal resolution. However, due to the limitation of spatial discreteness, it is difficult to accurately capture the spatial variation characteristics of wind speed in complex terrain areas (such as mountainous areas, urban areas, etc.); weather radar can detect the three-dimensional structural information of the wind field, as well as continuous, high-spatial-resolution regional wind speed data, but there is an underlying surface blind spot in radar observations in urban areas, and there is a certain gap between the observation results and the surface wind conditions.
[0003] Statistics show that the number of surveillance cameras in my country far exceeds 500 million, and most of them are deployed in urban areas. The surveillance audio data collected by the widely distributed surveillance cameras can continuously and dynamically record the changes in surface wind speed, providing high temporal and spatial resolution wind speed measurement. How to monitor the wind sound from the surveillance audio and measure the wind speed has become the focus of the present invention. However, affected by human activities, the surveillance audio data often contains complex environmental sounds (such as vehicle horns, conversations, vehicle noise, etc.), which increases the difficulty of accurately measuring wind speed.
[0004] To this end, the present invention adopts a deep learning method and introduces a dual attention mechanism. First, the audio data is analyzed from the monitoring data; then, the audio features of wind sounds at different speeds are analyzed to construct a monitoring audio feature model of wind sounds; finally, the audio features of the monitored wind sounds are used as input, and the time-frequency domain features of the wind sounds in the monitoring audio are mined based on convolutional neural networks and long short-term memory networks to suppress the interference of external environmental sounds, and a wind speed regression calculation deep learning model is constructed to achieve accurate measurement of surface wind speed. The patent of the present invention is easy to deploy and implement, and there is no need to disassemble and calibrate the installed monitoring cameras, which is convenient for introduction and application in meteorological, transportation, urban management and other departments. Compared with the patent (Jiangsu Meteorological Service Center. A method and system for identifying road meteorological conditions based on road noise frequency analysis [P]. Chinese Patent: CN115762565A.2023.03.07), this patent is aimed at qualitative identification of road conditions (such as water accumulation, snow accumulation, ice, etc.). The feature of the present invention is that it starts with the audio collected by the monitoring camera and adopts a deep learning method to achieve quantitative estimation of wind speed rather than qualitative judgment, which has a wider application scenario and practical value.
[0005] Through the above analysis, the problems and defects of the prior art are as follows:
[0006] (1) Due to the spatial discreteness of meteorological stations, it is difficult to accurately capture the spatial variation characteristics of wind speed in complex terrain areas (such as mountainous areas and urban areas);
[0007] (2) Weather radar observations in urban areas have underlying surface blind spots, and there is a certain gap between the observation results and the surface wind conditions.
[0008] (3) Affected by human activities, the audio data monitored by surveillance cameras often contains complex environmental sounds (such as vehicle horns, conversations, vehicle noise, etc.), which increases the difficulty of accurately measuring wind speed. Summary of the invention
[0009] In view of the problems existing in the prior art, the present invention provides a deep learning method and system for wind speed measurement based on audio data of surveillance cameras.
[0010] The present invention is implemented as follows: a deep learning method for wind speed measurement based on surveillance camera audio data, comprising:
[0011] Step 1: Separation and acquisition of monitoring audio data;
[0012] Step 2: extract the root mean square energy and MEL spectrum of the monitored wind sound as the time and frequency domain features of the wind sound;
[0013] Step three, introduce the dual attention mechanism, establish the SA-CNN wind sound frequency domain feature extraction deep learning model and the CA-CNN wind sound time domain feature extraction deep learning model, and then form a wind speed measurement deep learning model.
[0014] Further, the step 1 includes:
[0015] (1-1) Obtain the coordinate position of the surveillance camera and register the surveillance camera in the system;
[0016] (1-2) Separate the surveillance video and audio data, and convert the audio into .wav format.
[0017] Furthermore, the step 2 extracts the root mean square energy of the monitored wind sound as the time domain feature of the wind sound, which specifically includes the following steps:
[0018] (2-1) Frame the surveillance audio and perform windowing on the frame results using a Hamming window to smooth the signal while reducing information loss and continuity between adjacent frames;
[0019] (2-2) Perform discrete Fourier transform on each frame to convert the time domain waveform signal into the frequency domain and obtain the short-time energy spectrum , Delay frequency , unit: HZ;
[0020] (2-3) The energy spectrum The linear spectrum generated by the original sound signal is mapped to the Mel nonlinear spectrum based on auditory perception through a set of Mel-scale triangular filters. In Get log Get Log-Mel features:
[0021] (1)
[0022] In the formula, It is Log-Mel spectrum, is the preset value of the cepstrum feature, is the exponent of the cepstral coefficient, is the dimension of the result feature, realizing the extraction of the frequency domain features of the monitored wind sound, time The corresponding wind sound frequency domain characteristics are recorded as ;
[0023] (2-4) With time as the horizontal axis and the amplitude of the sound as the vertical axis, the amplitude envelope of the root mean square energy of the sound is plotted to visualize the change of the sound in the time dimension, which is used as the RMS time domain feature of the wind sound. The corresponding wind sound time domain characteristics are recorded as .
[0024] Furthermore, the establishment of the SA-CNN wind sound frequency domain feature extraction deep learning model in step 3 specifically includes the following steps:
[0025] (3-1) Introduce the spatial attention mechanism, which is combined with the CNN layer to form a frequency domain attention module;
[0026] (3-2) Connect the five frequency domain attention modules to form a SA-CNN network to deepen the network and extract high-dimensional frequency domain features;
[0027] (3-3) The wind sound frequency domain MEL features obtained in step (2-4) are input into the SA-CNN network, and the output feature vector is recorded as: .
[0028] Furthermore, the step 3 of constructing a CA-CNN wind sound time domain feature extraction deep learning model includes the following steps:
[0029] (4-1) Introduce the self-attention mechanism, which is combined with the CNN layer to form a temporal attention module;
[0030] (4-2) Connect the five temporal attention modules to form a CA-CNN network to deepen the network and extract high-dimensional temporal features;
[0031] (4-3) Input the wind sound time domain MEL features obtained in step (2-5) into the CA-CNN network, and the output feature vector is recorded as: .
[0032] Further, the step 3 of forming a wind speed measurement deep learning model includes:
[0033] (5-1) Using a fully connected neural network layer, the time-frequency features of the monitored wind sound extracted in step (3-3) and (4-3) are integrated to obtain a feature vector describing the wind speed;
[0034] (5-2) Use the linear regression layer to calculate the corresponding wind speed value and build a complete deep learning model.
[0035] Another object of the present invention is to provide a deep learning system for wind speed measurement based on surveillance camera audio data, the deep learning system for wind speed measurement based on surveillance camera audio data comprising:
[0036] Preprocessing module, used to separate and obtain monitoring audio data;
[0037] Frequency domain module, used to model and mine frequency domain features;
[0038] Time domain module, used to model and mine time domain features;
[0039] The prediction module is used to realize wind speed prediction based on monitoring audio.
[0040] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the deep learning method for wind speed measurement based on surveillance camera audio data.
[0041] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the deep learning method for wind speed measurement based on surveillance camera audio data.
[0042] Another object of the present invention is to provide an information data processing terminal, which is used to implement the wind speed measurement deep learning system based on surveillance camera audio data.
[0043] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:
[0044] First, the present invention effectively suppresses the interference of external environmental sounds by introducing the Attention mechanism, and deeply mines the wind speed information contained in the monitoring audio data;
[0045] With the generalization ability of deep learning methods, it can achieve rich and accurate measurements in complex monitoring scenarios, with good scalability and replicability;
[0046] The implementation of the method does not require disassembly of the surveillance camera and complex calibration, and the method is easy to deploy and implement;
[0047] The method uses urban surveillance cameras as carriers and can be deployed on existing urban surveillance resources without the need for additional hardware installation, with low maintenance and operation costs.
[0048] Compared with existing wind speed measurement methods, urban monitoring cameras have the advantages of large number, dense distribution, fast transmission and low cost. They solve the current problem of high-resolution measurement of near-ground wind speed and have important scientific significance and practical application value for urban micrometeorological modeling, wind energy utilization and development, and low-altitude economic development.
[0049] Second, from the perspective of urban monitoring, the present invention uses artificial intelligence methods such as deep learning to realize the measurement of surface wind speed. Through the collaborative observation of multiple cameras, high-resolution observation of ground wind speed is achieved, which makes up for the shortcomings of existing wind speed observation methods (including radar and meteorological stations) in terms of time and spatial resolution.
[0050] The present invention solves the following technical problems existing in the prior art and achieves significant technical progress:
[0051] Limitations of traditional wind speed measurement methods: Traditional wind speed measurement usually relies on dedicated anemometers or sensors, which are expensive and complex to install. The measurement accuracy of traditional equipment is affected in severe weather conditions or complex terrain. Traditional methods have slow real-time performance and response speed in environments with frequent wind speed changes. Traditional methods fail to fully utilize the wind sound characteristics in audio data, resulting in a single measurement method. In terms of time domain and frequency domain feature extraction and fusion, there is a lack of effective methods to improve data utilization efficiency.
[0052] The significant technical advances achieved by the present invention are:
[0053] Improve measurement accuracy and real-time performance: By introducing the SA-CNN and CA-CNN deep learning models, the time and frequency domain features in the audio data are fully utilized to improve the accuracy and real-time performance of wind speed measurement. Combining spatial attention and self-attention mechanisms improves the effectiveness of feature extraction and the robustness of the model.
[0054] Improvement of equipment cost and scope of application: This method only relies on existing surveillance camera audio data, reducing the reliance on dedicated measurement equipment and reducing costs. Through intelligent analysis of surveillance audio data, this method can still maintain high-precision measurements in complex environments and severe weather conditions.
[0055] Improved data utilization efficiency: Fully mining and utilizing the wind sound features in audio data solves the problem of low data utilization efficiency of traditional methods. By integrating the time and frequency domain features through the fully connected neural network layer, efficient feature vector extraction of wind speed description is achieved, further improving the accuracy of the model.
[0056] Expansion of technical applicability: This method is applicable to a variety of monitoring scenarios, including urban monitoring, traffic monitoring, and field environment monitoring, and has broad application prospects. By using deep learning and attention mechanism, the intelligent and automated wind speed measurement is realized, which greatly improves the technical level.
[0057] Third, the expected benefits and commercial value of the technical solution of the present invention after transformation are: using the urban monitoring camera as a platform to achieve high-resolution measurement of urban ground wind, the ground wind data produced has important significance and application value for the development of urban wind energy, low-altitude wind navigation, urban transportation, and the development of low-altitude economy. In addition, the invention further expands the functional scope of urban monitoring cameras, and the ground wind observation network composed of monitoring cameras also provides a useful supplement to the existing wind speed observation methods.
[0058] The technical solution of the present invention fills the technical gap in the industry at home and abroad: from the new perspective of urban monitoring cameras, high-resolution perception of ground wind speed is achieved, solving the problem of high-resolution observation of urban ground wind speed in the field of meteorology. After consulting domestic and foreign papers and other research materials, similar work was found. The problem of high-temporal and spatial resolution perception of urban ground wind is solved.
[0059] Fourth, the deep learning method for wind speed measurement based on surveillance camera audio data provided by the present invention extracts the time-frequency domain features of wind sound by separating and acquiring multi-sensor data, and introduces a dual attention mechanism to construct a deep learning model to achieve accurate measurement of wind speed. Compared with traditional methods, this method can more accurately capture the time-frequency features of wind sound, thereby improving the accuracy and reliability of wind speed measurement.
[0060] Among them, this method monitors the separation and acquisition of audio data, converts the audio into .wav format, and extracts the root mean square energy and MEL spectrum of wind sound as time and frequency domain features. By introducing the dual attention mechanism, two deep learning models, SA-CNN and CA-CNN, are established respectively, which realizes the frequency domain and time domain feature extraction of wind sound, thus laying the foundation for wind speed measurement.
[0061] In the SA-CNN model, the spatial attention mechanism is introduced, and the SA-CNN network is formed by connecting five frequency domain attention modules to extract high-dimensional frequency domain features; in the CA-CNN model, the self-attention mechanism is introduced, and the CA-CNN network is formed by connecting five time attention modules to extract high-dimensional time domain features. The application of these two models effectively enhances the ability to extract wind sound features.
[0062] Finally, the method combines the frequency domain and time domain features through a fully connected neural network layer, and uses a linear regression layer to calculate the corresponding wind speed value to build a complete deep learning model. This innovative method has made significant technological progress in the field of wind speed measurement, improved the accuracy and stability of measurement, and has broad application prospects.
[0063] Fifth, in the field of wind speed measurement, traditional methods often rely on direct measurement of physical sensors, which is costly to deploy and maintain, and is restricted by geographical location and environment. With the popularization of surveillance cameras and the development of deep learning technology, using audio data captured by surveillance cameras to indirectly measure wind speed has become a new method. However, how to accurately extract wind speed-related features from complex audio data and build an effective deep learning model to achieve accurate wind speed measurement has become a technical problem that needs to be solved urgently.
[0064] This paper proposes a deep learning method for wind speed measurement based on surveillance camera audio data. This method first realizes the separation and acquisition of surveillance audio data, and innovatively extracts the root mean square energy and MEL spectrum of the monitored wind sound as the time and frequency domain features of the wind sound. Then, by introducing the dual attention mechanism, two deep learning models, SA-CNN and CA-CNN, are constructed to extract high-dimensional frequency domain and time domain features of wind sound, respectively. Finally, the features extracted by the two models are fused, and the corresponding wind speed value is calculated using a linear regression layer to achieve accurate measurement of wind speed.
[0065] The significant technical advances of the present invention are: first, by extracting root mean square energy and MEL spectrum as features, the characteristics of wind sound in the time domain and frequency domain are effectively captured; second, by introducing a dual attention mechanism, the focus of the deep learning model on key features is increased, thereby improving the accuracy of wind speed measurement; third, the audio data of the surveillance camera is used to realize indirect measurement of wind speed, which reduces costs and expands the application scenarios of wind speed measurement; fourth, a complete deep learning model is constructed to realize the direct conversion from audio data to wind speed values, thereby improving the automation and intelligence level of wind speed measurement.
[0066] In summary, the deep learning method for wind speed measurement based on surveillance camera audio data proposed in the present invention solves the problems of high cost and environmental restrictions of traditional wind speed measurement methods through innovative feature extraction methods and deep learning model design, and realizes accurate and efficient measurement of wind speed, bringing new technological breakthroughs and development opportunities to the field of wind speed measurement. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] Figure 1 is a flow chart of a deep learning method for wind speed measurement based on surveillance camera audio data provided by an embodiment of the present invention;
[0068] Figure 2 is a SA-CNN structure diagram provided by an embodiment of the present invention;
[0069] Figure 3 is a CA-CNN structure diagram provided by an embodiment of the present invention;
[0070] Figure 4 is an RMS comparison diagram of audio signals of different wind speeds provided by an embodiment of the present invention;
[0071] Figure 5 is a MEL characteristic diagram of audio with different wind speeds provided by an embodiment of the present invention;
[0072] Figure 6 It is a module diagram of a deep learning system for wind speed measurement based on surveillance camera audio data provided by an embodiment of the present invention;
[0073] Figure 7 It is a schematic diagram of the operating interface of the monitoring audio wind speed measurement method provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0074] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0075] Two specific application examples of the embodiment of the present invention are:
[0076] Example 1: Wind speed measurement in urban road monitoring
[0077] In urban road monitoring, understanding real-time wind speed is helpful for traffic management and emergency response to emergencies. For example, when strong winds lead to road closures and increased traffic accidents, early warning can improve the efficiency and safety of urban traffic management.
[0078] Surveillance camera audio data acquisition: In the urban road monitoring system, surveillance cameras with audio capture function are installed to acquire surveillance video and audio data, and convert the audio data into .wav format.
[0079] Audio feature extraction: The audio data is processed by dividing into frames and performing Hamming windowing. Each frame is subjected to discrete Fourier transform to obtain the short-time energy spectrum. The linear spectrum is mapped to the Mel nonlinear spectrum through the Mel-scale triangular filter bank, and the log is taken to obtain the Log-Mel feature. The amplitude envelope of the sound root mean square energy is extracted as the time domain feature of the wind sound.
[0080] Deep learning model construction: Introduce spatial attention mechanism and self-attention mechanism, build SA-CNN and CA-CNN networks, respectively extract the frequency domain and time domain features of wind sound. Input the extracted time domain and frequency domain features into the fully connected neural network layer, perform feature fusion, and obtain the feature vector describing the wind speed.
[0081] Wind speed value calculation: Use the linear regression layer to calculate the corresponding wind speed value. Feedback the wind speed measurement results to the urban traffic management system to achieve real-time monitoring and early warning.
[0082] Example 2: Wind speed measurement in field environment monitoring
[0083] In field environment monitoring, real-time wind speed measurement is very important for meteorological monitoring and ecological environment protection. For example, in forest fire early warning, wind speed is one of the key factors, and real-time monitoring helps prevent and control fires.
[0084] Surveillance camera audio data acquisition: In a field environment, install a surveillance camera with audio capture function (such as on a surveillance tower in a forest). Acquire surveillance video and audio data, and convert the audio data into .wav format.
[0085] Audio feature extraction: The audio data is processed by dividing into frames and performing Hamming windowing. Each frame is subjected to discrete Fourier transform to obtain the short-time energy spectrum. The linear spectrum is mapped to the Mel nonlinear spectrum through the Mel-scale triangular filter bank, and the log is taken to obtain the Log-Mel feature. The amplitude envelope of the sound root mean square energy is extracted as the time domain feature of the wind sound.
[0086] Deep learning model construction: Introduce spatial attention mechanism and self-attention mechanism, build SA-CNN and CA-CNN networks, respectively extract the frequency domain and time domain features of wind sound. Input the extracted time domain and frequency domain features into the fully connected neural network layer, perform feature fusion, and obtain the feature vector describing the wind speed.
[0087] Wind speed value calculation: Use the linear regression layer to calculate the corresponding wind speed value. The wind speed measurement results are fed back to the ecological environment monitoring system to achieve real-time monitoring and early warning.
[0088] like Figure 1 As shown, the deep learning method for wind speed measurement based on surveillance camera audio data provided by an embodiment of the present invention includes the following steps:
[0089] S101, separation and acquisition of monitoring audio data;
[0090] S102, extracting the root mean square energy and MEL spectrum of the monitored wind sound as the time and frequency domain features of the wind sound;
[0091] S103, introduce the dual attention mechanism, establish the SA-CNN wind sound frequency domain feature extraction deep learning model and the CA-CNN wind sound time domain feature extraction deep learning model, and then form a wind speed measurement deep learning model.
[0092] The first step provided in the embodiment of the present invention includes:
[0093] (1-1) Obtain the coordinate position of the surveillance camera and register the surveillance camera in the system;
[0094] (1-2) Separate the surveillance video and audio data, and convert the audio into .wav format;
[0095] Step 2 provided in the embodiment of the present invention extracts the root mean square energy of the monitored wind sound as the time domain feature of the wind sound, and specifically includes the following steps:
[0096] (2-1) Frame the surveillance audio and perform windowing on the frame results using a Hamming window to smooth the signal while reducing information loss and continuity between adjacent frames;
[0097] (2-2) Perform a Discrete Fourier Transform (DFT) operation on each frame to convert the time domain waveform signal into the frequency domain and obtain the short-time energy spectrum. . Delay frequency , unit: HZ;
[0098] (2-3) The energy spectrum Through a set of Mel-scale triangular filter banks, according to the rule described in equation (1), the linear spectrum generated by the original sound signal is mapped to the Mel nonlinear spectrum based on auditory perception. In. And Get log Get Log-Mel features.
[0099] (1)
[0100] In the formula, It is Log-Mel spectrum, is the preset value of the cepstrum feature, is the exponent of the cepstral coefficient, is the dimension of the result feature. So far, the extraction of wind noise frequency domain features has been realized. The corresponding wind sound frequency domain characteristics are recorded as :
[0101] (2-4) With time as the horizontal axis and the amplitude of the sound as the vertical axis, the amplitude envelope of the root mean square energy of the sound is plotted to visualize the change of the sound in the time dimension, which is used as the RMS time domain feature of the wind sound. The corresponding wind sound time domain characteristics are recorded as .
[0102] like Figure 2 As shown, the step three provided in the embodiment of the present invention provides a SA-CNN wind sound frequency domain feature extraction deep learning model, which specifically includes the following steps:
[0103] (3-1) Construction of frequency domain attention module
[0104] By introducing the spatial attention mechanism, the model can identify the most important information by assigning different weights (or attention scores) to different parts of the input data. The attention mechanism is combined with the CNN layer to form a frequency domain attention module, which is implemented as follows:
[0105] Get the input feature map through the CNN network (Dimensions are , For long, For width, for the channel), The implementation of the layer convolutional neural network is as follows:
[0106] (2)
[0107] in, is the activation function, and are weights and biases respectively, is the convolution operation.
[0108] Extract the channel maximum value of each pixel to form a new feature map (Dimensions are ); Extract the channel average of each pixel to form another feature map (Dimensions are ).
[0109] Stack the two feature maps to get the feature map (Dimensions are );
[0110] Using the convolution operation, Adjust the number of channels to 1, and we get (Dimensions are );
[0111] The Sigmoid function normalizes the weight of each position to between 0 and 1 as the weight (Dimensions are );
[0112] The input feature map Multiply by weight , and obtain a new feature map.
[0113] (3-2) SA-CNN network construction. Figure 2 As shown in the figure, five frequency domain attention modules are connected to form a SA-CNN network to deepen the network and extract high-dimensional frequency domain features for mining wind speed information.
[0114] (3-3) The wind sound frequency domain MEL features obtained in step (2-4) are input into the SA-CNN network, and the output feature vector is recorded as: ;
[0115] like Figure 3 As shown, the step 3 of building a CA-CNN wind sound time domain feature extraction deep learning model provided by the embodiment of the present invention includes the following steps:
[0116] (4-1) Construction of temporal attention module
[0117] The self-attention mechanism is introduced to enable the model to identify the most important information by assigning different weights (or attention scores) to different parts of the input data. The attention mechanism is combined with the CNN layer to form a temporal attention module, which is implemented as follows:
[0118] The input feature map is obtained by CNN network processing (Dimensions are ), p. The implementation of the layer convolutional neural network is as shown in formula (2);
[0119] right Perform global pooling (maximum / average) to obtain a feature vector (Dimensions are );
[0120] Through the full connection operation The dimension is reduced to dimension, and add a ReLU activation layer to get a new feature vector ;
[0121] Using the full connection operation again will Dimension restored to dimension, and obtain a new feature vector (Dimensions are );
[0122] The Sigmoid function will The weight is normalized to between 0 and 1 as the weight of each pixel. (Dimensions are );
[0123] The input feature map Multiply by weight , and obtain a new feature map.
[0124] (4-2) CA-CNN network construction. Figure 3 As shown in the figure, five time attention modules are connected to form a CA-CNN network to deepen the network and extract high-dimensional time domain features for wind speed information extraction and discrimination.
[0125] (4-3) Input the wind sound time domain MEL features obtained in step (2-5) into the CA-CNN network, and the output feature vector is recorded as: ;
[0126] The step 3 of forming a wind speed measurement deep learning model provided by the embodiment of the present invention includes:
[0127] (5-1) Using a fully connected neural network layer, the time-frequency features of the monitored wind sound extracted in step (3-3) and (4-3) are integrated to obtain a feature vector describing the wind speed;
[0128] (5-2) Use the linear regression layer to calculate the corresponding wind speed value and build a complete deep learning model.
[0129] Figure 4 This is the RMS comparison chart of audio at different wind speeds; Figure 5 It is the MEL characteristic diagram of audio with different wind speeds.
[0130] like Figure 6 As shown, the wind speed measurement deep learning system based on surveillance camera audio data provided by an embodiment of the present invention includes:
[0131] Preprocessing module, used to separate and obtain monitoring audio data;
[0132] Frequency domain module, used to model and mine frequency domain features;
[0133] Time domain module, used to model and mine time domain features;
[0134] The prediction module is used to realize wind speed prediction based on monitoring audio.
[0135] The deep learning method for wind speed measurement based on surveillance camera audio data provided by an application embodiment of the present invention is applied to a computer device, wherein the computer device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the deep learning method for wind speed measurement based on surveillance camera audio data.
[0136] The deep learning method for wind speed measurement based on surveillance camera audio data provided by the application embodiment of the present invention is applied to an information data processing terminal, and the information data processing terminal is used to implement the deep learning system for wind speed measurement based on surveillance camera audio data.
[0137] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.
[0138] The operating interface of the monitoring audio wind speed measurement method developed by Nanjing University is shown in the figure. Figure 7 In the method shown, a surveillance camera installed at Nanjing Institute of Technology is taken as an example (coordinates: 119°049′N, 32°25′E). It is assumed that wind speed data needs to be obtained from the camera data:
[0139] The first step is to determine the location of the surveillance camera. Surveillance camera 2 is located at the north gate, and the actual coordinates are (118.6128754, 32.077598);
[0140] The second step is to separate the data sent back by the monitoring to obtain monitoring video and audio data respectively. The core code of the audio separation method is as follows:
[0141] # Capture surveillance video
[0142] my_clip = mp.VideoFileClip(video_path)
[0143] # Extract audio (generate .wav audio file in the video path after running)
[0144] my_clip.audio.write_audiofile(f'{video_path}.wav')
[0145] # Capture video
[0146] cap = cv2.VideoCapture(video_path)
[0147] # Use the cv2.WINDOW_NORMAL flag to create a window that can be closed by the mouse
[0148] cv2.namedWindow('video', cv2.WINDOW_NORMAL)
[0149] while cap.isOpened():
[0150] ret, frame = cap.read()
[0151] cv2.imshow('video', frame)
[0152] key = cv2.waitKey(25)
[0153] if key == 27 or cv2.getWindowProperty('video', cv2.WND_PROP_VISIBLE) < 1:
[0154] break # Release hardware resources
[0155] cap.release()
[0156] cv2.destroyAllWindows()
[0157] The third step is to extract the MEL and RMS changes of the monitored wind sound, and use the fully connected network to aggregate the root mean square energy map and MEL map of the wind sound. The core code for MEL feature extraction is as follows:
[0158] # Load the extracted audio
[0159] y, sr = librosa.load(f'{video_path}.wav', sr=None)
[0160] VOICE_LEN=32000 #Get the length of N_FFT
[0161] print("sr:" , sr)
[0162] N_FFT = getNearestLen(0.25,sr)
[0163] print("N_FFT:" , N_FFT)
[0164] y=normalizeVoiceLen(y,VOICE_LEN)
[0165] print("y.shape:" , y.shape)
[0166] librosa.display.waveshow(y, sr)
[0167] mel_data=librosa.feature.mel(y=y, sr=sr,S=None, n_mfcc=20, dct_type=2, norm='ortho',n_fft=N_FFT,hop_length=int(N_FFT / 4)) #Get wind sound mel features
[0168] rms_data=librosa.feature.rms(y=y, sr=sr,S=None, n_mfcc=20, dct_type=2, norm='ortho',n_fft=N_FFT,hop_length=int(N_FFT / 4)) #Get the rms features of wind sound
[0169] The fourth step is to build a parallel wind speed measurement model to serve the calculation of wind speed. The structure is as follows: Figure 1 The core code of the wind speed measurement model is as follows.
[0170] import torch.nn as nn
[0171] class ChannelAttention(nn.Module): # Temporal feature extraction CA-CNN module
[0172] def __init__(self, channels, reduction=16):
[0173] super(ChannelAttention, self).__init__()
[0174] self.avg_pool = nn.AdaptiveAvgPool2d(1)
[0175] self.fc1 = nn.Conv2d(channels, channels / / reduction, kernel_size=1, stride=1, padding=0)
[0176] self.relu = nn.ReLU(inplace=True)
[0177] self.fc2 = nn.Conv2d(channels / / reduction, channels, kernel_size=1, stride=1, padding=0)
[0178] self.sigmoid = nn.Sigmoid()
[0179] def forward(self, x):
[0180] out = self.avg_pool(x)
[0181] out = self.fc1(out)
[0182] out = self.relu(out)
[0183] out = self.fc2(out)
[0184] out = self.sigmoid(out)
[0185] out = x * out
[0186] return out
[0187] class SpatialAttention(nn.Module): # Frequency domain feature extraction SA-CNN module
[0188] def __init__(self):
[0189] super(SpatialAttention, self).__init__()
[0190] self.conv = nn.Conv2d(2, 1, kernel_size=7, stride=1, padding=3)
[0191] self.sigmoid = nn.Sigmoid()
[0192] def forward(self, x):
[0193] avg_out = torch.mean(x, dim=1, keepdim=True)
[0194] max_out, _ = torch.max(x, dim=1, keepdim=True)
[0195] out = torch.cat([avg_out, max_out], dim=1)
[0196] out = self.conv(out)
[0197] out = self.sigmoid(out)
[0198] out = x * out
[0199] return out
[0200] Step 6: Other cameras perform the above operations in sequence until all surveillance cameras are traversed.
[0201] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.
Claims
1. A deep learning method for wind speed measurement based on surveillance camera audio data, characterized in that: include: Step 1: Separation and acquisition of monitoring audio data; Step 2: extract the root mean square energy and MEL spectrum of the monitored wind sound as the time and frequency domain features of the wind sound; Step 3: Introduce the dual attention mechanism, establish the SA-CNN wind sound frequency domain feature extraction deep learning model and the CA-CNN wind sound time domain feature extraction deep learning model, and then form a wind speed measurement deep learning model; The step 3 of establishing the SA-CNN wind sound frequency domain feature extraction deep learning model specifically includes the following steps: (3-1) Introduce the spatial attention mechanism, which is combined with the CNN layer to form a frequency domain attention module; (3-2) Connect the five frequency domain attention modules to form a SA-CNN network to deepen the network and extract high-dimensional frequency domain features; (3-3) The wind sound frequency domain MEL features obtained in step (2-4) are input into the SA-CNN network, and the output feature vector is recorded as: ; The step 3 of constructing a CA-CNN wind sound time domain feature extraction deep learning model includes the following steps: (4-1) Introduce the self-attention mechanism, which is combined with the CNN layer to form a temporal attention module; (4-2) Connect the five temporal attention modules to form a CA-CNN network to deepen the network and extract high-dimensional temporal features; (4-3) Input the wind sound time domain MEL features obtained in step (2-5) into the CA-CNN network, and the output feature vector is recorded as: ; The step 2 extracts the root mean square energy of the monitored wind sound as the time domain feature of the wind sound, which specifically includes the following steps: (2-1) Frame the surveillance audio and perform windowing on the frame results using a Hamming window to smooth the signal while reducing information loss and continuity between adjacent frames; (2-2) Perform discrete Fourier transform on each frame to convert the time domain waveform signal into the frequency domain and obtain the short-time energy spectrum , Delay frequency , unit: HZ; (2-3) The energy spectrum The linear spectrum generated by the original sound signal is mapped to the Mel nonlinear spectrum based on auditory perception through a set of Mel-scale triangular filters. In Get log Get Log-Mel features: (1) is the dimension of the result feature, realizing the extraction of the frequency domain features of the monitored wind sound, time The corresponding wind sound frequency domain characteristics are recorded as ; (2-4) With time as the horizontal axis and the amplitude of the sound as the vertical axis, the amplitude envelope of the root mean square energy of the sound is plotted to visualize the change of the sound in the time dimension, which is used as the RMS time domain feature of the wind sound. The corresponding wind sound time domain characteristics are recorded as ; The step 3 of forming a wind speed measurement deep learning model includes: (5-1) Using a fully connected neural network layer, the time-frequency features of the monitored wind sound extracted in step (3-3) and (4-3) are integrated to obtain a feature vector describing the wind speed; (5-2) Use the linear regression layer to calculate the corresponding wind speed value and build a complete deep learning model.
2. The deep learning method for wind speed measurement based on surveillance camera audio data according to claim 1 is characterized in that: The step one comprises: (1-1) Obtain the coordinate position of the surveillance camera and register the surveillance camera in the system; (1-2) Separate the surveillance video and audio data, and convert the audio into .wav format.
3. A deep learning system for wind speed measurement based on surveillance camera audio data that implements the deep learning method for wind speed measurement based on surveillance camera audio data as described in any one of claims 1-2, characterized in that: The wind speed measurement deep learning system based on surveillance camera audio data includes: Preprocessing module, used to separate and obtain monitoring audio data; Frequency domain module, used to model and mine frequency domain features; Time domain module, used to model and mine time domain features; The prediction module is used to realize wind speed prediction based on monitoring audio.
4. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the deep learning method for wind speed measurement based on surveillance camera audio data as described in any one of claims 1-2.
5. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the processor executes the steps of the deep learning method for wind speed measurement based on surveillance camera audio data as described in any one of claims 1-2.
6. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the wind speed measurement deep learning system based on surveillance camera audio data as described in claim 3.
