Unmanned aerial vehicle audio identification method, system and device based on lightweight residual network and RK3568, and medium

Abstract

The invention belongs to the technical field of audio signal processing, and particularly discloses an unmanned aerial vehicle audio recognition method, system and device based on a lightweight residual network and RK3568 and a medium. The method disclosed by the invention comprises the following steps: firstly, preprocessing an original audio signal acquired by a microphone by using a CPU (Central Processing Unit) of an RK3568 chip, and extracting a Mel-frequency spectrogram as a time-frequency characteristic; then, constructing a lightweight deep neural network model containing a similar residual network architecture; and then, inputting the Mel spectrogram into a lightweight deep neural network model for analysis, inferring whether the unmanned aerial vehicle exists in the current area or not, and presenting an inference result on a display interface. Through deep optimization and hardware collaborative design of the residual network, the problem that a deep learning model is difficult to deploy at an embedded front end is solved, and high-precision, low-delay and low-power-consumption real-time identification of the audio of the unmanned aerial vehicle is realized. The method can be widely applied to audio recognition of the unmanned aerial vehicle.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of audio signal processing, and specifically relates to a UAV audio recognition method, system, device and medium based on a lightweight residual network and RK3568. BACKGROUND

[0002] With the rapid development and wide application of UAV technology, its role in civil fields such as logistics, mapping, agriculture and military fields such as reconnaissance and surveillance is increasingly prominent, and it also brings new challenges to airspace safety, privacy protection and important area protection. Therefore, fast, accurate and reliable non-cooperative detection and identification of UAV targets have become an urgent technical requirement. As a passive detection method, audio recognition has the advantages of not relying on visible light, no electromagnetic radiation, and all-weather work.

[0003] Traditional UAV audio recognition methods mostly use machine learning algorithms such as support vector machines or Gaussian mixture models. The generalization ability and recognition accuracy of these methods are limited. In recent years, deep neural networks have shown excellent performance in audio classification tasks. However, these models are usually computationally complex and have large parameter quantities, making it difficult to directly deploy them on embedded devices with limited computing resources, storage resources and power consumption, and unable to meet the needs of real-time and on-site deployment.

[0004] Therefore, a solution is needed to deeply adapt high-precision audio recognition algorithms to embedded hardware to achieve efficient and low-power real-time UAV audio monitoring. SUMMARY

[0005] The purpose of the present application is to provide a UAV audio recognition method based on a lightweight residual network and RK3568, which solves the problem of difficulty in deploying deep learning models on embedded frontends by deeply optimizing and co-designing the residual network with hardware, and realizes high-precision, low-delay and low-power real-time recognition of UAV audio; The second purpose of the present application is to provide a UAV audio recognition system for executing a UAV audio recognition method based on a lightweight residual network and RK3568; The third purpose of the present application is to provide a terminal device that can implement a UAV audio recognition method based on a lightweight residual network and RK3568 when executing its own program; The fourth purpose of the present application is to provide a computer-readable storage medium for storing a corresponding computer program of a UAV audio recognition method based on a lightweight residual network and RK3568.

[0006] To achieve the above purposes, the technical solutions adopted by the present application are as follows: The application discloses an unmanned aerial vehicle audio recognition method based on a lightweight residual network and an RK3568. S1, using a CPU of an RK3568 chip to pre-process original audio signals collected by a microphone, and extracting a mel spectrum graph as a time-frequency feature; S2, constructing a lightweight deep neural network model containing a residual network architecture; S3, inputting the mel spectrum graph into the lightweight deep neural network model for analysis to infer whether an unmanned aerial vehicle exists in a current area; S4, presenting an inference result on a display interface.

[0007] As a limitation, the pre-processing of the original audio signals comprises: performing pre-emphasis, framing and windowing processing on the collected original audio signals in sequence; calculating a short-time Fourier transform for each frame of signals to obtain a spectrum; filtering the spectrum through a mel filter bank, and taking a logarithm of the energy of each channel to generate a mel spectrum graph.

[0008] As a second limitation, the lightweight deep neural network model is deployed on an NPU of the RK3568 chip; the CPU and the NPU of the RK3568 chip exchange memory of input and output data through an RKNN runtime library.

[0009] As a further limitation, the residual network architecture comprises an initial feature extraction module for performing preliminary feature extraction on an input mel spectrum graph, four improved residual blocks connected in series, and a classification output module; the initial feature extraction module is composed of a 3*3 convolution layer, a batch normalization layer and a ReLU activation function connected in series; each improved residual block contains a multi-scale feature extraction module and a two-stage channel attention weighting mechanism, and realizes residual learning through identity mapping or projection connection; the classification output module is used for performing global average pooling on the output of the last residual block, accessing a fully connected layer, and outputting an inference result of whether an unmanned aerial vehicle exists.

[0010] As a further limitation, the multi-scale feature extraction module is arranged in a main path of the residual block and comprises four parallel branches: a first branch: a single 3*3 convolution layer; a second branch: a convolution sequence composed of two 3*3 convolution layers connected in series; a third branch: a 1*7 convolution layer; a fourth branch: a 7*1 convolution layer; The first to fourth branch output feature maps are spliced in the channel dimension to form fused features.

[0011] Preferably, the two-stage channel attention weighting mechanism comprises a first-stage channel attention module and a second-stage channel attention module. The first-stage channel attention module is a CAM applied to each parallel branch output end of the multi-scale feature extraction module, and is used for independently recalibrating the channels of each branch feature. The second-stage channel attention module is an ECA applied to the feature map after splicing the plurality of branch features, and is used for depth fusion of the overall feature.

[0012] An unmanned aerial vehicle audio recognition system for implementing the unmanned aerial vehicle audio recognition method based on a lightweight residual network and an RK3568, characterized by being integrated on an RK3568 development board and comprising an RK3568 chip, a memory, an audio acquisition module, a preprocessing module, a neural network inference module, and a Qt graphical display module. The RK3568 chip, as the core of the system, comprises a CPU core and an NPU core. The memory is used to store an operating system, application programs, and a neural network model. The audio acquisition module controls a microphone to acquire an original audio signal in an environment and outputs the original audio signal to the preprocessing module. The preprocessing module runs on the CPU of the RK3568 chip, pre-processes the original audio signal, generates a mel spectrogram, and outputs the mel spectrogram to the neural network inference module and the Qt graphical display module. The neural network inference module is driven by the NPU of the RK3568 chip, inputs the mel spectrogram into a lightweight deep neural network model, outputs an unmanned aerial vehicle inference result, and outputs the inference result to the Qt graphical display module. The Qt graphical display module runs on the CPU of the RK3568 chip, receives the mel spectrogram and the inference result, draws an audio waveform in real time, and displays the inference result.

[0013] As a limitation, the running code on the RK3568 development board is written in C++ language. The lightweight deep neural network model is trained and verified on a server side using Python language and PyTorch framework.

[0014] A terminal device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the unmanned aerial vehicle audio recognition method based on a lightweight residual network and an RK3568 when executing the computer program.

[0015] A computer readable storage medium, the computer readable storage medium has a computer program stored therein, the computer program is executed by a processor to implement the unmanned aerial vehicle audio recognition method based on the lightweight residual network and RK3568.

[0016] The application has the technical progress compared with the prior art, which lies in that: (1) The method of the application cooperates and optimizes the lightweight residual network and the RK3568 chip, adopts the mel spectrum graph to compress the time-frequency features, combines the deep neural network of the residual architecture to improve the feature extraction capability, so that the model significantly reduces the calculation complexity while maintaining high precision, and the CPU hardware acceleration of the RK3568 further guarantees the real-time performance, meets the on-site monitoring requirements of low power consumption and low delay, realizes the embedded efficient deployment of the unmanned aerial vehicle audio recognition, and finally forms a closed-loop solution from signal acquisition, intelligent analysis to result visualization, effectively solving the contradiction between the poor generalization of the traditional method and the difficulty of landing of the deep model; (2) In the preprocessing process of the method of the application, the high-frequency attenuation is compensated by pre-enhancement, the spectrum leakage is suppressed by framing and windowing, the time-frequency dynamics are accurately captured by short-time Fourier transform, the mel filter bank simulates the human auditory characteristics, and the energy distribution is combined with logarithmic compression, so that the generated mel spectrum graph not only retains the key frequency domain features, but also reduces the data dimension, significantly improves the feature extraction quality, and provides high-discrimination and low-redundancy input features for the subsequent lightweight residual network, so that more efficient unmanned aerial vehicle voiceprint recognition is realized on the embedded hardware; (3) The lightweight deep neural network model of the application is deployed on the NPU of the RK3568 chip, and the efficient cooperation of CPU and NPU is realized through the RKNN runtime library, the NPU is specially designed for neural network operation, its parallel computing capability significantly accelerates model inference, and the optimized memory management mechanism of the RKNN library reduces the data transmission overhead between CPU and NPU, so as to reduce the overall power consumption while guaranteeing the real-time performance, which can fully exert the advantages of heterogeneous computing hardware, this deployment scheme of soft and hard depth adaptation not only solves the problem of insufficient CPU computing resources, but also avoids the complexity and cost of external accelerator, and finally realizes the high-performance and low-delay operation of the unmanned aerial vehicle audio recognition system on the embedded device; (4) The method of the present application uses the cooperative design of multi-scale feature extraction and double-stage attention mechanism, four parallel branches focus on local details (3*3 convolution), deep semantics (convolution sequence), long time-frequency domain (1*7 convolution) and spatial correlation (7*1 convolution), and the CAM module recalibrates the independent channels of the branch output to strengthen the key frequency band features, and the ECA module eliminates the redundancy between multi-scale features through global fusion, finally makes the residual network have stronger feature discrimination and robustness while keeping lightweight, significantly improves the capture ability of the model to the drone voiceprint features; (5) The system of the present application adopts a C++ / Python heterogeneous programming architecture, the lightweight model trained by Python is deployed on NPU after optimization by RKNN tool chain, which fully utilizes its parallel computing advantage; the system bottom code written in C++ precisely controls microphone acquisition, real-time preprocessing and Qt graphic rendering, ensures low-delay interaction, this kind of software and hardware deep combination scheme not only retains the flexibility of PyTorch in the model training stage, but also realizes efficient resource scheduling of embedded system through C++, finally forms a complete closed loop from acoustic signal acquisition, intelligent analysis to visual feedback on RK3568 development board, meets the strict requirements of real-time performance, reliability and low power consumption in the unmanned aerial vehicle monitoring scene.

[0017] The present application belongs to the technical field of audio signal processing, through deep optimization and hardware cooperative design of residual network, the problem of difficulty in deploying deep learning model in embedded front end is solved, and high-precision, low-delay and low-power real-time identification of unmanned aerial vehicle audio is realized. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, which together with the embodiments of the present application, are used to explain the present application, and do not constitute a limitation of the present application.

[0019] In the drawings: Figure 1 is the process flowchart of embodiment 1 of the present application; Figure 2 is the lightweight deep neural network model structure schematic diagram of embodiment 1 of the present application; Figure 3 is the improved residual block structure schematic diagram of embodiment 1 of the present application; Figure 4 is the system overall architecture block diagram of embodiment 2 of the present application. DETAILED DESCRIPTION

[0020] The preferred embodiments of the present application will be described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and do not limit the present application.

[0021] Embodiment 1 The embodiment is a method for recognizing the sound of a UAV based on a lightweight residual network and RK3568, as shown in the figure, comprising the following steps in sequence: Figure 1 S1, using the CPU of the RK3568 chip to pre-process the original audio signal collected by the microphone, and extracting the mel spectrum as the time-frequency feature.

[0022] The pre-processing of the original audio signal includes: The collected original audio signal is sequentially pre-processed, framed and windowed; the short-time Fourier transform is calculated for each frame of signal to obtain the frequency spectrum; The frequency spectrum is filtered by a mel filter bank, and the energy of each channel is logarithmically transformed to generate a two-dimensional mel spectrum as an input feature, which can effectively represent the periodic modulation acoustic pattern generated by the rotation of the UAV propeller.

[0023] S2, constructing a lightweight deep neural network model containing a residual network architecture.

[0024] The lightweight deep neural network model is deployed on the NPU of the RK3568 chip. The CPU and NPU work together: the CPU is responsible for audio acquisition, pre-processing and result output tasks; the NPU independently undertakes neural network inference calculation, fully exerting its special AI acceleration capability. The CPU and NPU share the memory buffer through the RKNN runtime library, achieving efficient data transfer and avoiding performance loss caused by frequent copying.

[0025] As shown in the figure, the lightweight deep neural network model includes an initial feature extraction module for preliminary feature extraction of the input mel spectrum, four improved residual blocks connected in series, and a classification output module. Figure 2

[0026] Among them, the initial feature extraction module is composed of a 3x3 convolution layer, a batch normalization layer and a ReLU activation function connected in series, which is used to extract basic features with low overhead; Each of the improved residual blocks contains a multi-scale feature extraction module and a two-stage channel attention weighting mechanism, and realizes residual learning through identity mapping or projection connection.

[0027] The classification output module is used to perform global average pooling on the output of the last residual block, access the fully connected layer, and output the inference result of whether the UAV exists.

[0028] As shown in the figure, the lightweight deep neural network model includes an initial feature extraction module for preliminary feature extraction of the input mel spectrum, four improved residual blocks connected in series, and a classification output module. Figure 3 ​​As shown, the multi-scale feature extraction module is arranged in the main path of the residual block, and the input features are synchronously processed by four parallel convolution branches: the first branch adopts a single 3x3 convolution to capture local detailed features at low cost; the second branch is composed of two cascaded 3x3 convolutions to expand the receptive field and extract deeper nonlinear features with little increase in parameters; the third branch and the fourth branch respectively adopt asymmetric convolution pairs of 1x7 and 7x1, which can decompose the standard 7x7 large convolution, thus effectively modeling the long-range context dependence in the frequency dimension and the time dimension at a very low parameter cost, especially suitable for capturing the long-range stripe pattern of the UAV sound in the time-frequency spectrum. Finally, the output feature maps of the four branches are spliced in the channel dimension to realize the fusion of multi-scale features, enhancing the model's representation ability for complex and variable acoustic patterns.

[0029] On the basis of the multi-scale feature extraction structure, the improved residual block is also configured with a two-level channel attention weighting mechanism for realizing hierarchical channel reweighting of branch-level and fusion-level features. The first-level channel attention module is a CAM, which is applied to the output end of each parallel branch of the multi-scale feature extraction module, for independently reweighting the channel of each branch feature. The second-level channel attention module is an ECA, which is applied to the feature map after splicing the multi-branch features, for deep fusion of the overall features, realizing global channel weight distribution and strengthening key feature response.

[0030] After the model is trained on the server side, the RKNN-Toolkit2 tool is used for model conversion and INT8 quantization. By performing calibration on a representative audio dataset, an inference model adapted to the NPU instruction set of the RK3568 chip is generated, which significantly reduces the model size and computational overhead and improves the edge running efficiency.

[0031] S3, input the mel spectrum into the lightweight deep neural network model for analysis to infer whether there is a UAV in the current area.

[0032] S4, present the inference result on the display interface.

[0033] The original audio signals in this embodiment are 1400 wav format recordings, with a time length of 4 seconds, mainly composed of 500 UAV sounds and 900 non-UAV sounds selected from the UrbanSound8K dataset.

[0034] By comparing this embodiment with the method using other models, the results are shown in Table 1: Table 1 Comparison results of different model methods As can be seen from the above table, the improved model achieves high accuracy while keeping a small parameter size, proving the effectiveness of the improved method.

[0035] In summary, by depth optimizing the residual network and co-designing with hardware, the embodiment solves the problem of difficulty in deploying deep learning models on embedded frontends, and realizes high-precision, low-delay and low-power real-time recognition of UAV audio.

[0036] Embodiment 2 The embodiment is a UAV audio recognition system for implementing the UAV audio recognition method based on a lightweight residual network and RK3568 in embodiment 1, as shown in Figure 4 The UAV audio recognition system is integrated on an RK3568 development board, and includes an RK3568 chip, a memory, an audio acquisition module, a preprocessing module, a neural network inference module and a Qt graphical display module.

[0037] The RK3568 chip, as the core master unit of the hardware system, integrates a quad-core ARM Cortex-A55 CPU and a dual-core NPU.

[0038] The memory is used to store an operating system, application programs and neural network models. The on-board eMMC memory is used to store neural network model weights and system programs, and the DDR memory provides runtime data caching.

[0039] The audio acquisition module is externally connected to a microphone array for acquiring original audio signals in the environment and outputting the original audio signals to the preprocessing module.

[0040] The preprocessing module runs on the CPU of the RK3568 chip, pre-processes the original audio signals, generates a mel spectrum graph, and outputs the mel spectrum graph to the neural network inference module and the Qt graphical display module.

[0041] The neural network inference module is driven by the NPU of the RK3568 chip, and the mel spectrum graph is sent to the NPU where the lightweight deep neural network model is located through the API interface provided by RKNN, and inference calculation is triggered, and the inference result of the UAV is output, and the inference result is output to the Qt graphical display module; The Qt graphical display module runs on the CPU of the RK3568 chip, receives the mel spectrum graph and the inference result, draws the audio waveform in real time and displays the inference result.

[0042] As a limitation, the running code on the RK3568 development board is written in C++ language; The system runs on the Ubuntu 18.04 operating system based on the Linux kernel. The development environment includes Python, the PyTorch deep learning framework, NumPy, Librosa and other audio processing libraries. The model deployment stage uses the C++ language to develop the runtime program, relies on the RKNN-Toolkit2 model conversion tool, the RKNNAPI interface library and the ALSA audio driver library to ensure full-link compatibility from training to inference.

[0043] The lightweight deep neural network model is trained on a server equipped with a GPU using the Python language and the PyTorch framework. The model is trained end-to-end using the cross-entropy loss function and the Adam optimizer.

[0044] In use, after the system is powered on and started, the application automatically loads the model weights into the NPU and initializes the modules. Then, the microphone continuously collects environmental audio, and the preprocessing module generates mel spectrum picture segments in real time. Next, the neural network inference module sends the feature data to the NPU for model inference, and the NPU outputs the detection results after completing the calculation, and the Qt graphical display module updates the interface state.

[0045] Embodiment 3 The embodiment is a terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it is used to implement the unmanned aerial vehicle audio recognition method based on the lightweight residual network and the RK3568 of embodiment 1.

[0046] Embodiment 4 The embodiment is a computer-readable storage medium, and the computer-readable storage medium in the embodiment stores a computer program. When the computer program is executed by a processor, it is used to implement the unmanned aerial vehicle audio recognition method based on the lightweight residual network and the RK3568 of embodiment 1.

[0047] Among them, the computer readable storage medium can be a computer storage medium, but also a communication medium. Communication medium includes any medium that facilitates the transfer of computer program from one place to another. The computer storage medium can be any available medium that can be accessed by a general or special purpose computer. For example, the computer readable storage medium is coupled to the processor, so that the processor can read information from the computer readable storage medium, and can write information to the computer readable storage medium. Of course, the computer readable storage medium can also be a component of the processor. The processor and the computer readable storage medium can be located in an application specific integrated circuit (ASIC). In addition, the ASIC can be located in the user equipment. Of course, the processor and the computer readable storage medium can also exist as discrete components in the communication device. Specifically, the computer readable storage medium can be realized by any type of volatile or nonvolatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk, etc. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

Claims

1. A method for UAV audio recognition based on lightweight residual networks and RK3568, characterized in that, This includes the following steps performed sequentially: S1. The CPU using the RK3568 chip preprocesses the raw audio signal acquired by the microphone and extracts the Mel spectrogram as time-frequency features. S2. Construct a lightweight deep neural network model that includes a residual network-like architecture; S3. Input the Mel spectrum into a lightweight deep neural network model for analysis to infer whether there are drones in the current area; S4. Present the reasoning results on the display interface.

2. The UAV audio recognition method based on lightweight residual network and RK3568 according to claim 1, characterized in that, The preprocessing of the original audio signal includes: The acquired raw audio signal is sequentially pre-emphasized, framed, and windowed; the short-time Fourier transform of each frame is calculated to obtain the spectrum; The spectrum is filtered through a Mel filter bank, and the logarithm of the energy of each channel is taken to generate a Mel spectrum.

3. A method for UAV audio recognition based on a lightweight residual network and RK3568 according to claim 1 or 2, characterized in that, The lightweight deep neural network model is deployed on the NPU of the RK3568 chip; On the RK3568 chip, the CPU and NPU exchange input and output data in memory through the RKNN runtime library.

4. The UAV audio recognition method based on lightweight residual network and RK3568 according to claim 3, characterized in that, The residual network architecture includes: an initial feature extraction module that performs preliminary feature extraction on the input Mel spectrogram, four cascaded improved residual blocks, and a classification output module; The initial feature extraction module consists of a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function connected in series. Each of the improved residual blocks includes a multi-scale feature extraction module and a two-level channel attention weighting mechanism, and residual learning is achieved through identity mapping or projection connection; The classification output module is used to perform global average pooling on the output of the last residual block, connect it to the fully connected layer, and output the inference result of whether the drone exists or not.

5. The UAV audio recognition method based on lightweight residual network and RK3568 according to claim 4, characterized in that, The multi-scale feature extraction module is located in the main path of the residual block and includes four parallel branches: First branch: a single 3×3 convolutional layer; The second branch consists of a convolutional sequence of two cascaded 3×3 convolutional layers; Third branch: 1×7 convolutional layer; Fourth branch: 7×1 convolutional layer; The output feature maps of the first to fourth branches are spliced ​​together along the channel dimension to form a fused feature.

6. The UAV audio recognition method based on lightweight residual network and RK3568 according to claim 5, characterized in that, The dual-channel attention weighting mechanism includes a first-level channel attention module and a second-level channel attention module; The first-level channel attention module is CAM, which is applied to the output of each parallel branch of the multi-scale feature extraction module to perform independent channel recalibration on the features of each branch. The second-level channel attention module, ECA, is applied to the feature map after multiple branch features are concatenated, and is used to perform deep fusion of the overall features.

7. A drone audio recognition system, used to implement the drone audio recognition method based on a lightweight residual network and RK3568 as described in any one of claims 1 to 6, characterized in that, The UAV audio recognition system is integrated on the RK3568 development board and includes: an RK3568 chip, a memory, an audio acquisition module, a preprocessing module, a neural network inference module, and a Qt graphics display module; The RK3568 chip serves as the system core, including the CPU core and NPU core. Memory, used to store the operating system, applications, and neural network models; The audio acquisition module controls the microphone to acquire raw audio signals from the environment and outputs the raw audio signals to the preprocessing module; The preprocessing module, running on the CPU of the RK3568 chip, preprocesses the raw audio signal, generates a Mel spectrogram, and outputs the Mel spectrogram to the neural network inference module and the Qt graphics display module. The neural network inference module, driven by the NPU of the RK3568 chip, inputs the Mel spectrogram into the lightweight deep neural network model, outputs the UAV inference results, and outputs the inference results to the Qt graphics display module; The Qt graphics display module runs on the CPU of the RK3568 chip, receives Mel spectrograms and inference results, draws audio waveforms in real time, and displays the inference results.

8. The UAV audio recognition system according to claim 7, characterized in that, The code running on the RK3568 development board is written in C++. The lightweight deep neural network model was trained and validated on the server side using the Python language and the PyTorch framework.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a drone audio recognition method based on a lightweight residual network and RK3568 as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, is used to implement the UAV audio recognition method based on a lightweight residual network and RK3568 as described in any one of claims 1 to 6.

Images