Lightweight multi-spectrum passive acoustic identification edge inference method and system

Abstract

The application discloses a lightweight multi-spectrum passive acoustic identification edge inference method and system, first, the original passive acoustic signal is cut into fixed time length segments, a shared Mel filter bank is used to generate a log-Mel spectrogram and an Fbank energy diagram, and after normalization, the two are spliced into a fixed size double-channel multi-spectrum acoustic input tensor; second, a pre-main convolution compression module is arranged before a RepViT lightweight main stem, and a regular convolution operator is used to perform front-end compression and local feature aggregation on the fixed size double-channel multi-spectrum acoustic input tensor; third, after training, the model is re-parameterized, and the multi-branch structure during training is fused into a single-path convolution structure during inference; finally, a consistent deployment process is constructed through ONNX, TVM Relay, Vitis AI and DPU target architecture, operators supported by DPU can be unloaded to FPGA logic, and operators not supported are scheduled to ARM CPU for execution, thereby realizing heterogeneous edge inference.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a lightweight multispectral passive acoustic recognition edge reasoning method and system. Background Technology

[0002] Passive acoustic sensing is a crucial capability for unmanned underwater systems to perform maritime surveillance, target situational awareness, marine environmental monitoring, and long-endurance stealth missions. Compared to active sonar, passive sonar does not actively emit sound waves, resulting in better platform concealment and making it more suitable for extended underwater operations. However, ship-radiated noise exhibits strong non-stationarity, and the acquired underwater acoustic signals are easily affected by marine environmental noise, multipath propagation, platform motion, changes in acoustic channels, and variations in target operating conditions, leading to significant uncertainties in the engineering deployment of passive acoustic target identification tasks.

[0003] In recent years, deep learning models such as convolutional neural networks, recurrent neural networks, and Transformers have significantly improved the accuracy of underwater acoustic target recognition. However, these models are mostly evaluated offline on servers, and their computational load, parameter count, operator structure, and memory access patterns often do not fully consider the deployment constraints of edge devices. In unmanned underwater platforms, buoys, small unmanned surface vessels, or embedded acoustic sensing nodes, processor power consumption, heat dissipation space, onboard storage, and real-time performance are all strictly limited. Directly deploying complex deep networks can easily lead to excessive latency, excessive energy consumption, or operator incompatibility.

[0004] Therefore, the existing technology has at least the following shortcomings: (1) Single spectrum input is insufficient. When only log-Mel spectrum or energy characteristics of a single filter bank are used, it is difficult to simultaneously and fully represent the narrowband harmonic structure, broadband energy envelope and local energy disturbance in ship radiated noise.

[0005] (2) Multi-branch or multi-domain fusion structures are complex to deploy. Some multi-view methods require multiple input branches, dynamic alignment, cross-branch attention, or irregular tensor transformations, which can easily lead to DPUs not supporting operators and dynamic memory allocation, making them unsuitable for FPGA edge deployment.

[0006] (3) Lightweight models are disconnected from hardware execution links. Common lightweight networks are only validated on GPUs / CPUs for speed, lacking a consistent process from feature construction, training, calibration, quantization, compilation to on-board execution, which makes it impossible to reliably transfer algorithm accuracy to physical edge platforms.

[0007] (4) Operators that can run on a GPU in a conventional neural network are not necessarily suitable for an FPGA DPU. If the model contains dynamic shapes, runtime branches, complex rearrangements, or unsupported operators, it will cause a lot of CPU rollback, thereby reducing real-time performance and energy efficiency. Summary of the Invention

[0008] To overcome the shortcomings of existing technologies, this invention provides a lightweight multispectral passive acoustic recognition edge inference method and system. First, the original passive acoustic signal is segmented into fixed-duration segments, and a shared Mel filter bank is used to generate log-Mel spectra and Fbank energy maps respectively. These are then normalized and concatenated into a fixed-size dual-channel multispectral acoustic input tensor. Second, a pre-backbone convolutional compression module is set before the RepViT lightweight backbone to perform front-end compression and local feature aggregation on the fixed-size dual-channel multispectral acoustic input tensor using regular convolution operators. Third, after training, the model undergoes structural reparameterization, fusing the multi-branch structure during training into a single-path convolutional structure for inference. Finally, a consistent deployment process is built using ONNX, TVM Relay, Vitis AI, and DPU target architectures. Operators supported by the DPU are offloaded to FPGA logic, while unsupported operators are scheduled to be executed on the ARM CPU, achieving heterogeneous edge inference.

[0009] The technical solution adopted by this invention to solve its technical problem is as follows: Step 1: Acquire underwater passive acoustic signals, resample and segment the passive acoustic signals to obtain multiple acoustic segments of fixed duration; For each acoustic segment, the short-time power spectrum is calculated, and the log-Mel spectrum and Fbank energy map are generated using the same Mel filter bank. The log-Mel spectrum and Fbank energy map are normalized and their sizes are unified. The two are then stacked along the channel dimension to form a fixed-size dual-channel multispectral acoustic input tensor. Step 2: Feed the fixed-size dual-channel multispectral acoustic input tensor into a lightweight recognition network model that includes a pre-backbone convolutional compression module, a RepViT lightweight backbone, a global average pooling layer, and a linear classification layer to obtain the underwater target category probability or classification logits. Before deployment, the trained lightweight recognition network model is reparameterized; in this process, the pre-backbone convolutional compression module and the fusionable convolutional layers and batch normalization layers in the RepViT lightweight backbone are fused into equivalent convolutional layers, and the network structure during the training phase is converted into a single-path rule reasoning computation graph suitable for DPU execution. Step 3: Export the trained lightweight recognition network model as an ONNX model with a fixed input shape. Then, input the ONNX model into the TVM Relay for graph optimization and divide the subgraph according to the operator support range of the target DPU to obtain the DPU subgraph executed by the DPU and the CPU subgraph executed by the ARM CPU. Step 4: Use the fixed-size dual-channel multispectral acoustic input tensor obtained in Step 1 to construct a calibration set, perform INT8 quantization on the lightweight recognition network model, and compile it with Vitis AI to generate an executable file adapted to the target FPGA DPU; Step 5: Load the executable file and TVM Runtime graph executor generated by Vitis AI on the FPGA edge platform, perform DPU and ARM CPU collaborative inference on the fixed-size dual-channel multispectral acoustic input tensor obtained in Step 1, and output the underwater target category recognition result.

[0010] Preferably, step 1 specifically comprises: Let any fixed-duration acoustic segment obtained from underwater passive acoustic signal segmentation in step 1 be denoted as . Underwater passive acoustic signals The sampling rate is collected and divided into segments of length [missing information]. Non-overlapping segments; perform a short-time Fourier transform on each segment to obtain the short-time power spectrum. ,in Indicates frequency index, Indicates the time frame index; Based on short-time power spectrum Two spectral channels are constructed using the same Mel filter bank, and the two spectral channels correspond to different energy compression methods. The first spectral channel is the log-Mel spectrum. Through the Mel filter bank After projecting the short-time power spectrum onto the Mel frequency space and performing logarithmic compression, the calculation formula is as follows:

[0011] in, Indicates the Mel filter index. Small positive constants Indicated in frequency index Summing is performed on the above to obtain the first... The energy of the filter bank corresponding to each Mel filter; The second spectral channel is the Fbank energy map. It uses the same Mel filter bank and time framing as the first spectral channel, but employs square root compression, calculated as follows:

[0012] To each and Perform min-max normalization to map its numerical range to Then normalize the and Scaled to a uniform resolution; finally stacked along the channel dimension to form a fixed-size dual-channel multispectral acoustic input tensor. .

[0013] Preferably, step 2 specifically comprises: The lightweight recognition network model includes a pre-backbone convolutional compression module, a RepViT lightweight backbone, a global average pooling layer, and a linear classification layer. The pre-backbone convolutional compression module is located between the dual-channel multispectral acoustic input tensor and the RepViT lightweight backbone, and includes a first convolutional layer and a second convolutional layer; the stride of the first convolutional layer is 2, and the stride of the second convolutional layer is 1; both convolutional layers are set to padding of 1, and are sequentially connected to batch normalized (BN) and ReLU activation functions; The RepViT lightweight backbone is used to extract deep semantic features; the output of the RepViT lightweight backbone is then processed by a global average pooling layer to generate classification logits by a linear classification layer. Before deployment, the lightweight recognition network model, after training, merges the pre-backbone convolutional compression module and the convolutional layers adjacent to and fusionable with the BN layer in the RepViT lightweight backbone with the corresponding BN layer to form an equivalent convolutional layer for the inference stage; for weights of Bias is The convolutional layer, and the parameters are , , , and BN layer, fusion weights and fusion bias They are respectively:

[0014]

[0015] in, and These represent the kernel weights and biases of the convolutional layers to be merged, respectively; if the convolutional layers to be merged do not contain biases, then... Set to 0; and These represent the scaling and translation parameters of the corresponding BN layer, respectively. and These represent the mean and variance obtained during the training phase of the BN layer, respectively. is a small positive constant used for numerical stability.

[0016] Preferably, the unified resolution is .

[0017] Preferably, the fixed-size dual-channel multispectral acoustic input tensor .

[0018] Preferably, both the first convolutional layer and the second convolutional layer are... convolution.

[0019] Preferably, the categories of the classified logits include cargo ships, tugboats, passenger ships, and tankers.

[0020] A lightweight multispectral passive acoustic recognition edge reasoning system includes: Acoustic Acquisition and Preprocessing Unit: Used to receive passive acoustic signals acquired by hydrophones or underwater acoustic sensors, and to perform resampling, fixed-length segmentation and short-time power spectrum calculation; Multispectral feature construction unit: used to generate log-Mel spectra and Fbank energy maps based on the same Mel filter bank, complete normalization, size unification and channel stacking, and output a fixed-size dual-channel multispectral acoustic input tensor; Lightweight recognition network unit: includes a pre-backbone convolutional compression module, a RepViT lightweight backbone, a global average pooling layer, and a linear classification layer, used to classify multispectral features; Structure fusion and model conversion unit: used to perform structure reparameterization, ONNX export, graph optimization and operator compatibility checks on the trained lightweight recognition network model; Quantization compilation unit: used to perform INT8 calibration quantization based on real feature tensors and call the Vitis AI toolchain to generate compilation results adapted to the target DPU; FPGA heterogeneous inference unit: Deployed on the FPGA edge platform, it includes a DPU accelerator, ARM CPU, DDR memory and runtime graph executor, used to execute DPU subgraph and CPU subgraph; The result output unit is used to output the target category, confidence level, inference delay, energy consumption or other operating status information, and to provide the identification results to the mission planning or alarm module of the unmanned underwater platform.

[0021] The beneficial effects of this invention are as follows: (1) Improve the integrity of acoustic characterization. This invention integrates the log-Mel spectrum and Fbank energy map in the same Mel frequency space, which not only preserves the smooth global spectral envelope, but also preserves the local energy changes, and can better characterize the broadband components, harmonic components and local disturbances in the radiated noise of underwater targets.

[0022] (2) Improve edge deployability. This invention compresses the multispectral representation into a fixed-size dual-channel multispectral acoustic input tensor, avoiding complex multi-branch input pipelines and dynamic tensor operations, making the input features naturally suitable for DPU static shape inference.

[0023] (3) Improve network inference efficiency. This invention sets up a pre-stem convolutional compression module before the RepViT lightweight backbone to perform local aggregation and size compression on time-frequency input, thereby reducing the subsequent computational burden while ensuring accuracy.

[0024] (4) Improve hardware compatibility. This invention integrates the multi-branch structure in the training stage into an equivalent single-path convolutional structure in the inference stage through structural reparameterization, and further forms a single-path rule inference computation graph suitable for DPU execution; through the TVM / Vitis AI process, compatible operators are pushed down to the DPU subgraph for execution, reducing the risk of board-side running failure and performance instability.

[0025] (5) Reduce the loss of quantization deployment accuracy. This invention uses real multispectral feature tensors for INT8 calibration and maintains consistency in the feature extraction process during training, calibration and board-side execution, which helps to stably migrate server-side accuracy to the FPGA side.

[0026] (6) It has clear real-time performance and low power consumption. In the embodiment, the overall accuracy of the FP32 server reaches 95.8%, the overall accuracy of the ZCU102 FPGA INT8 board reaches 91.1%, the end-to-end latency is 8.82 ms, the throughput is 136 samples / s, the average power consumption is 8.3 W, and the energy consumption per sample is 0.061 J. Attached Figure Description

[0027] Figure 1 This is a diagram of the lightweight multispectral passive acoustic recognition edge inference system architecture of the present invention. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0029] 1. Multispectral passive acoustic recognition; This embodiment uses ship radiated noise collected by an unmanned underwater system as input. The system uses... The sampling rate is used to acquire the audio stream and divide it into continuous segments. Non-overlapping audio segments. Perform a short-time Fourier transform on each audio segment to obtain the short-time power spectrum. .

[0030] For each segment, use a shared Mel filter bank. The short-time power spectrum is mapped to the Mel frequency space. One approach involves generating a log-Mel spectrum by taking the logarithm of the filter bank energy. Another approach generates the Fbank energy map through square root compression. Both features are subjected to min-max normalization and then uniformly scaled to [a specific scale]. Then splice them together in the channel dimension. Fixed-size dual-channel multispectral acoustic input tensor.

[0031] The fixed-size dual-channel multispectral acoustic input tensor first enters the pre-backbone convolution compression module. The convolutional layer employs a stride of 2 and padding of 1 to reduce the tensor space size and extract local spectral texture; the second... The convolutional layers employ a stride of 1 and padding of 1 to further fuse local information. Both convolutional layers are followed by Batch Normalization (BN) and ReLU. The compressed features are input into the RepViT-M0.9 backbone network, which then passes through global average pooling and a linear classification layer to output four categories of ship target probabilities. The category corresponding to the highest probability is taken as the recognition result.

[0032] 2. Model training and reparameterization; This embodiment uses FP32 precision to train the model on the server side. The training data uses the same feature extraction routine as on the board to generate dual-channel input tensors, and is distributed according to the proportions of training set, validation set, and test set. The model is then partitioned. After training, structural reparameterization is performed, fusing the convolutional layers with their subsequent batch normalization (BN) layers into a single convolutional layer. The fused model retains equivalent forward computation results to the unfused model, but the inference graph is more regular with fewer branches, making it easier for the DPU to compile and execute.

[0033] This embodiment can also employ quantization-aware training or post-training quantization. A preferred implementation uses post-training uniform affine INT8 quantization and selects 128 real multispectral feature tensors as a calibration set to ensure that the quantization dynamic range is consistent with the actual underwater acoustic input distribution.

[0034] 3. FPGA deployment and inference; This embodiment uses the Xilinx ZCU102 FPGA platform and the DPUCZDX8G-zcu102 target. The trained PyTorch model is exported in ONNX format with a fixed input shape. The TVM Relay optimizes the ONNX graph and identifies DPU-compatible operators, dividing standard convolution, element-wise addition, and ReLU operators into DPU subgraphs, and dividing a small number of incompatible operators into ARM CPU subgraphs.

[0035] Subsequently, the Vitis AI compiler optimizes instruction scheduling and memory layout for the target DPU architecture, generating an on-board executable file. During inference, the TVM Runtime loads a graph executor on the ARM side, distributing the input tensors to the DPU and CPU subgraphs for execution. The DPU subgraph performs accelerated inference within the FPGA programmable logic, while the CPU subgraph executes operations unsuitable for the DPU, ultimately outputting a logits vector and obtaining the category result.

[0036] In this embodiment, the model, deployed on the ZCU102, achieves an overall accuracy of 91.1%, an end-to-end latency of 8.82 ms, a throughput of 136 samples / s, an average power consumption of 8.3 W, and an energy consumption of 0.061 J / sample, which can meet the real-time acoustic target perception requirements of unmanned underwater systems.

[0037] In this embodiment, the DeepShip underwater passive acoustic dataset was used for evaluation. This dataset contains recordings from 265 vessels, categorized into four types: cargo ships, tugboats, passenger ships, and tankers. Specifically, there are 110 cargo ship recordings with a total duration of 10 hours and 40 minutes; 70 tugboat recordings with a total duration of 11 hours and 17 minutes; 193 passenger ship recordings with a total duration of 12 hours and 22 minutes; and 240 tanker recordings with a total duration of 12 hours and 45 minutes. Continuous audio was divided into 3-second non-overlapping segments, and the training, validation, and test sets were divided in a 7:1.5:1.5 ratio.

[0038] FP32 training and server-side evaluation were performed on an Intel Xeon Silver 4110 CPU and an NVIDIA 2080Ti GPU; board-side evaluation was performed on a Xilinx ZCU102 FPGA platform, with the target DPU being a DPUCZDX8G-zcu102. The following table presents representative results for both server-side and FPGA-side evaluations.

[0039] Table 1 Comparison of FP32 performance on the server side

[0040] Table 2 Comparison of INT8 Deployment Performance on ZCU102 FPGA

[0041] Table 3 Ablation Experiments of Key Modules

[0042] As shown in Tables 1 to 3, the method of this invention achieves an overall accuracy of 95.8% and a Kappa coefficient of 0.946 on the server side. After INT8 quantization and deployment on the ZCU102 FPGA, it still maintains an overall accuracy of 91.1%, a Kappa coefficient of 0.908, and an end-to-end latency of 8.82 ms. The ablation results indicate that both multispectral fusion and pre-backbone convolutional compression modules have a positive effect on recognition accuracy and inference efficiency.

[0043] In the table, OA (%) represents the overall accuracy, which is the percentage of correctly classified samples out of the total test samples, used to evaluate the overall accuracy of the model's recognition. Kappa represents Cohen's Kappa coefficient, used to measure the consistency between the model's predicted classes and the true classes, after deducting the influence of random consistency. Compared with OA, Kappa more fully reflects the classification reliability under imbalanced class distribution. Higher OA and Kappa values ​​indicate better model recognition performance and classification consistency.

Claims

1. A lightweight multispectral passive acoustic recognition edge reasoning method, characterized in that, Includes the following steps: Step 1: Acquire underwater passive acoustic signals, resample and segment the passive acoustic signals to obtain multiple acoustic segments of fixed duration; For each acoustic segment, the short-time power spectrum is calculated, and the log-Mel spectrum and Fbank energy map are generated using the same Mel filter bank. The log-Mel spectrum and Fbank energy map are normalized and their sizes are unified. The two are then stacked along the channel dimension to form a fixed-size dual-channel multispectral acoustic input tensor. Step 2: Feed the fixed-size dual-channel multispectral acoustic input tensor into a lightweight recognition network model that includes a pre-backbone convolutional compression module, a RepViT lightweight backbone, a global average pooling layer, and a linear classification layer to obtain the underwater target category probability or classification logits. Before deployment, the trained lightweight recognition network model is reparameterized; in this process, the pre-backbone convolutional compression module and the fusionable convolutional layers and batch normalization layers in the RepViT lightweight backbone are fused into equivalent convolutional layers, and the network structure during the training phase is converted into a single-path rule reasoning computation graph suitable for DPU execution. Step 3: Export the trained lightweight recognition network model as an ONNX model with a fixed input shape. Then, input the ONNX model into the TVM Relay for graph optimization and divide the subgraph according to the operator support range of the target DPU to obtain the DPU subgraph executed by the DPU and the CPU subgraph executed by the ARM CPU. Step 4: Use the fixed-size dual-channel multispectral acoustic input tensor obtained in Step 1 to construct a calibration set, perform INT8 quantization on the lightweight recognition network model, and compile it with Vitis AI to generate an executable file adapted to the target FPGA DPU; Step 5: Load the executable file and TVM Runtime graph executor generated by Vitis AI on the FPGA edge platform, perform DPU and ARM CPU collaborative inference on the fixed-size dual-channel multispectral acoustic input tensor obtained in Step 1, and output the underwater target category recognition result.

2. The lightweight multispectral passive acoustic recognition edge reasoning method according to claim 1, characterized in that, Step 1 specifically involves: Let any fixed-duration acoustic segment obtained from underwater passive acoustic signal segmentation in step 1 be denoted as . Underwater passive acoustic signals The sampling rate is collected and divided into segments of length [missing information]. Non-overlapping segments; perform a short-time Fourier transform on each segment to obtain the short-time power spectrum. ,in Indicates frequency index, Indicates the time frame index; Based on short-time power spectrum Two spectral channels are constructed using the same Mel filter bank, and the two spectral channels correspond to different energy compression methods. The first spectral channel is the log-Mel spectrum. Through the Mel filter bank After projecting the short-time power spectrum onto the Mel frequency space and performing logarithmic compression, the calculation formula is as follows: in, Indicates the Mel filter index. Small positive constants Indicates frequency index Summing is performed on the above to obtain the first... The energy of the filter bank corresponding to each Mel filter; The second spectral channel is the Fbank energy map. It uses the same Mel filter bank and time framing as the first spectral channel, but employs square root compression, calculated as follows: To each and Perform min-max normalization to map its numerical range to Then normalize the and Scaled to a uniform resolution; finally stacked along the channel dimension to form a fixed-size dual-channel multispectral acoustic input tensor. .

3. The lightweight multispectral passive acoustic recognition edge reasoning method according to claim 2, characterized in that, Step 2 specifically involves: The lightweight recognition network model includes a pre-backbone convolutional compression module, a RepViT lightweight backbone, a global average pooling layer, and a linear classification layer. The pre-backbone convolutional compression module is located between the dual-channel multispectral acoustic input tensor and the RepViT lightweight backbone, and includes a first convolutional layer and a second convolutional layer; the stride of the first convolutional layer is 2, and the stride of the second convolutional layer is 1; both convolutional layers are set to padding of 1, and are sequentially connected to batch normalized (BN) and ReLU activation functions; The RepViT lightweight backbone is used to extract deep semantic features; the output of the RepViT lightweight backbone is then processed by a global average pooling layer to generate classification logits by a linear classification layer. Before deployment, the lightweight recognition network model, after training, merges the pre-backbone convolutional compression module and the convolutional layers adjacent to and fusionable with the BN layer in the RepViT lightweight backbone with the corresponding BN layer to form an equivalent convolutional layer for the inference stage; for weights of Bias is The convolutional layer, and the parameters are , , , and BN layer, fusion weights and fusion bias They are respectively: in, and These represent the kernel weights and biases of the convolutional layers to be merged, respectively; if the convolutional layers to be merged do not contain biases, then... Set to 0; and These represent the scaling and translation parameters of the corresponding BN layer, respectively. and These represent the mean and variance obtained during the training phase of the BN layer, respectively. is a small positive constant used for numerical stability.

4. The lightweight multispectral passive acoustic recognition edge reasoning method according to claim 3, characterized in that, The unified resolution is 96×224.

5. The lightweight multispectral passive acoustic recognition edge reasoning method according to claim 4, characterized in that, The fixed-size dual-channel multispectral acoustic input tensor .

6. The lightweight multispectral passive acoustic recognition edge reasoning method according to claim 5, characterized in that, Both the first and second convolutional layers are 3×3 convolutions.

7. The lightweight multispectral passive acoustic recognition edge reasoning method according to claim 6, characterized in that, The categories of the classified logits include cargo ships, tugboats, passenger ships, and tankers.

8. An edge inference system employing the lightweight multispectral passive acoustic recognition edge inference method as described in claim 1, characterized in that, include: Acoustic Acquisition and Preprocessing Unit: Used to receive passive acoustic signals acquired by hydrophones or underwater acoustic sensors, and to perform resampling, fixed-length segmentation and short-time power spectrum calculation; Multispectral feature construction unit: used to generate log-Mel spectra and Fbank energy maps based on the same Mel filter bank, complete normalization, size unification and channel stacking, and output a fixed-size dual-channel multispectral acoustic input tensor; Lightweight recognition network unit: includes a pre-backbone convolutional compression module, a RepViT lightweight backbone, a global average pooling layer, and a linear classification layer, used to classify multispectral features; Structure fusion and model conversion unit: used to perform structure reparameterization, ONNX export, graph optimization and operator compatibility checks on the trained lightweight recognition network model; Quantization compilation unit: used to perform INT8 calibration quantization based on real feature tensors and call the Vitis AI toolchain to generate compilation results adapted to the target DPU; FPGA heterogeneous inference unit: Deployed on the FPGA edge platform, it includes a DPU accelerator, ARM CPU, DDR memory and runtime graph executor, used to execute DPU subgraph and CPU subgraph; The result output unit is used to output target category, confidence level, inference delay, energy consumption or other operating status information, and to provide recognition results to the mission planning or alarm module of the unmanned underwater platform.

Images