Wireless underwater fish sound recognition system based on SOPC

Abstract

The present invention relates to the field of marine biometry, and more specifically to a SOPC-based wireless underwater fish sound recognition system. The system comprises a processing system end and a programmable logic end. The processing system end comprises a URAT communication module, an ICB bus control module, and a DTW algorithm module. The programmable logic end comprises an ADC control module, a DDR recording control module, an MFCC feature extraction module, a VAD module, a DTW algorithm acceleration module, a DMA control module, and a noise reduction module. The present invention utilizes SOPC-based hardware, increasing the portability of underwater fish sound signal recognition scenarios. The system can be easily deployed in actual outdoor testing environments, facilitating practical production activities. Furthermore, compared to pure MCU implementations of the DTW recognition algorithm, the present invention utilizes a software and hardware partitioning approach based on the SOPC architecture to accelerate specific computational components in the DTW algorithm, thereby increasing the speed of fish sound signal recognition.

Description

Technical Field

[0001] The present invention relates to the field of marine biological measurement, and in particular to a wireless underwater fish sound recognition system based on SOPC. Background Art

[0002] Passive acoustic technology, an emerging engineering technology in the field of marine biometry, has attracted widespread attention from researchers. Research on fish vocalizations primarily aims to facilitate information transfer between and within species, including the sounds of flocking during reproduction, alarm calls to avoid predators, tentative calls to find food, and calls to identify conspecifics. These characteristics are closely related to the species and physiological state of the fish and have specific biological significance. Automated detection and identification of underwater fish has significant research and practical application implications for the investigation and research of fishery resources, the monitoring and protection of aquatic environments, the management of aquaculture, and the analysis of patterns in aquatic biological activity.

[0003] FPGA-based SOPC systems leverage the programmable advantages of FPGAs. Through FPGAs, unique hardware accelerators and coprocessor logic can be designed, which can then work in conjunction with the MCU, breaking through hardware functional limitations and realizing a customized system-on-chip. Compared to pure hardware or pure software systems, FPGA-based SOPC systems offer greater flexibility and feasibility, and therefore have a greater market and application prospects. Technologies related to fish sound signal recognition often use a method where sensors in an anechoic tank collect signals and then process them on devices such as personal computers. This method has limited application scenarios. Furthermore, most existing audio recognition systems are designed based on human speech recognition models. Directly applying these systems to fish sound recognition significantly reduces recognition accuracy. Summary of the Invention

[0004] The embodiment of the present invention provides a wireless underwater fish sound recognition system based on SOPC, aiming to solve the technical problems of the existing fish sound recognition system in limited application scenarios and insufficient recognition accuracy.

[0005] An embodiment of the present invention provides a wireless underwater fish sound recognition system based on SOPC. The wireless underwater fish sound recognition system includes a processing system end and a programmable logic end. The processing system end includes a URAT communication module, an ICB bus control module, and a DTW algorithm module. The programmable logic end includes an ADC control module, a DDR recording control module, an MFCC feature extraction module, a VAD module, a DTW algorithm acceleration module, a DMA control module, and a noise reduction module.

[0006] The URAT communication module is used for data transmission between the wireless underwater fish sound identification system and an external wireless transmission module to achieve wireless communication;

[0007] The ICB bus control module is used to realize data transmission and hardware control in the wireless underwater fish sound identification system through the ICB bus;

[0008] The DTW algorithm module is used to send the audio to be recognized to the DTW algorithm acceleration module, and use the Euclidean distance result output by the DTW algorithm acceleration module to judge the fish sound, and output the fish sound judgment result;

[0009] The ADC control module is used to receive AD data of real-time underwater acoustic signals and provide a driving clock to the 8-bit high-speed ADC;

[0010] The noise reduction module is used to perform noise reduction processing on the real-time underwater acoustic signal before it is converted by the ADC control module;

[0011] The DDR recording control module is used to cache the AD data;

[0012] The DMA control module is used to realize cache acceleration between the ADC control module and the DDR recording control module by using DMA technology;

[0013] The MFCC feature extraction module is used to extract the MFCC feature parameters of the real-time underwater acoustic signal;

[0014] The VAD module is used to extract effective fish sound signals from the real-time underwater acoustic signal according to the MFCC feature parameters;

[0015] The DTW algorithm acceleration module is used to perform a sequence matching operation based on the DTW algorithm according to the effective fish sound signal and the audio to be identified, and output a Euclidean distance result.

[0016] Furthermore, the VAD module is a deep neural network model, which includes an input layer, a first hidden layer, a second hidden layer, a third hidden layer, a fourth hidden layer and an output layer, wherein Sigmoid is used as the activation function between the input layer and the first hidden layer, and ReLU is used as the activation function between the second hidden layer, the third hidden layer, the fourth hidden layer and the output layer, and the number of neurons in the input layer, the first hidden layer, the second hidden layer, the third hidden layer, the fourth hidden layer and the output layer are 39, 138, 64, 32, 16 and 10 respectively.

[0017] Furthermore, the first hidden layer, the second hidden layer, the third hidden layer, and the fourth hidden layer are all fully connected layers.

[0018] Furthermore, the DDR recording control module is read and written via the AXI protocol.

[0019] Furthermore, the MFCC feature extraction module is specifically used to:

[0020] Preprocessing the real-time underwater acoustic signal;

[0021] Performing short-time Fourier transform processing on the real-time underwater acoustic signal;

[0022] Processing the real-time underwater acoustic signal using a Mel filter bank to obtain a Mel frequency spectrum;

[0023] Calculating the logarithmic energy of the Mel frequency spectrum;

[0024] Performing discrete cosine transform processing on the logarithmic energy to obtain the MFCC feature parameters.

[0025] Furthermore, the process of preprocessing the real-time underwater acoustic signal by the MFCC feature extraction module includes:

[0026] Use FIR filter for pre-emphasis processing;

[0027] Divide the frames into frames according to the preset duration;

[0028] The Hamming window is used to perform windowing processing on the framed data.

[0029] Furthermore, in the process of the MFCC feature extraction module using a Mel filter to process the real-time underwater acoustic signal, the Mel filter group includes 26 triangular filters, and the coefficient of the Mel filter group is 16448.

[0030] Furthermore, the DTW algorithm module and the DTW algorithm acceleration module use an ICB bus for data exchange.

[0031] The beneficial effects achieved by the present invention are that the underwater fish sound signal recognition system is implemented using SOPC-based hardware, which increases the portability of the hardware in the underwater fish sound signal recognition scenario. The system can be conveniently deployed in actual outdoor detection environments to facilitate actual production activities such as aquaculture, fishery fishing, and environmental ecological research. In addition, compared with the pure MCU implementation of the DTW recognition algorithm, the software and hardware division method of the SOPC architecture of the present invention accelerates specific calculation parts in DTW, thereby improving the recognition speed of fish sound signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 1 is a schematic diagram of the structure of a wireless underwater fish sound recognition system based on SOPC provided by an embodiment of the present invention;

[0033] Figure 2is a schematic diagram of the structure of a deep neural network model provided by an embodiment of the present invention;

[0034] Figure 3 Schematic diagram of a Mel filter bank provided by an embodiment of the present invention;

[0035] Figure 4 This is a schematic diagram of Euclidean distance calculation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0037] The wireless underwater fish sound recognition system provided by the embodiment of the present invention includes a processing system end and a programmable logic end. The processing system end includes a URAT communication module, an ICB bus control module and a DTW algorithm module. The programmable logic end includes an ADC control module, a DDR recording control module, an MFCC feature extraction module, a VAD module, a DTW algorithm acceleration module, a DMA control module and a noise reduction module. In the embodiment of the present invention, the processing system end is implemented by a soft core of the Hummingbird E203 RISC-v, and the programmable logic end is implemented by an SOPC based on the XILINX XC7K32T-2FFG676 FPGA. The Hummingbird E203 RISC-v is a commonly used microprocessor that can be deployed in an SOPC system. Please refer to Figure 1 , Figure 1 : This is a schematic diagram of the structure of a wireless underwater fish sound recognition system based on SOPC provided by an embodiment of the present invention, wherein:

[0038] The URAT communication module is used for data transmission between the wireless underwater fish sound identification system and an external wireless transmission module to achieve wireless communication;

[0039] The ICB bus control module is used to realize data transmission and hardware control in the wireless underwater fish sound identification system through the ICB bus;

[0040] The DTW algorithm module is used to send the audio to be recognized to the DTW algorithm acceleration module, and use the Euclidean distance result output by the DTW algorithm acceleration module to judge the fish sound, and output the fish sound judgment result;

[0041] The ADC control module is used to receive AD data of real-time underwater acoustic signals and provide a driving clock to the 8-bit high-speed ADC;

[0042] The noise reduction module is used to perform noise reduction processing on the real-time underwater acoustic signal before it is converted by the ADC control module;

[0043] The DDR recording control module is used to cache the AD data;

[0044] The DMA control module is used to realize cache acceleration between the ADC control module and the DDR recording control module by using DMA technology;

[0045] The MFCC feature extraction module is used to extract the MFCC feature parameters of the real-time underwater acoustic signal;

[0046] The VAD module is used to extract effective fish sound signals from the real-time underwater acoustic signal according to the MFCC feature parameters;

[0047] The DTW algorithm acceleration module is used to perform a sequence matching operation based on the DTW algorithm according to the effective fish sound signal and the audio to be identified, and output a Euclidean distance result.

[0048] In addition, in an embodiment of the present invention, the Hummingbird E203 RISC-v microprocessor also includes an instruction tightly coupled memory (ITCM), a data tightly coupled memory (DTCM), an interrupt controller (PLIC and CLINT), a system storage bus, a ROM read-only memory, an off-chip Flash memory, a static random access memory (SRAM), a JTAG debugging interface, a private device bus, a fast IO interface, and corresponding peripheral modules, wherein:

[0049] The instruction tightly coupled memory is used to store instructions for the processor core to obtain instruction words, the data tightly coupled memory is used to store data for the processor core to obtain operands, the CLINT in the interrupt controller is used to generate software interrupts and timer interrupts, the PLIC is used for priority arbitration and dispatching of multiple external interrupt sources, the system storage bus is used for the processor to access the memory, the ROM read-only memory is used to store the solidified soft core files, the off-chip Flash memory is used to store the C language program running in the compiled soft core, the JTAG debugging interface is used for online debugging of the soft core program, the static random access memory is used to store the matching templates used in the DTW algorithm operation process, and the private device bus is used to control the data communication between the peripheral modules and the E203 processor.

[0050] The above components, as the storage structure of the processor, should be considered to be included in the SOPC-based wireless underwater fish sound recognition system provided in the embodiment of the present invention, and the present application can use the above components to perform data reading, writing and forwarding processes of each module.

[0051] Furthermore, the VAD module is a deep neural network model, please refer to Figure 2 , Figure 2: This is a structural diagram of a deep neural network model provided by an embodiment of the present invention, wherein the deep neural network model includes an input layer, a first hidden layer, a second hidden layer, a third hidden layer, a fourth hidden layer, and an output layer, wherein Sigmoid is used as an activation function between the input layer and the first hidden layer, and ReLU is used as an activation function between the second hidden layer, the third hidden layer, the fourth hidden layer, and the output layer, and the number of neurons in the input layer, the first hidden layer, the second hidden layer, the third hidden layer, the fourth hidden layer, and the output layer are 39, 138, 64, 32, 16, and 10, respectively.

[0052] Furthermore, the first hidden layer, the second hidden layer, the third hidden layer, and the fourth hidden layer are all fully connected layers. Their transfer functions are as follows:

[0053]

[0054] The input of the VAD module is the MFCC feature parameters of each frame of fish sound signal, and the output is the label corresponding to the frame. When the output is 0, it indicates that the frame is a non-fish sound frame; when the output is 1, it indicates that the frame is a valid fish sound frame.

[0055] Furthermore, the DDR recording control module is read and written through the AXI protocol, and after the clock domain and bit width conversion is performed by the AXI2ICB module, it is connected to the E203 processor core through the ICB bus.

[0056] Furthermore, the MFCC feature extraction module is specifically used to:

[0057] Preprocessing the real-time underwater acoustic signal;

[0058] Performing short-time Fourier transform processing on the real-time underwater acoustic signal;

[0059] The real-time underwater acoustic signal is processed using a Mel filter bank to obtain a Mel frequency spectrum, which is sensitive to low-frequency components but insensitive to high-frequency components. The conversion relationship between the signal frequency f and the Mel frequency m is as follows:

[0060]

[0061] Calculate the logarithmic energy of the Mel frequency spectrum, specifically:

[0062]

[0063] Performing discrete cosine transform processing on the logarithmic energy to obtain the MFCC feature parameters, the specific steps are as follows:

[0064]

[0065] Among them H m (k) is the frequency response of the triangular filter, and the calculation of the cosine function is achieved by the table lookup method.

[0066] Furthermore, the process of preprocessing the real-time underwater acoustic signal by the MFCC feature extraction module includes:

[0067] FIR filter is used for pre-emphasis processing to eliminate the high-frequency attenuation of the signal caused by underwater sound propagation. The pre-emphasis function is as follows:

[0068] y(n)=x(n)-α·x(n-1);

[0069] Where α is the pre-emphasis coefficient, which is 0.97.

[0070] The fish sound signal is framed according to a preset time length, and the frame processing is performed on the fish sound signal based on the short-time analysis principle. In the embodiment of the present invention, the preset time length is 20ms.

[0071] The Hamming window is used to perform windowing on the framed data. The formula is:

[0072]

[0073] In the hardware circuit for windowing processing, the operation of the cosine function is implemented using a table lookup method.

[0074] For further information, please refer to Figure 3 , Figure 3 : is a schematic diagram of the Mel filter group provided in an embodiment of the present invention. In the process in which the MFCC feature extraction module processes the real-time underwater acoustic signal using the Mel filter, the Mel filter group includes 26 triangular filters, and the coefficient of the Mel filter group is 16448.

[0075] Furthermore, the DTW algorithm module and the DTW algorithm acceleration module use an ICB bus for data exchange.

[0076] Specifically, in an embodiment of the present invention, the DTW algorithm module and the DTW algorithm acceleration module are respectively implemented based on the soft core of the Hummingbird E203 RISC-v and the SOPC of the XILINX XC7K32T-2FFG676 FPGA. Therefore, the overall process of the DYW algorithm is divided into software and hardware. In the SOPC, the DTW algorithm acceleration module is used to calculate the Euclidean distance between the fish sound signal to be identified and the matching template, wherein the matching template refers to pre-stored data of different fish sounds, which is used to match the signal to be identified. The matching template is stored in the static random access memory SRAM. Afterwards, the Euclidean distance calculation result is received by the DTW algorithm module in the Hummingbird E203 RISC-v to complete the fish sound recognition of the DTW algorithm. The hardware acceleration operation process using the DTW algorithm generally includes three steps:

[0077] 1. Calculate the distance matrix between the audio segment to be identified and the matching template audio segment;

[0078] 2. Search for the minimum path of the distance matrix. The sum of the values ​​on the minimum path is the Euclidean distance between the audio segment to be identified and the corresponding matching template, such as Figure 4 shown.

[0079] 3. Traverse all matching templates and compare the Euclidean distance between the audio segment to be recognized and each matching template. The label corresponding to the matching template with the smallest Euclidean distance is the recognition result.

[0080] In summary, the complete implementation process of the SOPC-based wireless underwater fish sound recognition system provided by the embodiment of the present invention is as follows:

[0081] After the system is powered on, the E203 processor first reads the stored C language program from the off-chip Flash memory and performs initialization operations. Then, according to the instructions, it accesses the UART interface device on the private device bus and uses AT commands through the UART interface to initialize and configure the external wireless transmission module, making it enter TCP transparent transmission mode.

[0082] The underwater acoustic signal is collected by peripheral hardware such as hydrophones, and converted into a digital signal by the 8-bit ADC control module. The digital fish sound signal is then cached in the 8-bit ADC control module, and the cached data is written to the DDR by the DMA control module.

[0083] Afterwards, the underwater acoustic signal is read out from the DDR according to the pre-processed frame number and input into the input register of the MFCC feature extraction module to obtain the MFCC corresponding to each frame of the underwater acoustic signal and cache it in the DDR.

[0084] While caching MFCC, the cached MFCC is read from the starting frame in DDR and written into the input register of the VAD module. It is then judged whether each frame signal belongs to a valid fish sound signal segment, thereby obtaining a valid fish sound signal and the corresponding MFCC in a section of the underwater acoustic signal.

[0085] Afterwards, according to the instruction, the MFCC of the valid fish sound segment is written into the input register of the DTW algorithm acceleration module. At the same time, the E203 processor core reads the MFCC matching template of the fish sound signal from the SRAM according to the instruction and transmits it to the ICB bus. Then, the DTW algorithm acceleration module reads the MFCC matching template from the ICB bus according to the instruction, and calculates the Euclidean distance between the valid fish sound segment and the MFCC matching template. After that, according to the instruction, the Euclidean distance value is transmitted to the DTW algorithm module of the E203 processor core through the ICB bus, and the above-mentioned DTW algorithm calculation process is repeated according to the instruction to calculate the Euclidean distance between the valid fish sound segment and multiple MFCC matching templates.

[0086] After receiving all the Euclidean distance values, the E203 processor core sorts them according to the instructions. The MFCC matching template label corresponding to the minimum Euclidean distance value is considered to be the recognition result of the frame signal and is saved in the output register.

[0087] Afterwards, all the above steps are repeated according to the instructions until all the extracted valid fish sounds are identified. At this time, the E203 processor core will obtain the mode value from the output register according to the instructions and calculate the percentage of the mode value. When the percentage is greater than 75%, the label corresponding to the mode value is considered to be the final recognition result. Otherwise, it is considered that the underwater acoustic signal collected by this recognition is a non-fish sound segment or no valid fish sound signal is collected;

[0088] Finally, according to the instruction execution, the recognition results are transmitted to the wireless transmission module through the UART serial port interface and other interfaces connected to the external hardware, and then sent to the GUI interface of the host computer through transmission protocols such as TCP for display, thus completing the underwater fish sound recognition process.

[0089] The beneficial effects achieved by the present invention are that the underwater fish sound signal recognition system is implemented using SOPC-based hardware, which increases the portability of the hardware in the underwater fish sound signal recognition scenario. The system can be conveniently deployed in actual outdoor detection environments to facilitate actual production activities such as aquaculture, fishery fishing, and environmental ecological research. In addition, compared with the pure MCU implementation of the DTW recognition algorithm, the software and hardware division method of the SOPC architecture of the present invention accelerates specific calculation parts in DTW, thereby improving the recognition speed of fish sound signals.

[0090] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0091] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0092] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0093] The embodiments of the present invention are described above in conjunction with the accompanying drawings. What is disclosed is only a preferred embodiment of the present invention. However, the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms and equivalent changes without departing from the scope of protection of the purpose of the present invention and the claims, which are all within the protection of the present invention.

Claims

1. A wireless underwater fish sound recognition system based on SOPC, characterized by: The wireless underwater fish sound recognition system includes a processing system end and a programmable logic end. The processing system end includes a URAT communication module, an ICB bus control module and a DTW algorithm module. The programmable logic end includes an ADC control module, a DDR recording control module, an MFCC feature extraction module, a VAD module, a DTW algorithm acceleration module, a DMA control module and a noise reduction module, wherein: The URAT communication module is used for data transmission between the wireless underwater fish sound identification system and an external wireless communication module to achieve wireless communication; The ICB bus control module is used to realize data transmission and hardware control in the wireless underwater fish sound identification system through the ICB bus; The DTW algorithm module is used to send the audio to be recognized to the DTW algorithm acceleration module, and use the Euclidean distance result output by the DTW algorithm acceleration module to judge the fish sound, and output the fish sound judgment result; The ADC control module is used to receive AD data of real-time underwater acoustic signals and provide a driving clock to the 8-bit high-speed ADC; The noise reduction module is used to perform noise reduction processing on the real-time underwater acoustic signal before it is converted by the ADC control module; The DDR recording control module is used to cache the AD data; The DMA control module is used to implement cache acceleration between the ADC control module and the DDR recording control module using DMA technology; The MFCC feature extraction module is used to extract the MFCC feature parameters of the real-time underwater acoustic signal; The VAD module is used to extract effective fish sound signals from the real-time underwater acoustic signal according to the MFCC feature parameters; The DTW algorithm acceleration module is used to perform a sequence matching operation based on the DTW algorithm according to the effective fish sound signal and the audio to be identified, and output a Euclidean distance result.

2. The wireless underwater fish sound recognition system based on SOPC as claimed in claim 1, characterized in that: The VAD module is a deep neural network model, which includes an input layer, a first hidden layer, a second hidden layer, a third hidden layer, a fourth hidden layer and an output layer, wherein Sigmoid is used as an activation function between the input layer and the first hidden layer, and ReLU is used as an activation function between the second hidden layer, the third hidden layer, the fourth hidden layer and the output layer. The number of neurons in the input layer, the first hidden layer, the second hidden layer, the third hidden layer, the fourth hidden layer and the output layer are 39, 138, 64, 32, 16 and 10 respectively.

3. The wireless underwater fish sound identification system based on SOPC as claimed in claim 2, characterized in that: The first hidden layer, the second hidden layer, the third hidden layer, and the fourth hidden layer are all fully connected layers.

4. The wireless underwater fish sound recognition system based on SOPC as claimed in claim 1, characterized in that: The DDR recording control module is read and written via the AXI protocol.

5. The wireless underwater fish sound recognition system based on SOPC as claimed in claim 1, characterized in that: The MFCC feature extraction module is specifically used for: Preprocessing the real-time underwater acoustic signal; Performing short-time Fourier transform processing on the real-time underwater acoustic signal; Processing the real-time underwater acoustic signal using a Mel filter bank to obtain a Mel frequency spectrum; Calculating the logarithmic energy of the Mel frequency spectrum; Performing discrete cosine transform processing on the logarithmic energy to obtain the MFCC feature parameters.

6. The wireless underwater fish sound recognition system based on SOPC as claimed in claim 5, characterized in that: The process of preprocessing the real-time underwater acoustic signal by the MFCC feature extraction module includes: Use FIR filter for pre-emphasis processing; Divide the frames into frames according to the preset duration; The Hamming window is used to perform windowing processing on the framed data.

7. The wireless underwater fish sound recognition system based on SOPC as claimed in claim 5, characterized in that: In the process of the MFCC feature extraction module using the Mel filter to process the real-time underwater acoustic signal, the Mel filter group includes 26 triangular filters, and the coefficient of the Mel filter group is 16448.

8. The wireless underwater fish sound recognition system based on SOPC as claimed in claim 1, characterized in that: The DTW algorithm module and the DTW algorithm acceleration module use the ICB bus to perform data exchange.

Images