A method, system and device for classifying fish feeding intensity

By extracting the sound signals of fish school and applying the deep convolutional neural network model, the feeding intensity of fish school is identified, and the water quality pollution caused by insufficient or excessive feeding in aquaculture is solved, and automatic feeding on demand is achieved, improving breeding efficiency and resource utilization rate.

CN116665701BActive Publication Date: 2024-11-19CHINA AGRI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310657873.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-06-06
Publication Date
2024-11-19
Estimated Expiration
2043-06-06

AI Technical Summary

Technical Problem

In aquaculture, artificial feeding of bait often leads to water quality pollution and waste of resources due to insufficient or excessive feeding, and there is a lack of effective identification of fish feeding needs.

Method used

A fish school feeding intensity classification method is adopted to extract the sound signals of the fish school, and combine the Mel spectrum, constant Q transformation and short-time Fourier transformation to generate the deep spectrum feature vector of fish school feeding, and use deep convolutional nerves. The network model determines the type of feeding intensity of fish schools and realizes automatic feeding on demand.

Benefits of technology

It realizes contactless and real-time identification of fish feeding intensity, avoids insufficient or excessive feeding, reduces water pollution and resource waste, and supports the development of self-demand feeding systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116665701B_ABST
    Figure CN116665701B_ABST
Patent Text Reader

Abstract

The present invention provides a method, system and equipment for classifying the feeding intensity of a school of fish, and relates to the field of aquaculture. The method comprises: extracting features from an audio segment to be tested, determining a fish feeding depth spectrum feature vector based on Mel spectrum, a fish feeding depth spectrum feature vector based on CQT, and a fish feeding depth spectrum feature vector based on STFT; fusing the fish feeding depth spectrum feature vector based on Mel spectrum, the fish feeding depth spectrum feature vector based on CQT, and the fish feeding depth spectrum feature vector based on STFT to generate a fused feature spectrogram; inputting the fused feature spectrogram into a deep convolutional neural network model constructed from historical audio segments corresponding to different types of feeding intensities, and determining the feeding intensity type corresponding to the audio segment to be tested. The present invention realizes automatic feeding on demand, and avoids water pollution and waste of resources caused by insufficient or excessive feeding.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of aquaculture, and in particular to a method, system and equipment for classifying the feeding intensity of fish schools. Background Art

[0002] In aquaculture, the amount of feed put in has always been an important issue that limits the economic benefits of aquaculture. Reasonable control of feed feeding is of great significance. At present, feeding decisions are mostly based on the experience of aquaculture personnel to set the amount of feed, ignoring the influence of factors such as fish feeding needs and water quality environment. Water pollution and resource waste are often caused by insufficient or overfeeding. Therefore, it is urgent to develop an automatic fish feeding intensity identification method based on fish feeding needs to achieve contactless and real-time feeding intensity classification, which is crucial for the further development of self-demand feeding systems. Summary of the invention

[0003] The purpose of the present invention is to provide a method, system and equipment for classifying the feeding intensity of fish schools, so as to solve the problem of water pollution and resource waste caused by insufficient or excessive feeding in artificial feeding.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] A method for classifying fish feeding intensity, comprising:

[0006] Extract features from the audio band to be tested, and determine the fish feeding depth spectrum feature vector based on Mel spectrum, the fish feeding depth spectrum feature vector based on CQT, and the fish feeding depth spectrum feature vector based on STFT;

[0007] Fusion of the fish feeding depth spectrum feature vector based on Mel spectrum, the fish feeding depth spectrum feature vector based on CQT and the fish feeding depth spectrum feature vector based on STFT to generate a fused feature spectrogram;

[0008] The fused feature spectrogram is input into a deep convolutional neural network model constructed by historical audio segments corresponding to different types of eating intensities to determine the type of eating intensity corresponding to the audio segment to be tested; the types of eating intensity include "strong", "medium", "weak" and "none".

[0009] Optionally, the process of extracting the fish feeding depth spectrum feature vector based on the Mel spectrum specifically includes:

[0010] A plurality of triangular filters are set within the frequency range of the fish feeding signal to form a triangular frequency filter group; the triangular frequency filter group includes a plurality of bandpass filters; the bandpass filter is a Mel filter; the transfer function of the bandpass filter is: Among them, H m (k) is a bandpass filter, m is the serial number of the Mel filter, M is the number of Mel filters, f(m) is the center frequency of the m-th Mel filter; f(m+1) is the center frequency of the m+1-th Mel filter; f(m-1) is the center frequency of the m-1-th Mel filter;

[0011] Using the triangular frequency filter group to perform fast Fourier transform processing on the sound signal in the audio segment to be tested, converting the sound signal from the time domain to the frequency domain, and generating a filtered sound signal;

[0012] determining an energy spectrum according to the filtered sound signal;

[0013] Determine the signal energy in each Mel filter according to the energy spectrum;

[0014] Determine a mel-spectrogram of the fish school feeding sound signal according to the signal energy;

[0015] A spectral feature vector of the fish feeding depth based on the Mel spectrum in the audio segment to be tested is extracted according to the Mel spectrum graph.

[0016] Optionally, the CQT-based fish feeding depth spectrum feature vector extraction process specifically includes:

[0017] Performing a constant-Q transformation on the sound signal in the audio segment to be measured to generate spectrum parameters after the constant-Q transformation;

[0018] Generate a constant Q transform spectrum diagram according to the spectrum parameters;

[0019] The CQT-based fish feeding depth spectral feature vector in the audio segment to be tested is extracted according to the constant Q transform spectrum diagram.

[0020] Optionally, the process of extracting the fish feeding depth spectrum feature vector based on STFT specifically includes:

[0021] Adding a short-time window function moving along the time axis to the sound signal in the audio segment to be tested, and intercepting the non-stationary signal at each moment by the short-time window function; the signal in the short-time window is a stationary signal;

[0022] Performing Fourier transform on the non-stationary signal to generate a time-frequency spectrum at each moment;

[0023] The STFT-based fish feeding depth spectral feature vector in the audio segment to be tested is extracted according to the time-frequency spectrum.

[0024] Optionally, the construction process of the deep convolutional neural network model specifically includes:

[0025] Obtain historical video clips and historical sound signals of fish before, during and after feeding;

[0026] Dividing different types of feeding intensities according to the historical video clips, and synchronously cutting the historical sound signals to determine historical audio segments corresponding to different types of feeding intensities;

[0027] The deep convolutional neural network model is constructed based on the historical audio segments.

[0028] A system for classifying the feeding intensity of fish schools, comprising:

[0029] A feature extraction module is used to extract features from the audio band to be tested, and determine a fish feeding depth spectrum feature vector based on Mel spectrum, a fish feeding depth spectrum feature vector based on CQT, and a fish feeding depth spectrum feature vector based on STFT;

[0030] A feature fusion module, used for fusing the fish feeding depth spectrum feature vector based on Mel spectrum, the fish feeding depth spectrum feature vector based on CQT and the fish feeding depth spectrum feature vector based on STFT to generate a fused feature spectrogram;

[0031] The eating intensity type determination module is used to input the fused feature spectrogram into a deep convolutional neural network model constructed by historical audio segments corresponding to different types of eating intensities, and determine the eating intensity type corresponding to the audio segment to be tested; the types of eating intensity include "strong", "medium", "weak" and "none".

[0032] Optionally, the process of extracting the fish feeding depth spectrum feature vector based on the Mel spectrum specifically includes:

[0033] The triangular frequency filter group setting unit is used to set a plurality of triangular filters within the frequency range of the fish feeding signal to form a triangular frequency filter group; the triangular frequency filter group includes a plurality of bandpass filters; the bandpass filter is a Mel filter; the transfer function of the bandpass filter is: Among them, H m (k) is a bandpass filter, m is the serial number of the Mel filter, M is the number of Mel filters, f(m) is the center frequency of the m-th Mel filter; f(m+1) is the center frequency of the m+1-th Mel filter; f(m-1) is the center frequency of the m-1-th Mel filter;

[0034] A fast Fourier transform processing unit, used for performing fast Fourier transform processing on the sound signal in the audio segment to be measured by using the triangular frequency filter group, converting the sound signal from the time domain to the frequency domain, and generating a filtered sound signal;

[0035] an energy spectrum determining unit, configured to determine an energy spectrum according to the filtered sound signal;

[0036] A signal energy determination unit, used to determine the signal energy in each Mel filter according to the energy spectrum;

[0037] A mel-spectrogram determination unit, used to determine the mel-spectrogram of the fish school feeding sound signal according to the signal energy;

[0038] The fish feeding depth spectrum feature vector extraction unit based on Mel spectrum is used to extract the fish feeding depth spectrum feature vector based on Mel spectrum in the audio segment to be tested according to the Mel spectrum graph.

[0039] Optionally, the CQT-based fish feeding depth spectrum feature vector extraction process specifically includes:

[0040] A frequency spectrum parameter generating unit after constant Q transformation, used for performing constant Q transformation on the sound signal in the audio segment to be measured to generate frequency spectrum parameters after constant Q transformation;

[0041] A constant Q transform spectrum diagram generating unit, used for generating a constant Q transform spectrum diagram according to the spectrum parameters;

[0042] The CQT-based fish feeding depth spectrum feature vector determination unit is used to extract the CQT-based fish feeding depth spectrum feature vector in the audio segment to be tested according to the constant Q transform spectrum diagram.

[0043] An electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the above-mentioned method for classifying the feeding intensity of a school of fish.

[0044] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned method for classifying the feeding intensity of a school of fish.

[0045] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: the present invention provides a method, system and device for classifying the feeding intensity of a school of fish, combines video clips with sound signals, extracts features of the combined audio segment to be tested based on Mel spectrum, constant-Q transform (CQT) and short-time Fourier transform (STFT), generates different fish feeding depth spectrum feature vectors, fuses different fish feeding depth spectrum feature vectors, inputs the fused feature spectrogram into a deep convolutional neural network model constructed by historical audio segments corresponding to different types of feeding intensities, and determines the feeding intensity type corresponding to the audio segment to be tested. The present invention combines video clips and sound signals, feeds fish according to the feeding intensity type according to the feeding needs of the fish, realizes automatic feeding on demand, and avoids water pollution and waste of resources caused by insufficient or excessive feeding. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 A flow chart of the fish feeding intensity classification method provided by the present invention;

[0048] Figure 2 This is a flow chart of the fish feeding intensity classification method based on CNN provided by the present invention;

[0049] Figure 3 This is a structural diagram of the experimental data acquisition system provided by the present invention;

[0050] Figure 4 The fish feeding data set provided by the present invention shows some sample graphs of different feeding intensity types;

[0051] Figure 5 The feeding intensity characteristic diagram of fish schools provided by the present invention;

[0052] Figure 6 This is a comparison chart of the traditional network structure and the improved CNN network model provided by the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] The purpose of the present invention is to provide a method, system and equipment for classifying the feeding intensity of fish schools, so as to realize automatic feeding on demand and avoid water pollution and waste of resources caused by insufficient or excessive feeding.

[0055] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] Sound recognition is a cutting-edge topic in today's pattern recognition theory, which covers many fields. At present, some sound recognition topics have made significant research progress, such as automatic speech recognition (ASR), music information retrieval (MIR), bird audio detection (BAD), environmental sound classification (ESC) and abnormal heart sound diagnosis. Fish and shrimp produce pulse sound wave signals during feeding, which can be used as a basis for judging feeding activities. The use of passive acoustic methods for detection will not have a negative impact on the feeding environment and feeding behavior of fish and shrimp, and can also provide a basis for formulating more effective feeding strategies, so that the feeding system meets the feeding needs of different fish populations. Therefore, the study of fish feeding behavior based on acoustic technology is an important means to quantify the feeding intensity of fish.

[0057] Feature fusion is a common fusion method, which is widely used in tasks such as image recognition, speech recognition and sound scene classification. In recent years, with the rapid development of deep learning, feature fusion has been widely used in many fields and has achieved good performance. At present, more scholars are paying attention to the fusion of image features or the combination of acoustic and image features. Studies have shown that using a convolutional neural network (CNN) model to fuse Mel-frequency cepstral coefficient (MFCC) features to create a fusion of acoustic features and visual features can achieve good results. Taking into account the advantages of the feature fusion algorithm, the present invention proposes a feeding intensity classification method based on the fusion of fish feeding sound signal features. This method has important guiding significance for the further development of self-demand feeding decision-making systems.

[0058] Embodiment 1

[0059] like Figure 1-Figure 2 As shown, the present invention provides a method for classifying the feeding intensity of a school of fish, comprising:

[0060] Step 101: extracting features from the audio segment to be tested, and determining a fish feeding depth spectrum feature vector based on Mel spectrum, a fish feeding depth spectrum feature vector based on CQT, and a fish feeding depth spectrum feature vector based on STFT.

[0061] In practical applications, before step 101, Figure 3 As shown, Hikvision visual cameras and omnidirectional hydrophones were used to obtain video clips and sound signals of fish before, during and after feeding.

[0062] According to the experience of aquaculture technicians and the existing feeding intensity classification standards, the fish feeding intensity videos were divided into four types: "strong", "medium", "weak", and "none" by reviewing the videos. Figure 4 Then, according to the classification intensity and time period of the video, the synchronized sound signal was cropped in the same way to obtain audio segments of four feeding intensity types.

[0063] For each type of audio segment of feeding intensity, we randomly selected audio segments, created training sets, validation sets, and test sets in a certain proportion, and built a deep convolutional neural network model based on the training set.

[0064] In practical applications, the extraction process of the fish feeding depth spectrum feature vector based on the Mel spectrum specifically includes:

[0065] Mel frequency is a nonlinear frequency inspired by the hearing characteristics of the human ear. Formula (1) describes the logarithmic relationship between sound frequency and Mel frequency, where f mel is the Mel frequency, and f is the actual frequency in Hz.

[0066]

[0067] The triangular frequency filter bank imitates the human ear to filter the speech signal, and M triangular filters are set within the frequency range of the fish feeding signal to form a triangular frequency filter bank.

[0068] The triangular frequency filter bank consists of 64 bandpass filters H m (k) is composed of a Mel filter, and its transfer function is shown in formula (2), wherein 1≤m≤M, m represents the serial number of the Mel filter, M is the number of Mel filters, f(m) is the center frequency of the m-th Mel filter; f(m+1) is the center frequency of the m+1-th Mel filter; and f(m-1) is the center frequency of the m-1-th Mel filter.

[0069]

[0070] As shown in formula (3), f l and f h are the lowest and highest frequencies of the filter, respectively, f s is the sampling frequency, N is the length of the Fast Fourier Transform (FFT), and f mel Mel senses the frequency, f mel In the present invention, f s Set to 22050, f l ≥0, f h Set to f s half of that, and N is set to 2048.

[0071]

[0072] After the triangular frequency filter bank is designed, the feeding signal y(n) is subjected to FFT to convert the time domain signal into the frequency domain. The feeding signal is the feeding sound signal in the audio segment to be tested. As shown in formula (4), k represents the kth spectral line in the frequency domain.

[0073]

[0074] The energy spectrum E(i, k) is obtained by square X(i, k) after FFT, and its expression is calculated as follows:

[0075] E(i,k)=[X(i,k)] 2 (5)

[0076] After that, the obtained energy spectrum passes through M Mel filter banks to obtain the signal energy in each Mel filter S(i, m):

[0077]

[0078] The above method and steps can be used to obtain an M×N matrix containing signal energy information, and the Mel spectrum of the fish feeding signal can be obtained by coloring according to the one-to-one mapping relationship between energy size and color depth.

[0079] In practical applications, the process of extracting the fish feeding depth spectrum feature vector based on STFT specifically includes:

[0080] In the field of digital signal processing, Short-Time Fourier Transform (STFT) is one of the commonly used signal processing methods and plays an important role in the field of time-frequency analysis.

[0081] STFT adds a short-time window function moving along the time axis to the signal. The short-time window captures the non-stationary signal near each moment. At this time, the signal in the short-time window can be regarded as a stationary signal, and the captured results are subjected to Fourier Transform (FT) to obtain the spectrum near each moment, that is, the real-time spectrum.

[0082] The signal processed by STFT has localized characteristics in the time domain and frequency domain, and can be used to analyze the time-frequency characteristics of the signal. STFT increases the time dimension by dividing the non-stationary signal into many frames containing quasi-stationary parts, and uses a window function to reduce the side lobes in the spectrum. As shown in equation (7), s[n] represents the audio signal with a window length of L, and w[t] represents the short-time window function. In this work, the sampling rate is set to 22050, L is set to 2048, the number of hops is set to 512, and the number of color levels expected to be output is 12.

[0083]

[0084] In practical applications, the CQT-based fish feeding depth spectrum feature vector specifically includes:

[0085] The Constant-Q Transform (CQT) uses a logarithmic frequency interval to keep the Q factor constant throughout the spectrum (the Q factor is the ratio of the center frequency to the bandwidth). Compared to the Fourier transform, the Constant-Q transform gives the low-frequency band in the spectrum a relatively high frequency resolution and the high-frequency band in the spectrum a relatively high time resolution; and the window length of the Constant-Q transform changes with the frequency.

[0086] The constant Q transform converts speech from the time domain to the frequency domain. The ratio of the center frequencies of two adjacent components in the constant Q transform remains unchanged. The center frequency of the kth component is shown in the following formula, where f k represents the center frequency of the kth component, f min represents the center frequency of the first component, that is, the tone with the lowest frequency in the entire spectrum, and β represents the number of spectrum lines in each octave. In this paper, β is set to 36, f min is set to 32.7Hz.

[0087]

[0088] The constant Q factor in the kth component is the ratio of the center frequency to the bandwidth. Since Q is a constant, it applies to all components in the spectrum. As shown in the following formula. k+1 -f k represents the bandwidth of the kth component, from which it can be seen that the value of the Q factor is only related to β.

[0089]

[0090] The window length N of the kth frequency band k Varies with frequency and is inversely proportional to the center frequency f of the k filters, N k Determined by the following formula, where f s represents the sampling frequency. In this paper, f s is set to 22050.

[0091]

[0092] Finally, the speech signal x(m) is subjected to constant Q transformation, and the frequency component of the kth octave of the Nth frame after transformation is determined by the following formula. represents the window function, x cqt (k) represents the spectrum parameters after constant Q transformation.

[0093]

[0094] The frequency range of human vocal organs is mostly concentrated in low frequencies. The commonly used time-frequency conversion method for signals is STFT. STFT will have problems such as period truncation at lower frequencies, resulting in low frequency resolution of speech. Compared with STFT, Mel spectrum and CQT provide frequency analysis on a logarithmic scale, which can solve this problem well, provide higher resolution for low frequencies, and more completely reflect the characteristics of the original sound. Based on the diversity of features, we believe that it is feasible to provide different features for classification. Figure 5 The STFT spectrogram, Mel spectrum, CQT spectrum and fusion feature graph are shown respectively. In addition, the Mel spectrum, CQT spectrum and STFT have not been used as part of the CNN model for fish feeding intensity recognition.

[0095] Step 102: Fusing the fish feeding depth spectrum feature vector based on Mel spectrum, the fish feeding depth spectrum feature vector based on CQT, and the fish feeding depth spectrum feature vector based on STFT to generate a fused feature spectrogram.

[0096] Step 103: Input the fused feature spectrogram into a deep convolutional neural network model constructed by historical audio segments corresponding to different types of eating intensities to determine the type of eating intensity corresponding to the audio segment to be tested; the types of eating intensity include "strong", "medium", "weak" and "none".

[0097] In practical applications, the fish feeding depth spectrum feature vector based on Mel spectrum, the fish feeding depth spectrum feature vector based on CQT and the fish feeding depth spectrum feature vector based on STFT are fused and classified using the improved CNN network model.

[0098] In practical applications, the transformed fusion feature spectrogram is used as the input of the pre-trained deep convolutional neural network module, and the improved CNN network is used to complete the task of fish feeding intensity classification. Figure 6 As shown in Figure 1, the SE attention mechanism (Squeeze-and-Excitation Networks) module of the Ghost-Blockneck module in the GhostNet model is replaced with a CA attention mechanism (Coordinate attention) module.

[0099] The present invention uses accuracy, precision, recall and F1-score to evaluate the classification of fish feeding activity intensity (Formula 12-15). Among them, true positive (TP) means that the positive class is judged as the positive class, false positive (FP) means that the negative class is judged as the positive class, false negative (FN) means that the positive class is judged as the negative class, and true negative (TN) represents the number of correctly classified negative samples. The four evaluation indicators are defined as follows:

[0100]

[0101]

[0102]

[0103]

[0104] The present invention first extracts the Mel, CQT and STFT features of feeding speech through the Librosa library, then fuses several different acoustic features, and inputs the fused feeding feature spectrogram into a fine-tuned convolutional neural network to extract deep features and classify them to obtain classification results. This method not only optimizes the fusion of different feeding audio features, but also improves the attention mechanism module of the Ghost-Blockneck module in the GhostNet model. Compared with the existing algorithms, the present invention significantly improves the accuracy of fish feeding sound recognition.

[0105] Embodiment 2

[0106] In order to execute the method corresponding to the above-mentioned embodiment 1 and realize the corresponding functions and technical effects, a fish feeding intensity classification system is provided below.

[0107] A system for classifying the feeding intensity of fish schools, comprising:

[0108] The feature extraction module is used to extract features from the audio band to be tested, and determine the fish feeding depth spectrum feature vector based on Mel spectrum, the fish feeding depth spectrum feature vector based on CQT, and the fish feeding depth spectrum feature vector based on STFT.

[0109] A feature fusion module is used to fuse the fish feeding depth spectrum feature vector based on Mel spectrum, the fish feeding depth spectrum feature vector based on CQT and the fish feeding depth spectrum feature vector based on STFT to generate a fused feature spectrogram.

[0110] The eating intensity type determination module is used to input the fused feature spectrogram into a deep convolutional neural network model constructed by historical audio segments corresponding to different types of eating intensities, and determine the eating intensity type corresponding to the audio segment to be tested; the types of eating intensity include "strong", "medium", "weak" and "none".

[0111] In practical applications, the extraction process of the fish feeding depth spectrum feature vector based on the Mel spectrum specifically includes:

[0112] The triangular frequency filter group setting unit is used to set a plurality of triangular filters within the frequency range of the fish feeding signal to form a triangular frequency filter group; the triangular frequency filter group includes a plurality of bandpass filters; the bandpass filter is a Mel filter; the transfer function of the bandpass filter is: Among them, H m(k) is a bandpass filter, m is the serial number of the Mel filter, M is the number of Mel filters, f(m) is the center frequency of the m-th Mel filter; f(m+1) is the center frequency of the m+1-th Mel filter; and f(m-1) is the center frequency of the m-1-th Mel filter.

[0113] The fast Fourier transform processing unit is used to perform fast Fourier transform processing on the sound signal in the audio segment to be tested by using the triangular frequency filter group, convert the sound signal from the time domain to the frequency domain, and generate a filtered sound signal.

[0114] The energy spectrum determining unit is used to determine the energy spectrum according to the filtered sound signal.

[0115] The signal energy determination unit is used to determine the signal energy in each Mel filter according to the energy spectrum.

[0116] The mel-spectrogram determining unit is used to determine the mel-spectrogram of the fish school feeding sound signal according to the signal energy.

[0117] The fish feeding depth spectrum feature vector extraction unit based on Mel spectrum is used to extract the fish feeding depth spectrum feature vector based on Mel spectrum in the audio segment to be tested according to the Mel spectrum graph.

[0118] In practical applications, the CQT-based fish feeding depth spectrum feature vector extraction process specifically includes:

[0119] A frequency spectrum parameter generating unit after constant Q transformation is used to perform constant Q transformation on the sound signal in the audio segment to be tested to generate frequency spectrum parameters after constant Q transformation; a constant Q transformation spectrum graph generating unit is used to generate a constant Q transformation spectrum graph according to the frequency spectrum parameters; a CQT-based fish feeding depth spectrum feature vector determining unit is used to extract the CQT-based fish feeding depth spectrum feature vector in the audio segment to be tested according to the constant Q transformation spectrum graph.

[0120] Embodiment 3

[0121] An embodiment of the present invention provides an electronic device including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the fish feeding intensity classification method provided in the first embodiment.

[0122] In practical applications, the above electronic device may be a server.

[0123] In practical applications, an electronic device includes: at least one processor, a memory, a bus and a communications interface.

[0124] Wherein: the processor, the communication interface, and the memory communicate with each other via a communication bus.

[0125] Communication interface, used to communicate with other devices.

[0126] The processor is used to execute the program, and specifically can execute the method described in the above embodiment.

[0127] Specifically, the program may include program codes including computer operation instructions.

[0128] The processor may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the electronic device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0129] The memory is used to store programs. The memory may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0130] Based on the description of the above embodiments, the embodiments of the present application provide a storage medium on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method described in any embodiment.

[0131] The fish feeding intensity classification system provided in the embodiments of the present application exists in various forms, including but not limited to:

[0132] (1) Mobile communication devices: These devices are characterized by having mobile communication functions and providing voice and data communications as their main purpose. These terminals include: smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones.

[0133] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access capabilities. These terminals include: PDA, MID and UMPC devices, such as iPad.

[0134] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players (such as iPods), handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0135] (4) Other electronic devices with data interaction functions.

[0136] Thus far, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired results. Additionally, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing may be advantageous.

[0137] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0138] For the convenience of description, the above device is described by being divided into various units according to their functions and described separately. Of course, when implementing the present application, the functions of each unit can be implemented in the same one or more software and / or hardware. It should be understood by those skilled in the art that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0139] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0140] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0141] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0142] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0143] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0144] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM),

[0145] Digital Versatile Disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, disk storage or other magnetic storage devices

[0146] Or any other non-transmission medium that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0147] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0148] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific transactions or implement specific abstract data types. The present application may also be practiced in distributed computing environments where transactions are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0149] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0150] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for classifying the feeding intensity of fish schools, It is characterized in that include: The audio segment to be tested is subjected to feature extraction to determine a fish feeding depth spectrum feature vector based on Mel spectrum, a fish feeding depth spectrum feature vector based on CQT, and a fish feeding depth spectrum feature vector based on STFT; the audio segment to be tested includes a video clip and a sound signal; STFT increases the time dimension by dividing a non-stationary signal into many frames containing quasi-stationary parts, and uses a window function to reduce the side lobes in the spectrum; as shown in formula (1), wherein STFT[f,t] is a fish feeding depth spectrum feature vector based on STFT, s[n] represents an audio signal with a window length of L, and w[t] represents a short-time window function; the sampling rate is set to 22050, L is set to 2048, the number of hops is set to 512, and the number of color levels expected to be output is 12; The constant Q transform converts speech from the time domain to the frequency domain. In the constant Q transform, the ratio of the center frequencies of two adjacent components remains unchanged. The center frequency of the kth component is shown in formula (2), where f k represents the center frequency of the kth component, f min represents the center frequency of the first component, that is, the sound with the lowest frequency in the entire spectrum, β represents the number of spectrum lines in each octave, β is set to 36, f min It is set to 32.7Hz; The constant Q factor in the kth component is the ratio of the center frequency to the bandwidth. Since Q is a constant, it applies to all components in the spectrum, as shown in equation (3), where f k+1 -f k represents the bandwidth of the kth component, and the value of the Q factor is only related to β; The window length N of the kth frequency band k Varies with frequency and is inversely proportional to the center frequency f of the k filters, N k Determined by the following formula, where f s represents the sampling frequency, f s is set to 22050; Finally, the speech signal x(m) is subjected to constant Q transformation, and the frequency component of the kth octave of the Nth frame after transformation is determined by equation (5), where: represents the window function, x cqt (k) represents the spectrum parameters after constant Q transformation; After that, the obtained energy spectrum passes through M Mel filter banks to obtain the signal energy in each Mel filter S(i, m): An M×N-order matrix containing the signal energy information is obtained, and the Mel spectrum of the fish feeding signal can be obtained by coloring it according to the one-to-one mapping relationship between the energy size and the color depth; Fusion of the fish feeding depth spectrum feature vector based on Mel spectrum, the fish feeding depth spectrum feature vector based on CQT and the fish feeding depth spectrum feature vector based on STFT to generate a fused feature spectrogram; The fused feature spectrogram is input into a deep convolutional neural network model constructed by historical audio segments corresponding to different types of feeding intensities to determine the feeding intensity type corresponding to the audio segment to be tested; the feeding intensity types include "strong", "medium", "weak" and "none"; the fused feature spectrogram is used as the input of a pre-trained deep convolutional neural network model, and an improved CNN network is used to complete the task of classifying the feeding intensity of fish schools; wherein, the improved CNN network is improved by replacing the SE attention mechanism module of the Ghost-Blockneck module in the GhostNet model with a CA attention mechanism module.

2. The method for classifying the feeding intensity of a school of fish according to claim 1, It is characterized in that The extraction process of the fish feeding depth spectrum feature vector based on the Mel spectrum specifically includes: A plurality of triangular filters are set within the frequency range of the fish feeding signal to form a triangular frequency filter group; the triangular frequency filter group includes a plurality of bandpass filters; the bandpass filter is a Mel filter; the transfer function of the bandpass filter is: Among them, H m (k) is a bandpass filter, m is the serial number of the Mel filter, M is the number of Mel filters, f(m) is the center frequency of the m-th Mel filter; f(m+1) is the center frequency of the m+1-th Mel filter; f(m-1) is the center frequency of the m-1-th Mel filter; Using the triangular frequency filter group to perform fast Fourier transform processing on the sound signal in the audio segment to be tested, converting the sound signal from the time domain to the frequency domain, and generating a filtered sound signal; determining an energy spectrum according to the filtered sound signal; Determine the signal energy in each Mel filter according to the energy spectrum; Determine a mel-spectrogram of the fish school feeding sound signal according to the signal energy; A spectral feature vector of the fish feeding depth based on the Mel spectrum in the audio segment to be tested is extracted according to the Mel spectrum graph.

3. The method for classifying the feeding intensity of a school of fish according to claim 1, It is characterized in that The CQT-based fish feeding depth spectrum feature vector extraction process specifically includes: Performing a constant-Q transformation on the sound signal in the audio segment to be measured to generate spectrum parameters after the constant-Q transformation; Generate a constant Q transform spectrum diagram according to the spectrum parameters; The CQT-based fish feeding depth spectral feature vector in the audio segment to be tested is extracted according to the constant Q transform spectrum diagram.

4. The method for classifying the feeding intensity of a school of fish according to claim 1, It is characterized in that The process of extracting the fish feeding depth spectrum feature vector based on STFT specifically includes: Adding a short-time window function moving along the time axis to the sound signal in the audio segment to be tested, and intercepting the non-stationary signal at each moment by the short-time window function; the signal in the short-time window is a stationary signal; Performing Fourier transform on the non-stationary signal to generate a time-frequency spectrum at each moment; The STFT-based fish feeding depth spectral feature vector in the audio segment to be tested is extracted according to the time-frequency spectrum.

5. The method for classifying the feeding intensity of a school of fish according to claim 1, It is characterized in that The construction process of the deep convolutional neural network model specifically includes: Obtain historical video clips and historical sound signals of fish before, during and after feeding; Dividing different types of feeding intensities according to the historical video clips, and synchronously cutting the historical sound signals to determine historical audio segments corresponding to different types of feeding intensities; The deep convolutional neural network model is constructed based on the historical audio segments.

6. A fish feeding intensity classification system, It is characterized in that include: The feature extraction module is used to extract features from the audio segment to be tested, and determine the fish feeding depth spectrum feature vector based on Mel spectrum, the fish feeding depth spectrum feature vector based on CQT, and the fish feeding depth spectrum feature vector based on STFT; the audio segment to be tested includes a video clip and a sound signal; STFT increases the time dimension by dividing the non-stationary signal into many frames containing quasi-stationary parts, and uses a window function to reduce the side lobes in the spectrum; as shown in formula (1), wherein STFT[f,t] is the fish feeding depth spectrum feature vector based on STFT, s[n] represents the audio signal with a window length of L, and w[t] represents the short-time window function; the sampling rate is set to 22050, L is set to 2048, the number of hops is set to 512, and the number of color levels expected to be output is 12; The constant Q transform converts speech from the time domain to the frequency domain. In the constant Q transform, the ratio of the center frequencies of two adjacent components remains unchanged. The center frequency of the kth component is shown in formula (2), where f k represents the center frequency of the kth component, f min represents the center frequency of the first component, that is, the sound with the lowest frequency in the entire spectrum, β represents the number of spectrum lines in each octave, β is set to 36, f min It is set to 32.7Hz; The constant Q factor in the kth component is the ratio of the center frequency to the bandwidth. Since Q is a constant, it applies to all components in the spectrum, as shown in equation (3), where f k+1 -f k represents the bandwidth of the kth component, and the value of the Q factor is only related to β; The window length N of the kth frequency band k Varies with frequency and is inversely proportional to the center frequency f of the k filters, N k Determined by the following formula, where f s represents the sampling frequency, f s is set to 22050; Finally, the speech signal x(m) is subjected to constant Q transformation, and the frequency component of the kth octave of the Nth frame after transformation is determined by equation (5), where: represents the window function, x cqt (k) represents the spectrum parameters after constant Q transformation; After that, the obtained energy spectrum passes through M Mel filter banks to obtain the signal energy in each Mel filter S(i, m): An M×N-order matrix containing the signal energy information is obtained, and the Mel spectrum of the fish feeding signal can be obtained by coloring it according to the one-to-one mapping relationship between the energy size and the color depth; A feature fusion module, used for fusing the fish feeding depth spectrum feature vector based on Mel spectrum, the fish feeding depth spectrum feature vector based on CQT and the fish feeding depth spectrum feature vector based on STFT to generate a fused feature spectrogram; A feeding intensity type determination module is used to input the fused feature spectrogram into a deep convolutional neural network model constructed by historical audio segments corresponding to different types of feeding intensities, and determine the feeding intensity type corresponding to the audio segment to be tested; the feeding intensity types include "strong", "medium", "weak" and "none"; the fused feature spectrogram is used as the input of the pre-trained deep convolutional neural network module, and the improved CNN network is used to complete the task of fish feeding intensity classification; wherein, the improved CNN network is improved by replacing the SE attention mechanism module of the Ghost-Blockneck module in the GhostNet model with the CA attention mechanism module.

7. The fish feeding intensity classification system according to claim 6, It is characterized in that The extraction process of the fish feeding depth spectrum feature vector based on the Mel spectrum specifically includes: The triangular frequency filter group setting unit is used to set a plurality of triangular filters within the frequency range of the fish feeding signal to form a triangular frequency filter group; the triangular frequency filter group includes a plurality of bandpass filters; the bandpass filter is a Mel filter; the transfer function of the bandpass filter is: Among them, H m (k) is a bandpass filter, m is the serial number of the Mel filter, M is the number of Mel filters, f(m) is the center frequency of the m-th Mel filter; f(m+1) is the center frequency of the m+1-th Mel filter; f(m-1) is the center frequency of the m-1-th Mel filter; A fast Fourier transform processing unit, used for performing fast Fourier transform processing on the sound signal in the audio segment to be measured by using the triangular frequency filter group, converting the sound signal from the time domain to the frequency domain, and generating a filtered sound signal; an energy spectrum determining unit, configured to determine an energy spectrum according to the filtered sound signal; A signal energy determination unit, used to determine the signal energy in each Mel filter according to the energy spectrum; A mel-spectrogram determination unit, used to determine the mel-spectrogram of the fish school feeding sound signal according to the signal energy; The fish feeding depth spectrum feature vector extraction unit based on Mel spectrum is used to extract the fish feeding depth spectrum feature vector based on Mel spectrum in the audio segment to be tested according to the Mel spectrum graph.

8. The fish feeding intensity classification system according to claim 6, It is characterized in that The CQT-based fish feeding depth spectrum feature vector extraction process specifically includes: A frequency spectrum parameter generating unit after constant Q transformation is used to perform constant Q transformation on the sound signal in the audio segment to be measured to generate frequency spectrum parameters after constant Q transformation; A constant Q transform spectrum diagram generating unit, used for generating a constant Q transform spectrum diagram according to the spectrum parameters; The CQT-based fish feeding depth spectrum feature vector determination unit is used to extract the CQT-based fish feeding depth spectrum feature vector in the audio segment to be tested according to the constant Q transform spectrum diagram.

9. An electronic device, It is characterized in that The electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the method for classifying the feeding intensity of a school of fish as described in any one of claims 1 to 5.

10. A computer-readable storage medium, It is characterized in that It stores a computer program, which, when executed by a processor, implements the method for classifying the feeding intensity of a school of fish as described in any one of claims 1 to 5.