Multi-scale underwater fish detection system and detection method based on attention module

Abstract

The present invention provides a multi-scale underwater fish detection system and method based on an attention module, which relates to the field of underwater detection technology. The present invention performs multi-scale fusion of sonar signals and image information. The depth of the fish school is determined by sonar and an acoustic wave feature vector is generated. An unmanned underwater robot is used to collect low-light images, edge features are generated in a grayscale image, color features are generated in an HSV color space, and the edge features and color features are corrected. The acoustic wave features, edge features, and color features are weightedly fused through feature alignment and attention modules to generate a comprehensive feature vector. Finally, a fish detection model is constructed based on the deep feature vector using a deep learning network to achieve real-time detection of the category and quantity of fish schools. This method utilizes multi-scale information and an attention mechanism to improve the accuracy and reliability of fish school detection.

Description

Technical Field

[0001] The present invention relates to the field of underwater detection technology, and in particular to a multi-scale underwater fish school detection system and a detection method based on an attention module. Background Art

[0002] With the rapid development of aquaculture and marine ecological research, underwater fish detection technology plays an important role in the fields of fishery resource assessment, fish behavior research, and underwater ecological monitoring. At present, common fish detection methods mainly rely on sonar equipment and image processing technology. Sonar equipment can provide the depth and distribution information of fish schools through echo intensity, but its resolution is limited by the frequency of sound wave emission and signal processing capabilities, and it is difficult to accurately describe the morphological characteristics of fish schools. The image-based detection method uses underwater cameras to capture the visual information of fish schools, but in complex underwater environments, due to factors such as insufficient light, high turbidity, and water absorption of light, the image quality is often low, resulting in reduced detection accuracy. In addition, the single data source method in the existing technology faces the problem of incomplete information. How to combine multi-scale features to improve the accuracy of detection has become a difficult problem facing current technology.

[0003] In the prior art, publication number CN114170497B discloses a multi-scale underwater fish detection method based on an attention module. By acquiring underwater fish images and performing enhanced preprocessing on the images, based on the optimized YOLO V4 detection framework, the ECA attention mechanism is added to enable the network to overcome background interference and focus on learning fish body features. The PANet connection is improved to increase multi-scale information and strengthen feature extraction to improve detection accuracy.

[0004] The main problems with the above methods are: they only rely on image data of underwater fish schools for detection. In a complex underwater environment, images are easily disturbed by environmental factors, and it is difficult to accurately extract fish school features even with image enhancement; the feature learning ability is enhanced through the EAC attention mechanism and the optimized YOLO V4 framework, but the basis is still image data processing at a single scale, and feature extraction is limited by the resolution and quality of the image, and the complementarity between different data sources cannot be utilized; when it comes to fish school detection tasks, only the fish body features are learned, and complex tasks such as fish school quantity estimation are not further considered, and the usage scenarios are limited.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0006] The purpose of the present invention is to provide a multi-scale underwater fish detection system and detection method based on an attention module to solve the problems raised in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A multi-scale underwater fish detection system based on an attention module, specifically comprising:

[0009] The equipment installation module is used to fix the sonar vertically downward on the underwater platform, set the frequency to 30kHz, transmit the sonar signal underwater, collect the time delay from the sound wave being sent from the sonar to the sound wave returning to the sonar, and generate the depth of the fish school;

[0010] The acoustic wave feature module is used to receive the reflected acoustic wave signal. The sonar system converts the acoustic wave signal into an electrical signal to generate echo intensity, and generates an acoustic wave feature vector based on the echo intensity.

[0011] An image acquisition module is used to acquire images at the depth of the school of fish detected by the sonar through an unmanned underwater robot equipped with a low-light camera, and to copy the acquired original images into two identical groups of images, one group of images is grayed to generate a first recognition image, and the other group of images is converted from the RGB color space to the HSV color space to generate a second recognition image;

[0012] An edge feature module is used to extract edge pixel points of the first recognition image through a sobel operator, generate an edge intensity feature of the first recognition image based on the gradient amplitude of each edge pixel point, generate an edge density feature of the first recognition image based on the edge intensity feature of the first recognition image and the number of edge pixels, generate an edge feature vector based on the edge intensity feature and the edge density feature, and generate an edge feature retention coefficient and a gradient amplitude compensation coefficient to correct the edge feature vector to generate a corrected edge feature vector;

[0013] A color feature module is used to extract the hue, saturation and brightness of each pixel in the second recognition image, calculate the average value and standard deviation of the hue, saturation and brightness of all pixels, generate the hue feature, saturation feature and brightness feature of the second recognition image based on the average value and standard deviation of the hue, saturation and brightness, generate a color feature vector based on the hue feature, saturation feature and brightness feature, and generate a color feature retention coefficient and a standard deviation compensation coefficient to correct the color feature vector to generate a corrected color feature vector;

[0014] A comprehensive feature module is used to perform linear transformation on the sound wave feature vector, the corrected edge feature vector and the corrected color feature vector through a fully connected layer, map them to the same dimension, generate aligned sound wave feature vectors, edge feature vectors and color feature vectors, splice the aligned feature vectors into a whole, generate the attention weight of each feature vector through a shared multi-layer perceptron with an attention module, and generate a comprehensive feature vector based on the aligned feature vector and the attention weights of each feature vector;

[0015] The model training module is used to build a deep learning network, take the comprehensive feature vector of the known category and number of fish as input, and the category and number of fish as labels to train the fish detection model;

[0016] The output judgment module is used to input the comprehensive feature vector generated based on the real-time detection data into the fish school detection model to generate the category and quantity of the fish school detected in real time.

[0017] Furthermore, the formula for generating the depth of the fish school is:

[0018] ;

[0019] ;

[0020] in, Indicates delay, Indicates the time it takes for the sonar to receive the returning sound wave. Indicates the time it takes for the sonar to transmit sonar signals underwater. Indicates the depth of the fish school, Indicates the speed at which sonar signals travel through water.

[0021] Furthermore, the principle for generating the acoustic wave feature vector is:

[0022] The formula used to generate the echo strength is:

[0023] ;

[0024] in, Indicates the echo strength, Indicates the power of the sound wave signal received by the sonar;

[0025] The acoustic wave eigenvector is:

[0026] ;

[0027] in, represents the acoustic wave feature vector.

[0028] Furthermore, the principle for generating edge feature vectors is:

[0029] The principle for extracting edge pixels of the first recognition image is:

[0030] For each pixel in the first recognition image, the pixel and the neighboring pixel are convolved with the horizontal template and the vertical template of the Prewitt operator to generate the grayscale difference of the pixel in the horizontal and vertical directions. The formula is:

[0031] ;

[0032] ;

[0033] ;

[0034] ;

[0035] in, Represents the horizontal template of the Prewitt operator, Represents the vertical template of the Prewitt operator, Represents the horizontal difference of the pixel. Represents the vertical difference of the pixel. Represents the coordinates of the pixel;

[0036] The gradient amplitude of each pixel is generated based on the grayscale difference in the horizontal and vertical directions. The formula is:

[0037] ;

[0038] in, The coordinates are The gradient amplitude of the pixel point;

[0039] Traverse all the gradient amplitudes, select a gradient amplitude that can most clearly reflect the edge features of the image and set it as the edge threshold. Pixels whose gradient amplitude is greater than the edge threshold are regarded as edge pixels.

[0040] The formula for generating edge strength features is:

[0041] ;

[0042] in, represents the edge strength feature, represents the set of all edge points, Represents the gradient magnitude of the edge point;

[0043] The formula for generating edge density features is:

[0044] ;

[0045] in, represents the edge density feature, Indicates the number of edge points;

[0046] The generated edge feature vector is:

[0047] ;

[0048] in, represents the edge feature vector;

[0049] The formula for generating the corrected edge feature vector is:

[0050] ;

[0051] in, represents the corrected edge feature vector, represents the edge feature retention coefficient, represents the gradient amplitude compensation coefficient, and , , represents the index of edge pixels in the first recognition image, Indicates The gradient amplitude of the edge pixel.

[0052] Furthermore, the principle for generating the color feature vector is:

[0053] The formula for generating the hue feature is:

[0054] ;

[0055] ;

[0056] ;

[0057] ;

[0058] in, Indicates the average hue value. Indicates the first The normalized value of the hue of each pixel. Indicates the first The hue of each pixel, Indicates the number of pixels. Represents the index of the pixel. represents the hue standard deviation, Indicates hue characteristics;

[0059] According to the same calculation method as the hue feature, the saturation feature and lightness feature are calculated, respectively:

[0060] ;

[0061] ;

[0062] in, Represents the saturation characteristic, represents the average saturation value, represents the standard deviation of saturation, Represents brightness characteristics, represents the average brightness, represents the standard deviation of brightness;

[0063] The resulting color feature vector is:

[0064] ;

[0065] in, represents the color feature vector;

[0066] The formula for generating the corrected color feature vector is:

[0067] ;

[0068] ;

[0069] in, Represents the pixel standard deviation combination vector in HSV color space, represents the corrected color feature vector, represents the color feature retention coefficient, represents the standard deviation compensation coefficient, and , .

[0070] Furthermore, the formula for generating the aligned sound wave feature vector, edge feature vector and color feature vector is as follows:

[0071] ;

[0072] ;

[0073] ;

[0074] in, Respectively represent the aligned sound wave feature vector, edge feature vector and color feature vector, represents the acoustic wave feature vector, represents the corrected edge feature vector, represents the corrected color feature vector, Represent the sound wave, edge and color linear transformation matrices respectively, Represent the sound wave, edge and color bias terms respectively.

[0075] Furthermore, the principle for generating the comprehensive feature vector is:

[0076] Concatenate the aligned sound wave feature vector, edge feature vector, and color feature vector into an overall vector:

[0077] ;

[0078] in, represents the overall vector;

[0079] The formula for generating the attention weights of each feature vector in the multi-layer perceptron is:

[0080] ;

[0081] in, represents the attention weight of the ith aligned feature vector, i is the index of the aligned feature vector, and , which correspond to the aligned sound wave feature vector, edge feature vector, and color feature vector respectively. Represents the value of the i-th input to the multilayer perceptron;

[0082] The comprehensive feature vector generated based on the aligned feature vector and the attention weights of each feature vector is:

[0083] ;

[0084] in, represents the comprehensive feature vector.

[0085] The present invention also provides a multi-scale underwater fish detection method based on an attention module, the method is performed by the multi-scale underwater fish detection system based on the attention module, and the specific steps include:

[0086] Step 1: Fix the sonar vertically downward on the underwater platform, set the frequency to 30kHz, emit sonar signals underwater, collect the time delay from the sonar to the return of the sound wave, and generate the depth of the fish school;

[0087] Step 2: Receive the reflected sound wave signal, and the sonar system converts the sound wave signal into an electrical signal to generate echo intensity, and generates a sound wave feature vector based on the echo intensity;

[0088] Step 3: Use an unmanned underwater robot equipped with a low-light camera to collect images at the depth of the fish school detected by the sonar, and copy the collected original images into two identical sets of images. One set of images is gray-scaled to generate a first recognition image, and the other set of images is converted from the RGB color space to the HSV color space to generate a second recognition image.

[0089] Step 4: extract edge pixels of the first recognition image by using the Sobel operator, generate edge intensity features of the first recognition image based on the gradient amplitude of each edge pixel, generate edge density features of the first recognition image based on the edge intensity features of the first recognition image and the number of edge pixels, generate an edge feature vector based on the edge intensity features and the edge density features, generate an edge feature retention coefficient and a gradient amplitude compensation coefficient to correct the edge feature vector, and generate a corrected edge feature vector;

[0090] Step 5: extract the hue, saturation and brightness of each pixel in the second recognition image, calculate the average value and standard deviation of the hue, saturation and brightness of all pixels, generate the hue feature, saturation feature and brightness feature of the second recognition image based on the average value and standard deviation of the hue, saturation and brightness, generate a color feature vector based on the hue feature, saturation feature and brightness feature, generate a color feature retention coefficient and a standard deviation compensation coefficient to correct the color feature vector, and generate a corrected color feature vector;

[0091] Step 6: linearly transform the sound wave feature vector, the corrected edge feature vector and the corrected color feature vector through a fully connected layer, map them to the same dimension, generate aligned sound wave feature vectors, edge feature vectors and color feature vectors, splice the aligned feature vectors into a whole, generate the attention weights of each feature vector through a shared multi-layer perceptron with an attention module, and generate a comprehensive feature vector based on the aligned feature vectors and the attention weights of each feature vector;

[0092] Step 7: Establish a deep learning network, take the comprehensive feature vector of known fish categories and numbers as input, and the fish categories and numbers as labels to train the fish detection model;

[0093] Step 8: Input the comprehensive feature vector generated based on the real-time detection data into the fish school detection model to generate the real-time detected fish school category and quantity.

[0094] Compared with the prior art, the present invention has the following beneficial effects:

[0095] The present invention generates the depth and sound wave characteristics of fish schools through sonar, which makes up for the inaccurate detection problem of single visual detection in low-light and turbid environments, increases the diversity of data, and detects the approximate position and depth of fish schools through sonar in advance, reduces the number of blind image acquisition times, and reduces working energy consumption; when processing the collected images, edge features are generated based on edge intensity and edge density in grayscale images, realizing multi-level characterization of image edge information, thereby deeply mining the morphology of fish schools and improving the recognition ability of fish school contours and boundaries; calculates the average values ​​of hue, saturation and brightness in HSV color space And standard deviation, can comprehensively extract the statistical characteristics of color distribution in the image, so as to accurately describe the overall characteristics and local changes of fish color. Compared with the color analysis method based on RGB, HSV space is more adaptable to lighting changes and underwater environment, and can still extract effective color features when the brightness is insufficient or there is a lot of background interference. The color feature vector and the edge feature vector form a complementary relationship, and the extracted edge feature vector and color feature vector are corrected to improve the detection accuracy. The edge feature mainly describes the morphological characteristics, while the color feature reflects the appearance characteristics. The combination of the two can improve the comprehensive performance of fish detection.

[0096] The present invention also maps the sound wave, edge and color feature vectors to the same dimension through a fully connected layer, solves the problem of inconsistent feature dimensions of different scales, facilitates the effective fusion of subsequent features, introduces a shared multi-layer perceptron to generate attention weights, and can dynamically adjust the importance of each feature, so that the model can assign appropriate weights according to different scenes and fish characteristics, generate a comprehensive feature vector, and contain information on multi-scale features. It is also optimized through dynamic weight allocation and can describe the characteristics of fish schools more comprehensively and efficiently. Compared with single-scale detection, multi-scale fusion combined with the attention mechanism significantly improves the detection accuracy and robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 This is a schematic diagram of the system modules of an embodiment of the present invention;

[0098] Figure 2 The figure is a schematic diagram of the method flow of an embodiment of the present invention. DETAILED DESCRIPTION

[0099] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0100] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0101] Example:

[0102] See also Figure 1 , the present invention provides a technical solution:

[0103] A multi-scale underwater fish detection system based on an attention module, specifically comprising:

[0104] The equipment installation module is used to fix the sonar vertically downward on the underwater platform, set the frequency to 30kHz, transmit the sonar signal underwater, collect the time delay from the sound wave being sent from the sonar to the sound wave returning to the sonar, and generate the depth of the fish school;

[0105] In this embodiment, the formula for generating the depth of the fish school is:

[0106] ;

[0107] ;

[0108] in, Indicates delay, Indicates the time it takes for the sonar to receive the returning sound wave. Indicates the time it takes for the sonar to transmit sonar signals underwater. Indicates the depth of the fish school, Indicates the speed at which sonar signals travel through water.

[0109] The propagation speed of sonar signals in water varies with underwater temperature, salinity and detection depth. Generally, the higher the water temperature, the higher the salinity and the deeper the detection depth, the faster the sonar signal propagates.

[0110] Setting the sonar frequency to a medium frequency of 30kHz can detect schools of fish at a longer distance while having better resolution, making it suitable for the detection of a variety of fish species.

[0111] The acoustic wave feature module is used to receive the reflected acoustic wave signal. The sonar system converts the acoustic wave signal into an electrical signal to generate echo intensity, and generates an acoustic wave feature vector based on the echo intensity.

[0112] In this embodiment, the principle for generating the acoustic wave feature vector is:

[0113] The formula used to generate the echo strength is:

[0114] ;

[0115] in, Indicates the echo strength, Indicates the power of the sound wave signal received by the sonar;

[0116] The echo intensity reflects the size, number and relative position data of the target fish school. The larger the fish school, the more sound waves are reflected and the stronger the echo intensity produced.

[0117] The acoustic wave eigenvector is:

[0118] ;

[0119] in, represents the acoustic wave feature vector.

[0120] An image acquisition module is used to acquire images at the depth of the school of fish detected by the sonar through an unmanned underwater robot equipped with a low-light camera, and to copy the acquired original images into two identical groups of images, one group of images is grayed to generate a first recognition image, and the other group of images is converted from the RGB color space to the HSV color space to generate a second recognition image;

[0121] In this embodiment, the formula for grayscale processing is:

[0122] ;

[0123] in, Represents the gray value of a pixel. Respectively represent the red, green and blue channel values ​​of the pixel;

[0124] The formula for converting an image from RGB color space to HSV color space is:

[0125] The principle of converting an image from RGB color space to HSV color space is:

[0126] For any point in the image, its RGB color space is , the HSV color space is , first convert the R, G, B values ​​to In scope:

[0127] ;

[0128] ;

[0129] ;

[0130] based on generate , the formula based on is:

[0131] ;

[0132] ;

[0133] ;

[0134] in, Represents hue, saturation, and brightness respectively.

[0135] An edge feature module is used to extract edge pixel points of the first recognition image through a sobel operator, generate an edge intensity feature of the first recognition image based on the gradient amplitude of each edge pixel point, generate an edge density feature of the first recognition image based on the edge intensity feature of the first recognition image and the number of edge pixels, generate an edge feature vector based on the edge intensity feature and the edge density feature, and generate an edge feature retention coefficient and a gradient amplitude compensation coefficient to correct the edge feature vector to generate a corrected edge feature vector;

[0136] In this embodiment, the principle for generating edge feature vectors is:

[0137] The principle for extracting edge pixels of the first recognition image is:

[0138] For each pixel in the first recognition image, the pixel and the neighboring pixel are convolved with the horizontal template and the vertical template of the Prewitt operator to generate the grayscale difference of the pixel in the horizontal and vertical directions. The formula is:

[0139] ;

[0140] ;

[0141] ;

[0142] ;

[0143] in, Represents the horizontal template of the Prewitt operator, Represents the vertical template of the Prewitt operator, Represents the horizontal difference of the pixel. Represents the vertical difference of the pixel. Represents the coordinates of the pixel;

[0144] The gradient amplitude of each pixel is generated based on the grayscale difference in the horizontal and vertical directions. The formula is:

[0145] ;

[0146] in, The coordinates are The gradient amplitude of the pixel point;

[0147] Traverse all the gradient amplitudes, select a gradient amplitude that can most clearly reflect the edge features of the image and set it as the edge threshold. Pixels whose gradient amplitude is greater than the edge threshold are regarded as edge pixels.

[0148] The formula for generating edge strength features is:

[0149] ;

[0150] in, represents the edge strength feature, represents the set of all edge points, Represents the gradient magnitude of the edge point;

[0151] The gradient amplitude of all edge points is summed up, and the result is the edge intensity feature of the image;

[0152] The formula for generating edge density features is:

[0153] ;

[0154] in, represents the edge density feature, Indicates the number of edge points;

[0155] The edge density is generated by dividing the edge strength by the area, which reflects the edge strength per unit area. The edge density feature is proportional to the edge strength feature and inversely proportional to the area.

[0156] The generated edge feature vector is:

[0157] ;

[0158] in, represents the edge feature vector;

[0159] The formula for generating the corrected edge feature vector is:

[0160] ;

[0161] in, represents the corrected edge feature vector, represents the edge feature retention coefficient, represents the gradient amplitude compensation coefficient, and , , represents the index of edge pixels in the first recognition image, Indicates The gradient amplitude of the edge pixel.

[0162] Edge feature retention coefficient Used to control the weight of the original edge features in the correction result, the gradient amplitude compensation coefficient It is used to reflect the influence of gradient amplitude on the original edge features. By analyzing the gradient amplitude distribution after edge extraction, the variance of edge gradient is calculated, and then the and , the formula based on is:

[0163] ;

[0164] ;

[0165] ;

[0166] ;

[0167] in, Represents the average value of the gradient amplitude of edge pixels. Represents the variance of the gradient amplitude of edge pixels, represents the edge feature retention coefficient, represents the gradient amplitude compensation coefficient; If it is too large, the correction result is more dependent on the original edge features, which is suitable for the case where the noise is small and the edge extraction is accurate. Small, the correction result depends more on the compensation effect of the gradient amplitude, which is suitable for the situation of large noise or incomplete edge detection. , .

[0168] A color feature module is used to extract the hue, saturation and brightness of each pixel in the second recognition image, calculate the average value and standard deviation of the hue, saturation and brightness of all pixels, generate the hue feature, saturation feature and brightness feature of the second recognition image based on the average value and standard deviation of the hue, saturation and brightness, generate a color feature vector based on the hue feature, saturation feature and brightness feature, and generate a color feature retention coefficient and a standard deviation compensation coefficient to correct the color feature vector to generate a corrected color feature vector;

[0169] The principle for generating color feature vectors is:

[0170] The formula for generating the hue feature is: ;

[0171] ;

[0172] ;

[0173] ;

[0174] in, Indicates the average hue value. Indicates the first The normalized value of the hue of each pixel. Indicates the first The hue of each pixel, Indicates the number of pixels. Represents the index of the pixel. represents the hue standard deviation, Indicates hue characteristics;

[0175] According to the same calculation method as the hue feature, the saturation feature and lightness feature are calculated, respectively:

[0176] ;

[0177] ;

[0178] in, Represents the saturation characteristic, represents the average saturation value, represents the standard deviation of saturation, Represents brightness characteristics, represents the average brightness, represents the standard deviation of brightness;

[0179] The resulting color feature vector is:

[0180] ;

[0181] in, represents the color feature vector;

[0182] The formula for generating the corrected color feature vector is:

[0183] ;

[0184] ;

[0185] in, Represents the pixel standard deviation combination vector in HSV color space, represents the corrected color feature vector, represents the color feature retention coefficient, represents the standard deviation compensation coefficient, and , ;

[0186] Color feature retention coefficient Used to control the weight of the original color feature in the correction result, standard deviation compensation coefficient It is used to adjust the influence of the standard deviation of the hue, saturation and brightness of the pixel points in the HSV color space on the color characteristics; the formula for calculating the color characteristic retention coefficient and the standard deviation compensation coefficient is as follows:

[0187] ;

[0188] ;

[0189] in, represents the color feature retention coefficient, represents the base of natural logarithms, represents the standard deviation compensation coefficient;

[0190] When the standard deviation of the color feature is large, Reduce, reduce the weight of the original color feature, and rely more on the compensation item for correction; The larger the value, the more the correction result depends on the original color features. It is suitable for scenes with uniform color distribution and small lighting changes. The smaller it is, the more the correction result depends on standard deviation compensation, which is suitable for scenes with large lighting changes and diverse color distribution. , .

[0191] A comprehensive feature module is used to perform linear transformation on the sound wave feature vector, the corrected edge feature vector and the corrected color feature vector through a fully connected layer, map them to the same dimension, generate aligned sound wave feature vectors, edge feature vectors and color feature vectors, splice the aligned feature vectors into a whole, generate the attention weight of each feature vector through a shared multi-layer perceptron with an attention module, and generate a comprehensive feature vector based on the aligned feature vector and the attention weights of each feature vector;

[0192] In this embodiment, the formula for generating the aligned sound wave feature vector, edge feature vector and color feature vector is as follows:

[0193] ;

[0194] ;

[0195] ;

[0196] in, Respectively represent the aligned sound wave feature vector, edge feature vector and color feature vector, represent the acoustic wave feature vector, edge feature vector and color feature vector respectively, Represent the sound wave, edge and color linear transformation matrices respectively, Represent the sound wave, edge and color bias terms respectively.

[0197] Linear transformation matrix and the bias term These are parameters learned through the model training process. Before the model starts, the linear transformation matrix is ​​initialized with Xavier, and the bias term is initialized to a zero vector. In the forward propagation, the original feature vector of the input is aligned through the fully connected layer to generate an aligned feature vector. The loss function is established based on the error between the aligned feature vector and the true label. In the back propagation, for each weight matrix and bias term, the gradient of the loss function to these parameters is calculated based on the chain rule, and the parameters are updated based on the Adam optimization algorithm. The linear transformation matrix and bias term are updated through multiple rounds of iterative training until the loss function is minimized, and the linear transformation matrix and bias term corresponding to the minimized loss function are output.

[0198] The principle for generating the comprehensive feature vector is:

[0199] Concatenate the aligned sound wave feature vector, edge feature vector, and color feature vector into an overall vector:

[0200] ;

[0201] in, represents the overall vector;

[0202] The formula for generating the attention weights of each feature vector in the multi-layer perceptron is:

[0203] ;

[0204] in, represents the attention weight of the ith aligned feature vector, i is the index of the aligned feature vector, and , which correspond to the aligned sound wave feature vector, edge feature vector, and color feature vector respectively. Represents the value of the i-th input to the multilayer perceptron;

[0205] The exp function represents the natural exponential function. The exponential function amplifies the difference between input values ​​while ensuring that the output is positive, so that larger values ​​have higher weights in the softmax output, thereby highlighting important features.

[0206] The comprehensive feature vector generated based on the aligned feature vector and the attention weights of each feature vector is:

[0207] ;

[0208] in, represents the comprehensive feature vector.

[0209] The model training module is used to build a deep learning network, take the comprehensive feature vector of the known category and number of fish as input, and the category and number of fish as labels to train the fish detection model;

[0210] In this embodiment, the result of the deep learning network is:

[0211] Input layer: contains 1 neuron, which is used to receive the input comprehensive feature vector;

[0212] The first hidden layer: contains 256 neurons, activated by the ReLU function;

[0213] The second hidden layer: contains 128 neurons, activated by the ReLU function;

[0214] Classification output layer: The number of neurons is equal to the number of fish categories, and is activated by the softmax function;

[0215] Regression output layer: contains 1 neuron, which is used to output the number of fish.

[0216] The output judgment module is used to input the comprehensive feature vector generated based on the real-time detection data into the fish school detection model to generate the category and quantity of the fish school detected in real time.

[0217] See also Figure 2 The present invention also provides a multi-scale underwater fish detection method based on an attention module, which is performed by the multi-scale underwater fish detection system based on an attention module, and specifically includes:

[0218] Step 1: Fix the sonar vertically downward on the underwater platform, set the frequency to 30kHz, emit sonar signals underwater, collect the time delay from the sonar to the return of the sound wave, and generate the depth of the fish school;

[0219] Step 2: Receive the reflected sound wave signal, and the sonar system converts the sound wave signal into an electrical signal to generate echo intensity, and generates a sound wave feature vector based on the echo intensity;

[0220] Step 3: Use an unmanned underwater robot equipped with a low-light camera to collect images at the depth of the fish school detected by the sonar, and copy the collected original images into two identical sets of images. One set of images is gray-scaled to generate a first recognition image, and the other set of images is converted from the RGB color space to the HSV color space to generate a second recognition image.

[0221] Step 4: extract edge pixels of the first recognition image by using the Sobel operator, generate edge intensity features of the first recognition image based on the gradient amplitude of each edge pixel, generate edge density features of the first recognition image based on the edge intensity features of the first recognition image and the number of edge pixels, generate an edge feature vector based on the edge intensity features and the edge density features, generate an edge feature retention coefficient and a gradient amplitude compensation coefficient to correct the edge feature vector, and generate a corrected edge feature vector;

[0222] Step 5: extract the hue, saturation and brightness of each pixel in the second recognition image, calculate the average value and standard deviation of the hue, saturation and brightness of all pixels, generate the hue feature, saturation feature and brightness feature of the second recognition image based on the average value and standard deviation of the hue, saturation and brightness, generate a color feature vector based on the hue feature, saturation feature and brightness feature, generate a color feature retention coefficient and a standard deviation compensation coefficient to correct the color feature vector, and generate a corrected color feature vector;

[0223] Step 6: linearly transform the sound wave feature vector, the corrected edge feature vector and the corrected color feature vector through a fully connected layer, map them to the same dimension, generate aligned sound wave feature vectors, edge feature vectors and color feature vectors, splice the aligned feature vectors into a whole, generate the attention weights of each feature vector through a shared multi-layer perceptron with an attention module, and generate a comprehensive feature vector based on the aligned feature vectors and the attention weights of each feature vector;

[0224] Step 7: Establish a deep learning network, take the comprehensive feature vector of known fish categories and numbers as input, and the fish categories and numbers as labels to train the fish detection model;

[0225] Step 8: Input the comprehensive feature vector generated based on the real-time detection data into the fish school detection model to generate the real-time detected fish school category and quantity.

[0226] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0227] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. Those skilled in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software methods depends on the specific application and design constraints of the technical solution.

[0228] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0229] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A multi-scale underwater fish detection system based on attention module, characterized in that: Specifically include: The equipment installation module is used to fix the sonar vertically downward on the underwater platform, set the frequency to 30kHz, transmit the sonar signal underwater, collect the time delay from the sound wave being sent from the sonar to the sound wave returning to the sonar, and generate the depth of the fish school; The acoustic wave feature module is used to receive the reflected acoustic wave signal. The sonar system converts the acoustic wave signal into an electrical signal to generate echo intensity, and generates an acoustic wave feature vector based on the echo intensity. An image acquisition module is used to acquire images at the depth of the school of fish detected by the sonar through an unmanned underwater robot equipped with a low-light camera, and to copy the acquired original images into two identical groups of images, one group of images is grayed to generate a first recognition image, and the other group of images is converted from the RGB color space to the HSV color space to generate a second recognition image; An edge feature module is used to extract edge pixel points of the first recognition image through a sobel operator, generate an edge intensity feature of the first recognition image based on the gradient amplitude of each edge pixel point, generate an edge density feature of the first recognition image based on the edge intensity feature of the first recognition image and the number of edge pixels, generate an edge feature vector based on the edge intensity feature and the edge density feature, and generate an edge feature retention coefficient and a gradient amplitude compensation coefficient to correct the edge feature vector to generate a corrected edge feature vector; A color feature module is used to extract the hue, saturation and brightness of each pixel in the second recognition image, calculate the average value and standard deviation of the hue, saturation and brightness of all pixels, generate the hue feature, saturation feature and brightness feature of the second recognition image based on the average value and standard deviation of the hue, saturation and brightness, generate a color feature vector based on the hue feature, saturation feature and brightness feature, and generate a color feature retention coefficient and a standard deviation compensation coefficient to correct the color feature vector to generate a corrected color feature vector; A comprehensive feature module is used to perform linear transformation on the sound wave feature vector, the corrected edge feature vector and the corrected color feature vector through a fully connected layer, map them to the same dimension, generate aligned sound wave feature vectors, edge feature vectors and color feature vectors, splice the aligned feature vectors into a whole, generate the attention weight of each feature vector through a shared multi-layer perceptron with an attention module, and generate a comprehensive feature vector based on the aligned feature vector and the attention weights of each feature vector; The model training module is used to build a deep learning network, take the comprehensive feature vector of the known category and number of fish as input, and the category and number of fish as labels to train the fish detection model; The output judgment module is used to input the comprehensive feature vector generated based on the real-time detection data into the fish school detection model to generate the category and quantity of the fish school detected in real time.

2. The multi-scale underwater fish detection system based on the attention module according to claim 1, characterized in that: The formula for generating the depth of the fish school in the equipment installation module is: ； ； in, Indicates delay, Indicates the time it takes for the sonar to receive the returning sound wave. Indicates the time it takes for the sonar to transmit sonar signals underwater. Indicates the depth of the fish school, Indicates the speed at which sonar signals travel through water.

3. The multi-scale underwater fish detection system based on the attention module according to claim 1, characterized in that: The principle for generating the acoustic wave feature vector in the acoustic wave feature module is: The formula used to generate the echo strength is: ； in, Indicates the echo strength, Indicates the power of the sound wave signal received by the sonar; The acoustic wave eigenvector is: ； in, represents the acoustic wave feature vector.

4. The multi-scale underwater fish detection system based on the attention module according to claim 1, characterized in that: The principle for generating edge feature vectors in the edge feature module is: The principle for extracting edge pixels of the first recognition image is: For each pixel in the first recognition image, the pixel and the neighboring pixel are convolved with the horizontal template and the vertical template of the Prewitt operator to generate the grayscale difference of the pixel in the horizontal and vertical directions. The formula is: ； ； ； ； in, Represents the horizontal template of the Prewitt operator, Represents the vertical template of the Prewitt operator, Represents the horizontal difference of the pixel. Represents the vertical difference of the pixel. Represents the coordinates of the pixel; The gradient amplitude of each pixel is generated based on the grayscale difference in the horizontal and vertical directions. The formula is: ； in, The coordinates are The gradient amplitude of the pixel point; Traverse all the gradient amplitudes, select a gradient amplitude that can most clearly reflect the edge features of the image and set it as the edge threshold. Pixels whose gradient amplitude is greater than the edge threshold are regarded as edge pixels. The formula for generating edge strength features is: ； in, represents the edge strength feature, represents the set of all edge points, Represents the gradient magnitude of the edge point; The formula for generating edge density features is: ； in, represents the edge density feature, Indicates the number of edge points; The generated edge feature vector is: ； in, represents the edge feature vector; The formula for generating the corrected edge feature vector is: ； in, represents the corrected edge feature vector, represents the edge feature retention coefficient, represents the gradient amplitude compensation coefficient, and , , represents the index of edge pixels in the first recognition image, Indicates The gradient amplitude of the edge pixel.

5. The multi-scale underwater fish detection system based on the attention module according to claim 1, characterized in that: The principle for generating the color feature vector in the color feature module is: The formula for generating the hue feature is: ； ； ； ； in, Indicates the average hue value. Indicates the first The normalized value of the hue of each pixel. Indicates the first The hue of each pixel, Indicates the number of pixels. Represents the index of the pixel. represents the hue standard deviation, Indicates hue characteristics; According to the same calculation method as the hue feature, the saturation feature and lightness feature are calculated, respectively: ； ； in, Represents the saturation characteristic, represents the average saturation value, represents the standard deviation of saturation, Represents brightness characteristics, represents the average brightness, represents the standard deviation of brightness; The resulting color feature vector is: ； in, represents the color feature vector; The formula for generating the corrected color feature vector is: ； ； in, Represents the pixel standard deviation combination vector in HSV color space, represents the corrected color feature vector, represents the color feature retention coefficient, represents the standard deviation compensation coefficient, and , .

6. The multi-scale underwater fish detection system based on the attention module according to claim 1, characterized in that: The formula for generating the aligned sound wave feature vector, edge feature vector and color feature vector in the comprehensive feature module is: ； ； ； in, Respectively represent the aligned sound wave feature vector, edge feature vector and color feature vector, represents the acoustic wave feature vector, represents the corrected edge feature vector, represents the corrected color feature vector, Represent the sound wave, edge and color linear transformation matrices respectively, Represent the sound wave, edge and color bias terms respectively.

7. The multi-scale underwater fish detection system based on the attention module according to claim 6, characterized in that: The principle for generating the comprehensive feature vector is: Concatenate the aligned sound wave feature vector, edge feature vector, and color feature vector into an overall vector: ； in, represents the overall vector; The formula for generating the attention weights of each feature vector in the multi-layer perceptron is: ； in, represents the attention weight of the ith aligned feature vector, i is the index of the aligned feature vector, and , which correspond to the aligned sound wave feature vector, edge feature vector, and color feature vector respectively. Represents the value of the i-th input to the multilayer perceptron; The comprehensive feature vector generated based on the aligned feature vector and the attention weights of each feature vector is: ； in, represents the comprehensive feature vector.

8. A multi-scale underwater fish detection method based on an attention module, characterized in that: The method is performed by the multi-scale underwater fish detection system based on the attention module according to any one of claims 1 to 7, and the specific steps include: Step 1: Fix the sonar vertically downward on the underwater platform, set the frequency to 30kHz, emit sonar signals underwater, collect the time delay from the sonar to the return of the sound wave, and generate the depth of the fish school; Step 2: Receive the reflected sound wave signal, and the sonar system converts the sound wave signal into an electrical signal to generate echo intensity, and generates a sound wave feature vector based on the echo intensity; Step 3: Use an unmanned underwater robot equipped with a low-light camera to collect images at the depth of the fish school detected by the sonar, and copy the collected original images into two identical sets of images. One set of images is gray-scaled to generate a first recognition image, and the other set of images is converted from the RGB color space to the HSV color space to generate a second recognition image. Step 4: extract edge pixels of the first recognition image by using the Sobel operator, generate edge intensity features of the first recognition image based on the gradient amplitude of each edge pixel, generate edge density features of the first recognition image based on the edge intensity features of the first recognition image and the number of edge pixels, generate an edge feature vector based on the edge intensity features and the edge density features, generate an edge feature retention coefficient and a gradient amplitude compensation coefficient to correct the edge feature vector, and generate a corrected edge feature vector; Step 5: extract the hue, saturation and brightness of each pixel in the second recognition image, calculate the average value and standard deviation of the hue, saturation and brightness of all pixels, generate the hue feature, saturation feature and brightness feature of the second recognition image based on the average value and standard deviation of the hue, saturation and brightness, generate a color feature vector based on the hue feature, saturation feature and brightness feature, generate a color feature retention coefficient and a standard deviation compensation coefficient to correct the color feature vector, and generate a corrected color feature vector; Step 6: linearly transform the sound wave feature vector, the corrected edge feature vector and the corrected color feature vector through a fully connected layer, map them to the same dimension, generate aligned sound wave feature vectors, edge feature vectors and color feature vectors, splice the aligned feature vectors into a whole, generate the attention weights of each feature vector through a shared multi-layer perceptron with an attention module, and generate a comprehensive feature vector based on the aligned feature vectors and the attention weights of each feature vector; Step 7: Establish a deep learning network, take the comprehensive feature vector of known fish categories and numbers as input, and the fish categories and numbers as labels to train the fish detection model; Step 8: Input the comprehensive feature vector generated based on the real-time detection data into the fish school detection model to generate the real-time detected fish school category and quantity.

Images