A recognition method for overlapping fish targets based on an improved yolov5 model

Abstract

The present invention discloses a method for identifying overlapping fish targets based on an improved Yolov5 model. Yolov5 is improved by introducing a self-attention layer (CBAM) before each of the three layers used in the pyramid structure. The activation function in the Conv layer is changed from SiLU to FReLU, which can extract two-dimensional spatial information. The Kalman method is used to establish the state change relationship between consecutive frames. A parameter variable distribution generated by the time series is used to assist in prediction. The prediction information for the next frame helps determine the overlap of fish, thereby achieving real-time identification of overlapping targets. The present invention utilizes a differential model of motion and a time series processing method to enhance the interpretability of the algorithm. The processing steps are fine-tuned for underwater scenes, achieving excellent detection results for overlapping underwater targets.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence image recognition, and in particular to a method for identifying overlapping fish targets based on an improved YOLOv5 model. Background Art

[0002] Many cities in my country have aquariums, housing a wide variety of underwater animals. Introducing a camera system that effectively identifies and labels fish within its field of view in real time can lower the learning curve for visitors and enhance their experience. Artificial intelligence is currently widely used in the field of object detection, with the YOLO series of algorithms recognized as leading. YOLOv5, consisting of a backbone and a head, enables end-to-end detection. However, while direct object recognition technology can achieve the selection and labeling of aquarium fish, it also faces challenges such as difficulty in underwater object recognition and the problem of overlapping fish.

[0003] Due to the complexity of underwater environments and the diversity of fish, existing convolutional neural network recognition results are less than ideal in terms of accuracy and real-time performance. The YOLO model can extract and train features from datasets, and learn and predict samples. However, due to the movement and living habits of fish, multiple fish often overlap, affecting recognition performance.

[0004] Existing object detection algorithms primarily include traditional object detection algorithms and deep learning-based object detection algorithms. Deep learning-based object detection algorithms can be further divided into anchor-based and anchor-free methods. Traditional object detection algorithms primarily rely on manual feature extraction, which suffers from drawbacks such as difficulty in feature extraction, lack of versatility, and time-consuming performance. Deep learning-based object detection algorithms offer high accuracy and robustness. Anchor-based methods include one-stage and two-stage detection algorithms, with two-stage algorithms offering higher accuracy and one-stage algorithms offering faster performance. Current anchor-free methods suffer from drawbacks such as extreme imbalance between positive and negative samples, semantic ambiguity (overlapping center points of two objects), and unstable detection results. For example, the detection of overlapping fish tails based on Faster-RCNN was improved through image enhancement and an improved Faster-RCNN network. However, due to the proneness of underwater images to color shift and low brightness, the detection success rate is low. Summary of the Invention

[0005] In order to solve the problem that the existing technology cannot efficiently distinguish and monitor the status and category of fish in the aquarium management and teaching process, the present invention provides a method for identifying overlapping fish targets based on an improved YOLOv5 model. The specific technical solution is as follows:

[0006] A method for identifying overlapping fish targets based on an improved yolov5 model comprises the following steps:

[0007] Step 1: Use a camera with a depth sensor to collect RGBD information of underwater video, extract images from the video, and complete the image information collection of the detection target;

[0008] Step 2: Use the Sea-Thru algorithm to reconstruct the underwater scene and repair the underwater image. Specifically, it uses the known distance to estimate the backscattering using a dark channel prior; then uses the local spatial average color to estimate the distance-dependent attenuation coefficient;

[0009] Step 3: Build and train an improved Yolov5 model. The improved Yolov5 model introduces a self-attention layer CBAM before each of the three layers used in the pyramid structure, and changes the activation function in the Conv layer from SiLU to FreLU that can extract two-dimensional spatial information. The input of the improved Yolov5 model is the restored underwater image, and the output is the detection box information.

[0010] Step 4: Sample the video stream at a certain frame sampling rate to obtain a single frame image, and after repairing it according to step 2, input the trained improved yolov5 model. The model outputs the information of the detection frame and takes the difference of the information of the detection frame to obtain the state vector of each target. Among them, x, y, w, and h are the coordinates of the center of mass position and the width and height of the rectangular candidate box respectively. is the difference between the corresponding states of two adjacent frames;

[0011] Step 5: For each frame, use the idea of Kalman filtering to track two consecutive frames of images to obtain the optimal prediction value of each target state vector;

[0012] Step 6: Assume that the landing point of the optimal prediction value of the same target between two adjacent frames satisfies the normal distribution, and use the difference between the prediction value and the detection value obtained in the previous iteration as a sample to calculate the variance σ of the normal distribution 2 , in order to calculate the spatial distribution law of each landing point;

[0013] Step 7: Determine the probability of overlap based on the degree of overlap of the spatial distribution laws of different landing points, and use the morphological characteristics of the overlap as the overlap criterion; if overlap occurs, the multi-target result predicted in step 5 shall prevail, and the output result of the trained improved yolov5 model shall be adjusted according to the final state vector; if no overlap occurs, the detection value of the output of the improved yolov5 model shall prevail.

[0014] Furthermore, the step five includes the following sub-steps:

[0015] (5.1) The information of the detection box output by the trained improved yolov5 model is used as the detection value, and the optimal prediction value of the state vector of the previous frame is used With the input vector u t-1 Substitution Calculate the predicted value of the current frame Where F is the state transfer matrix, B is the control matrix, and Δt is the time between two adjacent frames;

[0016]

[0017] (5.2) According to P t - =FP t-1 F T +Q uses the optimal estimate covariance matrix P of the previous frame t-1 And the hyperparameter process noise variance matrix Q introduces the covariance matrix P of the current frame t - ;

[0018] (5.3) According to K t =P t - H T (HP t - H T +R) -1 , using the covariance matrix P of the current frame t - and the observation noise variance matrix R to derive the Kalman gain K t ; H is the gain matrix;

[0019] (5.4) According to Use predicted value With the observed value z t Derive the optimal state vector estimate As the final state vector, and update the covariance matrix P of the current frame t - ;

[0020] (5.5) According to the covariance matrix P of the current frame t - and the Kalman gain K t , update the optimal covariance matrix P of the current frame t =(IK t H)P t - , used to calculate the covariance matrix of the next frame;

[0021] (5.6) The predicted value calculated from the previous frame and the detection value corresponding to the detection frame output by the trained improved yolov5 model are both normalized and then subjected to inner product operation. The inner product operation value is compared with the threshold. When the inner product operation value is greater than the threshold, the predicted value and the detection value are similar and are the same target; otherwise, they are different targets, thereby achieving target tracking.

[0022] Furthermore, the morphological feature is: a detection value is close to x and y of multiple prediction vectors, while w and h are greater than a single prediction vector.

[0023] Furthermore, in step seven, if overlap occurs, the multi-target result predicted in step five shall prevail, and the output result of the trained improved yolov5 model shall be adjusted as follows according to the final state vector:

[0024] For detection boxes whose distance d between the detection box edge and the image edge is less than 10% of the detection box width or height L, reduce non-maximum suppression NMS, specifically:

[0025] NMS=NMS×f(d / L)

[0026] f(d / L)=Ae -((d / L) / B)lnA .

[0027] Furthermore, A=0.1, B=0.5.

[0028] The beneficial effects of the present invention are as follows:

[0029] The present invention is a method for identifying overlapping fish targets based on an improved YOLOv5 model. First, the YOLOv5 model is improved by introducing a self-attention layer and a FReLU activation function that can effectively extract two-dimensional spatial information. At the same time, the Kalman method is used to establish a state change relationship between consecutive frames, and auxiliary prediction is performed based on the distribution of parameter variables generated by the time series, thereby achieving real-time identification of overlapping targets and outputting the position and classification information of the target to be detected. At the same time, when using the Kalman method to establish the state change relationship between consecutive frames, a motion differential model and a time series processing method are applied to enhance the interpretability of the entire identification method, utilize information between adjacent frames, and thereby improve the recognition effect of overlapping targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The flowchart of the identification method for overlapping fish targets based on the improved yolov5 model of the present invention.

[0031] Figure 2 Schematic diagram of the improved yolov5 model.

[0032] Figure 3Schematic diagram of PReLU used in the present invention. DETAILED DESCRIPTION

[0033] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments, and the purpose and effects of the present invention will become more apparent. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0034] This implementation captures the captured video stream at a fixed frame rate, uses the Kalman method to track objects between adjacent frames, calculates the probability distribution of object locations using time series statistics, and combines morphological criteria to detect overlap. This not only enhances recognition performance in overlapping object scenarios and improves information utilization, but also provides better interpretability.

[0035] Step 1: Use a camera with a depth sensor to collect RGBD information of underwater video, extract images from the video, and complete the image information collection of the detection target;

[0036] As one implementation method, a probe with a depth sensor arranged in an aquarium is directly used to collect underwater videos.

[0037] Step 2: Use the Sea-Thru algorithm to reconstruct the underwater scene and repair the underwater image. Specifically, it includes: using the known distance to estimate the backscattering in a dark channel prior manner; then using the local spatial average color to estimate the distance-dependent attenuation coefficient.

[0038] Underwater images are preprocessed using the Sea-Thru algorithm. Since the signal attenuation coefficient is non-uniform in the entire underwater scene, it depends on the distance and reflectivity of the object, and the coefficient that controls backscattering is different from the coefficient that controls signal attenuation, that is:

[0039] I c =D c +B c

[0040] Among them, I c represents the color distorted image captured by the camera, D c Indicates distance information, B c Indicates the backscattering information caused by the reflection of light from suspended particles in water. c and B c Distance attenuation coefficient and the backscatter coefficient Decision, that is

[0041]

[0042] Where z is the distance between the camera and the object, B when z = ∞ c 、J c is the unattenuated scene captured by the camera at the same position without attenuation along the z direction. D ={z,ρ,E,S c ,β},v B ={E,S c , b, β}, z is the distance, ρ is the reflectivity, E is the spectrum of the ambient light, D c is the spectral response of the camera, b and β are the physical scattering and beam attenuation coefficients of water, and they are all functions of wavelength λ.

[0043] Based on a given RGBD image, the backscattering is estimated using a known distance map with a dark channel prior. Then, the distance-dependent attenuation coefficient is estimated for the input image using the local spatial average color. This allows the underwater scene to be reconstructed and the image to be restored. For more information, see Akkaynak D, Treibitz T. Sea-thru: A method for removing water from underwater images [C] / / Proc of the IEEE IntConf on CVPR: Hawaii, 2019: 1682-1691.

[0044] Compared with models based on algorithms such as DCP (dark channel prior), sea-thru can more accurately estimate B_c (backscattered information) and J_c (real scene without attenuation), thereby better reconstructing underwater scenes.

[0045] Step 3: Construct and train an improved yolov5 model. The improved yolov5 model introduces a self-attention layer CBAM before each of the three layers used in the pyramid structure, and changes the activation function in the Conv layer from SiLU to FreLU that can extract two-dimensional spatial information; the input of the improved yolov5 model is the restored underwater image, and the output is the detection box information.

[0046] Extract the target image from the restored underwater image. The target image is represented by an image containing the target to be detected. Annotate the target image to generate an annotation file in the improved Yolov5 model training format. The annotation file is divided into a data set, a test set, and a validation set.

[0047] like Figure 2As shown in the figure, the improved YOLOv5 model introduces a self-attention layer (CBAM) before each of the three layers of the pyramid structure. This allows the network to focus more on important features and suppress unnecessary features, thereby better extracting key information from the image and passing it to the next layer and the upper layers of the pyramid. Specifically, the 4th, 6th, and 9th layers of the original network structure are the layers that input the pyramid structure. Adding a self-attention layer (CBAM) before these three layers allows for better extraction of useful information.

[0048] like Figure 3 As shown in the improved YOLOv5 model, the activation function in the Conv layer is changed from SiLU to FReLU, which can extract two-dimensional spatial information. The FReLU activation function expression is where x c，i，j Represents the value of the c channel, (i, j), Indicates x c，i，j The pooling window centered on is the pooling window coefficient. Because the spatial condition overhead added by the FReLU activation function is almost negligible, the FReLU function can effectively extract two-dimensional spatial information, thereby better completing visual tasks.

[0049] The improved YOLOv5 model is trained using the dataset and validation set. The initial weights are the weights of the crowdhuman training set training results, which can be downloaded from deepakcrk / yolov5-crowdhuman: Head and Person detection using yolov5. Detection from crowd. (github.com). After training, the weighted model is obtained.

[0050] The improved YOLOv5 model is verified using the dataset and the validation set to obtain the weighted model.

[0051] Step 4: Sample the video stream at a certain frame sampling rate to obtain a single frame image, and after repairing it according to step 2, input the trained improved yolov5 model. The model outputs the information of the detection frame and takes the difference of the information of the detection frame to obtain the state vector of each target. Among them, x, y, w, and h are the coordinates of the center of mass position and the width and height of the rectangular candidate box respectively. is the difference between the corresponding states of two adjacent frames.

[0052] Step 5: For each frame, use the Kalman filter to track two consecutive frames to obtain the optimal prediction value of each target state vector. This includes the following sub-steps:

[0053] (5.1) The information of the detection box output by the trained improved yolov5 model is used as the detection value, and the optimal prediction value of the state vector of the previous frame is used With the input vector u t-1 Substitution Calculate the predicted value of the current frame Where F is the state transfer matrix, B is the control matrix, and Δt is the time between two adjacent frames;

[0054]

[0055] (5.2) According to P t - =FP t-1 F T +Q uses the optimal estimate covariance matrix P of the previous frame t-1 And the hyperparameter process noise variance matrix Q introduces the covariance matrix P of the current frame t - ;

[0056] (5.3) According to K t =P t - H T (HP t - H T +R) -1 , using the covariance matrix P of the current frame t - and the observation noise variance matrix R to derive the Kalman gain K t ; H is the gain matrix;

[0057] (5.4) According to Use predicted value With the observed value z t Derive the optimal state vector estimate As the final state vector, and update the covariance matrix P of the current frame t - ;

[0058] (5.5) According to the covariance matrix P of the current frame t - and the Kalman gain K t , update the optimal covariance matrix P of the current frame t =(IK t H)P t - , used to calculate the covariance matrix of the next frame;

[0059] (5.6) The predicted value calculated from the previous frame and the detection value corresponding to the detection frame output by the trained improved yolov5 model are both normalized and then subjected to inner product operation. The inner product operation value is compared with the threshold. When the inner product operation value is greater than the threshold, the predicted value and the detection value are similar and are the same target; otherwise, they are different targets, thereby achieving target tracking.

[0060] Step 6: Assume that the landing point of the optimal prediction value of the same target between two adjacent frames satisfies the normal distribution, and use the difference between the prediction value and the detection value obtained in the previous iteration as a sample to calculate the variance σ of the normal distribution 2 , in order to calculate the spatial distribution law of each landing point.

[0061] At the beginning of the run, due to lack of data, you can choose not to calculate the relevant parameters first, or calculate σ after a delay. 2 .

[0062] Step 7: Determine the probability of overlap based on the degree of overlap of the spatial distribution laws of different landing points, and use the morphological features of the overlap (for example, a detection value is close to the x and y of multiple prediction vectors, while w and h are greater than a single prediction vector) as the overlap criterion; if no overlap occurs, the detection value output by the improved yolov5 model shall prevail; if overlap occurs, the multi-target result predicted in step 5 shall prevail, and the output result of the trained improved yolov5 model shall be adjusted as follows according to the final state vector:

[0063] For detection boxes whose distance d between the detection box edge and the image edge is less than 10% of the detection box width or height L, reduce non-maximum suppression NMS, specifically:

[0064] NMS=NMS×f(d / L)

[0065] As one implementation method, A=0.1, B=0.5.

[0066] The distribution law obeys the two-dimensional normal distribution N(x,μ x ,σ x ,y,μ y ,σ y ), where x, y are the coordinates of the center position at the current moment, μ x With μ y is the predicted value of the position coordinate at the next moment, and the variance σ in the x and y directions x and σ y The calculation method is as above. Then the probability of the distribution at the coordinate point (x′, y′) is

[0067]

[0068] Select an appropriate threshold according to the specific application scenario as the probability criterion for overlap: if the probability that two targets appear at the coordinate point (x′, y′) at the same time exceeds the preset threshold, overlap is determined.

[0069] Those skilled in the art will understand that the foregoing descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art will still be able to modify the technical solutions described in the foregoing examples or substitute equivalents for some of the technical features therein. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the invention shall be included within the scope of protection of the invention.

Claims

1. A method for identifying overlapping fish targets based on an improved yolov5 model, characterized in that: The method comprises the following steps: Step 1: Use a camera with a depth sensor to collect RGBD information of underwater video, extract images from the video, and complete the image information collection of the detection target; Step 2: Use the Sea-Thru algorithm to reconstruct the underwater scene and repair the underwater image. Specifically, it uses the known distance to estimate the backscattering using a dark channel prior; then uses the local spatial average color to estimate the distance-dependent attenuation coefficient; Step 3: Build and train an improved Yolov5 model. The improved Yolov5 model introduces a self-attention layer CBAM before each of the three layers used in the pyramid structure, and changes the activation function in the Conv layer from SiLU to FreLU to extract two-dimensional spatial information. The input of the improved Yolov5 model is the restored underwater image, and the output is the detection box information. Step 4: Sample the video stream at the frame sampling rate to obtain a single frame image, and after repairing it according to step 2, input the trained improved yolov5 model. The model outputs the information of the detection frame and takes the difference of the information of the detection frame to obtain the state vector of each target. Among them, x, y, w, and h are the coordinates of the center of mass position and the width and height of the rectangular candidate box respectively. is the difference between the corresponding states of two adjacent frames; Step 5: For each frame, use the idea of Kalman filtering to track two consecutive frames of images to obtain the optimal prediction value of each target state vector; Step 6: Assume that the landing point of the optimal prediction value of the same target between two adjacent frames satisfies the normal distribution, and use the difference between the prediction value and the detection value obtained in the previous iteration as a sample to calculate the variance σ of the normal distribution 2 , in order to calculate the spatial distribution law of each landing point; Step 7: Determine the probability of overlap based on the degree of overlap of the spatial distribution laws of different landing points, and use the morphological characteristics of the overlap as the overlap criterion; if overlap occurs, the multi-target result predicted in step 5 shall prevail, and the output result of the trained improved yolov5 model shall be adjusted according to the final state vector; if no overlap occurs, the detection value output by the improved yolov5 model shall prevail; The step five includes the following sub-steps: (5.1) The information of the detection box output by the trained improved yolov5 model is used as the detection value, and the optimal prediction value of the state vector of the previous frame is used With the input vector u t-1 Substitution Calculate the predicted value of the current frame Where F is the state transfer matrix, B is the control matrix, and Δt is the time between two adjacent frames; (5.2) According to P t - =FP t-1 F T +Q uses the optimal estimate covariance matrix P of the previous frame t-1 And the hyperparameter process noise variance matrix Q introduces the covariance matrix P of the current frame t - ; (5.3) According to K t =P t - H T (HP t - H T +R) -1 , using the covariance matrix P of the current frame t - and the observation noise variance matrix R to derive the Kalman gain K t ; H is the gain matrix; (5.4) According to Use predicted value With the observed value z t Infer the optimal state vector estimate As the final state vector, and update the covariance matrix P of the current frame t - ; (5.5) According to the covariance matrix P of the current frame t - and the Kalman gain K t , update the optimal covariance matrix P of the current frame t =(IK t H)P t - , used to calculate the covariance matrix of the next frame; (5.6) The predicted value calculated from the previous frame and the detection value corresponding to the detection frame output by the trained improved yolov5 model are both normalized and then subjected to inner product operation. The inner product operation value is compared with the threshold. When the inner product operation value is greater than the threshold, the predicted value and the detection value are similar and are the same target; otherwise, they are different targets, thereby achieving target tracking.

2. The method for identifying overlapping fish targets based on the improved yolov5 model according to claim 1, wherein: The morphological feature is that a certain detection value is close to x and y of multiple prediction vectors, while w and h are greater than a single prediction vector.

3. The method for identifying overlapping fish targets based on the improved yolov5 model according to claim 1, wherein: In step seven, if overlap occurs, the multi-target result predicted in step five shall prevail, and the output result of the trained improved yolov5 model shall be adjusted as follows according to the final state vector: For detection boxes whose distance d between the detection box edge and the image edge is less than 10% of the detection box width or height L, reduce non-maximum suppression NMS, specifically: NMS=NMS×f(d / L) f(d / L)=Ae -((d / L) / B)lnA 。 4. The method for identifying overlapping fish targets based on the improved yolov5 model according to claim 3, characterized in that: A=0.1, B=0.5.

Images