Machine Vision-Based Prediction System and Method for the Spawning Behavior of Cichlasoma synspilum

Abstract

Machine vision-based prediction system and method for the spawning behavior of Cichlasoma synspilum, including: an image collection module for acquiring video images of Cichlasoma synspilum; an image processing module for processing the video images and constructing an image dataset of Cichlasoma synspilum; an object detection unit for training on the image dataset of Cichlasoma synspilum; an object tracking unit for tracking the object of Cichlasoma synspilum to obtain the swimming trajectory of Cichlasoma synspilum; and a color recognition unit for extracting and comparing the body color of Cichlasoma synspilum. According to the behavior of Cichlasoma synspilum before spawning, the present invention predicts the spawning time of Cichlasoma synspilum through non-contact detection, bringing convenience to breeders and saving time and effort.

Description

Technical Field

[0001] The present invention relates to the technical field of purple firemouth fish farming, and specifically relates to a system and method for predicting the spawning behavior of purple firemouth fish based on machine vision. Background Art

[0002] In order to improve the hatching rate of purple firemouth fish eggs, it is necessary to artificially isolate and cultivate the newly laid purple firemouth fish eggs. Purple firemouth fish have the habit of incubating fish eggs in their mouths. To avoid the female fish holding the fish eggs in its mouth and reducing the survival rate of the fish eggs, it is necessary to collect the fish eggs immediately after the female fish lays eggs. At the same time, many enterprises need to collect and screen the fish eggs after the female fish has just laid eggs in order to cultivate high-quality purple firemouth fish fry. In order to grasp the timing of purple firemouth fish spawning, it is necessary to rely on manual observation, which is time-consuming and laborious. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a system and method for predicting the spawning behavior of purple firemouth fish based on machine vision, which predicts the spawning time of purple firemouth fish through non-contact detection according to the behavior of purple firemouth fish before spawning, bringing convenience to farmers and saving time and effort.

[0004] The technical solution adopted by the present invention is as follows:

[0005] A system for predicting the spawning behavior of purple firemouth fish based on machine vision, the system includes:

[0006] An image collection module, which acquires video images of purple firemouth fish;

[0007] An image processing module, which processes the video images and constructs an image dataset of purple firemouth fish;

[0008] An object detection unit, which is used to train the image dataset of purple firemouth fish;

[0009] An object tracking unit, which is used to track the purple firemouth fish to obtain the swimming trajectory of the purple firemouth fish;

[0010] A color recognition unit, which is used to extract and compare the body color of the purple firemouth fish.

[0011] The image collection module shoots the water tank where the purple firemouth fish is located through a high-definition camera to obtain the original dataset of the video images of the purple firemouth fish.

[0012] The image processing module performs frame processing on the original dataset of the video images to obtain purple firemouth fish images; preprocesses the purple firemouth fish images, including horizontal flipping, vertical flipping, and mosaic processing to expand the data pictures; constructs an image dataset of purple firemouth fish with the preprocessed images.

[0013] The target detection unit replaces the yolov4 feature extraction network CSPdarknet53 with the lightweight network Mobilenetv3 series, replaces all convolutions in yolov4 with depthwise separable convolutions, and obtains the lightweight network Mobilenetv3-yolov4.

[0014] The CBAM attention mechanism is added to the lightweight network Mobilenetv3-yolov4, and Mobilenetv3-yolov4 is used to train the purple firemouth fish image dataset.

[0015] The target tracking unit classifies the image dataset according to different postures and swimming directions of the purple firemouth fish, retrains REID in deepsort, and obtains the deep features of the purple firemouth fish; uses Mobilenetv3-yolov4 and deepsort to complete the tracking of the purple firemouth fish, and obtains the swimming trajectory of the purple firemouth fish.

[0016] The color recognition unit extracts the body colors of the purple firemouth fish in normal behavior and before spawning, and conducts a comparison of body colors. The method for predicting the spawning behavior of purple firemouth fish based on machine vision includes the following steps:

[0017] Step 1: Obtain the video images of purple firemouth fish through the image collection module, and process the video images through the image processing module to construct a purple firemouth fish image dataset.

[0018] Step 2: Use the constructed purple firemouth fish image dataset to train the target detection unit Mobilenetv3-yolov4 in the image processing module.

[0019] Step 3: Complete the target tracking of the purple firemouth fish through the target tracking unit Mobilenetv3-yolov4+deepsort in the image processing module.

[0020] Step 4: Extract the body color of the purple firemouth fish through the color recognition unit HSV in the image processing module.

[0021] The said Step 1 includes the following steps:

[0022] S1.1: First, use the Hikvision CCD high-definition camera (DS-2CD6233F-SDI) to photograph the water tank where the purple firemouth fish is located.

[0023] S1.2: Perform frame processing on the captured video images to obtain purple firemouth fish images.

[0024] S1.3: Preprocess the purple firemouth fish images. Horizontally flip, vertically flip the collected purple firemouth images, and perform mosaic processing to expand the data pictures, and perform image enhancement. Use the processed images to construct a purple firemouth fish image dataset.

[0025] Step 2 includes the following steps:

[0026] S2.1: Replace the yolov4 feature extraction network CSPdarknet53 with the lightweight network Mobilenetv3 series, and replace all convolutions of yolov4 with depthwise separable convolutions (DW convolutions) to obtain the lightweight network Mobilenetv3-yolov4;

[0027] S2.2: Add the CBAM attention mechanism to the lightweight network Mobilenetv3-yolov4;

[0028] S2.3: Use Mobilenetv3-yolov4 to train the purple firemouth fish image dataset.

[0029] Step 3 includes the following steps:

[0030] S3.1: Classify the purple firemouth fish image dataset according to different postures and swimming directions of the purple firemouth fish;

[0031] S3.2: Retrain REID in deepsort to obtain the deep features of the purple firemouth fish;

[0032] S3.3: Use Mobilenetv3-yolov4+deepsort to complete the tracking of the purple firemouth fish to obtain the swimming trajectory of the purple firemouth fish; Set a crossing line count on the spawning board to record the number of times the purple firemouth fish swims back and forth in the spawning area.

[0033] Step 4 includes the following steps:

[0034] S4.1: Complete the object detection of the purple firemouth fish through the yolov4 object detection unit;

[0035] S4.2: Then extract the normal behavior body color and pre-spawning body color of the purple firemouth fish through HSV and perform body color comparison.

[0036] For a purple firemouth fish spawning behavior prediction system and method based on machine vision of the present invention, the technical effects are as follows:

[0037] 1) Through step one of the present invention, the number and quality of samples can be improved, the training effect of the model can be improved, and the generalization and anti-interference ability of the model can be increased

[0038] 2) In the second step of the present invention, the lightweight network improves the detection speed, reduces the memory occupancy of the model, facilitates the deployment of the model, and at the same time adds the CBAM attention mechanism to improve the detection accuracy of the model.

[0039] 3) In the third step of the present invention, retraining REID can improve the target tracking effect in case of occlusion and reduce the problem of target ID jumping at the same time. Description of the Drawings

[0040] Figure 1 It is a picture of the spawning board taken facing the water tank directly.

[0041] Figure 2 It is a schematic diagram of the overall structure of Mobilenetv3.

[0042] Figure 3 It is a schematic diagram of the implementation of the CBAM attention mechanism.

[0043] Figure 4 It is a diagram of the channel attention mechanism.

[0044] Figure 5 It is a diagram of the spatial attention mechanism.

[0045] Figure 6 It is a picture of a chessboard.

[0046] Figure 7 It is a schematic diagram of the process of the embodiment of the present invention.

[0047] Figure 8 It is a schematic diagram of Mobilenetv3-yolov4 with the CBAM attention mechanism added.

[0048] Figure 9 It is a schematic diagram of tracking the number of times the purple firemouth fish swims back and forth in the spawning area. Detailed Embodiments

[0049] The purple firemouth fish spawning behavior prediction system based on machine vision uses the Hikvision CCD high-definition camera DS-2CD6233F-SDI to take pictures of the water tank where the purple firemouth fish is located. The image is transmitted through the high-definition camera to the image processing module on the PC side. The image processing module includes an improved target detection (Mobilenetv3-yolov4) unit, a target tracking (deepsort) unit, and a color recognition (HSV) unit connected in sequence.

[0050] The image collection module acquires the video images of the purple firemouth fish;

[0051] The image processing module is used to process the video images and construct an image dataset of the purple firemouth fish;

[0052] A target detection unit for training on the Amphilophus labiatus image dataset;

[0053] A target tracking unit for tracking the Amphilophus labiatus to obtain the swimming trajectory of the Amphilophus labiatus;

[0054] A color recognition unit for extracting and comparing the body color of the Amphilophus labiatus.

[0055] The image collection module captures the water tank where the Amphilophus labiatus is located through a high-definition camera to obtain the original dataset of the video images of the Amphilophus labiatus.

[0056] The image processing module performs frame splitting on the original dataset of video images to obtain Amphilophus labiatus images; preprocesses the Amphilophus labiatus images, including horizontal flipping, vertical flipping, and mosaic processing to expand the data pictures; constructs the Amphilophus labiatus image dataset with the preprocessed images.

[0057] The target detection unit uses the lightweight network Mobilenetv3 series to replace the yolov4 feature extraction network CSPdarknet53, and replaces all convolutions of yolov4 with depthwise separable convolutions to obtain the lightweight network Mobilenetv3-yolov4. As Figure 8 shown, the feature map is passed into Mobilenetv3-yolov4, first passing through the convolutional layer in Mobilenetv3 to extract features, and outputting three feature layers, namely 52×52×40, 26×26×112, 13×13×160. The 13×13×160 feature layer will continue to pass through the SPP max pooling layer and the convolutional layer, and then the three feature layers enter the PANet for enhanced feature extraction, and finally 3 Yolo heads are output.

[0058] The CBAM attention mechanism is added to the lightweight network Mobilenetv3-yolov4, and Mobilenetv3-yolov4 is used to train the Amphilophus labiatus image dataset.

[0059] The target tracking unit classifies the image dataset according to different postures and swimming directions of the Amphilophus labiatus, retrains REID in deepsort to obtain the deep features of the Amphilophus labiatus; uses Mobilenetv3-yolov4 and deepsort to complete the tracking of the Amphilophus labiatus to obtain the swimming trajectory of the Amphilophus labiatus.

[0060] The color recognition unit extracts the normal behavior body color and the body color before spawning of the Amphilophus labiatus, and conducts a body color comparison. A method for predicting the spawning behavior of Amphilophus labiatus based on machine vision includes the following steps:

[0061] Step 1: Obtain the video images of the purple firemouth cichlid through the image collection module, and process the video images through the image processing module to construct a purple firemouth cichlid image dataset;

[0062] Step 2: Use the constructed purple firemouth cichlid image dataset to train the object detection unit Mobilenetv3-yolov4 in the image processing module;

[0063] Step 3: Complete the object tracking of the purple firemouth cichlid through the object tracking unit Mobilenetv3-yolov4+deepsort in the image processing module;

[0064] Step 4: Extract the body color of the purple firemouth cichlid through the color recognition unit HSV in the image processing module.

[0065] The said Step 1 includes the following steps:

[0066] S1.1: First, use the Hikvision CCD high-definition camera (DS-2CD6233F-SDI) to photograph the water tank where the purple firemouth cichlid is located;

[0067] S1.2: Perform frame splitting on the photographed video images to obtain purple firemouth cichlid images;

[0068] S1.3: Preprocess the purple firemouth cichlid images, perform horizontal flipping, vertical flipping, and mosaic processing on the collected purple firemouth images to expand the data pictures, and use the processed images to construct a purple firemouth cichlid image dataset.

[0069] In Step 1, use the Hikvision CCD high-definition camera DS-2CD6233F-SDI to photograph the water tank where the purple firemouth cichlid is located. The high-definition camera is placed facing the spawning board of the water tank. Perform frame splitting on the video through python-opencv, and select a total of 1221 clear pictures. Use python-opencv to perform horizontal and vertical flipping on some clear images, and add mosaics and other operations to expand the dataset, obtaining a total of 3200 pictures. Use labelme to annotate the pictures, and then divide the 3200 annotated pictures into a dataset, dividing them into a training set, a validation set, and a test set according to the ratio of 8:1:1. The said Step 2 includes the following steps:

[0070] S2.1: Replace the yolov4 feature extraction network CSPdarknet53 with the lightweight network Mobilenetv3 series, and replace all convolutions in yolov4 with depthwise separable convolutions (DW convolutions) to obtain the lightweight network Mobilenetv3-yolov4; replace the yolov4 feature extraction network CSPdarknet53 with the lightweight network Mobilenetv3. Mobilenetv3 has the following characteristics: ①. It has an inverted residual structure with a linear bottleneck; ②. Depthwise separable convolutions. ③. A lightweight attention model. ④. Use h-swish to replace the swish function to reduce the computational load and improve performance. The overall structure diagram of Mobilenetv3 is as shown in Figure 2 shown. Replace all convolutions in SPP of yolov4 and PANet with separable convolutions (DW convolutions).

[0071] S2.2: Add the CBAM attention mechanism to the lightweight network Mobilenetv3-yolov4;

[0072] Add the CBAM attention mechanism to the lightweight network Mobilenetv3-yolov4. CBAM combines the channel attention mechanism and the spatial attention mechanism, and can achieve better results compared to the SENet that only focuses on the channel attention mechanism. The implementation schematic diagram is as shown in Figure 3 shown. CBAM will perform the processing of the channel attention mechanism and the spatial attention mechanism on the input feature layer respectively.

[0073] The specific implementation methods of the channel attention mechanism and the spatial attention mechanism:

[0074] Figure 4 This is the channel attention mechanism. The implementation of the channel attention mechanism can be divided into two parts. First, perform global average pooling and global max pooling on the input single feature layer respectively. Then, process the results of average pooling and max pooling using a shared fully connected layer, then add the two processed results, and then take a sigmoid to obtain the weight value (between 0 and 1) of each channel of the input feature layer. After obtaining this weight value, multiply this weight value by the original input feature layer.

[0075] Figure 5It is a spatial attention mechanism that takes the maximum value and average value of the input feature layer on the channels of each feature point. After that, these two results are stacked, the number of channels is adjusted by using a convolution with 1 channel once, and then a sigmoid is taken to obtain the weight value (between 0 and 1) of each feature point of the input feature layer. After obtaining this weight value, this weight value is multiplied by the original input feature layer.

[0076] S2.3: Use Mobilenetv3-yolov4 to train the purple firemouth fish image dataset.

[0077] The training steps and structure of Mobilenetv3-yolov4 are as follows:

[0078] 1): The input picture is processed through adaptive sizing and adjusted to a 3-channel picture of 224×224. Through the feature extraction of 16-layer convolutional modules, the picture is adjusted to a 16-channel picture of 112×112, and then normalized and activated by the rule6 function and sent to the bneck module for the next step of processing.

[0079] 2): The above picture is first reduced in dimension through 1×1 convolution and 3×3 convolution, and then through the SE attention module, the output is superimposed with the original input to obtain the output. Then the output passes through multiple bneck structures to obtain the effective feature layers of the last three shapes, which are 52×52×40, 26×26×112, and 13×13×112 respectively.

[0080] 3): The last layer of feature layer is input into the SPP layer, and feature fusion is carried out by using maximum pooling in different-sized pooling layers of 5×5, 9×9, and 13×13, and then enters the enhanced feature extraction network PANet, and the CBAM attention mechanism is added to PANet.

[0081] 4): In the prediction module, the three feature layers passing through PANet predict three different bounding boxes for each target in the training image, and the prediction boxes with an IOU greater than or equal to 0.5 are output. The dataset is loaded into the Mobilenetv3-yolov4, yolov4, fasterrcnn, and SSD300 models with the CBAM attention mechanism added for training. The comparison of the model memory occupancy, average detection accuracy (map), and frames per second (FPS) is shown in Table 1:

[0082] Table 1 Memory occupancy, average detection accuracy (map), and frames per second (FPS) of the Mobi lenetv3-yolov4, yolov4, fasterrcnn, and SSD300 models

[0083]

[0084] As can be seen from Table 1, the model of Mobilenetv3-yolov4 has the smallest proportion, the highest average detection accuracy (map), and its inference speed on the PC side is better than that of yolov4.

[0085] Step 3 includes the following steps:

[0086] S3.1: Classify the purple firemouth fish image dataset according to different postures and swimming directions of the purple firemouth fish, specifically as follows:

[0087] Classify 3,200 pictures into different swimming directions such as swimming left, swimming right, swimming up, etc., and different posture classifications, obtaining a total of 121 classes of different swimming directions and different postures.

[0088] S3.2: Retrain REID in DeepSort to obtain the deep features of the purple firemouth fish;

[0089] Input the 121 classes into a convolutional neural network for feature classification to obtain the deep features of the purple firemouth fish. During the multi-object tracking process of DeepSort, a Kalman filter is used to predict the motion trend of the next frame of the purple firemouth fish target. By first setting the maximum threshold A max , that is, the maximum number of consecutive detection frames. Then for each purple firemouth fish target to be tracked, record the number of frames a between the last detection result and the current frame after the tracking result is successfully matched. This number increases during the prediction of the Kalman filter. When the tracking result and the detection result match, it is reset to 0. If a continues to increase and exceeds the set maximum threshold A max , at this time, the target is considered lost or the tracking ends. When the existing tracker has been unable to match the target in the detection result, a new target is considered to have appeared, and a new tracker is added. In the next three consecutive frames of subsequent tracking, if the predicted position of the newly added tracker can correctly match the detection result, a new target is determined to have appeared; if not, it is considered a false detection and is deleted.

[0090] S3.3: Use Mobilenetv3-yolov4+DeepSort to complete the tracking of the purple firemouth fish to obtain the swimming trajectory of the purple firemouth fish; set a line crossing count on the spawning board to record the number of times the purple firemouth fish swims back and forth in the spawning area. Specifically as follows:

[0091] Use Mobilenetv3 - yolov4 + deepsort to track the purple firemouth cichlid and obtain its swimming trajectory. Calibrate the camera using the Hikvision CCD high - definition camera DS - 2CD6233F - SDI. The purple firemouth cichlid is photographed by an uncalibrated camera. Therefore, each frame in the video is the projection of the real space on the plane through projective transformation. Since the ratio between points in the plane cannot be maintained in projective transformation to conform to the true ratio of real objects, a planar region needs to be selected. By establishing a geometric model of camera imaging, the perspective projection matrix of points in the image domain and points in the real world is estimated, and the point x = [x, y, 1] in the image domain T is mapped to the point X = [X, Y, 1] in the corrected domain, and the mapping formula is:

[0092]

[0093] Use a chessboard as shown Figure 6 to calibrate the camera parameters as an actual marker. First, regard the plane where the marker is located as the (x, y, 1) plane. Then, according to the perspective projection formula, solve the camera parameters and distortion coefficients. Finally, use the camera parameters and distortion coefficients to obtain the corrected picture of the purple firemouth cichlid. According to the camera parameters, map the points of the detection box in the image domain to the points in the real space. According to the actual positions of the two detection boxes and the time between image frames, calculate the pixel speed of the purple firemouth cichlid. According to the actual size of the water tank, restore the pixel speed. The calculation formula for the scale factors in the x and y directions of the image is:

[0094]

[0095] where W represents the actual width of the water tank, H represents the actual length of the water tank in the image domain, and w and h are the length and width resolutions of the image. The swimming speed estimation formula of the purple firemouth cichlid is as shown:

[0096]

[0097] where represents the pixel speed of the purple firemouth cichlid detection box in the x - direction, represents the pixel speed of the purple firemouth cichlid detection box in the y - direction.

[0098] After obtaining the positions of each purple firemouth cichlid in the image domain from the Kalman filter, obtain the corresponding real - world position coordinates through the camera calibration parameters. Then, calculate the time interval between two frames of images through the number of frames per second of the video, and further calculate the actual swimming speed of the purple firemouth cichlid. The speed calculation formula of the purple firemouth cichlid is:

[0099]

[0100] Where: s t represents the speed of the purple firemouth fish at time t; fr is the frame rate of the video;

[0101] cx t and cy t respectively represent the x and y positions of the center point of the detection frame of the purple firemouth fish at the t-th moment;

[0102] cx t-1 and cy t-1 respectively represent the x and y positions of the center point of the detection frame of the purple firemouth fish at the (t - 1)-th moment;

[0103] Bx and By respectively represent the ratios of the image distance to the actual distance on the x-axis and y-axis;

[0104] Set a vertical line on the spawning board, as Figure 9 shown, Figure 9 The three "0"s in the upper left corner in represent the number of times the purple firemouth fish passes through the vertical line from left to right, the number of times it passes through the vertical line from right to left, and the total number of times it passes through the vertical line back and forth; starting from 0. Each time the purple firemouth fish passes through the vertical line, it is recorded once, and the number of times the purple firemouth fish swims back and forth on the spawning board within a certain period of time is obtained.

[0105] Step 4 includes the following steps:

[0106] S4.1: Obtain the purple firemouth fish image through yolov4 object detection;

[0107] S4.2: Then extract the normal body color and the body color before spawning of the purple firemouth fish through HSV, and conduct a body color comparison. Specifically as follows:

[0108] Perform body color recognition on the purple firemouth detected by yolov4. During the spawning period, the body color of the purple firemouth deepens and there are obvious changes in the body color. Use HSV to recognize the body color of the purple firemouth, and compare it with the normal body color extracted from the clear data set pictures collected in step one to identify whether the purple firemouth is in the spawning period.

[0109] Example:

[0110] 1. First, use the Hikvision CCD high-definition camera DS-2CD6233F-SDI to photograph the tank where the purple firemouth fish is located. Then, use python-opencv to perform frame-by-frame operations on the video and select 1221 clear pictures. Use python-opencv to horizontally and vertically flip some of the clear images and add mosaics to expand the dataset, obtaining a total of 3200 pictures. Use labelme to annotate the pictures, and then divide the 3200 annotated pictures into a dataset according to the ratio of 8:1:1 for the training set, validation set, and test set.

[0111] 2. Replace the feature extraction network CSPdarknet53 of yolov4 with the lightweight network Mobilenetv3, and replace all the convolutions in SPP of yolov4 and PANet with separable convolutions (DW convolutions). And add the CBAM attention mechanism to the lightweight network Mobilenetv3-yolov4.

[0112] 3. Input the dataset into the lightweight network Mobilenetv3-yolov4 to start training. The experimental platform is a desktop computer with an Intel(R) Core(TM) i7-8700 3.20GHz CPU, 12GB of video memory, an NVIDIA TITAN V GPU, a windows 10 system version, a python version of 3.7, a pytorch version of 1.2, and a CUDA version of 10.0. Adopt the strategy of transfer learning and use the pre-trained weights of the Mobilenetv3 network trained on VOC. The training process is divided into: 1) Freeze the backbone part of the network, set the batch_size to 32, and the number of training epochs to 50. 2) Unfreeze the training, set the batch_size to 8, and the number of training epochs to 50. The optimizer used in the training is the Adam optimizer.

[0113] 4. Classify the 3200 pictures into different swimming directions such as swimming left, swimming right, swimming up, etc., and different postures, obtaining a total of 121 classes for different swimming directions and different postures. Then input the 121 classes into a convolutional neural network for feature classification to obtain the depth features of the purple firemouth fish. Use the trained weights of Mobilenetv3-yolov4 and the REID depth features to complete the tracking of the purple firemouth. Set a vertical line in the center of the spawning board. Whenever the purple firemouth passes through the vertical line, it will be counted, and the number of times the purple firemouth fish passes back and forth through the spawning board will be recorded. While counting, the movement trajectory of the purple firemouth fish will be obtained through tracking.

[0114] 5. Camera calibration is performed on the Hikvision CCD high-definition camera (DS-2CD6233F-SDI). The purple firemouth fish is photographed by an uncalibrated camera. Therefore, each frame in the video is a projection of the real space on the plane through projective transformation. The chessboard is used as an actual marker to calibrate the camera parameters. Then, according to the perspective projection formula, the camera parameters and distortion coefficients are solved. Finally, the corrected picture of the purple firemouth fish is obtained using the camera parameters and distortion coefficients. According to the camera parameters, the points of the detection box in the image domain are mapped to the points in the real space, and the pixel speed of the purple firemouth fish is calculated based on the actual positions of the two detection boxes and the time between image frames.

[0115] 6. The body color of the purple firemouth fish is extracted by using the difference in body color between the spawning period and the non-spawning period. First, the body color of the clear pictures of the purple firemouth fish dataset in the non-spawning period is extracted using HSV based on python-opencv. When the purple firemouth fish enters the spawning period, the hue H, saturation S, and value V in HSV will change significantly.

[0116] 7. Behaviors of the purple firemouth fish when entering the spawning period: 1) Nest-building period: The purple firemouth fish will clean the spawning board to create a clean spawning environment, that is, it will hover around the spawning board. 2) Entering the spawning period: The body color of the purple firemouth fish changes. 3) Spawning period: The purple firemouth fish will swim around the spawning board, and the male purple firemouth fish will chase the female fish and butt the female fish's belly. At this time, the swimming speed of the purple firemouth fish continues to change and is faster than the swimming speed before the spawning period.

[0117] 8. Through machine vision, the spawning prediction of the purple firemouth fish is completed:

[0118] 1) Through Mobilenetv3-yolov4+deepsort, a vertical line is set in the center of the spawning board to record the number of times the purple firemouth fish swims back and forth on the spawning board. By observing the change in the number of times the purple firemouth fish swims back and forth on the spawning board over a period of time, mainly a significant increase compared to usual, it is judged whether the purple firemouth fish enters the nest-building period, and at this time, a three-level warning is initiated.

[0119] 2) The change in the body color of the purple firemouth fish is judged by HSV. When the body color of the purple firemouth fish changes significantly, a second-level warning is initiated.

[0120] 3) The swimming trajectory of the purple firemouth fish is obtained through Mobilenetv3-yolov4+deepsort, and the pixel speed of the purple firemouth fish is calculated based on the actual positions of the two detection boxes and the time between image frames. When the swimming speed of the purple firemouth fish is constantly changing and a second-level warning is in effect, a first-level warning is issued to remind professionals that the purple firemouth fish is about to spawn and take corresponding measures. The specific flowchart is as Figure 7 shown.

Claims

1. A prediction system for the spawning behavior of Paratheraps synspilus based on machine vision, characterized in that The system includes: An image collection module that acquires video images of Cichlasoma synspilum; An image processing module that processes the video images and constructs an image dataset of Cichlasoma synspilum; A target detection unit that trains on the image dataset of Cichlasoma synspilum; A target tracking unit that performs target tracking on Cichlasoma synspilum to obtain the swimming trajectory of Cichlasoma synspilum; A color recognition unit that extracts and compares the body colors of Cichlasoma synspilum; The method for predicting the spawning behavior of Cichlasoma synspilum using the said prediction system includes the following steps: Step 1: Acquire video images of Cichlasoma synspilum through the image collection module, and process the video images through the image processing module to construct an image dataset of Cichlasoma synspilum; Step 2: Use the constructed image dataset of Cichlasoma synspilum to train the target detection unit Mobilenetv3-yolov4 in the image processing module; Step 3: Complete the target tracking of Cichlasoma synspilum through the target tracking unit Mobilenetv3-yolov4+deepsort in the image processing module; Step 4: Extract the body color of Cichlasoma synspilum through the color recognition unit HSV in the image processing module; The said Step 2 includes the following steps: S2.1: Replace the yolov4 feature extraction network CSPdarknet53 with the lightweight network Mobilenetv3 series, and replace all convolutions of yolov4 with depthwise separable convolutions to obtain the lightweight network Mobilenetv3-yolov4; S2.2: Add the CBAM attention mechanism to the lightweight network Mobilenetv3-yolov4; S2.3: Use Mobilenetv3-yolov4 to train on the image dataset of Cichlasoma synspilum; The said Step 4 includes the following steps: S4.1: Complete the target detection of Cichlasoma synspilum through the yolov4 target detection unit; S4.2: Then extract the body colors of Cichlasoma synspilum in normal behavior and before spawning through HSV, and conduct a comparison of the body colors.

2. The system for predicting the spawning behavior of the purple firemouth fish based on machine vision according to claim 1, wherein: The said target detection unit replaces the yolov4 feature extraction network CSPdarknet53 with the lightweight network Mobilenetv3 series, and replaces all convolutions of yolov4 with depthwise separable convolutions to obtain the lightweight network Mobilenetv3-yolov4; Add the CBAM attention mechanism to the lightweight network Mobilenetv3-yolov4, and use Mobilenetv3-yolov4 to train on the image dataset of Cichlasoma synspilum.

3. The machine vision-based prediction system for the spawning behavior of the purple firemouth fish according to claim 1, wherein: The said target tracking unit classifies the image dataset according to different postures and swimming directions of Cichlasoma synspilum, retrains REID in deepsort to obtain the deep features of Cichlasoma synspilum; uses Mobilenetv3-yolov4 and deepsort to complete the tracking of Cichlasoma synspilum to obtain the swimming trajectory of Cichlasoma synspilum.

4. The machine vision-based prediction system for the spawning behavior of the purple firemouth fish according to claim 1, characterized in that: The said color recognition unit extracts the body colors of Cichlasoma synspilum in normal behavior and before spawning, and conducts a comparison of the body colors.

5. The system for predicting the spawning behavior of the purple firemouth fish based on machine vision according to claim 1, characterized in that: Step 1 includes the following steps: S1.1: First, use a CCD high-definition camera to photograph the water tank where the purple firemouth fish is located; S1.2: Perform frame division on the captured video image to obtain the purple firemouth fish image; S1.3: Preprocess the purple firemouth fish image, horizontally flip, vertically flip, and perform mosaic processing on the collected purple firemouth image to expand the data picture, and perform image enhancement. Use the processed image to construct the purple firemouth fish image dataset.

6. The prediction system for the spawning behavior of the purple firemouth fish based on machine vision according to claim 1, characterized in that: Step 3 includes the following steps: S3.1: Classify the purple firemouth fish image dataset according to different postures and swimming directions of the purple firemouth fish; S3.2: Retrain REID in deepsort to obtain the depth features of the purple firemouth fish; S3.3: Use Mobilenetv3-yolov4+deepsort to complete the tracking of the purple firemouth fish to obtain the swimming trajectory of the purple firemouth fish; set a crossing line count on the spawning board to record the number of times the purple firemouth fish swims back and forth in the spawning area.

7. The method for tracking the swimming trajectory of a purple firemouth fish using the prediction system as described in claim 1, characterized in that: Map the points of the detection box in the image domain to the points in the real space, and calculate the pixel speed of the purple firemouth fish according to the actual positions of the two detection boxes and the time between image frames; According to the actual size of the water tank, perform size restoration on the pixel speed. The scale factors in the x and y directions of the image are calculated by the formula: ； where W represents the actual width of the water tank, H represents the actual length of the water tank in the image domain, and w and h are the length and width resolutions of the image; the swimming speed estimation formula of the purple firemouth fish is as shown: ； Among them, represents the pixel velocity of the detection frame of the purple firemouth fish in the x direction, represents the pixel velocity of the detection frame of the purple firemouth fish in the y direction; After obtaining the positions of each purple firemouth fish in the image domain from the Kalman filter, obtain the corresponding real position coordinates through the camera calibration parameters, and then calculate the time interval between two frames of images through the number of frames per second of the video, and further calculate the actual swimming speed of the purple firemouth fish; the speed calculation formula of the purple firemouth fish: Set a vertical line on the spawning board, and record it every time the purple firemouth fish crosses the vertical line once to obtain the number of times the purple firemouth fish swims back and forth on the spawning board within a certain period of time.

Images