Fish behavior identification method and device

Abstract

The invention provides a fish behavior identification method and device. The method comprises the following steps: acquiring a fish motion image; inputting the fish motion image to the target detection model to obtain a fish target detection result; analyzing a fish target detection result to obtain a fish tracking trajectory; analyzing the fish tracking trajectory to obtain fish motion features; and inputting the fish motion features to the behavior recognition model to obtain a fish behavior recognition result. According to the fish behavior recognition method and device, the model framework integrating the motion feature extraction function and the behavior state recognition function is constructed, the defects that in the prior art, parameters are complex and recognition is difficult in fish behavior recognition can be overcome, and the fish behavior recognition precision and efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a fish behavior recognition method and device. Background Art

[0002] In aquaculture, monitoring and analysis of fish behavior can guide daily feeding, disease analysis and disease diagnosis. In actual breeding, common fish behaviors include feeding and coercion behaviors. Common characteristic indicators can be divided into motion characteristics, physical characteristics and environmental characteristics, among which motion characteristics are the most direct characteristic indicators that reflect changes in fish behavior.

[0003] Initial studies on fish behavior relied on direct observation and recording by observers. This method was subject to subjective influences and could not accurately identify fish behavior. With the development of technology, various methods have been proposed to extract fish behavior information, including machine vision technology, acoustic technology, and bio-indicator technology. In most cases, bio-indicator technology requires the implantation of biological recorders into animals through certain means, thereby affecting the normal behavior of animals; acoustic technology is easily affected by water surface and water wave noise; machine vision technology is a new technology developed on the basis of multiple technologies such as image processing, artificial intelligence, and signal processing. It has gradually become the first choice of aquaculture practitioners and researchers with its advantages of being non-destructive, accurate, and easy to deploy.

[0004] At present, the fish behavior recognition methods based on machine vision often have problems such as complex structure, large number of parameters, and large amount of calculation in the extraction of fish movement characteristics. Such algorithms cannot meet the needs in actual scenarios. Secondly, it is also difficult to accurately quantify the movement characteristics of fish and combine multiple movement characteristics to identify and classify their behavior states. The above problems make it difficult for current fish behavior recognition methods to ensure accuracy and efficiency.

[0005] Therefore, it is an urgent problem to propose a model framework that can integrate motion feature extraction and behavior state recognition functions, effectively reducing the model size and inference time while ensuring accuracy. Summary of the invention

[0006] The present invention provides a fish behavior recognition method and device, which are used to solve the defects of complex parameters and difficult recognition in the prior art of fish behavior recognition, and to improve the accuracy and efficiency of fish behavior recognition.

[0007] The present invention provides a fish behavior recognition method, comprising:

[0008] Obtain images of fish movement;

[0009] Inputting the fish motion image into the target detection model to obtain the fish target detection result;

[0010] Analyze the fish target detection result to obtain the fish tracking trajectory;

[0011] Analyzing the fish tracking trajectory to obtain fish movement characteristics;

[0012] Inputting the fish movement characteristics into a behavior recognition model to obtain a fish behavior recognition result;

[0013] Among them, the target detection model is trained based on fish motion image samples and the fish target detection results corresponding to the fish motion image samples as labels; the behavior recognition model is trained based on fish motion feature samples and the fish behavior recognition results corresponding to the fish motion feature samples.

[0014] According to a fish behavior recognition method provided by the present invention, the training step of the target detection model includes:

[0015] Acquire fish motion image samples, and perform data enhancement processing on the fish motion image samples;

[0016] Annotating the fish target detection result corresponding to the fish motion image sample as a label;

[0017] Dividing the fish motion image samples into a fish motion image sample training set, a fish motion image sample verification set, and a fish motion image sample test set;

[0018] The target detection model is obtained by training with the fish motion image sample training set, the fish target detection result of the target detection model is verified with the fish motion image sample verification set, and after obtaining the network weight of the target detection model, the target detection model is tested with the fish motion image sample test set.

[0019] According to a fish behavior recognition method provided by the present invention, the target detection model is obtained by training based on the LDW-YOLOv8 model. Before the target detection model is obtained by training the fish motion image sample training set, the method further includes:

[0020] Adjusting the basic network feature fusion structure of the LDW-YOLOv8 model to reduce the learning of redundant features;

[0021] Replace the backbone downsampling convolution of the LDW-YOLOv8 model with full-dimensional dynamic convolution;

[0022] The bounding box loss function CIoU_Loss of the LDW-YOLOv8 model is replaced by Wise-IoUv3 Loss.

[0023] According to a fish behavior recognition method provided by the present invention, the analyzing the fish target detection result to obtain the fish tracking trajectory includes:

[0024] Divide the fish target detection result into a high-score detection frame and a low-score detection frame according to a confidence threshold in the target detection result, put the high-score detection frame into a high-score detection frame set, and put the low-score detection frame into a low-score detection frame set;

[0025] Performing trajectory matching on the high-score detection frame set, placing unsuccessfully matched trajectories into a first remaining trajectory set, and placing unsuccessfully matched high-score detection frames in the high-score detection frame set into a remaining detection frame set;

[0026] Matching the low-score detection frame set with the first remaining track set, placing unsuccessfully matched tracks into a second remaining track set, and deleting the unsuccessfully matched low-score detection frames in the low-score detection frame set;

[0027] Update the Kalman filter of the successfully matched trajectory;

[0028] Generate a new track according to the high-score detection frame in the remaining detection frame set whose confidence is greater than the tracking score threshold, and delete the track with more than 30 frames in the second remaining track set;

[0029] A Kalman filter and an extended Kalman filter are constructed, and the new position of the trajectory with acceleration less than the acceleration threshold is predicted by the Kalman filter, and the new position of the trajectory with acceleration greater than the acceleration threshold is predicted by the extended Kalman filter to obtain the fish tracking trajectory.

[0030] According to a fish behavior recognition method provided by the present invention, before analyzing the fish tracking trajectory, the method further includes: performing Gaussian interpolation processing on the fish tracking trajectory, and the specific steps include:

[0031] The observation value of the i-th track in the fish tracking track at the x-frame is calculated according to the following formula: for:

[0032]

[0033] in, is the coordinate variable of trajectory i at the x frame, The mean is 0 and the variance is Gaussian distribution, f (i) ∈GP(0,k(x,x′)), l is a hyperparameter;

[0034] Through the known frame x1x2x3······x n and the frame to be predicted x *Compute the covariance matrix:

[0035]

[0036] K * =[k(x * ,x1) k(x * ,x2) … k(x * ,x n )]

[0037] The best estimate of Among them, y is The n×1 dimensional matrix composed of

[0038] By setting the hyperparameter σ to represent the maximum number of frame intervals for performing Gaussian interpolation, Gaussian interpolation is performed when the number of intervals is less than the hyperparameter σ.

[0039] According to a fish behavior recognition method provided by the present invention, the fish movement characteristics include: swimming speed, swimming acceleration and discreteness. In the process of analyzing the fish tracking trajectory, the image is corrected by inverse perspective transformation.

[0040] According to a fish behavior recognition method provided by the present invention, the training step of the behavior recognition model includes:

[0041] Obtain samples of fish movement characteristics;

[0042] Marking the fish behavior recognition result corresponding to the fish movement feature sample as a label;

[0043] Dividing the fish motion feature samples into a fish motion feature sample training set and a fish motion feature sample test set;

[0044] The behavior recognition model is obtained by training the fish motion feature sample training set, and the behavior recognition model is evaluated by the fish motion feature sample test set.

[0045] The present invention also provides a fish behavior identification device, comprising:

[0046] An image acquisition module, used for acquiring fish movement images;

[0047] A target detection module, used for inputting the fish motion image into a target detection model to obtain a fish target detection result;

[0048] A trajectory analysis module, used to analyze the fish target detection result to obtain the fish tracking trajectory;

[0049] A feature analysis module, used for analyzing the fish tracking trajectory to obtain fish movement features;

[0050] A behavior recognition module, used for inputting the fish movement characteristics into a behavior recognition model to obtain a fish behavior recognition result;

[0051] Among them, the target detection model is trained based on fish motion image samples and the fish target detection results corresponding to the fish motion image samples as labels; the behavior recognition model is trained based on fish motion feature samples and the fish behavior recognition results corresponding to the fish motion feature samples.

[0052] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned fish behavior identification methods when executing the program.

[0053] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the above-described fish behavior identification methods.

[0054] The present invention provides a method and device for identifying fish behavior. The method obtains fish motion images and inputs them into a target detection model to obtain fish target detection results, so as to identify fish target individuals from the original data of the fish motion images. The fish target detection results are analyzed to obtain fish tracking trajectories, that is, the motion trajectory routes of fish in the current state. The fish tracking trajectories are further analyzed, and fish motion features are extracted from the fish tracking trajectories by means of calculation and quantification to reflect various motion data of fish in the current state. The fish motion features are input into the behavior recognition model, and a comprehensive analysis is performed according to various fish motion features to obtain fish behavior recognition results, that is, the fish behavior in the current state, which provides an effective reference for fish breeding or diagnosis. Among them, the target detection model is obtained by training based on fish motion image samples and fish target detection results corresponding to the fish motion image samples as labels; the behavior recognition model is obtained by training based on fish motion feature samples and fish behavior recognition results corresponding to the fish motion feature samples. Based on machine vision and deep learning, the present invention constructs a model framework that integrates motion feature extraction and behavior state recognition functions, and effectively reduces the model size and reasoning time while ensuring accuracy. The defects of complex parameters and difficult identification in the prior art of fish behavior identification are solved, and the accuracy and efficiency of fish behavior identification are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 It is a flow chart of the fish behavior identification method provided by the present invention;

[0057] Figure 2 It is a structural schematic diagram of a target detection model of a fish behavior recognition method provided by the present invention;

[0058] Figure 3 It is a structural schematic diagram of the fish behavior identification device provided by the present invention;

[0059] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Combine the following Figure 1 The fish behavior recognition method according to the first embodiment of the present invention is described.

[0062] like Figure 1 As shown, a fish behavior recognition method of this embodiment specifically includes the following steps (the numbering of each step in the present invention is only used to distinguish the steps, and does not limit the specific execution order of each step):

[0063] Step S1: Acquire fish motion images.

[0064] In the process of acquiring fish motion images, video data of free motion of fish in actual breeding scenarios is collected as fish motion images. In this embodiment, the fish motion images collected are sea bass motion images.

[0065] Step S2: input the fish motion image into the target detection model to obtain the fish target detection result.

[0066] The fish motion image is input into the preset target detection model to obtain the fish target detection result, so as to identify the fish target individual from the raw data of the fish motion image and obtain the fish's location and behavior information.

[0067] Step S3: Analyze the fish target detection result to obtain the fish tracking trajectory.

[0068] The fish target detection results are analyzed and tracked to obtain the fish tracking trajectory, that is, the movement trajectory of the fish in the current state.

[0069] Step S4: Analyze the fish tracking trajectory to obtain fish movement characteristics.

[0070] Continue to analyze the fish tracking trajectory, and extract the fish movement characteristics from the fish tracking trajectory through calculation and quantification to reflect the various movement data of the fish in the current state and obtain the long-term free movement information of the target fish.

[0071] Step S5: input the fish movement characteristics into the behavior recognition model to obtain the fish behavior recognition result.

[0072] The fish movement characteristics are input into the behavior recognition model, and a comprehensive analysis is performed based on the various fish movement characteristics to obtain the fish behavior recognition results, that is, the fish behavior in the current state, providing an effective reference for fish breeding or diagnosis.

[0073] Among them, the target detection model is trained based on fish motion image samples and the fish target detection results corresponding to the fish motion image samples as labels; the behavior recognition model is trained based on fish motion feature samples and the fish behavior recognition results corresponding to the fish motion feature samples.

[0074] The target detection model is trained based on fish motion image samples and the corresponding fish target detection results of fish motion image samples as labels; the behavior recognition model is trained based on fish motion feature samples and the corresponding fish behavior recognition results of fish motion feature samples. The accurate recognition of fish target detection results and fish behavior recognition results is achieved through deep learning.

[0075] The first embodiment of the present invention provides a method for identifying fish behavior, which obtains fish target detection results by acquiring fish motion images and inputting them into a target detection model, so as to identify fish target individuals from the original data of fish motion images. The fish target detection results are analyzed to obtain fish tracking trajectories, that is, the motion trajectory routes of fish in the current state. The fish tracking trajectories are further analyzed, and fish motion features are extracted from the fish tracking trajectories by means of calculation and quantification to reflect various motion data of fish in the current state. The fish motion features are input into the behavior recognition model, and a comprehensive analysis is performed based on various fish motion features to obtain fish behavior recognition results, that is, the fish behavior in the current state, which provides an effective reference for fish breeding or diagnosis. Among them, the target detection model is obtained by training based on fish motion image samples and fish target detection results corresponding to the fish motion image samples as labels; the behavior recognition model is obtained by training based on fish motion feature samples and fish behavior recognition results corresponding to the fish motion feature samples. Based on machine vision and deep learning, the present invention constructs a model framework that integrates motion feature extraction and behavior state recognition functions, effectively reducing the model size and inference time while ensuring accuracy. The defects of complex parameters and difficult identification in the prior art of fish behavior identification are solved, and the accuracy and efficiency of fish behavior identification are improved.

[0076] In this embodiment, the training steps of the target detection model include:

[0077] Acquire fish motion image samples, and perform data enhancement processing on the fish motion image samples;

[0078] Annotating the fish target detection result corresponding to the fish motion image sample as a label;

[0079] Dividing the fish motion image samples into a fish motion image sample training set, a fish motion image sample verification set, and a fish motion image sample test set;

[0080] The target detection model is obtained by training with the fish motion image sample training set, the fish target detection result of the target detection model is verified with the fish motion image sample verification set, and after obtaining the network weight of the target detection model, the target detection model is tested with the fish motion image sample test set.

[0081] By controlling the water quality of the aquaculture environment, we collected video data of fish free movement under low temperature and hypoxia stress, and installed an automatic feeder to collect video data of fish free movement under feeding behavior. The collected video data of fish free movement is used as fish motion image samples. We use the image data enhancement library (Albumentation), image data annotation tool (Labelme) and multi-target tracking annotation tool (DarkLabel) to screen, enhance, annotate and classify fish motion image samples to form an effective data set.

[0082] First, the fish motion image samples are screened and data enhanced. The data enhancement methods include horizontal flip, vertical flip, blur, random brightness contrast, RGB translation, random rotation of 90 degrees, Gaussian noise and Gaussian blur. The fish target detection results corresponding to the fish motion image samples are annotated as labels, and after processing, they are saved as target detection and verification multi-target tracking effect datasets, and the fish motion image samples are divided into fish motion image sample training set, fish motion image sample verification set and fish motion image sample test set in a 6:2:2 manner.

[0083] The target detection model is trained with the fish motion image sample training set, the fish target detection result of the target detection model is verified with the fish motion image sample verification set, and after obtaining the network weights of the target detection model, the target detection model is tested with the fish motion image sample test set.

[0084] In this embodiment, the target detection model is obtained based on the LDW-YOLOv8 model training. Before the target detection model is obtained by training the fish motion image sample training set, the following steps are also included:

[0085] Adjusting the basic network feature fusion structure of the LDW-YOLOv8 model to reduce the learning of redundant features;

[0086] Replace the backbone downsampling convolution of the LDW-YOLOv8 model with full-dimensional dynamic convolution;

[0087] The bounding box loss function CIoU_Loss of the LDW-YOLOv8 model is replaced by Wise-IoUv3 Loss.

[0088] The target detection model of this embodiment is based on the LDW-YOLOv8 improved network model. The LDW-YOLOv8 model is improved based on the YOLOv8 model. L stands for Light, D stands for the use of full-dimensional dynamic convolution, and W stands for Wiou v3_Loss loss function. According to the characteristics of the data set, the size of the feature map is adjusted. First, the four downsamplings in the YOLOv8 backbone network are reduced to three times, so that the output feature map size is expanded from 20×20×512 to 40×40×512. At the same time, the one upsampling of the neck network and the feature fusion module are reduced, and the detection head part is redesigned to remove the large target detection head of the smallest layer of feature maps. By adjusting the basic network feature fusion structure of the target detection model, the learning of redundant features can be reduced.

[0089] The downsampling module in the YOLOv8 backbone network is improved, and the ordinary convolution of the three downsampling modules is changed to full-dimensional dynamic convolution ODConv. The full-dimensional dynamic convolution adopts a multi-dimensional attention mechanism along the four dimensions of the kernel space through a parallel strategy to learn complementary attention.

[0090] The two-dimensional convolution is described as: Output(x) = W*x, where x represents the input feature, W represents the convolution layer, and * represents the convolution operation. The dynamic convolution is described as: Output(x) = (α1w1+…+α n W n )*x, where W i represents the i-th group of convolution filters, α i W i The weight parameter of the full-dimensional dynamic convolution is Output(x)=(α w1 ⊙α f1 ⊙α c1 ⊙α s1 ⊙W1+…+α wn ⊙α fn ⊙α cn ⊙α sn ⊙W n )*x, where α wi Equivalent to α i , the newly introduced α fi , α ci and α si , represent the learnable weights of the output dimension, input dimension, and spatial dimension, respectively.

[0091] The bounding box loss function of the YOLOv8 object detection network is optimized, and the Wise-IoUv3 Loss based on the dynamic non-monotonic focusing mechanism is used to replace the original CIoU_Loss. This method can focus more on anchor boxes of normal quality and improve the overall performance of the detector.

[0092] Among them, β and r are controlled by hyperparameters α and δ; in, W g and H g Respectively represent the width and height of the minimum bounding box; the superscript * represents W g and H g It is separated from the computational graph; The final structural diagram of the target detection model in this embodiment is as follows Figure 2 shown.

[0093] The specific training process of the target detection model is:

[0094] The image size is scaled to 640*640, and the Stochastic Gradient Descent (SGD) algorithm is used as the optimizer. The basic parameter learning rate is set to 0.01. All experiments are performed on the GPU for 200 rounds. By conducting experiments based on the collection and preparation of data sets and comparing with other models, the number of parameters, amount of calculation, model size, FPS, inference time, accuracy, recall rate, AP50 and AP50-95 are used as the overall evaluation indicators of the model. The ablation experiment is shown in Table 1:

[0095] Table 1

[0096]

[0097]

[0098] The comparative experiment is shown in Table 2:

[0099] Table 2

[0100]

[0101]

[0102] In this embodiment, the analyzing the fish target detection result to obtain the fish tracking trajectory includes:

[0103] Divide the fish target detection result into a high-score detection frame and a low-score detection frame according to a confidence threshold in the target detection result, put the high-score detection frame into a high-score detection frame set, and put the low-score detection frame into a low-score detection frame set;

[0104] Performing trajectory matching on the high-score detection frame set, placing unsuccessfully matched trajectories into a first remaining trajectory set, and placing unsuccessfully matched high-score detection frames in the high-score detection frame set into a remaining detection frame set;

[0105] Matching the low-score detection frame set with the first remaining track set, placing unsuccessfully matched tracks into a second remaining track set, and deleting the unsuccessfully matched low-score detection frames in the low-score detection frame set;

[0106] Update the Kalman filter of the successfully matched trajectory;

[0107] Generate a new track according to the high-score detection frame in the remaining detection frame set whose confidence is greater than the tracking score threshold, and delete the track with more than 30 frames in the second remaining track set;

[0108] A Kalman filter and an extended Kalman filter are constructed, and the new position of the trajectory with acceleration less than the acceleration threshold is predicted by the Kalman filter, and the new position of the trajectory with acceleration greater than the acceleration threshold is predicted by the extended Kalman filter to obtain the fish tracking trajectory.

[0109] The target detection results output by the LDW-YOLOv8 model are calculated according to the confidence threshold τ high and τ lww Divide into high-score detection frames and low-score detection frames, and divide the confidence greater than τ high The high-score detection frames are put into the high-score detection frame set D high In the above example, the confidence level is less than τ high and greater than τ low The low-score detection frame is put into the low-score detection frame set D low middle.

[0110] The high-score detection box set D high Match with existing trajectories (when the algorithm detects in the first frame, there is no trajectory at this time, so it will create trajectory objects for all target detection boxes and store them. Starting from the second frame, the detection box is matched with the trajectory created in the first frame, and then the next frame is matched with the corresponding trajectory of the previous frame), calculate its IoU distance matrix, match it with the Hungarian algorithm, and put the trajectories that failed to match successfully into the first remaining trajectory set T remain , put the high-score detection frames that failed to match successfully in the high-score detection frame set into the remaining detection frame set D remain Then the low-score detection box set D low With the first remaining trajectory set T remain Match again and put the unmatched trajectories into the second remaining trajectory set T lost In the process, the low-score detection frames that failed to match successfully in the low-score detection frame set are deleted. For all the above successfully matched trajectories, their Kalman filters are updated and put into the current trajectory set.

[0111] For the remaining detection box set D remainMedium confidence is greater than the tracking score threshold τ track Create a new track for the high-score detection box and delete the second remaining track set T lost Trajectories with more than 30 frames.

[0112] Construct a Kalman filter (KF) and an extended Kalman filter (EKF) to predict that the acceleration is less than the acceleration threshold τ a The new position of the trajectory is predicted by the extended Kalman filter to be greater than the acceleration threshold τ a The new position of the trajectory is obtained to obtain the fish tracking trajectory.

[0113] Specifically, when the acceleration threshold of the trajectory is greater than τ a The extended Kalman filter is used to predict its new position. The extended Kalman filter performs Taylor expansion on the nonlinear function, linearizes the nonlinear function, and then predicts the fish tracking trajectory.

[0114] This embodiment solves the problem of the tracker being unable to make correct predictions due to mutual occlusion between freely moving fish, erroneous ID changes, and sudden acceleration of fish through the above method, thereby ensuring the accuracy of the fish tracking trajectory.

[0115] In this embodiment, before analyzing the fish tracking trajectory, the method further includes: performing Gaussian interpolation processing on the fish tracking trajectory, and the specific steps include:

[0116] The observation value of the i-th track in the fish tracking track at the x-frame is calculated according to the following formula: for:

[0117]

[0118] in, is the coordinate variable of trajectory i at the x frame, The mean is 0 and the variance is Gaussian distribution, f (i) ∈GP(0,k(x,x′)), l is a hyperparameter;

[0119] Through the known frame x1x2x3...x n and the frame to be predicted x * Compute the covariance matrix:

[0120]

[0121] K * =[k(x * ,x1) k(x *, x2) … k(x * , x n )]

[0122] The best estimate of Among them, y is The n×1 dimensional matrix composed of

[0123] By setting the hyperparameter σ to represent the maximum number of frame intervals for performing Gaussian interpolation, Gaussian interpolation is performed when the number of intervals is less than the hyperparameter σ.

[0124] is the observation value corresponding to the i-th trajectory at the x-th frame (x1x2x3…x n ), now we need to get x * The corresponding predicted value is This is the predicted value. The setting of hyperparameters represents a restriction relationship. Gaussian smoothing interpolation processing strategy will be executed only when the conditions are met. In the above way, Gaussian interpolation processing is performed on the fish tracking trajectory before analyzing it, which improves the analysis and recognition results of fish movement characteristics.

[0125] The specific process of evaluating the target tracking results for fish tracking trajectories:

[0126] The fish target detection results of 6 videos of fish free motion in different states were selected as the test set. The high-order tracking accuracy (HOTA), identification mean number ratio (IDF1), majority tracking (MT), main lost targets (ML), identity switching number (IDs), multi-target tracking accuracy (MOTA) and multi-target tracking precision (MOTP) were used for comprehensive evaluation to ensure the accuracy of the behavior recognition model. The evaluation results are shown in Table 3:

[0127] Table 3

[0128]

[0129] Only when the tracking accuracy is high enough can the error caused by extracting motion features through calculation be reduced.

[0130] In this embodiment, the fish movement characteristics include: swimming speed, swimming acceleration and discreteness. In the process of analyzing the fish tracking trajectory, the image is corrected through inverse perspective transformation.

[0131] In the process of acquiring and analyzing the movement characteristics of various fish, due to the perspective effect of the monocular camera, the image is corrected by inverse perspective mapping (IPM) to ensure that the pixel distance is evenly distributed in the entire image.

[0132] Fish movement characteristics include swimming speed, swimming acceleration and discreteness. Through inverse perspective transformation, the two-dimensional pixel point (u, v) is mapped to the corresponding world coordinate (X W ,Y W ,Z W ):

[0133] [uv 1] T =KRT[X w Y w Z w 1] T , where R is the rotation matrix, T is the translation matrix, K is the intrinsic matrix, h is the camera height, f is the focal length, ku and kv are the measured calibration values ​​in horizontal and vertical pixel units respectively, and c x , c y To correct the principal point displacement of the image plane optical axis.

[0134]

[0135]

[0136]

[0137] The discrete degree of the i-th frame can be expressed as: D i =det(cov(x, y)), where det(.) represents the determinant calculation of the center coordinates of the fish target detection frame position, and cov(.) is the covariance matrix operation of the matrix. i The smaller it is, the denser the fish distribution is, and vice versa, the more dispersed the fish distribution is.

[0138] The swimming speed of the individual with ID number i in the k+Δt frame and assuming that the total number of fish individuals in the video is N, the average swimming speed of the whole in the k+1 frame is:

[0139]

[0140] The swimming acceleration of the individual with ID number i at time t2 and assuming that the total number of fish individuals in the video is N, the average swimming acceleration of the whole at time t2 is:

[0141]

[0142] In this embodiment, the training steps of the behavior recognition model include:

[0143] Obtain samples of fish movement characteristics;

[0144] Marking the fish behavior recognition result corresponding to the fish movement feature sample as a label;

[0145] Dividing the fish motion feature samples into a fish motion feature sample training set and a fish motion feature sample test set;

[0146] The behavior recognition model is obtained by training the fish motion feature sample training set, and the behavior recognition model is evaluated by the fish motion feature sample test set.

[0147] The behavior recognition model is trained based on the LTSM (Long Short-Term Memory) model. During the training of the behavior recognition model, fish motion feature samples are obtained, and the fish behavior recognition results corresponding to the fish motion feature samples are annotated as labels. After obtaining the three fish motion features of swimming speed, swimming acceleration and discreteness, the fish behavior recognition results under low temperature, hypoxia, normal and feeding states are annotated respectively. The fish motion feature samples are divided into a fish motion feature sample training set and a fish motion feature sample test set. 70% of the fish motion feature sample data set is randomly selected as the fish motion feature sample training set, and 30% is used as the fish motion feature sample test set. The behavior recognition model is obtained by training the fish motion feature sample training set, and the behavior recognition model is evaluated by the fish motion feature sample test set.

[0148] The fish behavior identification device provided by the present invention is described below. The fish behavior identification device described below and the fish behavior identification method described above can be referenced to each other.

[0149] like Figure 3 As shown, the second embodiment of the present invention further provides a fish behavior recognition device, comprising:

[0150] An image acquisition module 210 is used to acquire fish movement images;

[0151] The target detection module 220 is used to input the fish motion image into the target detection model to obtain the fish target detection result;

[0152] A trajectory analysis module 230 is used to analyze the fish target detection result to obtain a fish tracking trajectory;

[0153] A feature analysis module 240 is used to analyze the fish tracking trajectory to obtain fish movement characteristics;

[0154] A behavior recognition module 250 is used to input the fish movement characteristics into a behavior recognition model to obtain a fish behavior recognition result;

[0155] Among them, the target detection model is trained based on fish motion image samples and the fish target detection results corresponding to the fish motion image samples as labels; the behavior recognition model is trained based on fish motion feature samples and the fish behavior recognition results corresponding to the fish motion feature samples.

[0156] The second embodiment of the present invention provides a fish behavior recognition device, which obtains fish target detection results by acquiring fish motion images and inputting them into a target detection model, so as to identify fish target individuals from the original data of fish motion images. The fish target detection results are analyzed to obtain fish tracking trajectories, that is, the motion trajectory routes of fish in the current state. The fish tracking trajectories are further analyzed, and fish motion features are extracted from the fish tracking trajectories by means of calculation and quantification to reflect various motion data of fish in the current state. The fish motion features are input into the behavior recognition model, and a comprehensive analysis is performed based on various fish motion features to obtain fish behavior recognition results, that is, the fish behavior in the current state, which provides an effective reference for fish breeding or diagnosis. Among them, the target detection model is obtained by training based on fish motion image samples and fish target detection results corresponding to the fish motion image samples as labels; the behavior recognition model is obtained by training based on fish motion feature samples and fish behavior recognition results corresponding to the fish motion feature samples. Based on machine vision and deep learning, the present invention constructs a model framework that integrates motion feature extraction and behavior state recognition functions, effectively reducing the model size and inference time while ensuring accuracy. The defects of complex parameters and difficult identification in the prior art of fish behavior identification are solved, and the accuracy and efficiency of fish behavior identification are improved.

[0157] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communications interface 320 and the memory 330 communicate with each other via the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the fish behavior recognition method, which includes:

[0158] Obtain images of fish movement;

[0159] Inputting the fish motion image into the target detection model to obtain the fish target detection result;

[0160] Analyze the fish target detection result to obtain the fish tracking trajectory;

[0161] Analyzing the fish tracking trajectory to obtain fish movement characteristics;

[0162] Inputting the fish movement characteristics into a behavior recognition model to obtain a fish behavior recognition result;

[0163] Among them, the target detection model is trained based on fish motion image samples and the fish target detection results corresponding to the fish motion image samples as labels; the behavior recognition model is trained based on fish motion feature samples and the fish behavior recognition results corresponding to the fish motion feature samples.

[0164] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0165] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the fish behavior identification method provided by the above methods, the method includes:

[0166] Obtain images of fish movement;

[0167] Inputting the fish motion image into the target detection model to obtain the fish target detection result;

[0168] Analyze the fish target detection result to obtain the fish tracking trajectory;

[0169] Analyzing the fish tracking trajectory to obtain fish movement characteristics;

[0170] Inputting the fish movement characteristics into a behavior recognition model to obtain a fish behavior recognition result;

[0171] Among them, the target detection model is trained based on fish motion image samples and the fish target detection results corresponding to the fish motion image samples as labels; the behavior recognition model is trained based on fish motion feature samples and the fish behavior recognition results corresponding to the fish motion feature samples.

[0172] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the fish behavior identification method provided by the above methods, the method comprising:

[0173] Obtain images of fish movement;

[0174] Inputting the fish motion image into the target detection model to obtain the fish target detection result;

[0175] Analyze the fish target detection result to obtain the fish tracking trajectory;

[0176] Analyzing the fish tracking trajectory to obtain fish movement characteristics;

[0177] Inputting the fish movement characteristics into a behavior recognition model to obtain a fish behavior recognition result;

[0178] Among them, the target detection model is trained based on fish motion image samples and the fish target detection results corresponding to the fish motion image samples as labels; the behavior recognition model is trained based on fish motion feature samples and the fish behavior recognition results corresponding to the fish motion feature samples.

[0179] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0180] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying fish behavior, characterized in that: include: Obtain images of fish movement; Inputting the fish motion image into the target detection model to obtain the fish target detection result; Analyze the fish target detection result to obtain the fish tracking trajectory; Analyzing the fish tracking trajectory to obtain fish movement characteristics; Inputting the fish movement characteristics into a behavior recognition model to obtain a fish behavior recognition result; The target detection model is obtained by training based on fish motion image samples and fish target detection results corresponding to the fish motion image samples as labels; The behavior recognition model is trained based on fish movement feature samples and fish behavior recognition results corresponding to the fish movement feature samples.

2. The fish behavior identification method according to claim 1, characterized in that: The training steps of the target detection model include: Acquire fish motion image samples, and perform data enhancement processing on the fish motion image samples; Annotating the fish target detection result corresponding to the fish motion image sample as a label; Dividing the fish motion image samples into a fish motion image sample training set, a fish motion image sample verification set, and a fish motion image sample test set; The target detection model is obtained by training with the fish motion image sample training set, the fish target detection result of the target detection model is verified with the fish motion image sample verification set, and after obtaining the network weight of the target detection model, the target detection model is tested with the fish motion image sample test set.

3. The fish behavior identification method according to claim 2, characterized in that: The target detection model is obtained by training based on the LDW-YOLOv8 model. Before the target detection model is obtained by training the fish motion image sample training set, the method further includes: Adjusting the basic network feature fusion structure of the LDW-YOLOv8 model to reduce the learning of redundant features; Replace the backbone downsampling convolution of the LDW-YOLOv8 model with full-dimensional dynamic convolution; The bounding box loss function CIoU_Loss of the LDW-YOLOv8 model is replaced by Wise-IoUv3 Loss.

4. The fish behavior identification method according to claim 1, characterized in that: The analyzing the fish target detection result to obtain the fish tracking trajectory includes: Divide the fish target detection result into a high-score detection frame and a low-score detection frame according to a confidence threshold in the target detection result, put the high-score detection frame into a high-score detection frame set, and put the low-score detection frame into a low-score detection frame set; Performing trajectory matching on the high-score detection frame set, placing unsuccessfully matched trajectories into a first remaining trajectory set, and placing unsuccessfully matched high-score detection frames in the high-score detection frame set into a remaining detection frame set; Matching the low-score detection frame set with the first remaining track set, placing unsuccessfully matched tracks into a second remaining track set, and deleting the unsuccessfully matched low-score detection frames in the low-score detection frame set; Update the Kalman filter of the successfully matched trajectory; Generate a new track according to the high-score detection frame in the remaining detection frame set whose confidence is greater than the tracking score threshold, and delete the track with more than 30 frames in the second remaining track set; A Kalman filter and an extended Kalman filter are constructed, and the new position of the trajectory with acceleration less than the acceleration threshold is predicted by the Kalman filter, and the new position of the trajectory with acceleration greater than the acceleration threshold is predicted by the extended Kalman filter to obtain the fish tracking trajectory.

5. The fish behavior identification method according to claim 4, characterized in that: Before analyzing the fish tracking trajectory, the method further includes: performing Gaussian interpolation processing on the fish tracking trajectory, and the specific steps include: The observation value of the i-th track in the fish tracking track at the x-frame is calculated according to the following formula: for: in, is the coordinate variable of trajectory i at the x frame, The mean is 0 and the variance is Gaussian distribution, f (i) ∈GP(0,k(x,x′)), l is a hyperparameter; Through the known frame x1x2x3······x n and the frame to be predicted x * Compute the covariance matrix: K * =[k(x * ,x1) k(x * ,x2) … k(x * ,x n )] The best estimate of Among them, y is The n×1 dimensional matrix composed of By setting the hyperparameter σ to represent the maximum number of frame intervals for performing Gaussian interpolation, Gaussian interpolation is performed when the number of intervals is less than the hyperparameter σ.

6. The fish behavior identification method according to claim 1, characterized in that: The fish movement characteristics include: swimming speed, swimming acceleration and discreteness. In the process of analyzing the fish tracking trajectory, the image is corrected through inverse perspective transformation.

7. The method for identifying fish behavior according to any one of claims 1 to 6, characterized in that: The training steps of the behavior recognition model include: Obtain samples of fish movement characteristics; Marking the fish behavior recognition result corresponding to the fish movement feature sample as a label; Dividing the fish motion feature samples into a fish motion feature sample training set and a fish motion feature sample test set; The behavior recognition model is obtained by training the fish motion feature sample training set, and the behavior recognition model is evaluated by the fish motion feature sample test set.

8. A fish behavior recognition device, characterized in that: include: An image acquisition module, used for acquiring fish movement images; A target detection module, used for inputting the fish motion image into a target detection model to obtain a fish target detection result; A trajectory analysis module, used to analyze the fish target detection result to obtain the fish tracking trajectory; A feature analysis module, used to analyze the fish tracking trajectory to obtain fish movement characteristics; A behavior recognition module, used for inputting the fish movement characteristics into a behavior recognition model to obtain a fish behavior recognition result; The target detection model is obtained by training based on fish motion image samples and fish target detection results corresponding to the fish motion image samples as labels; The behavior recognition model is trained based on fish movement feature samples and fish behavior recognition results corresponding to the fish movement feature samples.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the fish behavior recognition method as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the fish behavior recognition method according to any one of claims 1 to 7 is implemented.

Images