Fish behavior track determination method and device, electronic equipment and medium

Abstract

The invention provides a fish behavior track determination method and device, electronic equipment and a medium. The method comprises the steps that continuous multiple frames of to-be-recognized images collected in different directions corresponding to a to-be-detected fish are input into a target detection model, continuous multiple frames of target images and positioning coordinates corresponding to the target images are obtained, the target images comprise a target frame, and the target frame is used for determining the position of the to-be-detected fish in the to-be-recognized images; the positioning coordinates are pixel coordinates of a central point of the target frame; based on continuous multiple frames of target images and the positioning coordinates corresponding to the target images, performing three-dimensional trajectory reconstruction on the to-be-detected fish, and determining a behavior trajectory of the to-be-detected fish; the target detection model comprises a backbone network, a neck structure and a decoupling head structure, the backbone network comprises a spatial pyramid pooling module and a multi-head self-attention module, and the spatial pyramid pooling module is connected with the multi-head self-attention module. According to the fish behavior trajectory determination method provided by the invention, three-dimensional trajectory restoration of the to-be-detected fish can be realized.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, electronic equipment and medium for determining a fish behavior trajectory. Background Art

[0002] In recirculating aquaculture systems, ammonia is produced by the respiration and excretion of aquatic animals and the decomposition of unconsumed food by microorganisms, leading to deterioration of the water environment. Ammonia is usually measured as total ammonia nitrogen (TA-N) and exists in two forms in water: ionized ammonium (NH4 + ) and non-ionized ammonia (NH3). Ionized ammonium is non-toxic to aquatic animals, while non-ionized ammonia is highly toxic to aquatic animals. In intensive aquaculture, fish are raised at high stocking densities, especially when water circulation is insufficient or infrequent, which can lead to a continuous accumulation of ammonia in the water, causing economic losses to aquaculture.

[0003] The rapidly developing computer vision technology monitors the relevant indicators of abnormal behavior of aquatic animals such as fish, which indirectly reflects whether the ammonia nitrogen concentration in the water quality exceeds the standard. It is a new method and idea. The use of this technology can realize unmanned monitoring around the clock on the one hand, and objectively give early warnings based on the behavioral indicators of fish on the other hand to avoid aquaculture losses caused by abnormal water quality. However, due to the limitations of underwater living conditions and the higher complexity of the background, it brings certain difficulties to the target recognition and behavior tracking of the detection model to a certain extent. Summary of the invention

[0004] The present invention provides a method, device, electronic device and medium for determining fish behavior trajectory, so as to solve the problem that detection models in the prior art have difficulty in target recognition and behavior tracking.

[0005] The present invention provides a method for determining a fish behavior trajectory, comprising:

[0006] Inputting multiple frames of images to be identified collected from different directions corresponding to the fish to be detected into the target detection model to obtain multiple frames of target images and positioning coordinates corresponding to the target images, wherein the target image includes a target frame, and the target frame is used to determine the position of the fish to be detected in the image to be identified, and the positioning coordinates are the pixel coordinates of the center point of the target frame;

[0007] Based on the continuous multiple frames of target images and the positioning coordinates corresponding to the target images, a three-dimensional trajectory of the fish to be detected is reconstructed to determine the behavior trajectory of the fish to be detected;

[0008] Among them, the target detection model includes a backbone network, a neck structure and a decoupling head structure, the backbone network includes a spatial pyramid pooling module and a multi-head self-attention module, and the output end of the spatial pyramid pooling module is connected to the input end of the multi-head self-attention module.

[0009] In some embodiments, the step of inputting a plurality of consecutive frames of images to be identified and collected from different directions corresponding to the fish to be detected into a target detection model to obtain a plurality of consecutive frames of target images and positioning coordinates corresponding to the target images includes:

[0010] Based on the spatial pyramid pooling module, feature extraction is performed on the image to be identified to obtain a first feature map;

[0011] Inputting the first feature map into the multi-head self-attention module for feature encoding to obtain a second feature map;

[0012] Inputting the second feature map into the neck structure for feature fusion to obtain a third feature map;

[0013] The third feature map is input into the decoupling head structure to obtain the target image and the positioning coordinates corresponding to the target image.

[0014] In some embodiments, inputting the third feature map into the decoupling head structure to obtain the target image and the positioning coordinates corresponding to the target image includes:

[0015] Inputting the third feature map into the classification branch included in the decoupling head structure, and outputting the target image, wherein the target image is used to determine the category of the fish to be detected;

[0016] The third feature map is input into the regression branch included in the decoupling head structure, and the positioning coordinates are output.

[0017] In some embodiments, the three-dimensional trajectory of the fish to be detected is reconstructed based on the continuous multiple frames of target images and the positioning coordinates corresponding to the target images to determine the behavior trajectory of the fish to be detected, including:

[0018] Based on the KM algorithm, the feature point sets corresponding to two adjacent frames of target images are matched to determine multiple detection targets in the two adjacent frames of target images, and a tracking mark is assigned to each detection target, wherein the detection target is used to indicate the fish to be detected;

[0019] Based on the Kalman filter algorithm, the tracking identifier of the detection target and the positioning coordinates corresponding to the target image, the position of the detection target is predicted to obtain a position prediction result of the detection target;

[0020] Based on the position detection result, the corresponding points of the same detection target in the to-be-identified images collected in different directions are determined by direct linear transformation method to determine the three-dimensional coordinates of the detection target;

[0021] Based on the three-dimensional coordinates of the detection target, the behavior trajectory of the fish to be detected is determined, and the behavior trajectory is used to determine the movement state of the fish to be detected. The quantitative indicators corresponding to the movement state include at least one of the following: movement amount, movement speed and spatial distribution ratio.

[0022] In some embodiments, after performing position prediction on the detection target and obtaining the position prediction result of the detection target, the method further includes:

[0023] In the case where the motion state of the detection target cannot be detected in the continuous multiple frames of target images, based on the kernel correlation filtering algorithm, the detection target that disappears in the continuous multiple frames of target images is identified, and the disappeared detection target is compensated to determine the optimal tracking area;

[0024] Based on the optimal tracking area, determining the tracking position of the detection target in the continuous multiple frames of target images;

[0025] Based on the tracking position of the detection target, the behavior trajectory of the fish to be detected is determined.

[0026] In some embodiments, the training data set of the target detection model is obtained by:

[0027] The image data of the fish to be detected is obtained by a camera, wherein the image data includes a calibration plate; wherein the camera is installed in the front end direction and the top direction of the container where the fish to be detected is located, and a calibration plate is placed in the container, and the calibration plate is used to determine the inherent matrix and distortion parameters of the camera;

[0028] Based on the intrinsic matrix and the distortion parameter, the image data is corrected to obtain the data set.

[0029] The present invention also provides a device for determining fish behavior trajectory, comprising:

[0030] Inputting multiple frames of images to be identified collected from different directions corresponding to the fish to be detected into the target detection model to obtain multiple frames of target images and positioning coordinates corresponding to the target images, wherein the target image includes a target frame, and the target frame is used to determine the position of the fish to be detected in the image to be identified, and the positioning coordinates are the pixel coordinates of the center point of the target frame;

[0031] Based on the continuous multiple frames of target images and the positioning coordinates corresponding to the target images, a three-dimensional trajectory of the fish to be detected is reconstructed to determine the behavior trajectory of the fish to be detected;

[0032] Among them, the target detection model includes a backbone network, a neck structure and a decoupling head structure, the backbone network includes a spatial pyramid pooling module and a multi-head self-attention module, and the output end of the spatial pyramid pooling module is connected to the input end of the multi-head self-attention module.

[0033] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-described methods for determining fish behavior trajectories is implemented.

[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for determining the fish behavior trajectory as described in any one of the above is implemented.

[0035] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for determining the fish behavior trajectory as described in any one of the above is implemented.

[0036] The method, device, electronic device and medium for determining the fish behavior trajectory provided by the present invention determine multiple consecutive frames of target images and the positioning coordinates corresponding to the target images through a target detection model, thereby determining the position of the fish to be detected in the image to be identified and the positioning information corresponding to the fish to be detected, and combining with three-dimensional trajectory reconstruction to achieve three-dimensional trajectory restoration of the fish to be detected, and then the movement state of the fish to be detected can be analyzed, including quantitative indicators such as movement amount, speed, and spatial distribution ratio, which can further improve the level of intelligent animal breeding. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 This is one of the flow charts of the method for determining the fish behavior trajectory provided by the present invention;

[0039] Figure 2 It is a schematic diagram of the model structure of the method for determining the fish behavior trajectory provided by the present invention;

[0040] Figure 3It is a schematic diagram of the MHSA module structure of the fish behavior trajectory determination method provided by the present invention;

[0041] Figure 4 This is the second flow chart of the method for determining the fish behavior trajectory provided by the present invention;

[0042] Figure 5 It is a schematic diagram of the behavior trajectory of sturgeon in the method for determining the behavior trajectory of fish provided by the present invention;

[0043] Figure 6 It is a structural schematic diagram of a fish behavior trajectory determination device provided by the present invention;

[0044] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Combine the following Figure 1-Figure 7 The invention describes a method, device, electronic device and medium for determining fish behavior trajectories.

[0047] Figure 1 This is one of the flow charts of the method for determining the fish behavior trajectory provided by the present invention. Figure 1 The method for determining the fish behavior trajectory provided by the present invention includes: step 110 and step 120.

[0048] Step 110, inputting a plurality of consecutive frames of images to be identified collected from different directions corresponding to the fish to be detected into the target detection model, obtaining a plurality of consecutive frames of target images and positioning coordinates corresponding to the target images, wherein the target image includes a target frame, and the target frame is used to determine the position of the fish to be detected in the image to be identified, and the positioning coordinates are the pixel coordinates of the center point of the target frame;

[0049] Step 120: reconstruct the three-dimensional trajectory of the fish to be detected based on the continuous multiple frames of target images and the positioning coordinates corresponding to the target images, and determine the behavior trajectory of the fish to be detected;

[0050] Among them, the target detection model includes a backbone network, a neck structure and a decoupling head structure. The backbone network includes a spatial pyramid pooling module and a multi-head self-attention module. The output end of the spatial pyramid pooling module is connected to the input end of the multi-head self-attention module.

[0051] The execution subject of the method for determining the fish behavior trajectory provided by the present invention can be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited by the present invention.

[0052] The following takes the method for determining the fish behavior trajectory provided by the present invention executed by a computer as an example to describe the technical solution of the present invention in detail.

[0053] It should be noted that the fish to be detected can be replaced by other aquatic animals or terrestrial animals, such as shrimp, crab, horse, sheep or cattle, etc., and the present invention does not specifically limit this. The fish behavior trajectory determination method provided by the present invention will be described later using fish as an example.

[0054] In actual implementation, the image dataset corresponding to the fish to be detected is obtained in advance; then the target detection model is built based on the improved YOLOv8 model; the image dataset corresponding to the fish to be detected is input into the target detection model for training until a trained target detection model is obtained.

[0055] In this step, multiple consecutive frames of images to be identified corresponding to the fish to be detected are obtained; the multiple consecutive frames of images to be identified are input into the trained target detection model for detection, and the target detection model outputs the detection results, which are multiple consecutive frames including the target image of the target frame and the positioning coordinates corresponding to the target image.

[0056] Among them, the improved YOLOv8 model can include a backbone network, a neck structure and a decoupled head structure, such as Figure 2 shown.

[0057] The backbone network is a feature extraction network, and its function is to extract information from the image for use by the subsequent network. The neck structure is set between the backbone network and the decoupling head structure to better utilize the features extracted by the backbone network. The decoupling head structure can use the features extracted by the previous network to make identification and obtain target detection results.

[0058] The present invention improves the traditional YOLOv8 model and introduces a Multi Head Self-Attention (MHSA) module into the existing YOLOv8 model. The MHSA module can assign weights to each feature according to feature relevance and focus the model's attention on the object to be detected. The improved model can reduce the interference of irrelevant background during the detection process and improve the feature extraction capability of the network.

[0059] The MHSA module can assign weights to each feature according to feature relevance, so that the network's attention is focused on the object to be detected, reducing the interference of irrelevant background, thereby improving the network's feature extraction ability. The attention transformation of Q, K, and V is performed in the MHSA module, and the process is repeated k times independently. Finally, the results of the scaled dot product attention are spliced ​​and linearly transformed to obtain the output of the MHSA module.

[0060] The object detection model also uses a decoupled head structure. The YOLOv8 model replaces the coupled head structure in the YOLOv5 model with a decoupled head structure, realizing the separation of the detection head and the classification head. In addition, YOLOv8 replaces Anchor-Base (which requires anchor points) with Anchor-Free (which does not require anchor points), allowing the object to be located by the center of the fish to be detected, and then predicting the distance between the center point and the bounding box. Among them, the bounding box is the target box that determines the position of the fish to be detected in the image to be identified.

[0061] In actual implementation, after determining the detection results output by the target detection model, the three-dimensional trajectory of the fish to be detected is reconstructed based on multiple consecutive frames of target images and the positioning coordinates corresponding to the target images. The behavior trajectory of the fish to be detected can also be drawn based on the positioning coordinates, thereby determining the behavior trajectory of the fish to be detected.

[0062] In this step, the three-dimensional trajectory of the fish to be detected is reconstructed based on the Kalman filter algorithm, the Kuhn-Mankels algorithm and the kernel correlation filter algorithm.

[0063] It is understandable that the positioning coordinates that the target detection model can output are two-dimensional coordinates, and the positioning coordinates corresponding to the target images corresponding to different directions are not yet associated. The present invention associates the positioning coordinates corresponding to the target images corresponding to different directions through Kalman filtering, Kuhn-Mankels and kernel correlation filtering algorithms, thereby converting the two-dimensional coordinates into three-dimensional coordinates. Among them, the three-dimensional coordinates can be used to calculate quantitative indicators such as movement speed, movement amount and spatial distribution ratio, and then the three-dimensional trajectory can be reconstructed.

[0064] The Kalman filtering algorithm is an algorithm that uses the linear system state equation to optimally estimate the system state through system input and output observation data.

[0065] The Kuhn-Munkres Algorithm (KM) solves the optimal matching problem of weighted bipartite graphs.

[0066] The Kernel Correlation Filter (KCF) algorithm is a discriminative tracking algorithm that uses the given samples to train a discriminative classifier to determine whether the tracked object is the target or the surrounding background information. It mainly uses the circulant matrix to collect samples and the fast Fourier transform to accelerate the algorithm.

[0067] In the kernel correlation filter algorithm, the Kuhn-Mankels algorithm is used to compensate for the disappearance or overlap of objects during tracking, thereby reducing the errors caused by water surface fluctuations and illumination changes. The kernel correlation filter algorithm is used to compensate for the disappearance or occlusion of objects during tracking, thereby reducing the errors caused by occlusion. The Kalman filter can estimate the motion state of the fish to be detected through prior estimation and posterior estimation.

[0068] The method for determining the fish behavior trajectory provided by the present invention determines multiple consecutive frames of target images and the positioning coordinates corresponding to the target images through a target detection model, thereby determining the position of the fish to be detected in the image to be identified and the positioning information corresponding to the fish to be detected, and combines three-dimensional trajectory reconstruction to achieve three-dimensional trajectory restoration of the fish to be detected, and then can analyze the movement state of the fish to be detected, including quantitative indicators such as movement amount, speed, and spatial distribution ratio, which can further improve the level of intelligent animal breeding.

[0069] In some embodiments, step 110 may include:

[0070] Based on the spatial pyramid pooling module, feature extraction is performed on the image to be identified to obtain the first feature map;

[0071] Input the first feature map into the multi-head self-attention module for feature encoding to obtain a second feature map;

[0072] The second feature map is input into the neck structure for feature fusion to obtain a third feature map;

[0073] The third feature map is input into the decoupling head structure to obtain the target image and the positioning coordinates corresponding to the target image.

[0074] In actual implementation, the image to be recognized is input into Figure 2In the target detection model (improved YOLOv8 structure) shown, the image to be identified will be processed in sequence through the backbone network, the neck structure and the decoupling head structure.

[0075] In actual implementation, the improved YOLOv8 model introduces the MHSA module in the Transformer after the Spatial Pyramid PoolingFast (SPPF) module. The MHSA module structure is as follows: Figure 3 shown.

[0076] The image to be identified is processed by the module before the SPPF module to obtain a feature map, and then the feature map is input into the SPPF module for feature extraction to obtain the first feature map. The SPPF module can perform multi-scale feature extraction on the same feature map, which is conducive to improving the accuracy of the model.

[0077] The first feature map is input into the MHSA module for feature encoding. According to the feature encoding, three matrices of query Q, key K and value V are created. The matrix creation process refers to multiplying the three weight matrices obtained during the training process by the encoding; the three matrices of query Q, key K and value V are scaled and dot-producted; the above steps need to be done several times, each time representing an attention head; since the weight matrices of linear transformation of query Q, key K and value V are different each time, it is necessary to splice the results of several scaled dot-product operations, and finally, output the result of the MHSA module through linear transformation to obtain the second feature map.

[0078] The second feature map is input into the neck structure for feature fusion. The neck structure is constructed based on the path aggregation network (PAN) and feature pyramid network (FPN).

[0079] It is understandable that deepening the model network layer can obtain more feature information to improve the prediction effect, but it will also lead to a reduction in target position information. The premise of drawing the behavior trajectory of the fish to be detected is to obtain the positioning coordinates. Therefore, the neck structure uses PAN and FPN to perform multi-scale fusion on the second feature map to obtain the third feature map to eliminate the information loss caused by excessive convolution operations.

[0080] The third feature map is input into the decoupling head structure, and the detection result is output. The detection result is the target image and the positioning coordinates corresponding to the target image. The target frame will be displayed in the target image, and the target frame is used to locate the center point of the fish to be detected.

[0081] In some embodiments, the third feature map is input into the decoupling head structure to obtain the target image and the positioning coordinates corresponding to the target image, including:

[0082] Inputting the third feature map into the classification branch included in the decoupling head structure, outputting a target image, where the target image is used to determine the category of the fish to be detected;

[0083] The third feature map is input into the regression branch included in the decoupling head structure, and the positioning coordinates are output.

[0084] In actual implementation, the improved YOLOv8 model of the present invention replaces the coupling head in the YOLOv5 model with a decoupling head, thereby realizing the separation of the detection head and the classification head. Figure 2 It can be seen that the improved YOLOv8 model only retains the regression branch and the classification branch, and deletes the object branch. The classification branch is used for target recognition, and the regression branch is used to achieve target positioning.

[0085] In some embodiments, based on multiple consecutive frames of target images and the positioning coordinates corresponding to the target images, a three-dimensional trajectory of the fish to be detected is reconstructed to determine the behavior trajectory of the fish to be detected, including:

[0086] Based on the KM algorithm, the feature point sets corresponding to two adjacent target images are matched to determine multiple detection targets in the two adjacent target images, and a tracking mark is assigned to each detection target. The detection target is used to indicate the fish to be detected.

[0087] Based on the Kalman filter algorithm, the tracking mark of the detection target and the positioning coordinates corresponding to the target image, the position of the detection target is predicted to obtain the position prediction result of the detection target;

[0088] Based on the position detection results, the corresponding points of the same detection target in the images to be identified collected in different directions are determined by direct linear transformation method to determine the three-dimensional coordinates of the detection target;

[0089] Based on the three-dimensional coordinates of the detection target, the behavior trajectory of the fish to be detected is determined. The behavior trajectory is used to determine the movement state of the fish to be detected. The quantitative indicators corresponding to the movement state include at least one of the following: movement amount, movement speed and spatial distribution ratio.

[0090] In actual implementation, the KM algorithm, as a data matching algorithm, can solve the problem of matching feature points between detected targets in two adjacent frames of images.

[0091] All feature points between two adjacent target images can form a set M containing m feature elements and a set N containing n feature elements. The KM algorithm matches the weights of the target frames of the detection targets in two adjacent target images by matching the feature elements in the sets M and N, and determines multiple detection targets in the two adjacent target images. The detection targets are the fish to be detected, thereby achieving continuous tracking of multiple fish to be detected, and then assigning a unique tracking identifier to each detection target.

[0092] It can be understood that the role of the tracking mark is to distinguish each fish to be detected in the target image.

[0093] Based on the tracking mark of the detection target and the positioning coordinates corresponding to the target image, the position of the detection target is predicted by the Kalman filter algorithm to obtain the position prediction result of the detection target;

[0094] Based on the position detection result, the position prediction result of the detection target can be corrected.

[0095] It should be noted that the Kalman filter can estimate the motion state of the fish to be detected through a priori estimation and a posteriori estimation.

[0096] However, in the process of dealing with actual tracking problems, it is difficult for the Kalman filter to accurately determine the initial state. Therefore, it is necessary to provide an approximate value through prior knowledge and continuously adjust the value through a posteriori knowledge for correction. The initial state of the Kalman filter can be roughly estimated based on the movement speed of the fish to be detected and the position of the camera.

[0097] In the Kalman filter tracking system, the prediction covariance matrix represents the degree of trust in the current prediction state. The smaller its value, the higher its authenticity. With the iteration of the Kalman filter, before the tracking system reaches a stable state, the value of the prediction covariance matrix is ​​not always constant, and the H value converges to the minimum estimated variance matrix. As the tracking system decreases, the process noise P tends to converge. In addition, it is not appropriate for the value of the measurement noise R to be too large or too small. Therefore, during the parameter debugging process, the value of H will gradually increase, while the value of R will gradually decrease. The relationship between the predicted state vector and the uncertainty variance is shown in Equations (1) and (2).

[0098] e' ll-1 =Ge' l-1l-1 (1)

[0099] H ll-1 =GH l-1l-1 G T +P(2)In the formula, e is the state vector, G is the state transfer matrix, H is the covariance matrix, and P is the process noise matrix.

[0100] In actual implementation, the Kalman filter can be used to estimate the coordinate changes of the detection target, realize real-time dynamic tracking of the detection target, and then determine the corresponding points of the same detection target in the images to be identified collected in different directions through the direct linear transformation method to determine the three-dimensional coordinates of the detection target.

[0101] For example: different directions can be cameras set at different positions, such as a front camera and a top camera. Therefore, the corresponding points can be the horizontal coordinate of the detection target in the image captured by the front camera and the horizontal coordinate of the detection target in the image captured by the top camera, determining the three-dimensional coordinates of the detection target, thereby obtaining the behavior trajectory of the detection target.

[0102] It is understandable that the role of the improved YOLOv8 model in the present invention is to detect the positioning coordinates of the detection target in the two-directional cameras. The positioning coordinates are two-dimensional coordinates, and the positioning coordinates of the detection target determined by the front camera and the positioning coordinates of the detection target determined by the top camera are not yet associated. Kalman filtering, Kuhn-Mankels and kernel correlation filtering algorithms help to associate the positioning coordinates of the detection target determined by the two-directional cameras, so that the two-dimensional coordinates of the detection target become three-dimensional coordinates, and then the movement speed, movement amount and three-dimensional trajectory of the detection target can be determined. Indicators representing the movement state.

[0103] In some embodiments, after predicting the position of the detection target and obtaining the position prediction result of the detection target, the method for determining the fish behavior trajectory also includes:

[0104] When the motion state of the detection target cannot be detected in the continuous multi-frame target image, the detection target that disappears in the continuous multi-frame target image is identified based on the kernel correlation filtering algorithm, and the disappeared detection target is compensated to determine the optimal tracking area;

[0105] Based on the optimal tracking area, determine the tracking position of the detected target in multiple consecutive frames of target images;

[0106] Based on the tracking position of the detection target, the behavior trajectory of the fish to be detected is determined.

[0107] In actual implementation, the improved YOLOv8 model is used to train the image dataset to obtain the target detection model. The kernelized correlation filter algorithm is used to divide the background and target areas in the image dataset into negative samples and positive samples. The trained target detection model identifies the detection targets of multiple consecutive target images and finds the optimal tracking area.

[0108] If the detection target is not successfully identified in the continuous multi-frame target image, that is, the motion state of the detection target cannot be detected in the continuous multi-frame target image, it is considered that the detection target disappears due to occlusion.

[0109] The kernel correlation filtering algorithm is used to compensate for the disappeared detection target based on the predicted optimal tracking area, avoiding the phenomenon that the behavior trajectory of the fish to be detected cannot be continuously drawn due to the disappearance of the object.

[0110] In some embodiments, the training data set of the target detection model is obtained by:

[0111] The image data of the fish to be detected is obtained through a camera, and the image data includes a calibration plate; wherein the camera is installed in the front end direction and the top direction of the container where the fish to be detected is located, and the calibration plate is placed in the container, and the calibration plate is used to determine the inherent matrix and distortion parameters of the camera;

[0112] Based on the intrinsic matrix and distortion parameters, the image data is corrected to obtain a data set.

[0113] In the related art, unlike terrestrial animals, aquatic animals live in water and their movements are carried out in three-dimensional space. In addition, due to factors such as water surface fluctuations, changes in light, and the rapid movement of fish, the detection error is large, so it is necessary to analyze the three-dimensional movement of fish while ensuring the detection accuracy.

[0114] In actual implementation, the training dataset for the target detection model is obtained through the following steps:

[0115] Step 1: Data Acquisition

[0116] The living environment of fish is in three-dimensional space, so using only a single RGB camera is not enough to meet the requirements of obtaining three-dimensional information. In addition, when multiple fish swim in the water, occlusion is a common phenomenon, and setting cameras from different directions can partially alleviate this problem.

[0117] In order to obtain more spatial information of the object and accurately locate the position of the fish, the installation position of the camera is explained by taking the container where the fish to be detected is an aquarium as an example.

[0118] A camera was placed at the front and top of the aquarium to capture two-dimensional images of the fish from different angles and convert them into positional relationships in three-dimensional space. To synchronize the two cameras, a flash effect was created by quickly turning the ambient lighting on and off, and the brief appearance of the flash was used as the start-up time for the two cameras.

[0119] Based on the video data acquisition, all videos are cut at a speed of 1 second per frame to generate image data.

[0120] Taking sturgeon, perch and crucian carp as an example, 600 images are randomly selected from the image data of sturgeon, perch and crucian carp as the data set.

[0121] Due to the limited diversity of the experimental background environment, three data augmentation methods, rotation, mirroring, and chromaticity, were used to further enrich the image data. After data augmentation, the number of images in the dataset increased to 1,800, and then the images were divided into training, validation, and test sets in a ratio of 7:2:1. During the labeling process, the head of each fish was labeled using LabelImg software. The reason for not labeling the entire fish is that the head is less prone to deformation and easier to accurately locate than the body.

[0122] Step 2: Camera Calibration and Distortion Calibration

[0123] Due to the refraction or reflection in the underwater environment, errors are prone to occur in the process of converting pixel coordinates to world coordinates, so the camera needs to be calibrated. The front camera and the top camera are calibrated using Zhang's calibration method.

[0124] Before the experiment, a calibration plate (with 12×9 grids and each grid length and width of 25 mm) was immersed in water and photographed at different positions and angles by the front and top view cameras. The front and top view cameras were calibrated using multiple images containing the calibration plate to obtain the camera's intrinsic matrix and distortion parameters. The parameters obtained by camera calibration were used to correct radial and tangential distortion of the experimental images. The radial distortion correction equation and tangential distortion correction equation are shown in Equation (3) and Equation (4), respectively.

[0125]

[0126]

[0127] Where k1, k2, k3 are radial distortion coefficients, r is the radial distance from the image center to the pixel, and p1 and p2 are tangential distortion coefficients.

[0128] The fish behavior trajectory determination method provided by the present invention can be used to analyze the three-dimensional spatial behavior of fish during ammonia nitrogen stress and recovery.

[0129] like Figure 4 As shown in FIG. 1 , after obtaining the data set, the improved YOLOv8 model can be trained until the training end condition is met to obtain the target detection model.

[0130] In addition, a dynamic non-monotonic focusing mechanism, namely Wise-IoU, is added to the improved YOLOv8 model, and a gradient gain allocation strategy is proposed. The improved model can reduce the competitiveness of high-quality anchor boxes and also reduce the harmful gradients generated by low-quality samples.

[0131] The present invention also combines the improved YOLOv8 model with the Kalman filter, the Kuhn-Mankels pairing algorithm and the kernel correlation filter algorithm to perform three-dimensional trajectory reconstruction, which can analyze the behavior trajectory and movement amount of the fish to be detected from both qualitative and quantitative perspectives.

[0132] The fish behavior trajectory determination method provided by the present invention can provide a new method and idea for automated and intelligent research on aquatic animal behavior.

[0133] The following takes sturgeon as an example to illustrate the method for determining the fish behavior trajectory provided by the present invention.

[0134] 1. Recognition performance of target detection model

[0135] In order to verify the effectiveness of the target detection model provided by the present invention, the target detection model provided by the present invention is compared with mainstream YOLO series models such as YOLOv5 and YOLOv7. An ablation experiment is also conducted on the target detection model, and the specific experimental results are shown in Table 1.

[0136] Although the proposed target detection model did not achieve the optimal results in all indicators, it achieved the optimal results in accuracy and mAP@0.5, which can meet the requirements of the experiment. In addition, compared with the original YOLOv8 model, the performance of the target detection model proposed in this invention has been improved, which further verifies the effectiveness of the improved model.

[0137] Table 1 Comparison of different models and ablation test results

[0138]

[0139] (II) Behavior trajectory drawing

[0140] In the experiment, sturgeons showed regular exploratory behavior of swimming around the fish tank. Therefore, based on the unique movement characteristics of sturgeons, the movement trajectories of sturgeons in experiments with different ammonium chloride concentrations were plotted, such as Figure 5 shown. Figure 5 (a), (b) and (c) are the behavioral trajectories of sturgeons in normal environment, 30 mg / L ammonium chloride environment and restored normal environment, respectively.

[0141] from Figure 5 (a) and Figure 5 (c) It can be seen that under normal conditions, sturgeons exhibit the characteristic of swimming around the fish tank, but under ammonia nitrogen stress, the characteristic of sturgeons swimming around the fish tank is weakened and their behavior becomes relatively chaotic. Figure 5 (d) Figure 5 (e) and Figure 5(f) The behavioral trajectories of sturgeons in normal environment, 60 mg / L ammonium chloride environment and restored normal environment.

[0142] from Figure 5 It can be seen that the sturgeons kept swimming in the fish tank in all three stages of the experiment. However, compared with the normal two stages before and after ammonia nitrogen stress, the spatial distribution of the sturgeons' movement around the fish tank during ammonia nitrogen stress was more inclined to the upper part of the fish tank.

[0143] Figure 5 (g) Figure 5 (h) and Figure 5 (i) The behavioral trajectories of sturgeons in the normal environment, 90 mg / L ammonium chloride environment, and normal environment. In this experiment, the movement characteristics of sturgeons are similar to those in the 30 mg / L ammonium chloride environment. Figure 5 It can be clearly seen in the figure that under normal circumstances, sturgeons swim around in the fish tank, but under ammonia nitrogen stress, sturgeons swim up and down in a certain position in the fish tank. At the same time, under ammonia nitrogen stress, the phenomenon of sturgeons swimming around the fish tank has decreased.

[0144] (III) Exercise Statistics

[0145] In order to more intuitively reflect the movement characteristics of sturgeons under ammonia nitrogen stress and at various stages before and after stress, Table 2 summarizes the amount of sturgeons' movement under each experiment.

[0146] Under different concentrations of ammonium chloride stress, the amount of exercise of sturgeons under ammonia nitrogen stress was significantly lower than that in the normal stage before stress and the recovery stage after stress. Figure 5 The results are consistent.

[0147] Table 2 Exercise volume of sturgeons under different experiments

[0148]

[0149] The advantages of the present invention are:

[0150] (1) Automation: Deploying the target detection model proposed in the present invention to aquaculture can reduce the manpower and material resources required for manual detection of ammonia nitrogen concentration, and can achieve all-weather monitoring of fish and improve detection efficiency.

[0151] (2) Intelligence: The YOLOv8 target detection model is combined with the 3D trajectory reconstruction method to restore the 3D trajectory of fish and calculate the amount of motion. The fish status is further analyzed from quantitative indicators, which can further improve the intelligence level of aquaculture.

[0152] (3) The method proposed in the present invention can be applied not only to aquatic animals but also to terrestrial animals. It has certain theoretical and practical significance, broad application prospects and high application value, and can be industrialized.

[0153] The fish behavior trajectory determination device provided by the present invention is described below. The fish behavior trajectory determination device described below and the fish behavior trajectory determination method described above can be referenced to each other.

[0154] Figure 6 Schematic diagram of the structure of the fish behavior trajectory determination device provided by the present invention. Figure 6 The fish behavior trajectory determination device provided by the present invention comprises:

[0155] The first determination module 610 is used to input a plurality of consecutive frames of images to be identified collected from different directions corresponding to the fish to be detected into the target detection model to obtain a plurality of consecutive frames of target images and positioning coordinates corresponding to the target images, wherein the target image includes a target frame, and the target frame is used to determine the position of the fish to be detected in the image to be identified, and the positioning coordinates are the pixel coordinates of the center point of the target frame;

[0156] A second determination module 620 is used to reconstruct the three-dimensional trajectory of the fish to be detected based on the continuous multiple frames of target images and the positioning coordinates corresponding to the target images, so as to determine the behavior trajectory of the fish to be detected;

[0157] Among them, the target detection model includes a backbone network, a neck structure and a decoupling head structure, the backbone network includes a spatial pyramid pooling module and a multi-head self-attention module, and the output end of the spatial pyramid pooling module is connected to the input end of the multi-head self-attention module.

[0158] The fish behavior trajectory determination device provided by the present invention determines multiple consecutive frames of target images and the positioning coordinates corresponding to the target images through a target detection model, thereby determining the position of the fish to be detected in the image to be identified and the positioning information corresponding to the fish to be detected, and combines with three-dimensional trajectory reconstruction to achieve three-dimensional trajectory restoration of the fish to be detected, and then can analyze the movement state of the fish to be detected, including quantitative indicators such as movement amount, speed, and spatial distribution ratio, which can further improve the level of intelligent animal breeding.

[0159] In some embodiments, the first determining module 610 is specifically configured to:

[0160] Based on the spatial pyramid pooling module, feature extraction is performed on the image to be identified to obtain a first feature map;

[0161] Inputting the first feature map into the multi-head self-attention module for feature encoding to obtain a second feature map;

[0162] Inputting the second feature map into the neck structure for feature fusion to obtain a third feature map;

[0163] The third feature map is input into the decoupling head structure to obtain the target image and the positioning coordinates corresponding to the target image.

[0164] In some embodiments, the first determining module 610 is specifically configured to:

[0165] Inputting the third feature map into the classification branch included in the decoupling head structure, and outputting the target image, wherein the target image is used to determine the category of the fish to be detected;

[0166] The third feature map is input into the regression branch included in the decoupling head structure, and the positioning coordinates are output.

[0167] In some embodiments, the second determining module 620 is specifically configured to:

[0168] Based on the KM algorithm, the feature point sets corresponding to two adjacent frames of target images are matched to determine multiple detection targets in the two adjacent frames of target images, and a tracking mark is assigned to each detection target, wherein the detection target is used to indicate the fish to be detected;

[0169] Based on the Kalman filter algorithm, the tracking identifier of the detection target and the positioning coordinates corresponding to the target image, the position of the detection target is predicted to obtain a position prediction result of the detection target;

[0170] Based on the position detection result, the corresponding points of the same detection target in the to-be-identified images collected in different directions are determined by direct linear transformation method to determine the three-dimensional coordinates of the detection target;

[0171] Based on the three-dimensional coordinates of the detection target, the behavior trajectory of the fish to be detected is determined, and the behavior trajectory is used to determine the movement state of the fish to be detected. The quantitative indicators corresponding to the movement state include at least one of the following: movement amount, movement speed and spatial distribution ratio.

[0172] In some embodiments, the second determining module 620 is specifically configured to:

[0173] In the case where the motion state of the detection target cannot be detected in the continuous multiple frames of target images, based on the kernel correlation filtering algorithm, the detection target that disappears in the continuous multiple frames of target images is identified, and the disappeared detection target is compensated to determine the optimal tracking area;

[0174] Based on the optimal tracking area, determining the tracking position of the detection target in the continuous multiple frames of target images;

[0175] Based on the tracking position of the detection target, the behavior trajectory of the fish to be detected is determined.

[0176] In some embodiments, the training data set of the target detection model is obtained by:

[0177] The image data of the fish to be detected is obtained by a camera, wherein the image data includes a calibration plate; wherein the camera is installed in the front end direction and the top direction of the container where the fish to be detected is located, and a calibration plate is placed in the container, and the calibration plate is used to determine the inherent matrix and distortion parameters of the camera;

[0178] Based on the intrinsic matrix and the distortion parameter, the image data is corrected to obtain the data set.

[0179] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other via the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the fish behavior trajectory determination method, which includes:

[0180] Inputting multiple frames of images to be identified collected from different directions corresponding to the fish to be detected into the target detection model to obtain multiple frames of target images and positioning coordinates corresponding to the target images, wherein the target image includes a target frame, and the target frame is used to determine the position of the fish to be detected in the image to be identified, and the positioning coordinates are the pixel coordinates of the center point of the target frame;

[0181] Based on the continuous multiple frames of target images and the positioning coordinates corresponding to the target images, a three-dimensional trajectory of the fish to be detected is reconstructed to determine the behavior trajectory of the fish to be detected;

[0182] Among them, the target detection model includes a backbone network, a neck structure and a decoupling head structure, the backbone network includes a spatial pyramid pooling module and a multi-head self-attention module, and the output end of the spatial pyramid pooling module is connected to the input end of the multi-head self-attention module.

[0183] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0184] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the fish behavior trajectory determination method provided by the above methods, the method comprising:

[0185] Inputting multiple frames of images to be identified collected from different directions corresponding to the fish to be detected into the target detection model to obtain multiple frames of target images and positioning coordinates corresponding to the target images, wherein the target image includes a target frame, and the target frame is used to determine the position of the fish to be detected in the image to be identified, and the positioning coordinates are the pixel coordinates of the center point of the target frame;

[0186] Based on the continuous multiple frames of target images and the positioning coordinates corresponding to the target images, a three-dimensional trajectory of the fish to be detected is reconstructed to determine the behavior trajectory of the fish to be detected;

[0187] Among them, the target detection model includes a backbone network, a neck structure and a decoupling head structure, the backbone network includes a spatial pyramid pooling module and a multi-head self-attention module, and the output end of the spatial pyramid pooling module is connected to the input end of the multi-head self-attention module.

[0188] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the fish behavior trajectory determination method provided by the above methods, the method comprising:

[0189] Inputting multiple frames of images to be identified collected from different directions corresponding to the fish to be detected into the target detection model to obtain multiple frames of target images and positioning coordinates corresponding to the target images, wherein the target image includes a target frame, and the target frame is used to determine the position of the fish to be detected in the image to be identified, and the positioning coordinates are the pixel coordinates of the center point of the target frame;

[0190] Based on the continuous multiple frames of target images and the positioning coordinates corresponding to the target images, a three-dimensional trajectory of the fish to be detected is reconstructed to determine the behavior trajectory of the fish to be detected;

[0191] Among them, the target detection model includes a backbone network, a neck structure and a decoupling head structure, the backbone network includes a spatial pyramid pooling module and a multi-head self-attention module, and the output end of the spatial pyramid pooling module is connected to the input end of the multi-head self-attention module.

[0192] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0193] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for determining fish behavior trajectory, characterized in that: include: Inputting multiple frames of images to be identified collected from different directions corresponding to the fish to be detected into the target detection model to obtain multiple frames of target images and positioning coordinates corresponding to the target images, wherein the target image includes a target frame, and the target frame is used to determine the position of the fish to be detected in the image to be identified, and the positioning coordinates are the pixel coordinates of the center point of the target frame; Based on the continuous multiple frames of target images and the positioning coordinates corresponding to the target images, a three-dimensional trajectory of the fish to be detected is reconstructed to determine the behavior trajectory of the fish to be detected; Among them, the target detection model includes a backbone network, a neck structure and a decoupling head structure, the backbone network includes a spatial pyramid pooling module and a multi-head self-attention module, and the output end of the spatial pyramid pooling module is connected to the input end of the multi-head self-attention module.

2. The method for determining fish behavior trajectory according to claim 1, characterized in that: The method of inputting a plurality of consecutive frames of to-be-recognized images collected from different directions corresponding to the fish to be detected into the target detection model to obtain the plurality of consecutive frames of target images and the positioning coordinates corresponding to the target images comprises: Based on the spatial pyramid pooling module, feature extraction is performed on the image to be identified to obtain a first feature map; Inputting the first feature map into the multi-head self-attention module for feature encoding to obtain a second feature map; Inputting the second feature map into the neck structure for feature fusion to obtain a third feature map; The third feature map is input into the decoupling head structure to obtain the target image and the positioning coordinates corresponding to the target image.

3. The method for determining fish behavior trajectory according to claim 2, characterized in that: Inputting the third feature map into the decoupling head structure to obtain the target image and the positioning coordinates corresponding to the target image includes: Inputting the third feature map into the classification branch included in the decoupling head structure, and outputting the target image, wherein the target image is used to determine the category of the fish to be detected; The third feature map is input into the regression branch included in the decoupling head structure, and the positioning coordinates are output.

4. The method for determining fish behavior trajectory according to claim 1, characterized in that: The method of reconstructing the three-dimensional trajectory of the fish to be detected based on the continuous multiple frames of target images and the positioning coordinates corresponding to the target images to determine the behavior trajectory of the fish to be detected includes: Based on the KM algorithm, the feature point sets corresponding to two adjacent frames of target images are matched to determine multiple detection targets in the two adjacent frames of target images, and a tracking mark is assigned to each detection target, wherein the detection target is used to indicate the fish to be detected; Based on the Kalman filter algorithm, the tracking identifier of the detection target and the positioning coordinates corresponding to the target image, the position of the detection target is predicted to obtain a position prediction result of the detection target; Based on the position detection result, the corresponding points of the same detection target in the to-be-identified images collected in different directions are determined by direct linear transformation method to determine the three-dimensional coordinates of the detection target; Based on the three-dimensional coordinates of the detection target, the behavior trajectory of the fish to be detected is determined, and the behavior trajectory is used to determine the movement state of the fish to be detected. The quantitative indicators corresponding to the movement state include at least one of the following: movement amount, movement speed and spatial distribution ratio.

5. The method for determining fish behavior trajectory according to claim 4, characterized in that: After performing position prediction on the detection target and obtaining the position prediction result of the detection target, the method further includes: In the case where the motion state of the detection target cannot be detected in the continuous multiple frames of target images, based on the kernel correlation filtering algorithm, the detection target that disappears in the continuous multiple frames of target images is identified, and the disappeared detection target is compensated to determine the optimal tracking area; Based on the optimal tracking area, determining the tracking position of the detection target in the continuous multiple frames of target images; Based on the tracking position of the detection target, the behavior trajectory of the fish to be detected is determined.

6. The method for determining fish behavior trajectory according to claim 1, characterized in that: The training data set of the target detection model is obtained in the following way: The image data of the fish to be detected is obtained by a camera, wherein the image data includes a calibration plate; wherein the camera is installed in the front end direction and the top direction of the container where the fish to be detected is located, and a calibration plate is placed in the container, and the calibration plate is used to determine the inherent matrix and distortion parameters of the camera; Based on the intrinsic matrix and the distortion parameter, the image data is corrected to obtain the data set.

7. A device for determining fish behavior trajectory, characterized in that: include: A first determination module is used to input a plurality of consecutive frames of images to be identified collected from different directions corresponding to the fish to be detected into a target detection model to obtain a plurality of consecutive frames of target images and positioning coordinates corresponding to the target images, wherein the target image includes a target frame, and the target frame is used to determine the position of the fish to be detected in the image to be identified, and the positioning coordinates are the pixel coordinates of the center point of the target frame; A second determination module is used to reconstruct the three-dimensional trajectory of the fish to be detected based on the continuous multiple frames of target images and the positioning coordinates corresponding to the target images, so as to determine the behavior trajectory of the fish to be detected; Among them, the target detection model includes a backbone network, a neck structure and a decoupling head structure, the backbone network includes a spatial pyramid pooling module and a multi-head self-attention module, and the output end of the spatial pyramid pooling module is connected to the input end of the multi-head self-attention module.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for determining the fish behavior trajectory as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for determining the fish behavior trajectory as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for determining the fish behavior trajectory as described in any one of claims 1 to 6 is implemented.

Images