A fish activity amount anomaly early warning method and system, an electronic device, and a medium

Abstract

This invention discloses a method, system, electronic device, and medium for early warning of abnormal fish activity, relating to the field of intelligent fish farming technology. The method includes: inputting a target video into a multi-target fish tracking model to obtain the basic parameters of each fish in each frame of the target video; using the tracking displacement of each fish to correct the offset of the detection box of each fish, obtaining the bounding box of each fish; processing the center point coordinates of the bounding box of each fish using a data-for-association method to obtain a final trajectory set; obtaining the fish activity level based on the fish's mass and the center point coordinates of each bounding box in the final trajectory; summing the activity levels of each fish in the target video to obtain the activity level of the fish school in the target video; if the activity level of the fish school in the target video is not within a set activity level range, then the activity level of the fish school in the target video at the current moment is judged to be abnormal. This invention can replace manual observation and early warning of fish activity in real time.

Description

Technical Field

[0001] This invention relates to the field of intelligent fish farming technology, and in particular to a method, system, electronic device and medium for early warning of abnormal fish activity. Background Technology

[0002] Due to overfishing and environmental factors leading to a decline in wild fish resources, aquaculture has become crucial for food security. Therefore, intensive, large-scale, and facility-based production in aquaculture is an inevitable trend for improving aquatic product yield and efficiency. However, its high cost and high risk characteristics necessitate a focus on precision, scientific methods, and healthy practices in the production process. Factory farming allows for precise control of environmental conditions, water quality, and feeding practices, thereby increasing the yield and quality of farmed fish products. Monitoring the farming environment and fish species using intelligent monitoring technology in factory farming workshops is of great significance for achieving healthy aquaculture. In factory aquaculture, workers need to patrol the ponds 24 hours a day, promptly removing diseased and dead fish, and regularly transferring and cleaning the ponds to ensure healthy and efficient farming. Changes in fish movement are the most direct manifestation of water quality changes and can serve as a biological early warning indicator of changes in the aquaculture environment. They can quickly detect abnormalities when water quality deteriorates and play an important role in achieving intelligent environmental control. However, pond patrols are very arduous, and it is difficult for the human eye to observe whether the activity of all individual fish is abnormal in real time. It is impossible to determine the water quality status based on abnormal activity, thus making it impossible to take timely measures to address the aquaculture environment. Therefore, there is an urgent need for a method to replace the human eye in observing fish and to understand their activity levels in real time. Summary of the Invention

[0003] The purpose of this invention is to provide a method, system, electronic device, and medium for early warning of abnormal fish activity, which can replace manual observation and early warning of fish activity in real time.

[0004] To achieve the above objectives, the present invention provides the following solution: a method for early warning of abnormal fish activity, comprising: real-time acquisition of target video; wherein the target video includes multiple fish and the last frame of the target video is the current time.

[0005] The target video is input into the fish multi-target tracking model to obtain the basic parameters of each fish in each frame of the target video; the basic parameters include the center point coordinates of the detection box, the width of the detection box, the height of the detection box, the confidence level of the detection box, and the tracking displacement.

[0006] For any fish in any frame of the target video, the detection box of the fish in the frame is offset and corrected by the tracking displacement of the fish in the frame, thus obtaining the bounding box of the fish in the frame.

[0007] The data exchange association method is used to process the center point coordinates of the bounding box of each fish in each frame of the target video to obtain the final trajectory set; the final trajectory set includes the final trajectory of each fish in the target video.

[0008] For any fish in the target video, the activity level of the fish is obtained based on the fish's mass and the coordinates of the center points of each bounding box in the fish's final trajectory.

[0009] The activity levels of each fish in the target video are summed to obtain the activity level of the fish school in the target video; the fish school in the target video includes all fish in the target video.

[0010] If the activity level of the fish in the target video is not within the set activity level range, then the activity level of the fish in the target video at the current moment is determined to be abnormal.

[0011] A fish activity abnormality early warning system includes: an acquisition module for acquiring target video in real time; the target video includes multiple fish and the last frame of the target video is the current time.

[0012] The basic parameter output module is used to input the target video into the fish multi-target tracking model to obtain the basic parameters of each fish in each frame of the target video; the basic parameters include the center point coordinates of the detection box, the width of the detection box, the height of the detection box, the confidence level of the detection box, and the tracking displacement.

[0013] The offset correction module is used to perform offset correction on the detection box of the fish in any frame of the target video by using the tracking displacement of the fish in the frame of the target video, so as to obtain the bounding box of the fish in the frame of the target video.

[0014] The final trajectory set determination module is used to process the center point coordinates of the bounding box of each fish in each frame of the target video using the data exchange association method to obtain the final trajectory set; the final trajectory set includes the final trajectory of each fish in the target video.

[0015] The fish activity calculation module is used to calculate the activity of any fish in the target video based on the fish's mass and the coordinates of the center points of each bounding box in the fish's final trajectory.

[0016] The fish activity calculation module is used to sum the activity of each fish in the target video to obtain the activity of the fish school in the target video; the fish school in the target video includes all fish in the target video.

[0017] The activity level anomaly detection module is used to determine that the activity level of the fish in the target video is abnormal at the current moment if the activity level of the fish in the target video is not within the set activity level range.

[0018] An electronic device includes a memory and a processor, the memory storing a computer program and the processor running the computer program to cause the electronic device to perform the fish activity anomaly early warning method as described above.

[0019] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the fish activity abnormality early warning method as described above.

[0020] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: The present invention inputs the target video into a multi-target fish tracking model to obtain the basic parameters of each fish in each frame of the target video. The tracking displacement of each fish in each frame of the target video is used to offset the detection box of each fish in each frame of the target video to obtain the bounding box of each fish in each frame of the target video. The data-for-association method is used to process the center point coordinates of the bounding box of each fish in each frame of the target video to obtain the final trajectory set. The activity of the fish is obtained according to the mass of the fish and the center point coordinates of each bounding box in the final trajectory of the fish. The activity of each fish in the target video is accumulated to obtain the activity of the fish school in the target video. If the activity of the fish school in the target video is not within the set activity range, it is determined that the activity of the fish school in the target video at the current moment is abnormal. The video is used to automatically monitor the fish in the aquaculture pond, realizing non-subjective and non-contact 24-hour continuous monitoring, greatly reducing the workload of workers, and can also provide early warning based on the activity, replacing manual observation and early warning of fish activity in real time. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of the fish activity abnormality early warning method provided in Embodiment 1 of the present invention.

[0023] Figure 2 This is a flowchart of the fish activity abnormality early warning method provided in Embodiment 2 of the present invention.

[0024] Figure 3 The flowchart illustrates the method for determining and using a multi-target tracking model for fish provided in Embodiment 1 of the present invention.

[0025] Figure 4 This is the original FasterNet structure diagram.

[0026] Figure 5 The structural diagram of the fish multi-target tracking model provided in Embodiment 1 of the present invention.

[0027] Figure 6 This is a schematic diagram of the grid step size definition provided in Embodiment 2 of the present invention.

[0028] Figure 7 This is a schematic diagram of the fish activity abnormality early warning system provided in Embodiment 3 of the present invention.

[0029] Figure 8 This is a schematic diagram of the fish activity abnormality early warning device provided in Embodiment 3 of the present invention.

[0030] Figure 9 This is an overall architecture diagram of the fish activity abnormality early warning device provided in Embodiment 3 of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] Example 1: This embodiment of the invention provides a method for early warning of abnormal fish activity, such as... Figure 1 As shown, the method includes: Step 101: Real-time acquisition of target video. The target video includes multiple fish, and the last frame of the target video is the current time.

[0034] Step 102: Input the target video into the fish multi-target tracking model to obtain the basic parameters of each fish in each frame of the target video. The basic parameters include the center point coordinates of the detection box, the width of the detection box, the height of the detection box, the confidence level of the detection box, and the tracking displacement.

[0035] Step 103: For any fish in any frame of the target video, use the tracking displacement of the fish in the frame of the target video to perform offset correction on the detection box of the fish in the frame of the target video to obtain the bounding box of the fish in the frame of the target video.

[0036] Step 104: The center point coordinates of the bounding box of each fish in each frame of the target video are processed using the data-for-association method to obtain the final trajectory set. The final trajectory set includes the final trajectory of each fish in the target video.

[0037] Step 105: For any fish in the target video, obtain the activity level of the fish based on the fish's mass and the coordinates of the center points of each bounding box in the fish's final trajectory.

[0038] Step 106: Sum the activity levels of each fish in the target video to obtain the activity level of the fish school in the target video; the fish school in the target video includes all fish in the target video.

[0039] Step 107: If the activity level of the fish in the target video is not within the set activity level range, then it is determined that the activity level of the fish in the target video is abnormal at the current moment.

[0040] In practical applications, for any fish in any frame of the target video, the detection box of the fish in the target video is offset and corrected using the tracking displacement of the fish in that frame to obtain the bounding box of the fish in that frame. Specifically, for any fish in the nth frame of the target video, the center point coordinates of the detection box of the fish in the nth frame are offset and corrected using the tracking displacement of the fish in that frame to obtain the center point coordinates of the bounding box of the fish in the nth frame. The width and height of the bounding box of the fish in the nth frame are equal to the width and height of the detection box of the fish in the nth frame.

[0041] In practical applications, the method for early warning of abnormal fish activity further includes: for the nth frame of the target video, dividing all the fish in the nth frame of the target video into multiple fish groups corresponding to the nth frame of the target video based on the center point coordinates of the bounding box of each fish in the nth frame of the target video, the width of the bounding box, and the height of the bounding box; n is a positive integer greater than 1.

[0042] For any school of fish, the group attributes of the school are calculated based on the coordinates of the center point of the school corresponding to the nth frame of the target video, the speed of each fish in the school, and the coordinates of the center point of the bounding box of each fish in the school. The group attributes include the school radius, average speed, dispersion, cohesion, and relative direction.

[0043] The presence of out-of-group behavior in the fish population is determined based on the group attributes of the fish population and a set attribute threshold.

[0044] In practical applications, determining whether a fish group exhibits out-of-group behavior based on its group attributes and a set attribute threshold specifically includes: when the radius of the fish group is greater than the set attribute threshold corresponding to the fish group radius, the current fish group is determined to be in an unorganized state.

[0045] If both dispersion and cohesion are 0, it means that the current school of fish is a group and exhibits out-of-group behavior.

[0046] When the center point of the school and the center point of an individual fish coincide, and both dispersion and cohesion are not zero, then that individual fish is the key fish of the entire school; when the center point of the school and the center point of an individual fish coincide, and both dispersion and cohesion are zero, then that individual fish exhibits a tendency to leave the school.

[0047] When the relative direction is 1, it indicates that the fish are arranged very neatly, which means that the fish have a very strong organizational ability; when the relative direction is 1, it means that the fish are arranged very neatly, which means that the fish have a very strong organizational ability. A relative angle of 0.5 indicates that the fish are arranged in a relatively orderly manner and have a strong organizational ability; when the relative angle is less than 0.5, it indicates that the fish are not arranged in an orderly manner and are loosely organized.

[0048] In practical applications, the data-for-association method is used to process the center point coordinates of the bounding boxes of each fish in each frame of the target video to obtain the final trajectory set. Specifically, this includes: for the nth frame of the target video, a distance set is obtained based on the center point coordinates of the bounding boxes of all fish in the nth frame and the center points of each trajectory in the trajectory set corresponding to the (n-1)th frame; the distance set includes the distance between the center point of each trajectory in the trajectory set corresponding to the (n-1)th frame and the center point of the bounding box of each fish; the trajectory set includes multiple trajectories, and each trajectory includes multiple bounding boxes; the trajectory set corresponding to the first frame is all bounding boxes in the first frame of the target video with a confidence level greater than a set confidence threshold; n is a positive integer greater than 1.

[0049] Based on the distance set, the bounding boxes of all fish in the nth frame of the target video are assigned to the corresponding trajectories in the trajectory set of the (n-1)th frame. The complete trajectory set corresponding to the nth frame is then removed from the trajectory set corresponding to the (n-1)th frame. Unassigned bounding boxes are added to the trajectory set corresponding to the (n-1)th frame, resulting in the trajectory set corresponding to the nth frame. The nth frame is then updated. The process of obtaining the distance set based on the center point coordinates of the bounding boxes of all fish in the nth frame and the center points of the trajectories in the trajectory set corresponding to the (n-1)th frame is repeated until the trajectory set corresponding to the last frame of the target video and the complete trajectory set corresponding to the last frame of the target video are obtained. The trajectory set of the last frame of the target video and the complete trajectory set corresponding to all frames are determined as the final trajectory set. The complete trajectory set corresponding to the nth frame includes all unassigned trajectories in the trajectory set corresponding to the (n-1)th frame.

[0050] In practical applications, based on the distance set, the bounding boxes of all fish in the nth frame of the target video are assigned to the corresponding trajectories in the trajectory set corresponding to the (n-1)th frame. The complete trajectory set corresponding to the nth frame is then removed from the trajectory set corresponding to the (n-1)th frame, and the unassigned bounding boxes are added to the trajectory set corresponding to the (n-1)th frame, thus obtaining the trajectory set corresponding to the nth frame. Specifically, this includes: using the Jonker-Volgenant algorithm to assign the bounding boxes of all fish in the nth frame of the target video to the corresponding trajectories in the trajectory set corresponding to the (n-1)th frame based on the distance set.

[0051] In the nth frame of the target video, the bounding boxes of all fish that have not been assigned a trajectory are identified as new trajectories and added to the trajectory set corresponding to the (n-1)th frame. All unassigned trajectories in the trajectory set corresponding to the (n-1)th frame are identified as the complete trajectory set corresponding to the nth frame. The complete trajectory set corresponding to the nth frame is then removed from the trajectory set corresponding to the (n-1)th frame to obtain the trajectory set corresponding to the nth frame.

[0052] In practical applications, the activity level of the fish is obtained based on the fish's mass and the coordinates of the center points of each target bounding box in the fish's final trajectory; specifically, this includes establishing a grid coordinate system.

[0053] The coordinate system is transformed by performing coordinate system transformation on the center point coordinates of all target bounding boxes in the final trajectory of the fish to obtain the center point coordinates of all bounding boxes in the final trajectory of the fish in the grid coordinate system.

[0054] At the current loop count, based on the coordinates of the center point of the bounding box of the fish in the nth frame of the current loop count in the final trajectory of the fish in the grid coordinate system and the coordinates of the center point of the bounding box of the fish in the target frame of the current loop count in the final trajectory of the fish in the grid coordinate system, it is determined whether the fish moves between the nth frame of the current loop count and the target frame of the current loop count by a set grid step size; the target frame at the first loop count is the (n+1)th frame.

[0055] If the set grid step size is not moved, the target frame in the next loop count is determined to be the next frame after the target frame in the current loop count, and the loop count is updated to enter the next loop.

[0056] If the set grid step size is used, the movement distance of the fish in the current loop number is calculated based on the coordinates of the center point of the bounding box of the fish in the nth frame image of the final trajectory of the fish in the grid coordinate system and the coordinates of the center point of the bounding box of the fish in the target frame image of the final trajectory of the fish in the grid coordinate system. Then, the nth frame in the next loop number is determined as the target frame in the current loop number, and the target frame in the next loop number is the next frame after the target frame in the current loop number. The loop number is then updated to enter the next loop.

[0057] The activity level of the fish is obtained by calculating the fish's activity level based on its mass and the square of the distance it travels over all cycles.

[0058] In practical applications, such as Figure 3 As shown, the process of determining the multi-target tracking model for fish is as follows: S1: Collect videos of fish farming.

[0059] First, a camera was installed above the aquaculture ponds in the aquaculture farm to capture video of the fish farming from a top-down angle. During capture, the camera aimed to include the entire range of fish movement within the lens's coverage area. Then, the video was segmented into short videos of 10 to 50 seconds for data annotation. The annotations included the target detection box and the fish's ID for each fish (displacement could be obtained through inter-frame difference of the annotation box's center point. Confidence could be obtained using the Gaussian function of the annotation box's center point (Formula 5)). More than 6000 frames were annotated. Finally, the captured fish farming video was annotated frame-by-frame using the MOT format, and the annotated videos were divided into training and testing sets. In this embodiment, the camera was installed 1.5m directly above the aquaculture pond, capturing all individual fish in the pond. The camera used for video capture was a Hikvision3T86FWDV2-I3S (8 megapixels, 4mm focal length). The captured video resolution was 1920*2560, and the frame rate was 20fps.

[0060] S2: The feature extraction network, feature fusion network, and prediction network are trained using the training set to obtain a multi-target tracking model for fish.

[0061] (1) Feature Extraction Network: FasterNet is a backbone network that avoids excessive memory access through PConv computation, thus accelerating the model speed. To learn the displacement information of the tracked object, the improved FasterNet network in this invention, such as... Figure 4 As shown, an additional branch (Pre_Embedding branch) is added to embed information from the previous frame of video. The improved FasterNet network structure is shown in the diagram below. Figure 5As shown in the dashed box, two video frames are divided into blocks of the same size, and then the two are added pixel by pixel. The improved FasterNet network applies the idea of ​​a super-large kernel to the original FasterNet, changing the kernel size to 31 when embedding blocks (that is, in Embedding and Pre_Embedding), thus increasing the receptive field of the model.

[0062] (2) Feature Fusion Network: The feature fusion network is a deep aggregation network. Four image features at different scales can be obtained through the improved FasterNet network, and fused using an iterative deep aggregation method. This aggregation method can be expressed by formula (1): (1), where N(•) represents an aggregation node, IDA(•) represents iterative deep aggregation, and x n Let x1 represent the nth level feature, where n is the level number, and x1 represents the 1st level feature. As can be seen from formula (1), features from different levels are continuously aggregated through iteration. The definition of the aggregation node N(•) is as follows: (2), where BatchNorm(•) is the batch normalization function, and σ(•) is the activation function. is the weight of the i-th level feature, and b is the bias.

[0063] Different levels of iterative deep aggregation will obtain n feature maps of different scales. The n features of different scales generated by the improved FasterNet are aggregated and then predicted by the prediction network. The prediction network consists of four parallel prediction units, each of which is a 3×3 convolution, a ReLU activation function, and a 1×1 convolution.

[0064] (3) There are two main tasks in the training process of the fish multi-target tracking model: detection and tracking. Each task is trained using its own loss function. Multiple loss functions are combined, and the predicted and actual labeled values ​​are input into the loss function. Training is then performed using gradient descent to obtain the optimal solution for each task objective. The combination of objective functions for multiple tasks is shown in formula (3): (3), where, The loss function for the multi-target tracking model is... and These are the loss functions for the detection task and the tracking task, respectively. and These are their weights.

[0065] The detection task consists of two regression tasks: one predicts the center point of the detection box, and the other predicts the width and height of the detection box. The tracking task consists of one regression task. All three regression tasks use the Sparse Regression Loss function. As shown in formula (4), the loss function is only calculated when the center point of the detection box appears.

[0066] (4), where, These are the coordinates of the center point of the actually labeled detection box. K represents the coordinates of the center point of the predicted detection box, and K is the number of target box centers. It is the actual response heatmap of the center point coordinates (x, y) of the detection box.

[0067] (5), where G( ;σ) is a Gaussian kernel function with a kernel of size σ, (x k ,y k () represents the actual label of the center point of the detection frame.

[0068] In this specific implementation, the input image size is set to 960*1280, the batch size is 4, and the learning rate is 1.25×10⁻⁶. -4 The dropout value was 0.1, and the training lasted for a total of 70 epochs.

[0069] After the training process is complete, a multi-target tracking model for fish can be obtained. The performance of the obtained multi-target tracking model is evaluated using a test set.

[0070] In the training process described above, an improved FasterNet is used as the backbone network of the model, which is also a feature extraction network. A fusion feature network is used to fuse the four-layer feature maps output by the backbone network to obtain fused features. Then, a prediction network uses these fused features to predict the center point position of the detection box, the length and width of the detection box, and the tracking displacement. For example... Figure 5 As shown, Figure 5 The four cubes on the right are feature maps of four different scales generated by the improved FasterNet. They are fused together to form the first cube on the left. Using the fused features, the confidence score, the center point of the detection box, the length and width of the detection box, and the tracking displacement are predicted.

[0071] Example 2: This invention provides a more specific embodiment that details Example 1 above. First, it collects aquaculture videos and environmental data of fish in factory farming. Then, it trains a multi-target tracking model for fish to obtain their movement trajectories. Next, it proposes a grid step size-based method to estimate the activity level of the fish in the videos. Simultaneously, based on the connectivity of the bounding boxes of the fish in the videos, it divides the fish into groups and calculates the group attributes. Finally, it uses the calculation results to determine whether there are abnormalities in fish activity, whether the fish exhibit out-of-group behavior or a tendency to do so, providing intelligent early warnings for abnormal fish farming environments, abnormal activity levels, and out-of-group behavior.

[0072] like Figure 2 As shown, the embodiments of the present invention mainly include 5 steps: Step 1, collecting fish farming videos and growth environment data.

[0073] Video data of farmed fish is collected using surveillance cameras. The cameras are positioned vertically or at a certain angle above the fish ponds to capture images of the fish, ensuring that all areas potentially covered by the fish's movements are within the camera's monitoring range. Sensors are used to collect data on the fish's growth environment, including water temperature, dissolved oxygen, pH value, turbidity, and conductivity.

[0074] Step 2: Train the multi-target tracking model for fish and obtain its trajectory.

[0075] The acquired video data was labeled using the MOT format, and the labeled video data was used to train a multi-target tracking model for fish. Inputting unlabeled video data into the trained multi-target tracking model yielded the fish's identity (ID), the coordinates of the center point of the fish's bounding box in the video, the width and height of the bounding box, the fish's category, the confidence level of the bounding box, and the fish's tracking displacement.

[0076] The tracking process is carried out according to the following steps: ① When t=0, for any fish, the center point of the detection box of the fish is offset and corrected by the tracking displacement of the fish to obtain the bounding box, and the bounding box with a confidence level greater than the set confidence level threshold (set to 0.3) is determined as the trajectory starting point, where t represents the frame number.

[0077] ②When t≠0, after offset correction using the tracking displacement, the cost matrix including all calculated distances is obtained by calculating the distance between the center point of the bounding box of all fish and the center point of the trajectory of all fish based on the Euclidean distance calculation formula. The Euclidean distance is shown in formula (6).

[0078] (6), where, Let x represent the Euclidean distance between the center point of the bounding box of the i-th fish and the center point of the j-th trajectory under t. i,y i ) represents the coordinates of the center point of the bounding box of the i-th fish in the object set T, (x j ,y j ) is the coordinate of the center point of the j-th trajectory in the trajectory set D.

[0079] ③ The Jonker-Volgenant algorithm is used to linearly assign the bounding boxes and trajectories of the fish based on the cost matrix calculated in ②, resulting in an updated set of trajectories. If the assignment is successful, the bounding boxes are updated to trajectories; at the same time, trajectories that fail to match are terminated, and the bounding boxes that fail to match are set as the starting points of new trajectories.

[0080] Update t and repeat steps ② and ③ above to obtain the movement trajectory of each fish in the aquaculture video.

[0081] Step 3: Estimate fish activity based on grid step size.

[0082] The captured video is then sized appropriately according to the size of the fish (e.g., the grid size can be set to 1 / 8 of the image width × 1 / 8 of the image height ≈ the size of the fish's bounding box). The image region is then divided into a coordinate system composed of grids of the same size, generating grid coordinates. The actual pixel coordinates of the fish are then mapped to this grid coordinate system. Each grid cell represents a step size. Figure 6 As shown.

[0083] The process of mapping the actual pixel coordinates of fish to the grid coordinate system can be represented by formula (7): (7) In the formula, s(x,y) represents the actual pixel coordinates, grid(x',y') represents the grid coordinates, x and y are the horizontal and vertical coordinates of the actual pixel coordinates respectively, and w and h are the horizontal and vertical step sizes of the grid step size respectively. This indicates rounding down to the nearest integer.

[0084] The fish moves a very small distance in each frame. Calculating the distance frame by frame would waste time and computing power. Therefore, the Euclidean distance is calculated once for each grid step the fish moves in the grid coordinate system in the video. If the distance does not reach one grid step, the distance is not calculated. The length of its motion trajectory is obtained by accumulating the grid steps, as shown in formula (8).

[0085] (8).

[0086] The total activity of all fish targets in the video is summed to obtain the total activity of all fish in the video. The activity of fish can be statistically analyzed according to any time period. The total activity of all fish in the video is shown in formula (9).

[0087] (9), where, This represents the Euclidean distance traveled by the i-th fish within the time interval Δt. , Indicates the start time of the video. Indicates the end time of the video. This represents the total activity level of all fish in the video. Let N represent the mass of the i-th fish, and N represent the total number of individuals. and Let represent the coordinates of the i-th fish after grid transformation at time t and time t', respectively. Since the individual fish masses in a tank vary relatively little and are difficult to statistically analyze, ... It is usually set to 1.

[0088] The method for determining whether to move by one grid step is as follows: A grid mask is created. If the center point of the fish's bounding box touches the grid mask, the activity level is calculated and updated; if the center point of the fish's bounding box does not touch the grid mask, the activity level is not calculated. Since the center point of the fish's bounding box may not exactly touch the mask, a certain width of padding is added to the mask. The width can be set to [0, 1 / 4 × grid width / height]. Following the above process, the pseudocode for real-time calculation of fish target activity in the video is as follows.

[0089]

[0090] Step 4: Divide the fish into groups and calculate the group attributes. Steps 3 and 4 can be performed simultaneously.

[0091] In the video, overlapping bounding boxes of fish objects (indicating connectivity) are grouped into small schools. The school center and radius are obtained by fitting an ellipse to the connected regions; the radius includes a major axis and a minor axis. For each school, the average speed, dispersion, cohesion, and relative orientation of the fish are calculated using the following formulas: (10), where, Let t represent the average velocity of the school of fish at time t, and n represent the total number of fish in the school. Let represent the velocity of the i-th fish at time t. This indicates the time interval before and after a speed measurement. Indicates the position of the i-th fish at t+ The speed of time and Let x and y represent the horizontal and vertical components of the velocity of the i-th fish at time t, respectively.

[0092] The formula for calculating the dispersion of a fish school is as follows: (11), where x p y p Let x represent the x and y coordinates of the p-th fish. q yq Let x and y represent the x and y coordinates of the q-th fish.

[0093] The formula for calculating the cohesion of a school of fish is as follows: (12), where x c y c The x and y coordinates represent the center point of the fish school.

[0094] The formula for calculating the relative direction of a school of fish is as follows: (13), where, Let represent the angle of the p-th fish relative to the center point at time t, and its calculation formula is shown in formula (14): (14).

[0095] Step 5: Analyze the calculation results to provide intelligent early warnings for abnormalities in the fish farming environment, abnormal activity levels, and whether there is any herd behavior.

[0096] The system uses calculations to provide intelligent early warnings for abnormalities in the fish farming environment, abnormal activity levels, and the presence of out-of-group behavior. Specifically, it can be divided into two types: single-threshold early warning and comprehensive early warning. Based on the fish environmental parameters obtained in step one, and the fish activity levels and group attributes calculated in steps two, three, and four, an appropriate early warning threshold can be determined by observing the range of data changes and consulting relevant literature.

[0097] Single-threshold early warning involves setting a threshold for a specific variable. If an observed value exceeds this threshold, the device issues an alert and simultaneously activates environmental control equipment, such as turning on aerators. For example, excessively high fish activity over a period of time indicates fright or stress; excessively low activity over a period of time indicates disease or weakness. Specific thresholds are adjusted based on the actual condition of different fish species. Furthermore, if a fish consistently exhibits autistic behavior, it may be due to disease or weakness, and this can alert fish farmers to remove it.

[0098] The comprehensive early warning system adjusts the warning level to a higher level when multiple observed values ​​show abnormalities. Combining environmental data collected by sensors, if dissolved oxygen levels or water temperature are abnormal, and fish activity is also abnormal, a high-level alarm will be triggered. When a high-level alarm is triggered, it indicates that the fish are already exhibiting stress responses to the abnormal environment, and the warning program will immediately send a message to the aquaculture personnel for appropriate action.

[0099] Example 3: In addition to Example 2 above, this example also provides an early warning system for abnormal fish activity, such as... Figure 7 As shown, the fish activity abnormality early warning system provided in this embodiment mainly includes: a video and environmental parameter acquisition module for performing step one; a fish target tracking and trajectory acquisition module for performing step two; an activity estimation module for performing step three; a group division and group attribute calculation module for performing step four; and an early warning module for performing step five.

[0100] This invention also provides an early warning device, such as... Figure 8 As shown, it includes equipment control devices, video acquisition devices, environmental parameter acquisition devices, and environmental control equipment, such as... Figure 9 As shown, the environmental parameter acquisition device includes a temperature sensor, a dissolved oxygen sensor, a pH sensor, a conductivity sensor, and a turbidity sensor. These sensors are used to collect relevant water quality parameters, including water temperature, dissolved oxygen, pH value, conductivity, and turbidity. The video acquisition device is a camera. The video captured by the video acquisition device is transmitted to a cloud server, which is a remote computer used to deploy the early warning system and perform functions such as activity level and population attribute calculation, and sending early warning information. The cloud server required by this invention needs a dedicated graphics processing unit (GPU) to process the video information to ensure real-time performance. The environmental control device includes an STM32 core controller, a second PLC power line carrier module, and a smart switch. The first PLC power line carrier module (… Figure 9 The PLC power line carrier module (left side) includes a carrier and power lines, used to transmit collected environmental information to the STM32 core controller via an RS485 serial port. The STM32 core controller is the core component of the STM32 core board, consisting of an STM32 filter circuit, an STM32 microcontroller, a reset circuit, and a startup mode selection circuit. It receives information from the environmental acquisition device, sends it to the cloud server, and converts the server's calculations into control signals to control the smart switch. The core controller is an STM32F407 series development board, which features multiple serial ports, relatively fast ADC acquisition and SPI communication speeds, high real-time performance, and an Ethernet module. The communication interface can collect more external information and sensor data, such as temperature, dissolved oxygen, pH value, conductivity, and turbidity. The second PLC power line carrier module ( Figure 9 The PLC power line carrier module (on the right) is used to transmit signals received from the core controller to the intelligent switch. The intelligent switch refers to a circuit control system that utilizes a combination of control boards and electronic components. It receives information from the PLC power line carrier module from the core controller and controls the environmental control equipment, including circulating water pumps and aerators. These devices receive start / stop information from the intelligent switch and adjust parameters such as temperature and dissolved oxygen in the aquaculture water. Based on information provided by the early warning system, the equipment control device controls the environmental control equipment to perform environmental regulation.

[0101] Handheld devices are microcomputers that can be held in one's hand, such as mobile phones or other customized mobile terminals, used to receive early warning information sent by cloud servers.

[0102] The early warning system includes an acquisition module, a basic parameter output module, an offset correction module, a cost matrix calculation module, a trajectory set determination module, an activity calculation module, an activity anomaly judgment module, a connected component detection module, a group attribute calculation module, an outlier behavior judgment module, and an early warning information sending module. These modules are installed on a cloud server to process the collected environmental and video information. After obtaining the results, the cloud server sends them to the STM32 core controller via the network.

[0103] Example 4: This embodiment of the invention provides a fish activity abnormality early warning system corresponding to the method provided in Example 1 above. The system includes: an acquisition module for acquiring target video in real time; the target video includes multiple fish and the last frame of the target video is the current time.

[0104] The basic parameter output module is used to input the target video into the fish multi-target tracking model to obtain the basic parameters of each fish in each frame of the target video; the basic parameters include the center point coordinates of the detection box, the width of the detection box, the height of the detection box, the confidence level of the detection box, and the tracking displacement.

[0105] The offset correction module is used to perform offset correction on the detection box of the fish in any frame of the target video by using the tracking displacement of the fish in the frame of the target video, so as to obtain the bounding box of the fish in the frame of the target video.

[0106] The final trajectory set determination module is used to process the center point coordinates of the bounding box of each fish in each frame of the target video using the data exchange association method to obtain the final trajectory set; the final trajectory set includes the final trajectory of each fish in the target video.

[0107] The fish activity calculation module is used to calculate the activity of any fish in the target video based on the fish's mass and the coordinates of the center points of each bounding box in the fish's final trajectory.

[0108] The fish activity calculation module is used to sum the activity of each fish in the target video to obtain the activity of the fish school in the target video; the fish school in the target video includes all fish in the target video.

[0109] The activity level anomaly detection module is used to determine that the activity level of the fish in the target video is abnormal at the current moment if the activity level of the fish in the target video is not within the set activity level range.

[0110] This invention provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the fish activity abnormality early warning method according to the above method embodiment.

[0111] This invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the fish activity abnormality early warning method as described in the above method embodiments.

[0112] The beneficial effects of this invention specifically include: 1. This invention enables real-time multi-target tracking, calculation, and early warning of fish. The fish multi-target tracking model based on the improved FasterNet inherits the fast inference speed of FasterNet and has higher tracking accuracy than the original FasterNet.

[0113] 2. This invention calculates fish activity by using a grid step size, which avoids the problem of center point drift caused by changes in fish appearance, and also avoids the problem of large computational load caused by frame-by-frame calculation.

[0114] 3. This invention divides fish groups based on the connectivity of fish object detection boxes and calculates the group attributes of the fish groups. It does not require prior network training or prior knowledge of the number of fish or the distribution of individuals.

[0115] 4. This invention enables real-time monitoring and early warning of fish activity, which can not only detect abnormalities in fish activity and aquaculture water quality in a timely manner, reducing potential economic losses, but also has important significance for fish behavior research.

[0116] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0117] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for early warning of abnormal fish activity, characterized in that, include: Real-time acquisition of target video; The target video includes multiple fish, and the last frame of the target video is the current time. The target video is input into a multi-target fish tracking model to obtain the basic parameters of each fish in each frame of the target video. The basic parameters include the center point coordinates of the detection box, the width of the detection box, the height of the detection box, the confidence score of the detection box, and the tracking displacement. The multi-target fish tracking model includes an improved FasterNet network, a feature fusion network, and a prediction network connected in sequence. The improved FasterNet network includes a Pre_Embedding branch and a FasterNet network. The output of the Pre_Embedding branch and the output of the Embedding layer of the FasterNet network are added pixel by pixel. The sum is then connected to the input of the first FasterNetBlock module of the FasterNet network. The structure of the Pre_Embedding branch is the same as the structure of the Embedding layer of the FasterNet network. For any fish in any frame of the target video, the detection box of the fish in the frame of the target video is offset corrected by the tracking displacement of the fish in the frame of the target video to obtain the bounding box of the fish in the frame of the target video. The center point coordinates of the bounding box of each fish in each frame of the target video are processed using a data association method to obtain a final trajectory set; the final trajectory set includes the final trajectory of each fish in the target video. For any fish in the target video, the activity level of the fish is obtained based on the fish's mass and the coordinates of the center points of each bounding box in the fish's final trajectory, specifically including: Establish a grid coordinate system; The coordinate system is transformed by performing coordinate system transformation on the center point coordinates of all target bounding boxes in the final trajectory of the fish to obtain the center point coordinates of all bounding boxes in the final trajectory of the fish in the grid coordinate system; At the current loop count, based on the coordinates of the center point of the bounding box of the fish in the nth frame of the current loop count in the final trajectory of the fish in the grid coordinate system and the coordinates of the center point of the bounding box of the fish in the target frame of the current loop count in the final trajectory of the fish in the grid coordinate system, it is determined whether the fish moves between the nth frame of the current loop count and the target frame of the current loop count by a set grid step size; the target frame at the first loop count is the (n+1)th frame; If the set grid step size is not moved, the target frame in the next loop count is determined to be the next frame after the target frame in the current loop count, and the loop count is updated to enter the next loop; If the movement is set to a grid step size, then based on the coordinates of the center point of the bounding box of the fish in the nth frame image of the final trajectory of the fish in the grid coordinate system at the current loop number and the coordinates of the center point of the bounding box of the fish in the target frame image of the final trajectory of the fish in the grid coordinate system at the current loop number, the movement distance of the fish in the current loop number is calculated, then the nth frame of the next loop number is determined as the target frame of the current loop number, the target frame of the next loop number is the next frame of the target frame of the current loop number, and the loop number is updated to enter the next loop; The activity level of the fish is obtained by dividing the fish's mass by the square of the distance it moves over all cycles. The activity levels of each fish in the target video are summed to obtain the activity level of the fish school in the target video; the fish school in the target video includes all fish in the target video. If the activity level of the fish in the target video is not within the set activity level range, then the activity level of the fish in the target video at the current moment is determined to be abnormal.

2. The method for early warning of abnormal fish activity according to claim 1, characterized in that, Also includes: For the nth frame of the target video, all fish in the nth frame of the target video are divided into multiple fish groups corresponding to the nth frame of the target video based on the center point coordinates of the bounding box of each fish in the nth frame of the target video, the width of the bounding box, and the height of the bounding box; n is a positive integer greater than 1. For any school of fish, the group attributes of the school are calculated based on the coordinates of the center point of the school corresponding to the nth frame of the target video, the speed of each fish in the school, and the coordinates of the center point of the bounding box of each fish in the school. The group attributes include the school radius, average speed, dispersion, cohesion, and relative direction. The presence of out-of-group behavior in the fish population is determined based on the group attributes of the fish population and a set attribute threshold.

3. The method for early warning of abnormal fish activity according to claim 1, characterized in that, The data association method is used to process the center point coordinates of the bounding box of each fish in each frame of the target video to obtain the final trajectory set, which specifically includes: For the nth frame of the target video, a distance set is obtained based on the coordinates of the center points of the bounding boxes of all fish in the nth frame and the center points of each trajectory in the trajectory set corresponding to the (n-1)th frame. The distance set includes the distance between the center point of each trajectory in the trajectory set corresponding to the (n-1)th frame and the center point of the bounding box of each fish. The trajectory set includes multiple trajectories, and each trajectory includes multiple bounding boxes. The trajectory set corresponding to the first frame is all bounding boxes in the first frame of the target video with a confidence level greater than a set confidence threshold. n is a positive integer greater than 1. Based on the distance set, the bounding boxes of all fish in the nth frame of the target video are assigned to the corresponding trajectories in the trajectory set of the (n-1)th frame. The complete trajectory set corresponding to the nth frame is then removed from the trajectory set corresponding to the (n-1)th frame. Unassigned bounding boxes are added to the trajectory set corresponding to the (n-1)th frame, resulting in the trajectory set corresponding to the nth frame. The nth frame is then updated. The process of obtaining the distance set based on the center point coordinates of the bounding boxes of all fish in the nth frame and the center points of the trajectories in the trajectory set corresponding to the (n-1)th frame is repeated until the trajectory set corresponding to the last frame of the target video and the complete trajectory set corresponding to the last frame of the target video are obtained. The trajectory set of the last frame of the target video and the complete trajectory set corresponding to all frames are determined as the final trajectory set. The complete trajectory set corresponding to the nth frame includes all unassigned trajectories in the trajectory set corresponding to the (n-1)th frame.

4. The method for early warning of abnormal fish activity according to claim 3, characterized in that, Based on the distance set, the bounding boxes of all fish in the nth frame of the target video are assigned to the corresponding trajectories in the trajectory set of the (n-1)th frame. The complete trajectory set corresponding to the nth frame is then removed from the trajectory set corresponding to the (n-1)th frame. Unassigned bounding boxes are added to the trajectory set corresponding to the (n-1)th frame, resulting in the trajectory set corresponding to the nth frame. Specifically, this includes: The Jonker-Volgenant algorithm is used to assign the bounding boxes of all fish in the nth frame of the target video to the corresponding trajectories in the trajectory set of the (n-1)th frame based on the distance set; In the nth frame of the target video, the bounding boxes of all fish that have not been assigned a trajectory are identified as new trajectories and added to the trajectory set corresponding to the (n-1)th frame. All unassigned trajectories in the trajectory set corresponding to the (n-1)th frame are identified as the complete trajectory set corresponding to the nth frame. The complete trajectory set corresponding to the nth frame is then removed from the trajectory set corresponding to the (n-1)th frame to obtain the trajectory set corresponding to the nth frame.

5. The method for early warning of abnormal fish activity according to claim 1, characterized in that, The process of determining the multi-target tracking model for fish includes: Obtain a training set, which includes historical videos and detection boxes for each fish in each frame of the historical videos, as well as the ID of each fish; With the goal of minimizing the total loss function, a multi-target fish tracking model is obtained by training an improved FasterNet network, a feature fusion network, and a prediction network connected in sequence based on the training set.

6. A fish activity abnormality early warning system, characterized in that, include: The acquisition module is used to acquire the target video in real time; The target video includes multiple fish, and the last frame of the target video is the current time. The basic parameter output module is used to input the target video into the fish multi-target tracking model to obtain the basic parameters of each fish in each frame of the target video. The basic parameters include the center point coordinates of the detection box, the width of the detection box, the height of the detection box, the confidence score of the detection box, and the tracking displacement. The fish multi-target tracking model includes an improved FasterNet network, a feature fusion network, and a prediction network connected in sequence. The improved FasterNet network includes a Pre_Embedding branch and a FasterNet network. The output of the Pre_Embedding branch and the output of the Embedding layer of the FasterNet network are added pixel by pixel. After addition, the result is connected to the input of the first FasterNetBlock module of the FasterNet network. The structure of the Pre_Embedding branch is the same as the structure of the Embedding layer of the FasterNet network. The offset correction module is used to perform offset correction on the detection box of any fish in any frame of the target video by using the tracking displacement of the fish in the frame of the target video, so as to obtain the bounding box of the fish in the frame of the target video. The final trajectory set determination module is used to process the center point coordinates of the bounding box of each fish in each frame of the target video using a data association method to obtain the final trajectory set; the final trajectory set includes the final trajectory of each fish in the target video; The fish activity calculation module is used to calculate the activity level of any fish in the target video based on the fish's mass and the coordinates of the center points of each bounding box in the fish's final trajectory. Specifically, it includes: Establish a grid coordinate system; The coordinate system is transformed by performing coordinate system transformation on the center point coordinates of all target bounding boxes in the final trajectory of the fish to obtain the center point coordinates of all bounding boxes in the final trajectory of the fish in the grid coordinate system; At the current loop count, based on the coordinates of the center point of the bounding box of the fish in the nth frame of the current loop count in the final trajectory of the fish in the grid coordinate system and the coordinates of the center point of the bounding box of the fish in the target frame of the current loop count in the final trajectory of the fish in the grid coordinate system, it is determined whether the fish moves between the nth frame of the current loop count and the target frame of the current loop count by a set grid step size; the target frame at the first loop count is the (n+1)th frame; If the set grid step size is not moved, the target frame in the next loop count is determined to be the next frame after the target frame in the current loop count, and the loop count is updated to enter the next loop; If the movement is set to a grid step size, then based on the coordinates of the center point of the bounding box of the fish in the nth frame image of the final trajectory of the fish in the grid coordinate system at the current loop number and the coordinates of the center point of the bounding box of the fish in the target frame image of the final trajectory of the fish in the grid coordinate system at the current loop number, the movement distance of the fish in the current loop number is calculated, then the nth frame of the next loop number is determined as the target frame of the current loop number, the target frame of the next loop number is the next frame of the target frame of the current loop number, and the loop number is updated to enter the next loop; The activity level of the fish is obtained by dividing the fish's mass by the square of the distance it moves over all cycles. The fish activity calculation module is used to sum the activity of each fish in the target video to obtain the activity of the fish school in the target video; the fish school in the target video includes all fish in the target video; The activity level anomaly detection module is used to determine that the activity level of the fish in the target video is abnormal at the current moment if the activity level of the fish in the target video is not within the set activity level range.

7. An electronic device, characterized in that, include: A memory and a processor, the memory being used to store a computer program, the processor running the computer program to cause the electronic device to perform the fish activity abnormality early warning method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the fish activity abnormality early warning method as described in any one of claims 1 to 5.

Images