Method for predicting and tracking motion trail of fish by underwater robot based on LSTM (Long Short Term Memory)

Abstract

The invention discloses a method for predicting and tracking a fish movement track by an underwater robot based on LSTM (Long Short Term Memory). The method comprises the following steps: preprocessing a fish movement video and sonar data to obtain historical movement track data; constructing a double-layer LSTM neural network prediction model, and performing model training through historical motion track data; deploying the trained model to the underwater robot; the underwater robot collects a fish motion video and sonar data to obtain initial motion track data with a set length; inputting the initial motion trail data into a trained prediction model, and predicting the motion trail of the target fish; repeating the prediction step to obtain multiple groups of motion tracks, and calculating a confidence interval of track coordinate points at each moment; if the confidence coefficient of the coordinate point at a certain moment meets a threshold value, the underwater robot goes to the point on the future trajectory to perform pre-judgment tracking, otherwise, tracking is performed according to the fish position observed in the previous frame, the initial motion trajectory data is updated by adopting a sliding window method, and the step 5 and the step 6 are executed again.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent perception of underwater robots, and particularly relates to a fish motion trajectory prediction and tracking method based on LSTM for underwater robots. BACKGROUND

[0002] Traditional methods relying on manual experience or single device fixed-point observation have been difficult to meet the needs of smart ocean aquaculture in state evaluation and feeding regulation, and intelligent fishing in precise positioning and efficient operation. Underwater robots equipped with high-definition cameras and multi-beam imaging sonars can obtain real-time fish group images and position information, providing an effective means for realizing all-around, all-weather monitoring and fishing operation. However, fish movement has the characteristics of high nonlinearity, strong randomness and easy environmental interference, and instantaneous observation data cannot reliably predict its movement trajectory, resulting in low recognition rate and poor evaluation performance in aquaculture monitoring, and limiting the automation and precision of fishing operations. Therefore, there is an urgent need for a trajectory prediction and tracking algorithm suitable for underwater robots, which can learn the dynamic characteristics of the environment based on the historical motion sequence of fish and accurately predict their future trajectory. This technology will greatly improve the monitoring and decision-making capabilities of underwater robots in aquaculture waters, and has key significance for promoting the fine management of smart ocean aquaculture and realizing the efficiency and sustainability of intelligent fishing. SUMMARY

[0003] To solve the above technical problems, the purpose of the present application is to provide a fish motion trajectory prediction and tracking method based on LSTM for underwater robots.

[0004] The fish motion trajectory prediction and tracking method based on LSTM for underwater robots of the present application comprises:

[0005] Step 1: Collect fish motion video and sonar data, and pre-process to obtain historical motion trajectory data;

[0006] Step 2: Construct a double-layer LSTM neural network prediction model, and train the prediction model with historical motion trajectory data;

[0007] Step 3: Deploy the trained double-layer LSTM neural network prediction model to the underwater robot;

[0008] Step 4: The underwater robot tracks the target fish, continuously collects fish motion video and sonar data, until the initial motion trajectory data of a set length is obtained;

[0009] Step 5: Input the initial motion trajectory data of a set length into the trained double-layer LSTM neural network prediction model to predict the future T-frame motion trajectory of the target fish;

[0010] Step 6: Repeat step 5 to obtain multiple sets of target fish future T time motion trajectories, and calculate the confidence interval of each time trajectory coordinate point;

[0011] Step 7: If the confidence of the trajectory coordinate point at a certain time meets the set threshold, the underwater robot directly goes to the point on the future trajectory for predictive tracking, otherwise the underwater robot tracks the fish position observed in the last frame, updates the initial motion trajectory data using the sliding window method, and re-executes steps 5 and 6.

[0012] The fish motion trajectory prediction tracking method based on LSTM underwater robot has the following beneficial effects:

[0013] 1. The double-layer LSTM architecture is adopted, the first layer filters out sensor noise in real time and fuses position and speed features, and the second layer models long-term behavior patterns, which significantly improves the memory time and the prediction accuracy of complex motion trajectories.

[0014] 2. The multi-step rolling prediction mechanism is designed, and T frames of trajectory prediction are output in a single forward propagation, which greatly reduces the error accumulation compared with iterative prediction and improves the early decision-making ability of the robot.

[0015] 3. The confidence interval estimation is introduced, the prediction uncertainty is quantified by multiple random sampling, the robot can execute the risk-sensitive encirclement strategy, and the success rate is improved.

[0016] 4. After using the present application, the accuracy of trajectory prediction of isolated fish and dense fish schools in open water can be effectively improved, and the prediction error is greatly reduced. At the same time, the method can give the confidence interval for the underwater robot to avoid or encircle in advance, and realize the truly deployable "intelligent fishing".

[0017] 5. The present application realizes the automatic and normal precision statistics of the number and motion of fish in the breeding net cage or pond through the long-distance and non-contact data acquisition of the sensor carried by the underwater robot combined with the high-precision target tracking capability, which completely avoids the interference to the fish school.

[0018] 6. Through long-term monitoring, a correlation model between fish motion patterns and growth indicators can be established to realize growth trend prediction based on behavior data, replacing part of the cumbersome work of fishing measurement. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a flowchart of the fish motion trajectory prediction tracking method based on LSTM underwater robot. DETAILED DESCRIPTION

[0020] As Figure 1As shown, the fish motion trajectory prediction tracking method based on the LSTM underwater robot of the present application comprises:

[0021] Step 1: Collect fish motion video and sonar data, and pre-process to obtain historical motion trajectory data, specifically:

[0022] Step 1.1: Real-time observation of target fish groups is performed by using an underwater robot equipped with a high-definition camera and a multi-beam imaging sonar in an experimental water environment.

[0023] In specific implementation, the data used by the present application is obtained by real-time observation of landscape fish such as koi and goldfish using an underwater robot equipped with a high-definition camera and a multi-beam imaging sonar in a laboratory pool environment.

[0024] Step 1.2: Fish identification is performed on the collected video and sonar data using a deep learning-based target detection algorithm (such as the YOLO series).

[0025] Step 1.3: Multi-target tracking algorithm (such as DeepSORT) is used for cross-frame association to obtain continuous and smooth spatio-temporal trajectory sequences for each fish.

[0026] Step 1.4: For each spatio-temporal trajectory sequence, the 3D position coordinates x, y, z in the world coordinate system are calculated, and the velocity components v x , v y , v z are calculated by numerical differentiation.

[0027] Step 1.5: The position and velocity are combined into a 6-dimensional vector [x, y, z, v x , v y , v z ] as the basic input unit of the prediction model.

[0028] Step 1.6: Z-Score standardization processing is performed on all input features to make the mean value 0 and the standard deviation 1, to accelerate model training and improve stability.

[0029] Step 2: A double-layer LSTM neural network prediction model is constructed, and the prediction model is trained using historical motion trajectory data.

[0030] The double-layer LSTM neural network prediction model comprises an input layer, a first-layer LSTM network, a second-layer LSTM network, and an output layer; the input layer is used to receive input historical trajectory sequence X(t)={X t-k ,X t-k+1 ,...,X t}, which contains k time step observation values, wherein Xi = [x i , y i , z i ,v xi , v yi , v zi ] contains position and velocity components.

[0031] The first layer LSTM network, as the front-end processor of the network, the core role is to filter and primary feature abstraction of the input original trajectory sequence in real time, including the number of neurons d1=128, according to the short-term motion trajectory and real-time input to filter out sensor noise, extract motion features and transmit to the second layer LSTM network, denoted as:

[0032] Forget gate:

[0033]

[0034] wherein, is the output of the forget gate, is the hidden layer output of the first LSTM layer at the last time, is the input at the current time, is the weight matrix of the forget gate; is the bias matrix of the forget gate.

[0035] Input gate:

[0036]

[0037] wherein, is the output of the input gate, is the weight matrix of the input gate; is the bias matrix of the input gate.

[0038] Cell state update:

[0039]

[0040]

[0041] wherein, is the candidate cell state, is the cell state, is the weight matrix of the candidate gate, is the bias matrix of the candidate gate.

[0042] Output gate:

[0043]

[0044] wherein, is the output of the output gate, This is the weight matrix of the output gate; This is the bias matrix for the output gate.

[0045] Hidden layer state update:

[0046]

[0047] in, The 128-dimensional short-term motion feature vector is used as the input to the second-layer LSTM network.

[0048] Context-Based Dynamic Filtering: The first-layer LSTM network, through its gating mechanisms (especially the forget gate and input gate), does not employ a static filter with fixed parameters, but instead implements intelligent filtering based on motion context. For each new input data point, the LSTM compares it with the short-term historical states of the previous frames to determine whether the information represents a true motion trend or a transient jitter caused by sensor noise. If it is the latter, the forget gate tends to preserve the previous stable state, while the input gate restricts the update of the cell state by the noise information, thereby achieving real-time, adaptive noise suppression.

[0049] Short-term motion pattern extraction: While filtering out noise, this layer extracts higher-level, discriminative short-term motion features from the original 6-dimensional position and velocity data. For example, it can identify:

[0050] a. Continuity of the direction of motion (e.g., uniform linear motion).

[0051] b. Changes in motion state (such as acceleration or deceleration).

[0052] c. The initiation of a trend (such as an initial signal of a slight turn).

[0053] Feature vector generation and propagation: After the above processing, the hidden state output by the first LSTM layer at each time step It is no longer a simple physical quantity, but a 128-dimensional feature vector that encodes short-term motion features after noise filtering and abstraction. This sequence of feature vectors forms the input of the second LSTM layer, providing clean and information-rich underlying data for deeper temporal modeling.

[0054] The second LSTM network is the same as the first LSTM network, with 64 neurons (d2) and a memory capacity of 1000 time steps. It predicts the potential behavioral intentions of fish based on long-term movement trajectories. The second LSTM network outputs the final hidden state at each time step. It is no longer a simple feature, but a 64-dimensional, highly condensed code of behavioral intent.

[0055] The second-layer LSTM, serving as the network's advanced cognitive core, elevates the short-term motion features extracted from the first layer into an understanding of the fish's long-term behavioral intentions and macroscopic movement patterns. Its specific working mechanism is as follows:

[0056] Long-term context modeling: With its massive memory capacity of approximately 1000 time steps, this layer can span much longer time windows than the first layer, establishing long-range dependencies between behaviors. It can identify and memorize macroscopic patterns such as periodic patrols, diurnal migration patterns, and fixed routes under specific environments.

[0057] Inference and Encoding of Behavioral Intent: This layer receives the noise-filtered short-term feature sequence output from the first layer and performs in-depth analysis on it. By analyzing the evolution of the feature sequence, it can infer the potential behavioral intent of the fish. For example:

[0058] a. When a continuous, directionally stable high-speed motion feature is identified, it may be encoded as an intention of "escape behavior".

[0059] b. When slow motion features with slight changes in direction are detected, they may be encoded as an intention to "forage and patrol".

[0060] c. When a sharp, rapid turning feature is identified, it may be encoded as an intention to “suddenly evade”.

[0061] Generate behavioral summary state: After the above high-level analysis and integration, the final hidden state output by the second-layer LSTM at each time step. It is no longer a simple feature, but a 64-dimensional, highly condensed "behavioral intent code". This state vector encapsulates the system's overall interpretation of the target's behavior from the past to the present, as well as high-level expectations of future movement trends.

[0062] Output layer:

[0063]

[0064]

[0065] in, For displacement increment, This represents the position of the fish in frame t+1. The position of the fish in frame t. These are the output layer weights; This is the bias matrix of the output layer; this layer will ultimately hide the state. This is transformed into a specific predicted location vector.

[0066] In step 2, the number of trajectory prediction frames T is first determined, and then the historical motion trajectory data is input into a two-layer LSTM neural network prediction model to predict the future position of the fish. Multiple rolling predictions are continuously performed within the prediction time domain. Specifically, ... The historical trajectory is input into a two-layer LSTM neural network prediction model to predict forward. The position, and then with As a historical trajectory, let's predict forward again. The position of the frame is used to predict the future movement trajectory of the fish by scrolling.

[0067] In practice, the prediction model is trained using historical motion trajectory data, specifically employing an end-to-end supervised learning paradigm. The core objective is to minimize the error between the multi-step predicted trajectory output by the model and the actual trajectory by optimizing the network parameters. Mean squared error is used as the loss function, and the model training uses the Adam optimizer, combined with the backpropagation algorithm, to update all parameters in the network, including the weights and biases of the two LSTM layers and the output layer, thereby stably and quickly converging to the local optimum of the loss function.

[0068] Step 3: Deploy the trained two-layer LSTM neural network prediction model onto the underwater robot. This forms a real-time intelligent closed-loop system of "perception-prediction-decision-action".

[0069] Step 4: The underwater robot tracks the target fish and continuously collects video and sonar data of the fish's movement until it obtains an initial motion trajectory data of a set length, such as 30 frames.

[0070] Step 5: Input the initial motion trajectory data of a set length into the trained two-layer LSTM neural network prediction model to predict the motion trajectory of the target fish in future T frames.

[0071] Step 6: Repeat step 5 to obtain multiple sets of the target fish's motion trajectories for future T frames, and calculate the confidence interval of the trajectory coordinates at each time step. Based on this prediction, the robot plans and executes its actions.

[0072] In practice, the 95% confidence interval is calculated using the following formula:

[0073]

[0074]

[0075]

[0076] in, The mean position of the predicted i+T frames. CI represents the standard deviation of the predicted i+T frame position. i+TLet N be the confidence interval for frame i+T, and N be the number of repeated predictions.

[0077] Step 7: If the confidence level of the trajectory coordinates at a certain moment meets the set threshold, the underwater robot will directly go to that point on the future trajectory to perform predictive tracking. Otherwise, the underwater robot will track the fish position observed in the previous frame, update the initial motion trajectory data using the sliding window method, and re-execute steps 5 and 6.

[0078] Step 7 specifically involves the following steps: When the real data of a new moment arrives, the robot updates the initial motion trajectory data using the sliding window method, that is, discarding the oldest moment data and adding the latest moment data; the updated motion trajectory data is immediately sent to the prediction model, triggering a new round of prediction and updating the confidence interval. The robot dynamically adjusts its motion state according to the new prediction results, and this process is repeated to achieve adaptive tracking.

[0079] High confidence (narrow interval): The robot adopts an aggressive strategy, such as going directly to the predicted point for precise tracking, which is the most efficient.

[0080] Low confidence (wide range): The robot adopts a conservative strategy, tracking the fish's position based on the previous frame's observations to avoid decision-making errors. This risk-sensitive decision-making mechanism significantly improves the system's success rate and robustness.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the ideas of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for predicting and tracking the movement trajectory of fish using an underwater robot based on LSTM, characterized in that, include: Step 1: Collect fish movement videos and sonar data, and perform preprocessing to obtain historical movement trajectory data; Step 2: Construct a two-layer LSTM neural network prediction model and train the prediction model using historical motion trajectory data; Step 3: Deploy the trained two-layer LSTM neural network prediction model onto the underwater robot; Step 4: The underwater robot tracks the target fish, continuously collecting video and sonar data of the fish's movement until it obtains initial trajectory data of a set length; Step 5: Input the initial motion trajectory data of a set length into the trained two-layer LSTM neural network prediction model to predict the motion trajectory of the target fish in the next T frames; Step 6: Repeat step 5 to obtain the motion trajectory of the target fish in the next T frames, and calculate the confidence interval of the trajectory coordinates at each time step; Step 7: If the confidence level of the trajectory coordinates at a certain moment meets the set threshold, the underwater robot will directly go to that point on the future trajectory to perform predictive tracking. Otherwise, the underwater robot will track the fish position observed in the previous frame, update the initial motion trajectory data using the sliding window method, and re-execute steps 5 and 6.

2. The method for predicting and tracking the movement trajectory of fish using an underwater robot based on LSTM according to claim 1, characterized in that, Step 1 specifically involves: Step 1.1: In the experimental water environment, an underwater robot equipped with a high-definition camera and multi-beam imaging sonar was used to observe the target fish school in real time. Step 1.2: Use a deep learning-based target detection algorithm to identify fish based on the collected video and sonar data; Step 1.3: Then, a multi-target tracking algorithm is used to perform cross-frame correlation, thereby obtaining a continuous and smooth spatiotemporal trajectory sequence for each fish; Step 1.4: For each spatiotemporal trajectory sequence, calculate its 3D position coordinates x, y, z in the world coordinate system, and calculate its velocity component v using numerical differentiation. x , v y , v z ; Step 1.5: Combine position and velocity into a 6-dimensional vector [x, y, z, v] x , v y , v z [This serves as the basic input unit for the prediction model;] Step 1.6: Perform Z-Score standardization on all input features to make their mean 0 and standard deviation 1, in order to accelerate model training and improve stability.

3. The method for predicting and tracking the movement trajectory of fish using an underwater robot based on LSTM according to claim 1, characterized in that, The two-layer LSTM neural network prediction model includes: an input layer, a first-layer LSTM network, a second-layer LSTM network, and an output layer; the input layer is used to receive the input historical trajectory sequence X(t) = {X...} t-k ,X t-k+1 ,...,X t } contains observations at k time steps, where X i = [x i , y i , z i ,v xi , v yi , v zi Includes position and velocity components; The first-layer LSTM network, acting as the network's front-end processor, performs real-time filtering and preliminary feature abstraction on the input raw trajectory sequence. This includes having 128 neurons (d1=128). Based on the short-term motion trajectory and real-time input, it filters out sensor noise, extracts motion features, and transmits them to the second-layer LSTM network, as shown below: Forgotten Gate: in, For the output of the forget gate, This is the hidden layer output of the first LSTM layer at the previous time step. Input the current time. Here is the weight matrix for the forget gate; Here is the bias matrix for the forget gate; Input Gate: in, The output of the input gate, This is the weight matrix of the input gate; This is the bias matrix of the input gate; Cell status update: In the formula, Candidate cell state, In cellular state, Let be the weight matrix of the candidate gates. Here is the candidate gate bias matrix; Output gate: in, For the output of the output gate, This is the weight matrix of the output gate; This is the bias matrix for the output gate; Hidden layer state update: in, The 128-dimensional short-term motion feature vector is used as the input to the second-layer LSTM network. The second LSTM network is the same as the first LSTM network, with 64 neurons (d2) and a memory capacity of 1000 time steps. It predicts the potential behavioral intentions of fish based on long-term movement trajectories. The second LSTM network outputs the final hidden state at each time step. It is no longer a simple feature, but a 64-dimensional, highly condensed code of behavioral intent; Output layer: in, For displacement increment, This represents the position of the fish at time t+1. Let be the position of the fish at time t. These are the output layer weights; This is the bias matrix of the output layer; this layer will ultimately hide the state. This is transformed into a specific predicted location vector.

4. The method for predicting and tracking the movement trajectory of fish using an underwater robot based on LSTM according to claim 1, characterized in that, In step 2, the number of trajectory prediction frames T is first determined, and then the historical motion trajectory data is input into a two-layer LSTM neural network prediction model to predict the future position of the fish. Multiple rolling predictions are continuously performed within the prediction time domain. Specifically, ... The historical trajectory is input into a two-layer LSTM neural network prediction model to predict forward. Location, then with As a historical trajectory, let's predict forward again. By observing the position of the fish and making rolling predictions, the predicted future movement trajectory of the fish can be obtained.

5. The method for predicting and tracking the movement trajectory of fish using an underwater robot based on LSTM according to claim 1, characterized in that, The prediction model is trained using historical motion trajectory data, specifically using an end-to-end supervised learning paradigm. Its core objective is to minimize the error between the multi-step predicted trajectory output by the model and the actual trajectory by optimizing the network parameters. The mean squared error is used as the loss function. The model is trained using the Adam optimizer and combined with the backpropagation algorithm to update all parameters in the network, including the weights and biases of the two LSTM layers and the output layer, so as to stably and quickly converge to the local optimum of the loss function.

6. The method for predicting and tracking the movement trajectory of fish using an underwater robot based on LSTM according to claim 1, characterized in that, In step 6, the 95% confidence interval is calculated according to the following formula: in, The mean position of the predicted i+T frames. CI represents the standard deviation of the predicted i+T frame position. i+T Let N be the confidence interval for frame i+T, and N be the number of repeated predictions.

7. The method for predicting and tracking the movement trajectory of fish using an underwater robot based on LSTM according to claim 1, characterized in that, Step 7 specifically involves: When the real data of the new moment arrives, the robot uses the sliding window method to update the initial motion trajectory data, that is, discarding the oldest moment data and adding the latest moment data. The updated motion trajectory data is immediately sent to the prediction model, triggering a new round of prediction and updating the confidence interval. The robot dynamically adjusts its motion state according to the new prediction results, and repeats this cycle to achieve adaptive tracking and monitoring, and interception or capture when relevant needs arise.

Images