Underwater target tracking method, device, equipment, storage medium and program product

Abstract

The invention relates to an underwater target tracking method, device and equipment, a storage medium and a program product. The method comprises the following steps: acquiring a first motion track of each underwater robot, the first motion track being used for representing a motion track when each underwater robot tracks a target in a simplified underwater environment in which interference is ignored; a plurality of second movement tracks corresponding to the underwater robots are generated according to the first movement tracks and an imitation learning strategy, an offline track pool is generated according to the second movement tracks, and the second movement tracks are used for representing movement tracks when the underwater robots track the target in the simulated underwater environment; and for each underwater robot, a third motion track of the underwater robot is determined according to the offline track pool and the corresponding prediction model, and the prediction model is obtained by training a preset initial prediction model according to the offline track pool. By adopting the method, the motion trail of the underwater robot can be accurately determined.

Description

Technical Field

[0001] The present application relates to the field of target tracking technology, and in particular to an underwater target tracking method, device, equipment, storage medium and program product. Background Art

[0002] Tracking targets using autonomous underwater robots is an important research topic, especially multi-agent control.

[0003] In the existing methods, the target tracking strategy of underwater robots can be determined by the artificial potential field method. The main idea is to abstract the positive demands (such as target tracking, communication, etc.) into a gravitational field, and the negative demands (such as obstacle avoidance, etc.) into a repulsive field, thereby abstracting the movement of the intelligent body into passive movement under the field.

[0004] However, the existing technology has good performance in aspects such as path planning in traditional ideal environments, but it requires a lot of simplification of the environment and cannot cope with very complex requirements, that is, the accuracy of the underwater robot's motion trajectory determined in a complex environment is low. Summary of the invention

[0005] Based on this, it is necessary to provide an underwater target tracking method, device, equipment, storage medium and program product that can accurately determine the motion trajectory of an underwater robot in response to the above technical problems.

[0006] In a first aspect, the present application provides an underwater target tracking method, comprising:

[0007] Acquire a first motion trajectory of each underwater robot, where the first motion trajectory is used to represent the motion trajectory of each underwater robot when tracking a target in a simplified underwater environment where interference is ignored;

[0008] Generate multiple second motion trajectories corresponding to each underwater robot according to each first motion trajectory and the imitation learning strategy, and generate an offline trajectory pool according to each second motion trajectory, wherein the second motion trajectory is used to represent the motion trajectory of each underwater robot when tracking the target in the simulated underwater environment;

[0009] For each underwater robot, a third motion trajectory of the underwater robot is determined according to the offline trajectory pool and a corresponding prediction model, and the prediction model is obtained by training a preset initial prediction model according to the offline trajectory pool.

[0010] In one embodiment, generating a plurality of second motion trajectories corresponding to each underwater robot according to each first motion trajectory and the imitation learning strategy includes:

[0011] A generative adversarial imitation learning model is obtained, wherein the generative adversarial imitation learning model includes an initial global identification network and multiple initial first policy networks, wherein the number of the initial first policy networks is the same as the number of underwater robots, and each initial first policy network corresponds to each underwater robot one by one; according to each first motion trajectory, the initial global identification network and each initial first policy network are trained to obtain a global identification network and multiple first policy networks; and multiple second motion trajectories corresponding to each underwater robot are generated through each first policy network.

[0012] In one of the embodiments, the prediction model includes a posterior information extractor and a second policy network based on Transformer, and the third motion trajectory of the underwater robot is determined according to the offline trajectory pool and the corresponding prediction model, including: obtaining a motion trajectory segment of the underwater robot in a preset time length from the offline trajectory pool; determining a latent vector corresponding to the motion trajectory segment according to the motion trajectory segment and the posterior information extractor; inputting the motion trajectory segment and the latent vector into the second policy network based on Transformer, and determining the third motion trajectory of the underwater robot according to the output of the second policy network.

[0013] In one of the embodiments, the training process of the prediction model includes: determining a training sample based on the motion trajectory in the offline trajectory pool; obtaining a sample trajectory segment of the underwater robot from the training sample, the sample trajectory segment being the motion trajectory of the underwater robot within a certain time length; determining a latent vector corresponding to the sample trajectory segment based on the sample trajectory segment and the initial posterior information extractor; determining a predicted trajectory of the underwater robot based on the sample trajectory segment, the latent vector and the initial second policy network; determining a loss function based on the predicted trajectory and the sample trajectory corresponding to the training sample, and updating the network parameters of the initial posterior information extractor and the initial second policy network based on the loss function until the posterior information extractor and the second policy network are obtained.

[0014] In one embodiment, obtaining the first motion trajectory of each underwater robot includes: generating the first motion trajectory of each underwater robot according to a preset target tracking algorithm in a simplified underwater environment; wherein the preset target tracking algorithm includes: artificial potential field method, Lyapunov vector field method, model predictive control method.

[0015] In one embodiment, the third motion trajectory includes the state space and behavior space of the underwater robot, the state space includes the position information of the tracking target and the positions of other underwater robots, and the behavior space includes the forward speed, swing speed and angular velocity of the underwater robot.

[0016] In a second aspect, the present application also provides an underwater target tracking device, comprising:

[0017] An acquisition module, used for acquiring a first motion trajectory of each underwater robot, where the first motion trajectory is used for representing the motion trajectory of each underwater robot when tracking a target in a simplified underwater environment where interference is ignored;

[0018] A generation module is used to generate a plurality of second motion trajectories corresponding to each underwater robot according to each first motion trajectory and an imitation learning strategy, and to generate an offline trajectory pool according to each motion trajectory, wherein the second motion trajectory is used to represent the motion trajectory of each underwater robot when tracking a target in a simulated underwater environment;

[0019] The prediction module is used to determine the third motion trajectory of each underwater robot according to the offline trajectory pool and the corresponding prediction model, and the prediction model is obtained by training a preset initial prediction model according to the offline trajectory pool.

[0020] In one embodiment, the generation module is specifically used to obtain a generative adversarial imitation learning model, which includes an initial global identification network and multiple initial first policy networks, the number of the initial first policy networks is the same as the number of underwater robots, and each initial first policy network corresponds to each underwater robot one by one; according to each first motion trajectory, the initial global identification network and each initial first policy network are trained to obtain a global identification network and multiple first policy networks; and multiple second motion trajectories corresponding to each underwater robot are generated through each first policy network.

[0021] In one of the embodiments, the prediction model includes a posterior information extractor and a second policy network based on Transformer, and the prediction module is specifically used to obtain a motion trajectory segment of the underwater robot in a preset time length from an offline trajectory pool; determine a latent vector corresponding to the motion trajectory segment according to the motion trajectory segment and the posterior information extractor; input the motion trajectory segment and the latent vector into the second policy network based on Transformer, and determine the third motion trajectory of the underwater robot according to the output of the second policy network.

[0022] In one of the embodiments, the underwater target tracking device also includes a training module, which is used to determine a training sample based on the motion trajectory in the offline trajectory pool; obtain a sample trajectory segment of the underwater robot from the training sample, the sample trajectory segment is the motion trajectory of the underwater robot within a certain time length; determine the latent vector corresponding to the sample trajectory segment based on the sample trajectory segment and the initial posterior information extractor; determine the predicted trajectory of the underwater robot based on the sample trajectory segment, the latent vector and the initial second policy network; determine the loss function based on the predicted trajectory and the sample trajectory corresponding to the training sample, and update the network parameters of the initial posterior information extractor and the initial second policy network based on the loss function until the posterior information extractor and the second policy network are obtained.

[0023] In one of the embodiments, the acquisition module is specifically used to generate the first motion trajectory of each underwater robot in a simplified underwater environment according to a preset target tracking algorithm; wherein the preset target tracking algorithm includes: artificial potential field method, Lyapunov vector field method, and model predictive control method.

[0024] In one embodiment, the motion trajectory includes the state space and behavior space of the underwater robot, the state space includes the position information of the tracking target and the positions of other underwater robots, and the behavior space includes the forward speed, swing speed and angular velocity of the underwater robot.

[0025] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, any of the methods described in the first aspect is implemented.

[0026] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the methods described in the first aspect above.

[0027] In a fifth aspect, the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements any of the methods described in the first aspect above.

[0028] The underwater target tracking method, device, equipment, storage medium and program product are obtained by obtaining the first motion trajectory of each underwater robot, the first motion trajectory is used to represent the motion trajectory of each underwater robot when tracking the target in a simplified underwater environment ignoring interference; generating multiple second motion trajectories corresponding to each underwater robot according to each first motion trajectory and imitation learning strategy, and generating an offline trajectory pool according to each second motion trajectory, the second motion trajectory is used to represent the motion trajectory of each underwater robot when tracking the target in a simulated underwater environment; for each underwater robot, the third motion trajectory of the underwater robot is determined according to the offline trajectory pool and the corresponding prediction model, and the prediction model is obtained by training the preset initial prediction model according to the offline trajectory pool. Through imitation learning, multiple second motion trajectories are generated according to the first motion trajectory in the simulated underwater environment, so that the second motion trajectory obtained is more consistent with the motion trajectory in the underwater environment. At the same time, the prediction model can learn the dependency relationship of each second motion trajectory from the offline trajectory pool generated by the second motion trajectory, so that when the third motion trajectory of the underwater robot is determined according to the prediction model, the accuracy can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 A schematic diagram of a flow chart of an underwater target tracking method in one embodiment;

[0031] Figure 2 A schematic diagram of an underwater simulation environment in an embodiment;

[0032] Figure 3 A schematic diagram of a flow chart of a step of generating a plurality of second motion trajectories in one embodiment;

[0033] Figure 4 It is a schematic diagram of a flow chart of a step of determining a third motion trajectory of an underwater robot in one embodiment;

[0034] Figure 5 A flowchart of a prediction model training step in one embodiment;

[0035] Figure 6 is a flow chart of an underwater target tracking method in another embodiment;

[0036] Figure 7 is a structural block diagram of an underwater target tracking device in one embodiment;

[0037] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0039] Tracking targets using autonomous underwater robots is an important research topic, especially multi-agent control.

[0040] In the existing methods, the target tracking strategy of underwater robots can be determined by the artificial potential field method. The main idea is to abstract the positive demands (such as target tracking, communication, etc.) into a gravitational field, and the negative demands (such as obstacle avoidance, etc.) into a repulsive field, thereby abstracting the movement of the intelligent body into passive movement under the field.

[0041] However, the existing technology has good performance in aspects such as path planning in traditional ideal environments, but it requires a lot of simplification of the environment and cannot cope with very complex requirements, that is, the accuracy of the underwater robot's motion trajectory determined in a complex environment is low.

[0042] In view of this, the present application provides an underwater target tracking method that can accurately determine the motion trajectory of an underwater robot, and its execution subject can be an underwater target tracking device, which can be implemented by software, hardware, or a combination of software and hardware, and can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software. In the following method embodiments, the execution subject is taken as an example to illustrate, wherein the computer device can be a server, and the embodiments of the present application do not limit the specific type of the computer device.

[0043] In an exemplary embodiment, Figure 1 As shown, a method for tracking an underwater target is provided, comprising the following steps 101 to 103. Among them:

[0044] S101, obtaining a first motion trajectory of each underwater robot.

[0045] The first motion trajectory is used to represent the motion trajectory of each underwater robot when tracking the target in a simplified underwater environment where interference is ignored.

[0046] Optionally, the underwater robot can be an autonomous underwater vehicle (AUV), which can autonomously perform various tasks underwater, such as marine environment detection, underwater target search and tracking, etc., without external physical connection, relying on its own energy (such as batteries) and pre-programmed instructions or intelligent control systems.

[0047] Optionally, in a simplified underwater environment, any underwater interference and communication factors are ignored, and the tracking target and all AUVs are regarded as particles.

[0048] In one possible implementation, the first motion trajectory of each underwater robot may be preset based on expert experience.

[0049] In another possible implementation, the first motion trajectory of each underwater robot may be obtained by acquiring the historical motion trajectory of each underwater robot when tracking the target, and taking the historical motion trajectory of the underwater robot with the best performance as the first motion trajectory.

[0050] S102, generating a plurality of second motion trajectories corresponding to each underwater robot according to each first motion trajectory and the imitation learning strategy, and generating an offline trajectory pool according to each second motion trajectory.

[0051] The second motion trajectory is used to represent the motion trajectory of each underwater robot when tracking a target in a simulated underwater environment.

[0052] Optionally, in a simulated underwater environment, underwater dynamics and communication constraints are considered, and there are effects of disturbances and sensor detection fluctuations.

[0053] It should be noted that the arrangement of obstacles and the motion trajectory of the tracking target are consistent in the simplified underwater environment and the simulated underwater environment. The only difference is that the simplified underwater environment ignores interference, while the simulated underwater environment takes various influences into consideration.

[0054] Optionally, the established underwater environment model is introduced below: In this model, the task to be solved is that N (N>1) AUVs need to track a single moving target or multiple moving targets underwater. The embodiment of the present application takes tracking a single moving target as an example.

[0055] In this model, if Figure 2 As shown in Figure 1, it includes multiple sensors, underwater robots, targets and obstacles. The sensors are placed on the seabed. All AUVs and targets are assumed to be fixed in a horizontal plane with a depth of d meters underwater. The position of the target is recorded as , and the positions of other AUVs are recorded as ,in At the same time, there are M obstacles in the scene, recorded as Under this condition, it is hoped that multiple AUVs can collaborate to track dynamic targets while strictly avoiding collisions. The target tracking process should meet the following requirements:

[0056] First, the minimum requirement is that at least one AUV can track the target, that is, the distance between at least one AUV and the target is within the effective detection distance. In order to ensure the accuracy of target position detection and the robustness of tracking, it should also be ensured that as many AUVs as possible are within the effective detection distance of the target.

[0057] Second, effective communication between AUVs is ensured. The minimum requirement is that the communication map between AUVs is interconnected, that is, any two AUVs can communicate directly through an effective communication link (meeting the distance requirements so that the signal strength meets the communication constraints), or with the help of other AUVs as intermediaries to ensure that individual AUVs in the cluster will not be "left behind".

[0058] Third, all AUVs must meet the safety constraints of avoiding collisions, that is, an appropriate safety distance must be maintained between AUVs and obstacles, between AUVs, and between AUVs and targets to prevent collisions. This requires AUVs to form dynamic formations according to the environment.

[0059] Optionally, the dynamic model in the underwater environment model is introduced below:

[0060] In the underwater environment model, all AUVs and targets move in the same plane, ignoring the depth change, with 3 degrees of freedom. For AUV No. i, the velocity is defined in the body frame. , which includes the x / y component, also called the forward velocity Surge velocity and sway velocity (swayvelocity) and angular velocity , defines the position in the world frame According to Fossen hydrodynamics formula, we have:

[0061]

[0062] in, represents the inertia matrix including the additional mass due to the hydrodynamic forces, For speed The derivative of represents the Coriolis matrix, is the damping matrix describing the viscous fluid dynamics, is the additional matrix of the vertical force due to gravity and buoyancy, is the thruster terminal control quantity, and The relationship can be generated by the following kinematic formula:

[0063]

[0064] The transformation matrix J is expressed as:

[0065]

[0066] In order to apply it in practice, the relevant formulas are now discretized, namely:

[0067]

[0068]

[0069] in, , is the discretized time interval.

[0070] Optionally, the communication model in the underwater environment model is introduced below:

[0071] The hydroacoustic equation can be used to model the communication between AUVs. Now assume that in the underwater target tracking task, the communication between the AUV and the target depends on active sonar. The AUV detects the target or obstacle according to the formula of the two-way propagation loss of active sonar, which can be expressed as follows:

[0072]

[0073] Among them, SL is the sound source level, TL is the propagation loss, TS is the target strength, NL is the noise level, DI is the receiving directivity coefficient, DT is the successful detection threshold, and all formulas are in decibels (dB). When the echo margin EM > 0dB, the target can be successfully detected. The maximum detection distance can be expressed as:

[0074]

[0075] Communication between AUVs requires that the AUVs can passively receive each other's signals. The communication process conforms to the passive sonar equation, which is as follows:

[0076]

[0077] Among them, the TL propagation loss of water is modeled. The propagation loss TL is caused by two factors: the decrease in energy flux density caused by the expansion of the sound wave front and the acoustic absorption of the ocean. The formula can be expressed as:

[0078]

[0079] Where d represents the propagation distance, k is the coefficient affected by the wavefront diffusion mode, and here k is 1.5, which is the shallow sea sound propagation taking into account the seabed sound absorption. α represents the additional absorption coefficient, which is related to the sound wave frequency f and can be calculated by the following empirical formula:

[0080]

[0081] The unit of frequency is kHz and the unit of α is dB / km.

[0082] Optionally, the process of multiple AUVs tracking dynamic targets is modeled as a Markov decision process MDP, that is, the decision made by the AUV is only related to the current state of the environment. Specifically, the MDP involves the following four elements, forming the following quaternion:

[0083]

[0084] in, is the state space, A is the behavior space, which can also be called the action space, P represents the transition probability of the global state after executing a global action under the condition of a global state, and γ is the discount factor.

[0085] It is understandable that in the prior art, when underwater target tracking is performed by reinforcement learning methods, a reward function is designed to allow an agent to gradually find the behavior that can lead to maximizing the reward under continuous trial and error, thereby completing the requirements. However, in underwater target tracking scenarios with more task requirements, for example, in the embodiments of the present application, including tracking requirements, communication maintenance requirements and safety requirements, which may be further expanded in practice, the reward function will be difficult to design, which may lead to poor performance at best, and even cause the agent to ignore certain requirements, or even training to fail to converge. Therefore, the Markov decision process in the embodiments of the present application is different from the usual MDP. This MDP does not include a reward function. The embodiments of the present application do not rely on the reward function, but are driven according to the first motion trajectory and the imitation learning strategy.

[0086] Optionally, the imitation learning strategy may be to learn underwater target tracking in a simulated underwater environment by learning a first motion trajectory.

[0087] In one possible implementation, the imitation learning strategy may be a behavior cloning method, and a strategy network may be obtained by training according to the first motion trajectory. By inputting the current environment state S into the strategy network, a corresponding behavior A may be output.

[0088] In another possible implementation, the imitation learning strategy can be an inverse reinforcement learning method, and an optimization objective function can be constructed based on the first motion trajectory. The objective function is related to the estimation of the reward function and the degree of match with the first motion trajectory. Then, the parameters of the reward function are continuously adjusted through the optimization algorithm to make the optimization objective function optimal. In this way, the reward function can be solved. After determining the reward function, multiple second motion trajectories can be generated based on the reinforcement learning algorithm.

[0089] Optionally, the second motion trajectory is stored in an offline trajectory pool, and the offline trajectory pool includes a plurality of second motion trajectories.

[0090] S103: For each underwater robot, a third motion trajectory of the underwater robot is determined according to the offline trajectory pool and a corresponding prediction model, where the prediction model is obtained by training a preset initial prediction model according to the offline trajectory pool.

[0091] Optionally, the third motion trajectory of the underwater robot may be predicted by storing the second motion trajectory generated by the interaction between each underwater robot and the simulated underwater environment in an offline trajectory pool and randomly collecting a portion of data from the offline trajectory pool.

[0092] Optionally, the third motion trajectory may be a motion trajectory generated by each underwater robot according to the second motion trajectory when interacting with the simulated underwater environment, and the underwater robot may be driven to track the target according to the third motion trajectory.

[0093] Optionally, the prediction model can be obtained by training a preset initial prediction model through an offline trajectory pool through offline learning. The prediction model and the training process of the prediction model are not limited in the embodiments of the present application.

[0094] The underwater target tracking method is as follows: by obtaining the first motion trajectory of each underwater robot, the first motion trajectory is used to represent the motion trajectory of each underwater robot when tracking the target in a simplified underwater environment ignoring interference; generating multiple second motion trajectories corresponding to each underwater robot according to each first motion trajectory and the imitation learning strategy, and generating an offline trajectory pool according to each second motion trajectory, the second motion trajectory is used to represent the motion trajectory of each underwater robot when tracking the target in a simulated underwater environment; for each underwater robot, determining the third motion trajectory of the underwater robot according to the offline trajectory pool and the corresponding prediction model, the prediction model is obtained by training the preset initial prediction model according to the offline trajectory pool. By imitating learning, multiple second motion trajectories are generated according to the first motion trajectory in a simulated underwater environment, so that the second motion trajectory obtained is more consistent with the motion trajectory in the underwater environment. At the same time, the prediction model can learn the dependency relationship between the second motion trajectories from the offline trajectory pool generated by the second motion trajectory, so that when the third motion trajectory of the underwater robot is determined according to the prediction model, the accuracy can be improved.

[0095] In an exemplary embodiment, Figure 3 As shown, optionally, generating multiple second motion trajectories corresponding to each underwater robot according to each first motion trajectory and the imitation learning strategy includes the following steps 301 to 303. Among them:

[0096] S301, obtaining a generative adversarial imitation learning model.

[0097] Among them, the generative adversarial imitation learning model includes an initial global identification network and multiple initial first policy networks. The number of the initial first policy networks is the same as the number of underwater robots, and each initial first policy network corresponds to each underwater robot one by one.

[0098] Optionally, the generative adversarial imitation learning model can learn the first motion trajectory and find the mapping from the state to the behavior from the first motion trajectory. The global discriminant network in the generative adversarial imitation learning is denoted as , whose main goal is to distinguish the first motion trajectory from the motion trajectory generated by the first strategy network. Through the training process, the first motion trajectory and the motion trajectory generated by the first strategy network are made increasingly difficult to distinguish, that is, the motion trajectory generated by the first strategy network approaches the first motion trajectory. It can be understood that the first strategy network is used to generate the motion trajectory, and the global identification network is used to distinguish whether the motion trajectory is the first motion trajectory or the motion trajectory generated by the first strategy network.

[0099] S302 , training an initial global identification network and initial first strategy networks according to each first motion trajectory to obtain a global identification network and a plurality of first strategy networks.

[0100] Optionally, for each underwater robot, the initial first strategy network corresponding to it is trained to obtain the first strategy network corresponding to each underwater robot.

[0101] Optionally, when training the initial global identification network and each initial first policy network, the initial global identification network and each initial first policy network are trained alternately. The initial global identification network can be trained several times first so that it can better learn to distinguish between expert behavior (first motion trajectory) and generator behavior (motion trajectory generated by each initial first policy network), and then each initial first policy network is trained. This alternating training process will continue for multiple rounds until the behavior generated by the generator is difficult to distinguish in the global identification network, or the preset stopping conditions such as the number of training times and the convergence of the loss function value are reached.

[0102] It should be noted that the following formulas follow the following notation convention, that is, if a certain quantity is only for a single underwater robot, the symbol is not in bold, but in bold to represent the set of a certain quantity of all underwater robots. For example, the training behavior of underwater robot number i is , and the behavior of all underwater robots is .

[0103] Optionally, the initial first policy network can generate behaviors according to the current state of the underwater simulation environment, put them into the replay pool, and then collect a trajectory batch of size B from the replay pool. , ,in, The motion trajectory generated by the initial first policy network, is the first motion trajectory, Indicates the status, Representing the behavior, the above trajectory batch is used as input and input into the initial global identification network. The initial global identification network inputs a probability value to determine whether each state and the corresponding behavior is generated by the first policy network or belongs to the first motion trajectory.

[0104] Optionally, update the weight parameters of the initial global discriminator by minimizing a loss function, The expression is as follows:

[0105]

[0106] Among them, D(x) is the output of the global discriminator. Then, the gradient of the loss function with respect to the discriminator parameters is calculated by the back-propagation algorithm, and the parameters of the initial global discriminator are updated using an optimization algorithm. The optimization algorithm may include stochastic gradient descent, Adam, gradient penalty, and spectral regularization.

[0107] Optionally, the initial first policy network can be trained based on a reinforcement learning method, and when training each initial first policy network, a trajectory batch of size B is collected from the replay pool. , calculate the reward for each trajectory and put the reward into the trajectory, that is, , among which, rewards It can be calculated by the following formula:

[0108]

[0109] Optionally, the motion trajectory includes the state space and behavior space of the underwater robot. The state space includes the position information of the tracking target and the positions of other underwater robots. The behavior space includes the forward speed, swing speed and angular velocity of the underwater robot. The motion trajectory may also include rewards and the state space of the underwater robot at the next moment.

[0110] S303: Generate multiple second motion trajectories corresponding to each underwater robot through each first strategy network.

[0111] Optionally, multiple second motion trajectories can be obtained by interacting with the underwater simulation environment through the behaviors generated by each first strategy network, and each motion trajectory is placed in an offline trajectory pool, so that an expanded offline trajectory pool can be obtained, and the scale of the offline trajectory pool can be greater than 500,000 time steps.

[0112] The above-mentioned generative adversarial imitation learning model is obtained, and the generative adversarial imitation learning model includes an initial global identification network and multiple initial first policy networks. The number of initial first policy networks is the same as the number of underwater robots, and each initial first policy network corresponds to each underwater robot one by one; according to each first motion trajectory, the initial global identification network and each initial first policy network are trained to obtain a global identification network and multiple first policy networks; multiple second motion trajectories corresponding to each underwater robot are generated through each first policy network. In this way, with the help of imitation learning, the underwater robot control strategy that can simulate the underwater environment can be obtained without relying on the design of the reward function, which greatly reduces the dependence on human prior knowledge and can effectively improve the accuracy of determining the motion trajectory of the underwater robot.

[0113] In an exemplary embodiment, Figure 4 As shown, optionally, the prediction model includes a posterior information extractor and a second strategy network based on Transformer, and the third motion trajectory of the underwater robot is determined according to the offline trajectory pool and the corresponding prediction model, including the following steps 401 to 403. Among them:

[0114] S401, obtaining a motion trajectory segment of an underwater robot within a preset time length from an offline trajectory pool.

[0115] Optionally, after the preset time length is obtained, a motion trajectory segment of the preset time length may be randomly obtained from an offline trajectory pool.

[0116] Optionally, a batch of motion trajectory segments with a size of n can be collected from the offline trajectory pool, with a time length of K, and the corresponding time steps cover from t to t+K-1. It should be noted that if t+K-1 exceeds the end of the trajectory, that is, there is no corresponding trajectory segment, the excess part is set to empty.

[0117] S402, determining a latent vector corresponding to the motion trajectory segment according to the motion trajectory segment and the posterior information extractor.

[0118] Optionally, the posterior information extractor can discover hidden features and patterns that are not easily observed directly from the motion trajectory segments, namely latent vectors. The latent vectors can help the prediction model better understand the rated intrinsic structure of the data in the motion trajectory segments, thereby improving the generalization ability of the prediction model in different data sets and scenarios.

[0119] Optionally, the acquired motion trajectory segment is flipped, and the flipped motion trajectory segment is input into a posterior information extractor, so as to determine a latent vector corresponding to the motion trajectory segment according to an output of the posterior information extractor.

[0120] Optionally, the posterior information extractor can be implemented based on a recurrent neural network architecture, or based on a convolutional neural network architecture, or based on a Transformer architecture, which is not limited in the embodiments of the present application.

[0121] S403: Input the motion trajectory segment and the latent vector into a second strategy network based on Transformer, and determine a third motion trajectory of the underwater robot according to an output of the second strategy network.

[0122] Optionally, Transformer can be a deep learning architecture, mainly composed of multi-head attention mechanism (Multi-Head Attention), positional encoding (Positional Encoding) and feed-forward neural network (Feed-Forward Neural Network). Unlike traditional recurrent neural network (RNN) or convolutional neural network (CNN), Transformer processes sequence data completely based on the attention mechanism and can effectively capture long-distance dependencies in the sequence.

[0123] Optionally, the state vector and behavior vector within the historical K time steps in the motion trajectory segment are obtained, and then the state vector, behavior vector and latent vector within the historical K time steps are input into a second policy network based on Transformer, and the behavior at the next moment is predicted by the second policy network to obtain the third motion trajectory of the underwater robot.

[0124] Optionally, the obtained third motion trajectory may also be stored in the discrete trajectory pool to continuously predict the subsequent motion trajectory of the underwater robot.

[0125] The above method obtains a motion trajectory segment of the underwater robot in a preset time length from the offline trajectory pool; determines the latent vector corresponding to the motion trajectory segment according to the motion trajectory segment and the posterior information extractor; inputs the motion trajectory segment and the latent vector into the second policy network based on Transformer, and determines the third motion trajectory of the underwater robot according to the output of the second policy network. By obtaining the latent vector corresponding to the motion trajectory segment, and inputting the motion trajectory segment and the corresponding latent vector into the pre-trained second policy network, the third motion trajectory of the underwater robot can be accurately predicted.

[0126] In an exemplary embodiment, Figure 5 As shown, optionally, the training process of the prediction model includes the following steps 501 to 505. Among them:

[0127] S501: Determine a training sample according to the motion trajectory in the offline trajectory pool.

[0128] Optionally, all the motion trajectories in the offline trajectory pool may be used as training samples, or a batch of motion trajectories may be randomly collected from the offline trajectory pool as training samples, which is not limited in this embodiment of the present application.

[0129] S502, obtaining sample trajectory segments of the underwater robot from training samples.

[0130] Among them, the sample trajectory segment is the motion trajectory of the underwater robot within a certain time length.

[0131] Optionally, according to the set time length, a batch of sample trajectory segments are randomly collected from the training samples. For example, if the set time length is k, starting from time t, the time steps corresponding to each sample trajectory in the sample trajectory segment are from t to t+K-1, or the corresponding time steps are from t-K+1 to t.

[0132] Optionally, if there is no motion trajectory corresponding to the t+K-1 time step in the training sample in the randomly collected sample trajectory fragment, the excess part can be set to an empty vector and the corresponding attention mask can be set.

[0133] S503: Determine a latent vector corresponding to the sample trajectory segment according to the sample trajectory segment and the initial posterior information extractor.

[0134] Optionally, the network parameters of the initial posterior information extractor can be , the sample trajectory segment is input into the initial posterior information extractor, and the latent vector corresponding to the sample trajectory segment is determined according to the output of the initial posterior information extractor.

[0135] S504, determining a predicted trajectory of the underwater robot according to the sample trajectory segment, the latent vector and the initial second strategy network.

[0136] Optionally, the network parameters of the initial second strategy network can be , the state vector, behavior vector and latent vector in the sample trajectory segment are input into the second strategy network, and the second strategy network can predict the behavior of the underwater robot at the next moment.

[0137] S505, determining a loss function according to the predicted trajectory and the sample trajectory corresponding to the training sample, and updating network parameters of the initial posterior information extractor and the initial second policy network based on the loss function until the posterior information extractor and the second policy network are obtained.

[0138] Optionally, the loss function can be expressed by the following formula:

[0139]

[0140] in, is the corresponding behavior vector in the sample trajectory, is the behavior vector predicted by the initial second policy network.

[0141] Optionally, the network parameters of the initial and second initial strategies can be optimized by an optimization algorithm. Update, the optimization algorithm can include stochastic gradient descent, Adam, gradient penalty and spectral regularization, etc. At the same time, the network parameters of the initial posterior information extractor are updated by gradient reversal to update.

[0142] Optionally, the above network parameter updating process, i.e., step 502 to step 505, may be iterated until a preset number of training times is reached or the loss function converges, thereby obtaining a posterior information extractor and a second strategy network.

[0143] Optionally, through offline learning, a prediction model that can cope with various scenarios such as arbitrary obstacle arrangements can be trained based on the generated extensive offline trajectory pool, and the motion trajectory of the underwater robot at the next moment can be accurately determined.

[0144] It can be understood that according to the architecture proposed in the embodiment of the present application of transferring the first motion trajectory to the underwater simulation environment to obtain the second motion trajectory, and predicting the third motion trajectory of the underwater robot at the next moment based on the second motion trajectory, the motion trajectory of the underwater robot collected in the actual task of underwater target tracking can also be realized. By introducing the reinforcement learning domain conversion technology from simulation to reality, it can be directly applied to the real scene, which has good scalability.

[0145] In an exemplary embodiment, optionally, obtaining a first motion trajectory of each underwater robot includes:

[0146] In a simplified underwater environment, a first motion trajectory of each underwater robot is generated according to a preset target tracking algorithm.

[0147] Among them, the preset target tracking algorithms include: artificial potential field method, Lyapunov vector field method, and model predictive control method.

[0148] Optionally, the Artificial Potential Field Method is a path planning method based on a virtual force field. The movement of an underwater robot (or target) in an underwater environment can be compared to movement in a virtual potential field. This potential field is composed of a gravitational field and a repulsive field. The gravitational field is used to attract the robot to move toward the target position, and the repulsive field is used to make the robot avoid obstacles. The gravitational force can be determined according to the position of the underwater robot and the target position, and the repulsive force can be determined according to the position of the underwater robot and the position of the obstacle. The gravitational force and the repulsive force can be added to determine the resultant force acting on the underwater robot. Then, the first motion trajectory of the robot is generated according to the direction of the resultant force and the kinematic model of the underwater robot through numerical integration and other methods.

[0149] Optionally, in the Lyapunov vector field method, a Lyapunov function needs to be defined. The motion trajectory of the underwater robot tends to move in a direction that reduces the value of the Lyapunov function. Therefore, starting from the initial state, the state change of the underwater robot is gradually calculated along the Lyapunov vector field to obtain the first motion trajectory.

[0150] Optionally, in each control cycle, the model predictive control method predicts the motion trajectory of the underwater robot in the future based on the current state and dynamic model of the underwater robot, and then optimizes the predicted motion trajectory according to a given objective function to obtain a first motion trajectory. The objective function may be minimum path error, minimum heading angle deviation, etc., and may be set according to the specific scenario, and the embodiments of the present application are not limited to this.

[0151] As an optional implementation, Figure 6 As shown, the underwater target tracking method provided in the embodiment of the present application may include the following specific steps:

[0152] S601, in a simplified underwater environment, generating a first motion trajectory of each underwater robot according to a preset target tracking algorithm.

[0153] Among them, the preset target tracking algorithms include: artificial potential field method, Lyapunov vector field method, and model predictive control method.

[0154] S602, obtaining a generative adversarial imitation learning model.

[0155] Among them, the generative adversarial imitation learning model includes an initial global identification network and multiple initial first policy networks. The number of the initial first policy networks is the same as the number of underwater robots, and each initial first policy network corresponds to each underwater robot one by one.

[0156] S603 , training the initial global identification network and the initial first strategy networks according to the first motion trajectories to obtain a global identification network and a plurality of first strategy networks.

[0157] S604: Generate multiple second motion trajectories corresponding to each underwater robot through each first strategy network.

[0158] S605: Generate an offline trajectory pool according to each second motion trajectory.

[0159] The second motion trajectory is used to represent the motion trajectory of each underwater robot when tracking a target in a simulated underwater environment.

[0160] S606: Determine a training sample according to the motion trajectory in the offline trajectory pool.

[0161] S607, obtaining a sample trajectory segment of each underwater robot from the training samples, where the sample trajectory segment is a motion trajectory of the underwater robot within a certain time length.

[0162] S608: For each underwater robot, determine a latent vector corresponding to the sample trajectory segment according to the corresponding sample trajectory segment and the initial posterior information extractor.

[0163] S609: For each underwater robot, determine the predicted trajectory of the underwater robot according to the corresponding sample trajectory segment, latent vector and the initial second strategy network.

[0164] S610, for each underwater robot, determine the loss function according to the predicted trajectory and the sample trajectory corresponding to the training sample, and update the network parameters of the initial posterior information extractor and the initial second policy network based on the loss function until the posterior information extractor and the second policy network are obtained.

[0165] S611: For each underwater robot, obtain a motion trajectory segment of the underwater robot within a preset time length from an offline trajectory pool.

[0166] S612: For each underwater robot, determine a latent vector corresponding to the motion trajectory segment according to the motion trajectory segment and the posterior information extractor.

[0167] S613: For each underwater robot, the motion trajectory segment and the latent vector are input into a second strategy network based on Transformer, and a third motion trajectory of the underwater robot is determined according to an output of the second strategy network.

[0168] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0169] Based on the same inventive concept, the embodiment of the present application also provides an underwater target tracking device for implementing the underwater target tracking method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more underwater target tracking device embodiments provided below can refer to the limitations of the underwater target tracking method above, and will not be repeated here.

[0170] In an exemplary embodiment, Figure 7 As shown, an underwater target tracking device 700 is provided, comprising: an acquisition module 701, a generation module 702 and a prediction module 703, wherein:

[0171] An acquisition module 701 is used to acquire a first motion trajectory of each underwater robot, where the first motion trajectory is used to represent the motion trajectory of each underwater robot when tracking a target in a simplified underwater environment where interference is ignored;

[0172] A generation module 702 is used to generate a plurality of second motion trajectories corresponding to each underwater robot according to each first motion trajectory and the imitation learning strategy, and to generate an offline trajectory pool according to each motion trajectory, wherein the second motion trajectory is used to represent the motion trajectory of each underwater robot when tracking a target in a simulated underwater environment;

[0173] The prediction module 703 is used to determine the third motion trajectory of each underwater robot according to the offline trajectory pool and the corresponding prediction model, where the prediction model is obtained by training a preset initial prediction model according to the offline trajectory pool.

[0174] In an exemplary embodiment, the generation module 702 is specifically used to obtain a generative adversarial imitation learning model, which includes an initial global identification network and multiple initial first policy networks. The number of initial first policy networks is the same as the number of underwater robots, and each initial first policy network corresponds to each underwater robot one by one; according to each first motion trajectory, the initial global identification network and each initial first policy network are trained to obtain a global identification network and multiple first policy networks; and multiple second motion trajectories corresponding to each underwater robot are generated through each first policy network.

[0175] In an exemplary embodiment, the prediction model includes a posterior information extractor and a second policy network based on Transformer, and the prediction module 703 is specifically used to obtain a motion trajectory segment of the underwater robot in a preset time length from an offline trajectory pool; determine the latent vector corresponding to the motion trajectory segment according to the motion trajectory segment and the posterior information extractor; input the motion trajectory segment and the latent vector into the second policy network based on Transformer, and determine the third motion trajectory of the underwater robot according to the output of the second policy network.

[0176] In an exemplary embodiment, the underwater target tracking device 700 also includes a training module, which is used to determine training samples based on motion trajectories in an offline trajectory pool; obtain sample trajectory segments of the underwater robot from the training samples, and the sample trajectory segments are motion trajectories of the underwater robot within a certain time length; determine the latent vector corresponding to the sample trajectory segment based on the sample trajectory segment and the initial posterior information extractor; determine the predicted trajectory of the underwater robot based on the sample trajectory segment, the latent vector and the initial second policy network; determine the loss function based on the predicted trajectory and the sample trajectory corresponding to the training sample, and update the network parameters of the initial posterior information extractor and the initial second policy network based on the loss function until the posterior information extractor and the second policy network are obtained.

[0177] In an exemplary embodiment, the acquisition module 701 is specifically used to generate the first motion trajectory of each underwater robot in a simplified underwater environment according to a preset target tracking algorithm; wherein the preset target tracking algorithm includes: artificial potential field method, Lyapunov vector field method, and model predictive control method.

[0178] In an exemplary embodiment, the motion trajectory includes the state space and behavior space of the underwater robot. The state space includes the position information of the tracking target and the positions of other underwater robots. The behavior space includes the forward speed, swing speed and angular velocity of the underwater robot.

[0179] Each module in the underwater target tracking device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0180] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for tracking underwater targets is implemented.

[0181] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0182] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps described in any of the above method embodiments when executing the computer program.

[0183] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps described in any of the above method embodiments are implemented.

[0184] In one embodiment, a computer program product is provided, including a computer program, which implements the steps described in any of the above method embodiments when executed by a processor.

[0185] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0186] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0187] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for tracking an underwater target, characterized in that: The method comprises: Acquire a first motion trajectory of each underwater robot, where the first motion trajectory is used to represent the motion trajectory of each underwater robot when tracking a target in a simplified underwater environment where interference is ignored; Generate a plurality of second motion trajectories corresponding to each of the underwater robots according to each of the first motion trajectories and the imitation learning strategy, and generate an offline trajectory pool according to each of the second motion trajectories, wherein the second motion trajectories are used to represent the motion trajectories of each of the underwater robots when tracking a target in a simulated underwater environment; For each of the underwater robots, a third motion trajectory of the underwater robot is determined according to the offline trajectory pool and a corresponding prediction model, wherein the prediction model is obtained by training a preset initial prediction model according to the offline trajectory pool.

2. The method according to claim 1, characterized in that The step of generating a plurality of second motion trajectories corresponding to each of the underwater robots according to each of the first motion trajectories and the imitation learning strategy comprises: Acquire a generative adversarial imitation learning model, wherein the generative adversarial imitation learning model includes an initial global discrimination network and multiple initial first policy networks, the number of the initial first policy networks is the same as the number of the underwater robots, and each of the initial first policy networks corresponds to each of the underwater robots one by one; According to each of the first motion trajectories, the initial global identification network and each of the initial first strategy networks are trained to obtain a global identification network and a plurality of first strategy networks; A plurality of second motion trajectories corresponding to each of the underwater robots are generated through each of the first strategy networks.

3. The method according to claim 1, characterized in that The prediction model includes a posterior information extractor and a second strategy network based on Transformer, and determining the third motion trajectory of the underwater robot according to the offline trajectory pool and the corresponding prediction model includes: Acquire a motion trajectory segment of the underwater robot within a preset time length from the offline trajectory pool; Determining a latent vector corresponding to the motion trajectory segment according to the motion trajectory segment and the posterior information extractor; The motion trajectory segment and the latent vector are input into the second Transformer-based policy network, and a third motion trajectory of the underwater robot is determined according to an output of the second policy network.

4. The method according to claim 3, characterized in that The training process of the prediction model includes: Determining a training sample according to the motion trajectory in the offline trajectory pool; Acquire a sample trajectory segment of the underwater robot from the training sample, wherein the sample trajectory segment is a motion trajectory of the underwater robot within a certain time length; Determining a latent vector corresponding to the sample trajectory segment according to the sample trajectory segment and an initial posterior information extractor; Determining a predicted trajectory of the underwater robot according to the sample trajectory segment, the latent vector and an initial second strategy network; A loss function is determined according to the predicted trajectory and the sample trajectory corresponding to the training sample, and network parameters of the initial posterior information extractor and the initial second policy network are updated based on the loss function until the posterior information extractor and the second policy network are obtained.

5. The method according to claim 1, characterized in that The obtaining of the first motion trajectory of each underwater robot includes: In the simplified underwater environment, generating a first motion trajectory of each of the underwater robots according to a preset target tracking algorithm; Among them, the preset target tracking algorithms include: artificial potential field method, Lyapunov vector field method, and model predictive control method.

6. The method according to claim 1, characterized in that The motion trajectory includes the state space and behavior space of the underwater robot. The state space includes the position information of the tracking target and the positions of other underwater robots. The behavior space includes the forward speed, swing speed and angular velocity of the underwater robot.

7. An underwater target tracking device, characterized in that: The device comprises: An acquisition module, used for acquiring a first motion trajectory of each underwater robot, wherein the first motion trajectory is used for representing a motion trajectory of each underwater robot when tracking a target in a simplified underwater environment where interference is ignored; A generation module, used to generate a plurality of second motion trajectories corresponding to each of the underwater robots according to each of the first motion trajectories and an imitation learning strategy, and to generate an offline trajectory pool according to each of the motion trajectories, wherein the second motion trajectory is used to represent the motion trajectory of each of the underwater robots when tracking a target in a simulated underwater environment; A prediction module is used to determine, for each of the underwater robots, a third motion trajectory of the underwater robot according to the offline trajectory pool and a corresponding prediction model, wherein the prediction model is obtained by training a preset initial prediction model according to the offline trajectory pool.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images