Human-machine integrated ship control method based on shared panel

Abstract

A human-machine integrated ship control method based on a shared panel, the method comprising the following steps: our own ships in a fleet collecting multi-source sensing data by means of a multi-source sensor; fusing the multi-source sensing data to obtain fused data, and calculating a future track of an enemy target ship; sharing encrypted fused data to our own fleet, and synchronously receiving encrypted fused data of other shared panels; after receiving the encrypted fused data, our own ship determining a planned route and an intention of our own ship, and sharing the planned route and the intention; and a human commander changing a track or a combat objective of the own ship, and the rest of our own ships re-optimizing planned paths and then sharing same, so as to complete real-time human-machine integrated ship control. A multi-machine interaction mode based on shared panels enhances the cognitive capability and coordination of marine fleets on law enforcement and rights protection missions.

Description

A Human-Machine Integration Ship Control Method Based on Shared Panels Technical Field

[0001] This invention belongs to the field of human-computer interaction and information collaboration technology, specifically relating to a human-computer integrated ship control method based on a shared panel. Background Technology

[0002] In maritime law enforcement and rights protection missions, facing massive amounts of data and complex maneuvering plans, fleets require accurate and consistent full-range perception and display, real-time and easy-to-use course mapping, and collaborative global mission awareness with low cognitive load. Therefore, leveraging systematic and intelligent technologies to enhance the cognitive capabilities and coordination of maritime law enforcement and rights protection fleets has significant application value.

[0003] In existing maritime law enforcement and rights protection missions, methods based on single-entity perception cannot achieve large-scale global cognition, resulting in insufficient patrol and adaptability capabilities and low efficiency. Multi-entity perception lacks effective means of consistency and fusion, failing to quickly obtain accurate and reliable unified cognition results from complex and massive amounts of perception data. Information is isolated among fleet members, with poor sharing capabilities. The fleet cannot share the planned paths and intentions of other members in real time, and the coordination and collaboration for the same mission cannot be guaranteed. Human-machine separation in path planning and control leads to poor interactivity and is prone to path deviation in complex situations. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide a human-machine integrated ship control method based on a shared panel.

[0005] Technical Solution: The human-machine integrated ship control method based on a shared panel described in this invention, wherein the friendly fleet includes multiple friendly ships, each equipped with a shared panel, specifically includes the following steps:

[0006] Step 1: The friendly ships in the fleet use multi-source sensors to collect multi-source sensor data, which includes information about friendly warships, information about enemy target ships, and information about the surrounding environment.

[0007] Step 2: The multi-source sensor data collected from different friendly warships are fused to obtain fused data, and the future trajectory of the enemy target ship is calculated through a temporal neural network f. After obtaining the future trajectory of the enemy target ship, the tactical mode of the enemy target ship when it is on that trajectory is determined.

[0008] Step 3: Encrypt and compress the fused data to obtain encrypted fused data, and share the encrypted fused data with other shared panels in your own fleet, and simultaneously receive encrypted fused data from other shared panels;

[0009] Step 4: After receiving the encrypted fusion data, the friendly ship decrypts it to obtain decrypted fusion data. Using the decrypted fusion data and the mission objective, the friendly ship determines its planned route and intention, and shares the friendly ship's planned route and intention through the sharing panel.

[0010] Step 5: Human commanders change the course or combat objective of their own ships by hand-drawing a course map on the shared panel. Other friendly ships re-optimize the planned path by sharing the ship's intentions, target ship's intentions, and mission objectives on the shared panel, thus completing the real-time control of ships by human and machine integration.

[0011] Furthermore, the information of the friendly warships includes their position, course, and speed, while the information of the enemy target ships includes target ship detection, attribute identification, target ship tracking, target ship positioning, speed, acceleration, and direction. The target ship detection, attribute identification, and target ship tracking are achieved through cameras, while the surrounding environment information, target ship positioning, speed, acceleration, and direction are achieved through radar.

[0012] Furthermore, target ship detection includes the following steps:

[0013] Step 21: Collect various ship image data using cameras, divide them into training and test sets, and train them using transfer learning; Step 22: Use deep learning target detection algorithms to identify the model of the target ship, assign a unique identifier to the detected target ship, and retrieve the information of the corresponding ship model from the ship model database and display it on the shared panel.

[0014] The specific detection process is represented by the following formula: (P i1 ,P i2 ,...,P ij ) = g(network output) (x i ,y i ,w i ,h i = f(network output)

[0015] In the formula, P i,j Let x represent the probability that the i-th bounding box belongs to the j-th type of ship. i ,y i ,w i ,h i These are the coordinates of the ship's center point and its length and width, respectively. g and f are the decoding heads of the neural network. A unique identifier is assigned to the detected target, and information such as initial position and velocity is obtained. An extended Kalman filter is used to track the target ship's trajectory and output it to the shared panel.

[0016] Furthermore, target ship tracking is expressed by the following formula:

[0017] In the formula, It is the best estimate of the target at the current moment. It is an estimate of the target at the current moment, Z k These are observed values. It is the observation function, K k It is the Kalman gain.

[0018] Furthermore, in step 2, the multi-source sensor data from different friendly warships are fused using the DS evidence theory to obtain fused data, which is expressed by the following formula:

[0019] In the formula, m 12…n (A) represents the probability of merging n friendly ships, m n Let A be the final fusion result of our own ship, and B be the basic probability corresponding to the result of our nth friendly ship as evidence element. n It is the set of events supported by the sensor observations of the nth friendly ship.

[0020] Furthermore, in step two, the data fusion includes information on all friendly ships, their tracks and intentions, information on enemy target ships and their tracks, and information on the surrounding environment.

[0021] Furthermore, the future trajectory Y of the enemy target ship is represented by the following formula: Y=f(X,θ)

[0022] In the formula, Y is the future trajectory of the target ship, and X = [x1, x2, ..., x n ] represents the feature vector of the target ship, f is a temporal neural network, and θ is the network parameter.

[0023] The input consists of the features of the target ship, including position, speed, acceleration, model, and historical data mining patterns, forming a feature vector X = [x1, x2, ..., x]. n The output is the target ship's estimated position Y over a future period. After obtaining the predicted trajectory of the enemy ship, an expert system based on historical big data is established to determine the tactical mode (attack, retreat, breakout, decoy, reconnaissance, etc.) when the target ship is on that trajectory.

[0024] Furthermore, the planned route for our own ships in step three is expressed by the following formula:

[0025] in, Let θ be the gradient of the policy. s represents the expected trajectory of the sampled data. t Let a be the state at time t. t Let θ be the action at time t, where t represents time, θ is the policy parameter, π is the policy function, and A is the advantage function.

[0026] A reinforcement learning PPO algorithm model is constructed. The shared panel situation is gridded based on the current situational state. Strategies are formulated using the situational information on the shared panel, the target's intentions, and mission objectives. The gradient ascent method is used for optimization, and the calculated planning steps serve as the ship's tactical path. The situational information on the shared panel includes timestamps; the status, path planning information, and intentions of all friendly ships; the information and intentions of all ships in the target fleet; the overall maritime operational environment; and the overall friendly mission information.

[0027] Furthermore, in step five, the hand-drawn trajectory map is recognized online, the points and speeds when the trajectory is drawn are recorded, and the type of the line element is determined. The types of line elements include straight line elements, curved line elements, and circular arc elements. Straight line elements, curved line elements, and circular arc elements are respectively fitted to straight lines, polylines, and circular arcs through the points on the line element.

[0028] The hand-drawn flight path recognition uses an online recognition method, which records the position and speed when the flight path is drawn, and performs line element identification to determine whether the line segment is a straight line, curve or arc. After determining the category of the line element, the points on the line element need to be fitted with straight lines, polylines or arcs.

[0029] Furthermore, the straight line fitting uses the least squares method, expressed by the following formula:

[0030] In the formula, σ is the error, and x i and y i Let be the i-th coordinate point on the hand-drawn track map, n be the number of coordinate points on the hand-drawn track map, and A and B be the parameters of the straight line fitting equation.

[0031] Porous line fitting is performed by calculating the coordinates of the intersection points, as expressed by the following formula:

[0032] In the formula, x and y are the coordinates of the intersection of the broken lines, a1, b1 and c1 are the parameters of the first line equation, and a2, b2 and c2 are the parameters of the second line equation.

[0033] Circular arc fitting uses the least squares method to fit the arc, expressed by the following formula:

[0034] In the formula, σ² is the error, and x j and y j Let R be the j-th coordinate point on the hand-drawn flight path map, R be the radius of the arc, and X and Y be the coordinates of the center point of the arc.

[0035] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: The human-machine fusion ship control method based on shared panels of the present invention adopts multi-source information fusion theory and interaction technology based on shared panels, and at the same time solves the problems of incomplete information collection by a single ship and poor accuracy of cognitive results under complex multi-source information, as well as the lack of information sharing and poor interactivity and coordination among ships in the fleet to perform missions. Attached Figure Description

[0036] Figure 1 is a flowchart illustrating the human-machine integration ship control method based on a shared panel;

[0037] Figure 2 shows the display effect of the shared panel;

[0038] Figure 3 is a schematic diagram of the interaction and collaboration logic between human commanders and shared panels, ships, and fleets. Detailed Implementation

[0039] Example 1

[0040] As shown in Figure 1, the human-machine integrated ship control method based on a shared panel of the present invention includes the following steps:

[0041] Step S1: Collect sensor data from each ship in the fleet, including using GPS to locate the position of friendly ships; using an inertial measurement unit (IMU) to calculate the speed, acceleration, and direction of friendly ships; using camera data for target ship detection, attribute identification, and tracking; using radar data to detect the surrounding marine environment and the location, speed, acceleration, and direction of target ships; and using wind sensors and weather station information to detect changes in the marine environment.

[0042] Multi-source sensor data is received in real time, and the camera and radar information collected by an individual includes different perspectives of the same sensing area from other ships.

[0043] Image data of target ships is collected using cameras, divided into training and testing sets, and trained using transfer learning. A deep learning target detection algorithm is used to identify the models of common ships at sea, and these models are linked to a ship model database. The attributes, equipment, and other information of the corresponding ship models are then displayed on a shared panel. i,j Let x represent the probability that the i-th bounding box belongs to the j-th type of ship. i ,y i ,w i ,h i Here, are the coordinates of the ship's center point and its length and width, respectively. g and f are the decoding heads of the neural network. This detection process can be represented as: (P i1 ,P i2 ,...,P ij ) = g(network output) (x i ,y i ,wi ,h i = f(network output)

[0044] A unique identifier is assigned to the detected target, initial position and velocity information are obtained, and an extended Kalman filter is used to track the target ship's trajectory and output it to the shared panel. Let... It is the best estimate of the target at the current moment. It is an estimate of the target at the current moment, Z k These are observed values. It is the observation function, K k It is the Kalman gain, and the tracking formula is:

[0045] Step S2: After obtaining the perception results of a single ship, it is necessary to fuse the perception results of different ships targeting the same target. For basic information such as the position, speed, and type of friendly ships, the results of that ship are used as the basis. For the perception information of the target ship, because the data collected from different ships' cameras, positions, speeds, and types will differ, the DS evidence theory is used to fuse uncertain information from different sources to provide a decision-level fusion result. Assume there are n friendly ship results as evidence elements, with corresponding basic probability allocations of m. n Then the probability distribution formula after merging is:

[0046] Multi-source sensor data processing utilizes intelligent perception algorithms to identify target ships' position, type, heading, speed, acceleration, and other information. The processing incorporates both current-moment perception information and prior information from historical moments. Multi-source information fusion algorithms are used to align perception information from multiple ships targeting the same object, eliminating feature ambiguity and ensuring consistent fleet perception of the same target within the same coordinate system at the same time.

[0047] Step S3: The data is encrypted and compressed using RSA technology, and the data stream is compressed in near-lossless real-time using pulse coding. Data sharing and synchronous reception are achieved through a 5G network and edge cloud technology. The perception results after multi-source sensor fusion are displayed on a shared panel, including information on all friendly ships, their tracks and intentions, information on detectable enemy ships and their tracks, and overall environmental information. A temporal neural network f is used to calculate the future track of each enemy ship. The input is the characteristics of the target ship, including position, speed, acceleration, model, and historical data mining patterns forming a feature vector X = [x1, x2, ..., x...]. n The output is the target ship's estimated position Y over a future period, which can be represented as: Y = f(X, θ), where θ are the network parameters:

[0048] After obtaining predictions about the enemy ship's trajectory, an expert system based on historical big data is established to determine the tactical mode (attack, retreat, breakout, decoy, reconnaissance, etc.) of the target ship on that trajectory. Commanders or intelligent algorithms can then formulate our own strategies based on the target's trajectory and tactics.

[0049] The shared panel display effect is shown in Figure 2, which includes functions such as analysis of sea surface weather; analysis of the target ship's attributes, status, track, prediction and intent; the attributes, status, track, prediction and intent of friendly ships; overall mission objectives; view switching; and hand-drawn path conversion display.

[0050] Step S4: Based on the situational information obtained in the previous step and combined with the mission objective, the human commander can manually plan the ship's future trajectory and tactical intentions on a shared panel to facilitate tactical coordination with other ships. If no specific instructions are issued, a reinforcement learning PPO algorithm model can be constructed. The panel is rasterized according to the current situational state, and strategies are formulated using the situational information, the target's intentions, and the mission objective. The gradient ascent method is used for optimization, and the calculated planning steps serve as the ship's tactical path.

[0051] Figure 3 illustrates the interactive and collaborative logic between human commanders, the shared control panel, ships, and the fleet. Commanders can control the navigation status of their own ships through the local interface on the shared control panel, automatically controlling the navigation by drawing a route map or selecting mission objectives. During the control process, this information is uploaded to the shared control panel in real time. Other ships, upon receiving this information, adjust their strategies accordingly, choosing between automatic and manual adjustment modes.

[0052] Step S5: After the initial planning, the commander can modify the ship's course by hand, sharing the route planning or modified drawing data recorded on the shared panel, distinguishing them by different colors based on the ship's origin. Once the commander changes a ship's course, all other ships can obtain this information through the shared panel.

[0053] The hand-drawn flight path recognition uses an online method, recording the points and speeds at the time of drawing the path. It then determines whether a line segment is a straight line, curve, or arc by analyzing line elements. After classifying the line element, it performs line fitting, polyline fitting, or arc fitting on the points on that line element. If the line element is a straight line, the least squares method is used to fit the straight line equation, as shown in the following formula, where σ is the error and x... i and y i Here are the hand-drawn coordinate points, and A and B are the parameters of the line fitting equation:

[0054] The broken line is fitted by calculating the coordinates of the intersection point. The calculation method is to solve the coordinates of the intersection point by solving the equations of two straight lines simultaneously, and then correct the hand-drawn track results.

[0055] When the curvature of a straight line is greater than a threshold, the least squares method is used to fit the circular arc, as shown in the following expression, where x i and y i Let X be a point on the arc, R be the radius of the arc, and X and Y be the coordinates of the center point of the arc.

[0056] Step S6: Periodically, the shared panel will synchronize information and check for any modified ship tracks or mission information. If the information has not been modified, the paths of all ships will be updated again.

[0057] When the information is modified, if the manually controlled ships do not change their tracks, the other automatically controlled ships will output coordinated actions using reinforcement learning. During the update, the input is updated by adding the current time and future tracks of the other ships, as well as the previously planned track of the current ship, compared to the initial determined track. This allows the automatically controlled ships to select their next output action based on the formula: a=arg maxπ(a|s)←

[0058] Where a is the action, π is the policy function, and A is the action space.

[0059] The autonomous ship path is updated. To avoid conflicts between the algorithm-calculated path and the commander's planned path, the commander's command has the highest priority. The other autonomously controlled ships will automatically coordinate. The human commander can switch control modes.

[0060] Step S7: After calculating the planned path for the subsequent missions of the local ship, upload it again to the shared panel so that other ships can coordinate or modify the planned route, thus realizing the function of iterative update.

Claims

1. A human-machine fusion ship control method based on a shared panel, a friendly fleet comprising a plurality of friendly ships, each of which is provided with a shared panel, characterized in that: The method comprises the following steps: Step one, the multi-source sensor of the fleet is used to collect multi-source sensing data, which includes information of the own warship, information of the enemy target ship and surrounding environment information; Step two, the multi-source sensing data collected by different own warships is fused to obtain fusion data, and the future track of the enemy target ship is calculated through a time sequence neural network f, and the tactical mode of the enemy target ship on the track is determined; Step three, the fusion data is encrypted and compressed to obtain encrypted fusion data, and the encrypted fusion data is shared with other sharing panels in the own fleet, and the encrypted fusion data of other sharing panels is synchronously received; Step four, the encrypted fusion data is decrypted to obtain decrypted fusion data after being received by the own warship, and the planning route and intention of the own warship are determined by using the decrypted fusion data and the task target, and the planning route and intention of the own warship are shared through the sharing panel; Step five, the human commander changes the track or combat target of the own warship through hand-drawing track chart on the sharing panel, and the rest of the own warships reoptimize the planning path through the own warship intention, the enemy target ship intention and the task target of the sharing panel, and then share, to complete the real-time control of the man-machine fusion warship.

2. The man-machine fusion ship control method based on a shared panel according to claim 1, characterized in that: The information of the own warship includes the position, heading and speed of the own warship, the information of the enemy target ship includes the detection, attribute identification, tracking, positioning, speed, acceleration and direction of the enemy target ship, the detection, attribute identification and tracking of the enemy target ship are realized through a camera, and the surrounding environment information, the positioning, speed, acceleration and direction of the enemy target ship are realized through a radar.

3. The man-machine fusion ship control method based on a shared panel according to claim 2, characterized in that: The detection of the enemy target ship comprises the following steps: Step 21, various ship image data are collected by using a camera, are divided into a training set and a test set, and are trained by using transfer learning; Step 22, a deep learning target detection algorithm is used to identify the model of the enemy target ship, a unique identifier is assigned to the detected enemy target ship, and the information of the corresponding model ship is called from a ship model database and displayed to the sharing panel.

4. The man-machine fusion ship control method based on a shared panel according to claim 2, characterized in that: The enemy target ship tracking is represented by the following equation: In the formulae, is the best estimate of the target's current time, is the estimate of the target current time, Z k is the observation, is the observation function, K k is the Kalman gain.

5. The man-machine fusion ship control method based on a shared panel according to claim 1, characterized in that: In step 2, the multi-source sensor data of different enemy warships is fused by using D-S evidence theory to obtain fused data, which is expressed by the following formula: where m 12…n (A) is the probability of the fusion result of n own ships, m n is the basic probability of the result of the nth own ship as evidence element, A is the final fusion result of the own warship, B n is the event set supported by the sensor observation result of the nth own ship.

6. The man-machine fusion ship control method based on a shared panel according to claim 1, characterized in that: The fusion data in step two includes the information, track and intention of all own warships, the information and track of the enemy target ship, and the surrounding environment information.

7. The man-machine fusion ship control method based on a shared panel according to claim 6, characterized in that: The future track Y of the enemy target ship is represented by the following formula: Y=f(X, θ) In the formula, Y is the future track of the enemy target ship, X = [x1, x2, …, x n ] is the feature vector of the enemy target ship, f is the time series neural network, and θ is the network parameter.

8. The man-machine fusion ship control method based on a shared panel according to claim 1, characterized in that: The planned route of the own ship in step three is expressed by the following equation: wherein the gradient of the strategy θ, s t a t a a 9. The man-machine fusion ship control method based on a shared panel according to claim 1, characterized in that: In step five, the hand-drawing track chart is online identified, the point position and speed during drawing the track are recorded, the type of the line element is judged, the type of the line element includes straight line element, curved line element and circular arc element, and the straight line fitting, polyline fitting and circular arc fitting are respectively performed on the points on the line element.

10. The man-machine fusion ship control method based on a shared panel according to claim 9, characterized in that: The straight line fitting is performed using the least square method, and is expressed by the following equation: where σ1is the error, x i and y i are the horizontal and vertical coordinates of the i-th coordinate point on the hand-drawn track chart, n is the number of coordinate points on the hand-drawn track chart, and A and B are the parameters of the straight line fitting equation. The broken line fitting is fitted by calculating the intersection point coordinates, and is expressed by the following equation: In the formula, (x, y) is the coordinate of the intersection point of the polyline, a1, b1 and c1 are parameters of the first straight line equation, and a2, b2 and c2 are parameters of the second straight line equation. The circle arc fitting uses a least square method to fit the circle arc, which is expressed by the following formula: where σ2is the error, x j and y j are the horizontal and vertical coordinates of the jth coordinate point on the hand drawn track plot, R is the radius of the circular arc, and X and Y are the coordinates of the center point of the circular arc.

Images