A human-machine fusion ship control method based on shared panel

Abstract

The present invention discloses a human-machine fusion ship control method based on a shared panel, comprising the following steps: one's own ship uses multi-source sensors to collect multi-source sensor data; the multi-source sensor data is fused to obtain fused data, and the future track of the enemy target ship is calculated; the encrypted fused data is shared with the one's own fleet, and the encrypted fused data from other shared panels is synchronously received; after receiving the encrypted fused data, the one's own ship determines the planned route and intention of the one's own ship, and shares the planned route and intention; the human commander changes the track or combat target of the one's own ship, and the other one's own ships re-optimize the planned path and share it, completing the real-time control of the human-machine fusion ship. The present invention utilizes the theory of multi-source data fusion to fuse the information of one's own ship. Based on the multi-machine interaction mode of the shared panel, an iterative optimization strategy guided by mission objectives is proposed, which enhances the cognitive ability and coordination of the maritime fleet in the process of law enforcement, rights protection, etc.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-computer interaction and information collaboration, and in particular relates to a human-computer fusion ship control method based on a shared panel. Background Art

[0002] In maritime law enforcement and rights protection missions, facing massive amounts of data and complex maneuvering schemes, fleets require accurate and consistent full-range perception and display, real-time and convenient course plotting, and collaborative global mission awareness with low cognitive load. Therefore, utilizing systematic and intelligent technologies to enhance the cognitive capabilities and collaboration of maritime law enforcement and rights protection fleets has significant application value.

[0003] In existing maritime law enforcement and rights protection tasks, means based on single-subject perception cannot achieve large-scale global cognition, patrol and adaptability are insufficient, and efficiency is low; multi-subject perception lacks effective consistency fusion means, and cannot quickly obtain accurate and reliable unified cognitive results from complex and massive perception data; information between individuals in the fleet is isolated and sharing is poor, and the fleet cannot share the planned paths and intentions of other individuals in real time, and coordination and collaboration for the same task cannot be guaranteed; human-machine separation in path planning and control, poor interactivity, and path deviation is prone to occur under complex situations. Summary of the Invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a human-machine fusion ship control method based on a shared panel.

[0005] Technical solution: The human-machine fusion ship control method based on a shared panel described in the present invention comprises a fleet comprising multiple ships, each of which is provided with a shared panel, and specifically comprises the following steps:

[0006] Step 1: The ships in the fleet use multi-source sensors to collect multi-source sensor data, where the multi-source sensor data includes information about the ships, information about the enemy target ships, and information about the surrounding environment.

[0007] Step 2: Fusing the multi-source sensor data collected by different friendly warships to obtain fused data, and calculating the future track of the enemy target ship through the time series neural network f. After obtaining the future track of the enemy target ship, the tactical mode of the enemy target ship when it is on the track is determined;

[0008] Step 3: Encrypt and compress the fused data to obtain encrypted fused data, share the encrypted fused data with other shared panels in the own fleet, and synchronously receive the encrypted fused data from other shared panels;

[0009] Step 4: After receiving the encrypted fusion data, the own ship decrypts the data to obtain decrypted fusion data, uses the decrypted fusion data and the mission objective to determine the planned route and intention of the own ship, and shares the planned route and intention of the own ship through the sharing panel;

[0010] Step 5: The human commander changes the track or combat target of his own ship through a hand-drawn track chart on the shared panel. The other friendly ships re-optimize the planned path based on their own ship intention, target ship intention, and mission objectives on the shared panel and share it, completing real-time human-machine fusion ship control.

[0011] Furthermore, the information of the own warship includes the position, heading and speed of the own ship, and the information of the enemy target ship includes target ship detection, attribute identification, target ship tracking, target ship positioning, speed, acceleration and direction. The target ship detection, attribute identification and target ship tracking are achieved through cameras, and the surrounding environment information, target ship positioning, speed, acceleration and direction are achieved through radar.

[0012] Furthermore, target ship detection includes the following steps:

[0013] Step 21: Use the camera to collect image data of various ships, divide it into training sets and test sets, and use transfer learning for training; Step 22: Use the deep learning target detection algorithm to identify the model of the target ship, assign a unique identifier to the detected target ship, and retrieve the information of the corresponding model ship from the ship model database and display it on the shared panel.

[0014] The specific detection process is expressed by the following formula:

[0015] (P i1 ,P i2 ,...,P ij ) = g(network output)

[0016] (x i ,y i ,w i ,h i )=f(network output)

[0017] Where, P i,j represents the probability that the i-th bounding box belongs to the j-th type of ship, x i ,y i ,w i ,h i Where g and f are the center coordinates and length and width of the ship, respectively. g and f are the decoding heads of the neural network. A unique identifier is assigned to the detected target, and information such as its initial position and velocity is obtained. The extended Kalman filter is used to track the target ship and output it to the shared panel.

[0018] Furthermore, target ship tracking is expressed as follows:

[0019]

[0020] Where, is the best estimate of the target at the current moment, is the current estimate of the target, z k is the observed value, is the observation function, K k is the Kalman gain.

[0021] Furthermore, in step 2, the DS evidence theory is used to fuse the multi-source sensor data of different friendly warships to obtain the fused data, which is expressed by the following formula:

[0022]

[0023] Where m 12…n (A) is the probability of merging n friendly ships, m n is the basic probability corresponding to the result of the nth own ship as the evidence element, A is the final fusion result of the own warship, B n It is the set of events supported by the sensor observation results of the nth own ship.

[0024] Furthermore, the fused data in step 2 includes the information, tracks and intentions of all friendly ships, the information and tracks of enemy target ships, and the surrounding environment information.

[0025] Furthermore, the future track Y of the enemy target ship is expressed by the following formula:

[0026] Y=f(X,θ)

[0027] Where Y is the future track of the target ship, X=[x1,x2,...,x n ] is the feature vector of the target ship, f is the temporal neural network, and θ is the network parameter.

[0028] The input is the characteristics of the target ship, including position, speed, acceleration, model, and historical data mining rules to form a feature vector X = [x1, x2, ..., x n ], and the output is the estimated position information Y of the target ship in the future. After obtaining the trajectory prediction of the enemy ship, an expert system based on historical big data is established to determine the tactical mode (attack, retreat, breakout, decoy, reconnaissance, etc.) of the target ship when it is on that trajectory.

[0029] Furthermore, the planned route of our ship in step 3 is expressed as follows:

[0030]

[0031] in, is the gradient of policy θ, represents the expected trajectory of the sample, s t is the state at time t, a t is the action at time t, subscript t represents time, θ is the policy parameter, π is the policy function, and A is the advantage function.

[0032] A reinforcement learning PPO algorithm model is constructed. The shared panel is rasterized based on its current state. A strategy is formulated using the shared panel's situational information, the target party's intentions, and the mission objectives. This strategy is optimized using a gradient ascent method, and the calculated planning steps serve as the ship's tactical path. The shared panel's situational information includes timestamps; the state, path planning, and intentions of all our ships; information on the target fleet's ships and their intentions; the overall maritime combat environment; and our overall mission information.

[0033] Furthermore, the hand-drawn track map in step five is recognized online, the position and speed when drawing the track are recorded, and the type of the line element is determined. The types of line elements include straight line elements, curved line elements and circular arc line elements. Straight line elements, curved line elements and circular arc line elements are fitted by straight lines, broken lines and circular arcs respectively through the points on the line elements.

[0034] Hand-drawn track map recognition uses an online recognition method to record the point position and speed when drawing the track, and perform line element judgment on whether the line segment is a straight line, curve or arc. After judging the type of the line element, the points on the line element need to be fitted with a straight line, a broken line or an arc.

[0035] Furthermore, the linear fitting is performed using the least squares method, which is expressed as follows:

[0036]

[0037] Where σ is the error, x i and y i is the i-th coordinate point on the hand-drawn track chart, n is the number of coordinate points on the hand-drawn track chart, A and B are the parameters of the straight line fitting equation;

[0038] The broken line fitting is performed by calculating the intersection coordinates, which can be expressed as follows:

[0039]

[0040] Where x and y are the coordinates of the intersection of the broken lines, a1, b1, and c1 are the parameters of the first straight line equation, and a2, b2, and c2 are the parameters of the second straight line equation;

[0041] Arc fitting uses the least squares method to fit the arc, which is expressed by the following formula:

[0042]

[0043] Where σ2 is the error, x j and y j is the jth coordinate point on the hand-drawn track chart, R is the radius of the arc, and X and Y are the coordinates of the center point of the arc.

[0044] Beneficial effects: Compared with the existing technology, the present invention has the following significant advantages: The human-machine fusion ship control method based on the shared panel of the present invention adopts the multi-source information fusion theory and the interactive technology based on the shared panel, and at the same time solves the problems of incomplete information collection of a single ship and poor accuracy of cognitive results under complex multi-source information, as well as the problems of no information sharing between ships in the fleet performing tasks, poor interactivity and coordination. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 The figure is a flow chart of a human-machine fusion ship control method based on a shared panel;

[0046] Figure 2 Display effect diagram for shared panel;

[0047] Figure 3 Schematic diagram of the interaction and collaboration between human commanders and shared panels, ships, and fleets. DETAILED DESCRIPTION

[0048] Example 1

[0049] like Figure 1 As shown, the human-machine fusion ship control method based on the shared panel of the present invention includes the following steps:

[0050] Step S1: Collect sensor data from each ship in the fleet, using the Global Positioning System (GPS) to locate the position of the own ship; the Inertial Measurement Unit (IMU) to calculate the speed, acceleration, and direction of the own ship; camera data is used for target ship detection, attribute recognition, and tracking; radar data is used to detect the surrounding environment at sea and the positioning, speed, acceleration, and direction of the target ship; wind sensors and weather station information are used to detect changes in the marine environment.

[0051] Multi-source sensor data is received in real time, and the camera and radar information collected by individuals contains different perspectives of the same perception area of ​​other ships.

[0052] The camera is used to collect target ship image data, which is divided into training and test sets. Transfer learning is used for training, and a deep learning target detection algorithm is used to identify the models of common ships at sea. The model is then linked to the ship model database, and the attributes, equipment, and other information of the corresponding ship model are displayed on the shared panel. i,jrepresents the probability that the i-th bounding box belongs to the j-th type of ship, x i ,y i ,w i ,h i are the center coordinates and length and width of the ship respectively, g and f are the decoding heads of the neural network, and the detection process can be expressed as:

[0053] (P i1 ,P i2 ,...,P ij ) = g(network output)

[0054] (x i ,y i ,w i ,h i )=f(network output)

[0055] Assign a unique identifier to the detected target, obtain the initial position and speed information, and use the extended Kalman filter to track the target ship and output it to the shared panel. is the best estimate of the target at the current moment, is the current estimate of the target, Z k is the observed value, is the observation function, K k is the Kalman gain, and the tracking formula is:

[0056]

[0057] Step S2: After obtaining the perception results of a single ship, it is necessary to fuse the perception results of different ships for the same target. For the basic information of the own ship, such as the position, speed, and model, the results of the own ship shall prevail. For the perception information of the target ship, since the camera, position, speed, model and other data collected from different ships may be different, the DS evidence theory is used to fuse the uncertainty information from different sources to provide a decision-level fusion result. Assuming that there are n own ship results as evidence elements, the corresponding basic probability distribution is m n , then the probability distribution formula after merging is:

[0058]

[0059] Multi-source sensor data processing utilizes intelligent perception algorithms to identify target ship location, type, heading, speed, acceleration, and other information. This processing incorporates both current perception information and prior information from previous moments. A multi-source information fusion algorithm aligns the perception information of the same target from multiple ships, eliminating feature ambiguity and ensuring consistent fleet perception of the same target in the same coordinate system at the same time.

[0060] Step S3: Encrypt and compress the data using RSA technology, and use pulse coding methods to perform near-lossless real-time compression of the data stream. Data sharing and synchronous reception are achieved through ocean-going 5G networks and edge cloud technology. The perception results after multi-source sensor fusion are displayed on the shared panel: including the information, tracks and intentions of all friendly ships, the information and tracks of enemy detectable ships, and overall environmental information. The future track of each enemy ship is calculated through the time series neural network f. The input is the characteristics of the target ship, including position, speed, acceleration, model, and historical data mining rules to form a feature vector X = [x1, x2, ..., x n ], the output is the target ship position estimation information Y in the future period, which can be expressed as, where θ is the network parameter:

[0061] Y=f(X,θ)

[0062] After obtaining a prediction of the enemy ship's trajectory, an expert system based on historical big data is established to determine the target ship's tactical mode (attack, retreat, breakout, decoy, reconnaissance, etc.) when it is on that trajectory. Commanders or intelligent algorithms can then formulate their own strategy based on the target's trajectory and tactics.

[0063] The sharing panel display effect is as follows Figure 2 As shown, it includes analysis of sea level weather; analysis of the target ship's attributes, status, track, prediction and intention; analysis of the own ship's attributes, status, track, prediction and intention; overall mission objectives; perspective switching; hand-drawn path conversion display and other functions.

[0064] Step S4: Using the situational information obtained in the previous step and combined with the mission objectives, the human commander can hand-draw the future trajectory and tactical intent of their own ship on the shared panel, enabling tactical coordination with other ships. If no specific instructions are issued, a reinforcement learning PPO algorithm model can be constructed. The panel is rasterized based on the current situation state of the shared panel. A strategy is formulated using the situational information on the shared panel, the target party's intent, and the mission objectives. A gradient ascent method is used for optimization, and the calculated planning step is used as the ship's tactical path.

[0065]

[0066] The interactive coordination logic between human commanders and shared panels, ships, and fleets is as follows: Figure 3 As shown, the commander can control the navigation status of their own ship through the local interface in the shared panel, by drawing a route map or selecting the mission intention for automatic control. During the control process, this information is uploaded to the shared drawing board in real time. Other ships then obtain this information and make corresponding adjustments to their own strategies. There are two modes to choose from: automatic adjustment and manual adjustment.

[0067] Step S5: After the initial planning, the commander can modify the ship's route by hand-drawing, sharing the route plan or changing the plotted data on the panel. Different ships are distinguished by different colors. Once the commander changes the ship's route, other ships can access this information through the shared panel.

[0068] Hand-drawn track map recognition uses an online recognition method to record the point position and speed when drawing the track, and perform line element judgment to determine whether the line segment is a straight line, curve or arc. After judging the type of line element, the points on the line element need to be fitted with a straight line, a broken line or an arc. If the line element is a straight line, the least squares method is used to fit the straight line equation as follows, where σ is the error and x is the error. i and y i are hand-drawn coordinate points, and A and B are the parameters of the straight line fitting equation:

[0069]

[0070] The broken line is fitted by calculating the coordinates of the intersection point. The calculation method is to solve the coordinates of the intersection point by combining two straight line equations and correct the hand-drawn track results.

[0071]

[0072] When the curvature of the line is greater than the threshold, the arc is fitted using the least squares method. The expression is as follows, where x i and y i is the coordinate point on the arc, R is the arc radius, and X and Y are the coordinates of the center point of the arc:

[0073]

[0074] Step S6: Every once in a while, the shared panel performs information synchronization to detect any modified ship tracks or mission information. If the information has not been modified, the paths of all ships are updated again.

[0075] After the information is modified, if the manually controlled ship is selected not to change its track, the remaining automatically controlled ships will use reinforcement learning to output coordinated actions. The updated input is based on the initial determined ship track, adding the current and future tracks of the remaining friendly ships, as well as the last planned track of the ship. The next output action for the automated ships is determined based on the formula:

[0076] a=arg maxπ(a|s)←

[0077] Among them, a is the action, π is the policy function, and A is the action space.

[0078] Update the path of the autonomous ship. In order to avoid conflicts between the algorithm-calculated path and the commander-planned path, the commander's instructions have the highest priority. The other autonomously controlled ships automatically coordinate, and the human commander can switch the control mode.

[0079] Step S7: After calculating the planned path for the subsequent mission of the local ship, it is uploaded to the shared panel again for other ships to collaborate or modify the planned route to achieve the iterative update function.

Claims

1. A human-machine fusion ship control method based on a shared panel, wherein a fleet of a party includes multiple ships of the party, each of which is equipped with a shared panel, and is characterized by: The following steps are involved: Step 1: The ships in the fleet use multi-source sensors to collect multi-source sensor data, where the multi-source sensor data includes information about the ships, information about the enemy target ships, and information about the surrounding environment. Step 2: Fusing the multi-source sensor data collected by different friendly warships to obtain fused data, and calculating the future track of the enemy target ship through the time series neural network f. After obtaining the future track of the enemy target ship, the tactical mode of the enemy target ship when it is on the track is determined; Step 3: Encrypt and compress the fused data to obtain encrypted fused data, share the encrypted fused data with other shared panels in the own fleet, and synchronously receive the encrypted fused data from other shared panels; Step 4: After receiving the encrypted fusion data, the own ship decrypts the data to obtain decrypted fusion data, uses the decrypted fusion data and the mission objective to determine the planned route and intention of the own ship, and shares the planned route and intention of the own ship through the sharing panel; Step 5: The human commander changes the track or combat target of his own ship through a hand-drawn track chart on the shared panel. The other friendly ships re-optimize the planned path based on their own ship intentions, enemy target ship intentions, and mission objectives on the shared panel and share it, completing real-time control of human-machine fusion ships.

2. The human-machine fusion ship control method based on a shared panel according to claim 1 is characterized by: The information of the own warship includes the position, heading and speed of the own ship, and the information of the enemy target ship includes enemy target ship detection, attribute identification, enemy target ship tracking, enemy target ship positioning, speed, acceleration and direction. The enemy target ship detection, attribute identification and enemy target ship tracking are achieved through cameras, and the surrounding environment information, enemy target ship positioning, speed, acceleration and direction are achieved through radar.

3. The human-machine fusion ship control method based on a shared panel according to claim 2 is characterized by: Enemy target ship detection includes the following steps: Step 21: Use cameras to collect various ship image data, divide them into training sets and test sets, and use transfer learning for training; Step 22: Use a deep learning target detection algorithm to identify the model of the enemy target ship, assign a unique identifier to the detected enemy target ship, and retrieve the information of the corresponding model ship from the ship model database and display it on the shared panel.

4. The human-machine fusion ship control method based on a shared panel according to claim 2 is characterized by: The enemy target ship tracking is expressed by the following formula: Where, is the best estimate of the target at the current moment, is the current estimate of the target, Z k is the observed value, is the observation function, K k is the Kalman gain.

5. The human-machine fusion ship control method based on a shared panel according to claim 1 is characterized by: In step 2, the DS evidence theory is used to fuse the multi-source sensor data of different friendly warships to obtain the fused data, which can be expressed as follows: Where m 12…n (A) is the probability of merging n friendly ships, m n is the basic probability corresponding to the result of the nth own ship as the evidence element, A is the final fusion result of the own warship, B n It is the set of events supported by the sensor observation results of the nth own ship.

6. The human-machine fusion ship control method based on a shared panel according to claim 1 is characterized by: The fused data in step 2 includes the information, tracks and intentions of all friendly ships, the information and tracks of enemy target ships, and the surrounding environment information.

7. The human-machine fusion ship control method based on a shared panel according to claim 6 is characterized by: The future track Y of the enemy target ship is expressed by the following formula: Y=f(X,θ) Where Y is the future track of the enemy target ship, X=[x1,x2,...,x n ] is the feature vector of the enemy target ship, f is the temporal neural network, and θ is the network parameter.

8. The human-machine fusion ship control method based on a shared panel according to claim 1 is characterized by: The planned route of your own ship in step 3 is expressed by the following formula: in, is the gradient of policy θ, represents the expected trajectory of the sample, s t is the state at time t, a t is the action at time t, subscript t represents time, θ is the policy parameter, π is the policy function, and A is the advantage function.

9. The human-machine fusion ship control method based on a shared panel according to claim 1 is characterized by: In step 5, the hand-drawn track map is recognized online, the position and speed when drawing the track are recorded, and the type of line element is determined. The types of line elements include straight line elements, curved line elements and circular arc line elements. Straight line elements, curved line elements and circular arc line elements are respectively fitted by straight line fitting, broken line fitting and circular arc fitting through the points on the line element.

10. The human-machine fusion ship control method based on a shared panel according to claim 9 is characterized in that: The linear fitting adopts the least squares fitting method, which is expressed by the following formula: Where σ1 is the error, x i and y i are the horizontal and vertical coordinates of the i-th coordinate point on the hand-drawn track chart, n is the number of coordinate points on the hand-drawn track chart, A and B are the parameters of the straight line fitting equation; The broken line fitting is performed by calculating the intersection coordinates, which can be expressed as follows: Where (x, y) is the coordinate of the intersection of the broken lines, a1, b1, and c1 are the parameters of the first straight line equation, and a2, b2, and c2 are the parameters of the second straight line equation; Arc fitting uses the least squares method to fit the arc, which is expressed by the following formula: Where σ2 is the error, x j and y j are the horizontal and vertical coordinates of the jth coordinate point on the hand-drawn track chart, R is the radius of the arc, and X and Y are the coordinates of the center point of the arc.

Images