Man-machine fusion ship control method based on shared panel

Abstract

The invention discloses a man-machine fusion ship control method based on a shared panel. The man-machine fusion ship control method comprises the following steps that an own ship collects multi-source sensing data through a multi-source sensor; fusing the multi-source sensing data to obtain fused data, and calculating a future track of the enemy target ship; sharing the encrypted fusion data to the own fleet, and synchronously receiving the encrypted fusion data of other shared panels; after receiving the encrypted fusion data, the own ship determines a planned route and intention of the own ship, and shares the planned route and intention; and the human commander changes the track or combat target of the own ship, and the other own ships re-optimize the planned path and share the re-optimized planned path to complete the real-time control of the man-machine fusion ship. According to the method, own ship information is fused by using a multi-source data fusion theory. Based on a multi-machine interaction mode of a shared panel, an iterative optimization strategy with a task target as a guide is provided, and the cognitive ability and collaboration of a maritime fleet in the processes of law enforcement, right protection and the like are enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-machine interaction and information collaboration, and in particular relates to a human-machine fusion ship control method based on a shared panel. Background Art

[0002] In the maritime law enforcement and rights protection mission, facing massive data and complex maneuvering schemes, the fleet needs accurate and consistent full-range perception display, real-time and easy course drawing, and collaborative global task cognition with low cognitive load. Therefore, it is of great application value to use systematic and intelligent technology to enhance the cognitive ability and coordination of the maritime law enforcement and rights protection fleet.

[0003] In existing maritime law enforcement and rights protection tasks, means based on single-subject perception cannot achieve large-scale global cognition, and the patrol and change capabilities are insufficient and inefficient; the perception of multiple subjects lacks effective consistency fusion means, and cannot quickly obtain accurate and reliable unified cognitive results in complex and massive perception data; the information between individuals in the fleet is isolated and the sharing is poor. The fleet cannot share the planned paths and intentions of other individuals in real time, and the coordination and collaboration for the same task cannot be guaranteed; in path planning and control, humans and machines are separated, the interactivity is poor, and path deviation is easy to occur under complex situations. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a human-machine fusion ship control method based on a shared panel.

[0005] Technical solution: The human-machine fusion ship control method based on a shared panel described in the present invention, wherein the own fleet includes a plurality of own ships, each of which is provided with a shared panel, specifically comprises the following steps:

[0006] Step 1: The ships of the fleet collect multi-source sensing data using multi-source sensors, where the multi-source sensing data includes information of the ships of the fleet, information of the enemy target ships, and surrounding environment information;

[0007] Step 2: Fusing the multi-source sensor data collected by different friendly warships to obtain fused data, and calculating the future track of the enemy target ship through the time series neural network f, and after obtaining the future track of the enemy target ship, determining the tactical mode of the enemy target ship when it is on the track;

[0008] Step 3: Encrypt and compress the fused data to obtain encrypted fused data, and share the encrypted fused data with other shared panels in the own fleet, and synchronously receive the encrypted fused data from other shared panels;

[0009] Step 4: After receiving the encrypted fusion data, the own ship decrypts the data to obtain the decrypted fusion data, uses the decrypted fusion data and the mission objective to determine the planned route and intention of the own ship, and shares the planned route and intention of the own ship through the sharing panel;

[0010] Step 5: The human commander changes the track or combat target of his own ship through a hand-drawn track chart on the shared panel. The other friendly ships re-optimize the planned path through their own ship intentions, target ship intentions, and mission objectives on the shared panel and then share it, completing real-time control of human-machine fusion ships.

[0011] Furthermore, the information of the own warship includes the position, heading and speed of the own ship, and the information of the enemy target ship includes target ship detection, attribute recognition, target ship tracking, target ship positioning, speed, acceleration and direction. The target ship detection, attribute recognition and target ship tracking are achieved through cameras, and the surrounding environment information, target ship positioning, speed, acceleration and direction are achieved through radar.

[0012] Furthermore, target ship detection includes the following steps:

[0013] Step 21: Use the camera to collect various ship image data, divide them into training sets and test sets, and use transfer learning for training; Step 22: Use the deep learning target detection algorithm to identify the model of the target ship, assign a unique identifier to the detected target ship, and retrieve the information of the corresponding model of the ship from the ship model database and display it on the shared panel.

[0014] The specific detection process is expressed by the following formula:

[0015] (P i1 ,P i2 ,...,P ij ) = g(network output)

[0016] (x i ,y i ,w i ,h i ) = f(network output)

[0017] Where P i,j represents the probability that the i-th bounding box belongs to the j-th type of ship, x i ,y i ,w i ,h i are the center coordinates and length and width of the ship, respectively, and g and f are the decoding heads of the neural network. A unique identifier is assigned to the detected target, and the initial position and speed information are obtained. The extended Kalman filter is used to track the target ship and output it to the shared panel.

[0018] Furthermore, the target ship tracking is expressed as follows:

[0019]

[0020] In the formula, is the best estimate of the target at the current moment, is the current estimate of the target, z k is the observed value, is the observation function, K k is the Kalman gain.

[0021] Furthermore, in step 2, the DS evidence theory is used to fuse the multi-source sensor data of different friendly warships to obtain fused data, which is expressed by the following formula:

[0022]

[0023] In the formula, m 12…n (A) is the probability of merging n friendly ships, m n is the basic probability corresponding to the result of the nth own ship as the evidence element, A is the final fusion result of the own warship, B n It is the set of events supported by the sensor observations of the nth own ship.

[0024] Furthermore, the fused data in step 2 includes the information, tracks and intentions of all friendly ships, the information and tracks of enemy target ships, and the surrounding environment information.

[0025] Furthermore, the future track Y of the enemy target ship is expressed by the following formula:

[0026] Y=f(X,θ)

[0027] Where Y is the future track of the target ship, X = [x 1 ,x 2 ,...,x n ] is the feature vector of the target ship, f is the temporal neural network, and θ is the network parameter.

[0028] The input is the characteristics of the target ship, including position, speed, acceleration, model, and historical data mining rules to form a feature vector X = [x 1 ,x 2 ,...,x n ], and the output is the estimated position information Y of the target ship in the future. After obtaining the track prediction of the enemy ship, an expert system based on historical big data is established to determine the tactical mode (attack, retreat, breakout, bait, reconnaissance, etc.) of the target ship when it is on the track.

[0029] Furthermore, the planned route of the own ship in step 3 is expressed by the following formula:

[0030]

[0031] in, is the gradient of policy θ, represents the expected trajectory of the sample, s t is the state at time t, a t is the action at time t, the subscript t represents time, θ is the policy parameter, π is the policy function, and A is the advantage function.

[0032] The reinforcement learning PPO algorithm model is constructed, and the panel is rasterized according to the current shared panel situation status. The strategy is formulated using the situation information on the shared panel, the target party's intention, and the mission goal. The gradient ascent method is used for optimization, and the calculated planning step is used as the ship's tactical path. The situation information of the shared panel includes the timestamp; the status information, path planning information and intention of all our ships; the information and intention of all ships in the target fleet; the overall environment information of maritime operations; and our overall mission information.

[0033] Furthermore, in step five, the hand-drawn track map is recognized online, the position and speed when drawing the track are recorded, and the type of the line element is determined. The types of line elements include straight line elements, curved line elements and circular arc line elements. Straight line elements, curved line elements and circular arc line elements are respectively fitted by straight lines, broken lines and circular arcs through the points on the line elements.

[0034] Hand-drawn track map recognition adopts online recognition method, records the point position and speed when drawing the track, and performs line element judgment on whether the line segment is a straight line, curve or arc. After judging the type of line element, the points on the line element need to be fitted with a straight line, a broken line or an arc.

[0035] Furthermore, the linear fitting is performed using the least squares method, which is expressed by the following formula:

[0036]

[0037] In the formula, σ is the error, x i and i is the i-th coordinate point on the hand-drawn track map, n is the number of coordinate points on the hand-drawn track map, A and B are the parameters of the straight line fitting equation;

[0038] The polyline fitting is performed by calculating the coordinates of the intersection points, which can be expressed by the following formula:

[0039]

[0040] In the formula, x and y are the coordinates of the intersection of the broken lines, a 1 、b 1and c 1 is the parameter of the first straight line equation, a 2 , b 2 and c 2 are the parameters of the second straight line equation;

[0041] Arc fitting uses the least squares method to fit the arc, which is expressed by the following formula:

[0042]

[0043] In the formula, σ 2 is the error, x j and j is the jth coordinate point on the hand-drawn track chart, R is the radius of the arc, and X and Y are the coordinates of the center point of the arc.

[0044] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: The human-machine fusion ship control method based on a shared panel of the present invention adopts multi-source information fusion theory and interactive technology based on a shared panel, and at the same time solves the problems of incomplete information collection of a single ship and poor accuracy of cognitive results under complex multi-source information, as well as the lack of information sharing between ships in the fleet performing tasks, and poor interactivity and coordination. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flowchart of a human-machine fusion ship control method based on a shared panel;

[0046] Figure 2 Display effect diagram for shared panel;

[0047] Figure 3 Schematic diagram of the interactive collaboration logic between human commanders and shared panels, ships, and fleets. DETAILED DESCRIPTION

[0048] Example 1

[0049] like Figure 1 As shown, the human-machine fusion ship control method based on a shared panel of the present invention comprises the following steps:

[0050] Step S1: Collect sensor data from each ship in the fleet, using the global positioning system (GPS) to locate the position of the own ship; the inertial measurement unit (IMU) to calculate the speed, acceleration and direction of the own ship; camera data is used for target ship detection, attribute recognition and tracking; radar data is used to detect the surrounding environment at sea and the positioning, speed, acceleration and direction of the target ship; wind sensors and weather station information are used to detect changes in the marine environment.

[0051] Multi-source sensor data is received in real time, and the camera and radar information collected by individuals contains different perspectives of the same perception area of ​​other ships.

[0052] The target ship image data is collected by the camera and divided into training set and test set. The transfer learning training is used, and the deep learning target detection algorithm is used to identify the models of common ships at sea, and then it is linked to the ship model database, and the attributes, equipment and other information of the corresponding ship model are called out and displayed on the shared panel. i,j represents the probability that the i-th bounding box belongs to the j-th type of ship, x i ,y i ,w i ,h i are the center coordinates and length and width of the ship, respectively. g and f are the decoding heads of the neural network. The detection process can be expressed as:

[0053] (P i1 ,P i2 ,...,P ij ) = g(network output)

[0054] (x i ,y i ,w i ,h i ) = f(network output)

[0055] Assign a unique identifier to the detected target, obtain the initial position and speed information, and use the extended Kalman filter to track the target ship and output it to the shared panel. is the best estimate of the target at the current moment, is the current estimate of the target, Z k is the observed value, is the observation function, K k is the Kalman gain, and the tracking formula is:

[0056]

[0057] Step S2: After obtaining the perception results of a single ship, it is necessary to fuse the perception results of different ships of the same target. For the basic information such as the position, speed, and model of the own ship, the results of the own ship shall prevail. For the perception information of the target ship, because the camera, position, speed, model and other data collected from different ships will be different, the DS evidence theory is used to fuse the uncertainty information from different sources to provide a decision-level fusion result. Assuming that there are n own ship results as evidence elements, the corresponding basic probability distribution is m n , then the probability distribution formula after merging is:

[0058]

[0059] Multi-source sensor data processing uses intelligent perception algorithms to identify the location, type, heading, speed, acceleration and other information of target ships. The processing includes the perception information at the current moment and the prior information at the historical moment. The multi-source information fusion algorithm is used to align the perception information of the same target of multiple ships to eliminate feature ambiguity and ensure that the fleet has consistent perception of the same target in the same coordinate system at the same time.

[0060] Step S3: Encrypt and compress the data, encrypt it using RSA technology, and use pulse coding method to perform nearly lossless real-time compression of data stream. Through the ocean 5G network and edge cloud technology, data sharing and synchronous reception are realized. The perception results after multi-source sensor fusion are displayed in the shared panel: including the information, tracks and intentions of all ships of the own side, the information and tracks of detectable enemy ships, and overall environmental information. And calculate the future track of each enemy ship through the time series neural network f. The input is the characteristics of the target ship, including position, speed, acceleration, model, and historical data mining rules to form a feature vector X=[x 1 ,x 2 ,...,x n ], the output is the estimated position information Y of the target ship in the future, which can be expressed as, where θ is the network parameter:

[0061] Y=f(X,θ)

[0062] After obtaining the prediction of the enemy ship's track, an expert system based on historical big data is established to determine the tactical mode (attack, retreat, breakout, bait, reconnaissance, etc.) of the target ship when it is on that track. The commander or intelligent algorithm can formulate our strategy based on the target track and tactics.

[0063] The sharing panel display effect is as follows Figure 2 As shown, it includes analysis of sea level weather; analysis of target ship's attributes, status, track, prediction and intention; attributes, status, track, prediction and intention of one's own ship; overall mission objectives; perspective switching; hand-drawn path conversion display and other functions.

[0064] Step S4: Based on the situation information obtained in the previous step and combined with the mission objectives, the human commander can formulate the future track and tactical intention of the ship by hand-drawing on the shared panel so that other ships can coordinate tactically. If no specific instructions are issued, a reinforcement learning PPO algorithm model can be constructed to rasterize the panel according to the current shared panel situation state, formulate strategies using the situation information on the shared panel, the intention of the target party, and the mission objectives, and use the gradient ascent method for optimization, and the calculated planning step is used as the tactical path of the ship.

[0065]

[0066] The interactive coordination logic between human commanders and shared panels, ships, and fleets is as follows: Figure 3 As shown in the figure, the commander can control the navigation status of the ship through the local interface in the shared panel, by drawing a route map or selecting the mission intention for automatic control. During the control process, this information is uploaded to the shared drawing board in real time. After other ships obtain this information, they make corresponding adjustments to their own strategies. There are two modes to choose from: automatic adjustment and manual adjustment.

[0067] Step S5: After the initial planning, the commander can modify the ship's route by hand-drawing, sharing the route planning or changing the drawing data recorded on the panel, and distinguishing different sources of ships with different colors. Once the commander changes the ship's route, other ships can obtain this information through the shared panel.

[0068] Hand-drawn track map recognition uses an online recognition method to record the point position and speed when drawing the track, and perform line element judgment on whether the line segment is a straight line, curve or arc. After judging the type of line element, the points on the line element need to be fitted with a straight line, a broken line or an arc. If the line element is a straight line, the least squares method is used to fit the straight line equation as follows, where σ is the error, x is the error, and i and i are hand-drawn coordinate points, and A and B are the parameters of the straight line fitting equation:

[0069]

[0070] The broken line is fitted by calculating the coordinates of the intersection point. The calculation method is to solve the coordinates of the intersection point by combining two straight line equations to correct the hand-drawn track results.

[0071]

[0072] When the curvature of the line is greater than the threshold, the least squares method is used to fit the arc. The expression is as follows, where x i and i is the coordinate point on the arc, R is the radius of the arc, and X and Y are the coordinates of the center point of the arc:

[0073]

[0074] Step S6: At regular intervals, the shared panel performs information synchronization to detect the modified ship track or mission information. If the information has not been modified, the paths of all ships are updated again.

[0075] When the information is modified, if the manually controlled ship is selected not to change the track, the other automatically controlled ships use reinforcement learning to output coordinated actions. When updating, the input is compared with the initial determined ship track, and the current and future tracks of the other ships on the side and the last planned track of the ship are added. Thus, the next output action of the automatic ship is selected according to the formula:

[0076] a=arg maxπ(a|s)←

[0077] Among them, a is the action, π is the policy function, and A is the action space.

[0078] Update the path of the autonomous ship. In order to avoid conflicts between the path calculated by the algorithm and the path planned by the commander, the commander's command has the highest priority. The other autonomously controlled ships automatically coordinate, and the human commander can switch the control mode.

[0079] Step S7: After calculating the planned path for the subsequent mission of the local ship, upload it to the sharing panel again for other ships to collaborate or modify the planned route to achieve the iterative update function.

Claims

1. A human-machine fusion ship control method based on a shared panel, wherein the own fleet includes a plurality of own ships, each of which is provided with a shared panel, and is characterized in that: The following steps are involved: Step 1: The ships of the fleet collect multi-source sensing data using multi-source sensors, where the multi-source sensing data includes information of the ships of the fleet, information of the enemy target ships, and surrounding environment information; Step 2: Fusing the multi-source sensor data collected by different friendly warships to obtain fused data, and calculating the future track of the enemy target ship through the time series neural network f, and after obtaining the future track of the enemy target ship, determining the tactical mode of the enemy target ship when it is on the track; Step 3: Encrypt and compress the fused data to obtain encrypted fused data, and share the encrypted fused data with other shared panels in the own fleet, and synchronously receive the encrypted fused data from other shared panels; Step 4: After receiving the encrypted fusion data, the own ship decrypts the data to obtain the decrypted fusion data, uses the decrypted fusion data and the mission objective to determine the planned route and intention of the own ship, and shares the planned route and intention of the own ship through the sharing panel; Step 5: The human commander changes the track or combat target of his own ship through a hand-drawn track chart on the shared panel. The other friendly ships re-optimize the planned path through their own ship intentions, target ship intentions, and mission objectives on the shared panel and then share it, completing real-time control of human-machine fusion ships.

2. According to claim 1, the human-machine fusion ship control method based on the shared panel is characterized in that: The information of the own warship includes the position, heading and speed of the own ship, and the information of the enemy target ship includes target ship detection, attribute recognition, target ship tracking, target ship positioning, speed, acceleration and direction. The target ship detection, attribute recognition and target ship tracking are achieved through cameras, and the surrounding environment information, target ship positioning, speed, acceleration and direction are achieved through radar.

3. According to claim 2, the human-machine fusion ship control method based on the shared panel is characterized in that: Target ship detection includes the following steps: Step 21: Use cameras to collect various ship image data, divide them into training sets and test sets, and use transfer learning for training; Step 22: Use a deep learning target detection algorithm to identify the model of the target ship, assign a unique identifier to the detected target ship, and retrieve the information of the corresponding model of the ship from the ship model database and display it on the shared panel.

4. According to claim 2, the human-machine fusion ship control method based on the shared panel is characterized in that: Target ship tracking is expressed by the following formula: In the formula, is the best estimate of the target at the current moment, is the current estimate of the target, Z k is the observed value, is the observation function, K k is the Kalman gain.

5. According to claim 1, the human-machine fusion ship control method based on the shared panel is characterized in that: In step 2, the DS evidence theory is used to fuse the multi-source sensor data of different friendly warships to obtain the fused data, which is expressed by the following formula: In the formula, m 12…n (A) is the probability of merging n friendly ships, m n is the basic probability corresponding to the result of the nth own ship as the evidence element, A is the final fusion result of the own warship, B n It is the set of events supported by the sensor observations of the nth own ship.

6. According to claim 1, the human-machine fusion ship control method based on the shared panel is characterized in that: The fused data in step 2 includes the information, tracks and intentions of all friendly ships, the information and tracks of enemy target ships, and the surrounding environment information.

7. The human-machine fusion ship control method based on a shared panel according to claim 6 is characterized by: The future track Y of the enemy target ship is expressed by the following formula: Y=f(X,θ) Where Y is the future track of the target ship, X = [x1, x2, ..., x n ] is the feature vector of the target ship, f is the temporal neural network, and θ is the network parameter.

8. The human-machine fusion ship control method based on a shared panel according to claim 1 is characterized by: The planned route of your ship in step 3 is expressed by the following formula: in, is the gradient of policy θ, represents the expected trajectory of the sample, s t is the state at time t, a t is the action at time t, the subscript t represents time, θ is the policy parameter, π is the policy function, and A is the advantage function.

9. The human-machine fusion ship control method based on a shared panel according to claim 1 is characterized by: In step 5, the hand-drawn track map is recognized online, the position and speed when drawing the track are recorded, and the type of the line element is determined. The types of line elements include straight line elements, curved line elements and circular arc line elements. Straight line elements, curved line elements and circular arc line elements are respectively fitted by straight lines, broken lines and circular arcs through the points on the line elements.

10. The human-machine fusion ship control method based on a shared panel according to claim 9 is characterized in that: The linear fitting adopts the least squares fitting method, which is expressed by the following formula: In the formula, σ is the error, x i and i is the i-th coordinate point on the hand-drawn track map, n is the number of coordinate points on the hand-drawn track map, A and B are the parameters of the straight line fitting equation; The polyline fitting is performed by calculating the coordinates of the intersection points, which can be expressed by the following formula: Where x and y are the coordinates of the intersection of the broken lines, a1, b1 and c1 are the parameters of the first straight line equation, and a2, b2 and c2 are the parameters of the second straight line equation; Arc fitting uses the least squares method to fit the arc, which is expressed by the following formula: In the formula, σ2 is the error, x j and j is the jth coordinate point on the hand-drawn track chart, R is the radius of the arc, and X and Y are the coordinates of the center point of the arc.

Images