A method for enhancing the intelligent perception of unmanned ships by integrating virtual and real fusion into multiple twins

Abstract

The present invention provides a method for enhancing the virtual-reality fusion multi-twin intelligent perception of an unmanned ship, comprising: acquiring data information; performing data fusion processing on the acquired data information, storing the fusion results in a database and constructing a model database; transmitting the data in the model database to an electronic chart display platform, a three-dimensional vision display platform and a virtual-reality fusion display platform, the electronic chart display platform realizes chart operations and route corrections, and extracts the channel and buoy information of the electronic chart in real time, and sends the real-time extracted information to the three-dimensional vision platform to complete data supplement; the three-dimensional vision platform uses Unity as a development engine to construct a three-dimensional virtual scene, and realizes perception instantiation, weather system and risk warning functions; the virtual-reality fusion display platform renders the navigation pictures collected by the ship-borne camera, so that the shore operator can judge the navigation safety of the unmanned ship through intuitive guidance information and identification borders, thereby realizing the function of assisted driving.

Description

Technical Field

[0001] The present invention relates to the field of safe navigation assistance technology for unmanned ships, and in particular to a method for enhancing virtual-reality fusion multi-twin intelligent perception of unmanned ships. Background Art

[0002] In the field of ship navigation, drivers mostly obtain navigation environment, target ship parameters and navigation parameters by observing shipborne equipment (radar, AIS, electronic charts, etc.). For unmanned ships, as navigation equipment increases, it also brings information load to drivers. When taking on driving tasks, drivers also need to complete the integration and processing of navigation information. The current complex display system of shipborne equipment brings convenience to the safe navigation of ships, but also makes drivers tired of processing various information. Therefore, when providing navigation information to drivers, the effectiveness and timeliness of the information must be guaranteed.

[0003] At present, the navigation status display of unmanned ships mostly relies solely on electronic charts. Electronic charts are developed to display the basic charts such as the ship itself, target ship, route, re-integrated AIS information and radar echo. Although the electronic chart is very powerful, it is only a navigation aid. The limitations of the system itself, display errors and failures, inappropriate or incorrect system settings and use by users, sensor errors, and timely and effective use of backup arrangements all require users not to rely too much on it. Multiple sensors and multiple interfaces distract the operator's attention and are prone to accidents. Therefore, the traditional navigation aid system has certain shortcomings. Summary of the invention

[0004] According to the technical problems raised above, a method for enhancing the virtual-real fusion multi-twin intelligent perception of unmanned ships is provided. The present invention can improve the visual perception navigation system of unmanned ships, so that the operator can observe the navigation status of the unmanned ship more intuitively and in real time from the shore.

[0005] The technical means adopted by the present invention are as follows:

[0006] A method for enhancing the multi-twin intelligent perception of an unmanned ship by fusion of virtual and real aspects, comprising:

[0007] S1. Obtain data information;

[0008] S2, performing data fusion processing on the acquired data information, storing the fusion results in a database and constructing a model database;

[0009] S3. Transmit the data in the model database to the electronic chart display platform, the three-dimensional vision display platform and the virtual-reality fusion display platform to realize assisted driving of the unmanned ship.

[0010] Furthermore, in step S1, the acquired data information includes environmental information, own ship information and other ship information, wherein:

[0011] Environmental information includes weather information, ocean current information, waterway information, and buoy information; a variety of sensors and equipment jointly obtain environmental information for unmanned ship navigation. Sensor equipment for obtaining environmental information includes depth sounders for measuring water depth, wind speed and direction meters for measuring wind speed and direction, current meters for measuring water flow speed and direction, wave meters for measuring sea wave data, and electronic nautical charts for providing waterway information;

[0012] The ship's information includes data information generated when the ship is sailing, which is used to reflect the ship's navigation status; the equipment for obtaining the ship's information includes a gyrocompass for providing the unmanned ship's orientation, a GPS for positioning the ship and providing accurate longitude and latitude information, a speedometer for measuring the voyage and performing track calculation, and a rudder angle feedback device for obtaining feedback information from the unmanned ship's automatic rudder for optimization and adjustment;

[0013] Other ship information includes data information of target ships detected within a certain range, and the equipment for obtaining target ship data information includes radar and AIS.

[0014] Furthermore, in step S2, data fusion processing is performed on the acquired data information, mainly data fusion is performed on the redundant parts of the target ship data information acquired by radar and AIS, specifically as follows:

[0015] Space Unification:

[0016] The Mercator projection is used to transform the AIS information coordinates. The Mercator projection formula is as follows:

[0017]

[0018]

[0019] Among them, a represents the major axis of the ellipsoid, and the value of a is 6378137m; b represents the minor semi-axis of the spheroid, and its value is approximately 6356752.3142m; represents the base latitude; λ represents the longitude, in radians; represents latitude, the unit is unit radian; λ0 represents the central meridian and the central longitude is 0; represents the radius of curvature of the yoke; e represents the first eccentricity and e · represents the second eccentricity and

[0020] The radar information format is converted from the polar coordinate system to the rectangular coordinate system. The conversion formula is as follows:

[0021]

[0022] Where R represents the distance of the radar target at time k; θ(k) represents the azimuth of the radar target at time k; x(k) represents the x-axis component of the radar target at time k in the rectangular coordinate system; y(k) represents the y-axis component of the radar target at time k in the rectangular coordinate system;

[0023] Time Unification:

[0024] The linear interpolation algorithm is used to achieve time alignment, as follows:

[0025] On the interval [a,b], given n+1 nodes: a=t0<t1<…<t n-1 <t n =b and corresponding to y0,y1,…,y n , let h i =t i+1 -t i , The interpolation function is S h (t), which is a broken line function. In any small interval [t i ,t i+1 ] can be expressed as t i <t<t i+1 ;

[0026] The basis function method is used to construct a piecewise linear function. h (t) is Among them l j (t) is the Langrange interpolation basis function, and the formula of the Langrange interpolation basis function is as follows:

[0027]

[0028] The distance-speed comprehensive fine association algorithm is used to associate the same target based on its same motion characteristics within a certain period of time, as follows:

[0029] Select a track within a certain sampling time, and calculate the difference in position and speed between the corresponding points of the radar and AIS in the x and y directions. The target with the smallest difference is considered to be the associated target. The measurement matrix of the radar to the target is known to be: The measurement matrix of AIS to the target is: The calculation formula is as follows:

[0030]

[0031] in, Where Dx ij ,Dyij Respectively represent the average speed difference between radar target i and AIS target j in the x and y directions at N measurement points, The radar tracking target i and AIS tracking target j that meet the minimum requirement are determined to be associated; the track fusion algorithm is based on the statistical weighted criterion, and the extended Kalman filter is used to process the fusion result to make the track closer to the real track;

[0032] The accuracy of weight distribution for multi-sensor data fusion determines the accuracy of data fusion. Suppose there are n sensing devices, z i (i=1,2,3,...,n) is the measured value, is the weight, and the measurement value of the i-th sensor among n sensors is z i =x i +e i , where z i is the measured value, x i is the true value, e i is the error value. Since there is no interference between the sensors and they are independent of each other, they all obey the normal distribution, as shown in the following formula:

[0033]

[0034] Statistical weighting is to multiply the weighed and measured values ​​and add them up to get an estimate To achieve Unbiased estimation needs to satisfy get

[0035] The minimum mean square error is used to allocate weights. The formula is as follows:

[0036]

[0037] Convert it into a linear programming problem as follows:

[0038]

[0039]

[0040] For linear programming, the Lagrange multiplier method is used and an auxiliary function is constructed to solve the problem. The auxiliary function is set as The optimal weight and minimum variance are calculated as follows:

[0041]

[0042] Data Fusion:

[0043] The data of distance, direction, speed and heading are fused, and the fusion formula is as follows:

[0044]

[0045] in, is the radar and AIS distance variance, is the azimuth variance, They are speed variance and heading variance respectively;

[0046] In order to improve the fusion accuracy, the fusion result is processed by extended Kalman filtering. The extended Kalman filtering formula is as follows:

[0047]

[0048] in, is the state prediction mean, P k|k-1 is the state prediction variance, K k is the Kalman gain, is the mean of the target posterior, P k is the variance of the target posterior, and the state vector of the ship motion model is x = [X, Y, SOG, COG] T , X, Y, SOG, COG are the horizontal and vertical coordinates, speed, and heading respectively, and the ship motion system equation is as follows:

[0049]

[0050] Among them, the observation matrix is ​​z = [D, B, SOG, COG] T , D, B, SOG, COG are distance, azimuth, speed, and heading respectively. The observation equation is as follows:

[0051]

[0052] Furthermore, in step S3:

[0053] The electronic chart display platform is used as a basic navigation aid display system to realize chart operation and route correction, and to extract the channel and buoy information of the electronic chart in real time, and send the real-time extracted information to the three-dimensional visual platform to complete data supplement;

[0054] The 3D visual platform uses Unity as a development engine to build a 3D virtual scene, and implements perception instantiation, weather system and risk warning functions to achieve the effect of intelligent perception 3D visualization;

[0055] The virtual-reality fusion display platform renders the navigation images captured by the ship-borne camera, recognizes the ships within the observation range through the YOLOv3 algorithm, and renders the guiding ships and guiding routes, so that the shore operator can judge the navigation safety of the unmanned ship through intuitive guidance information and identification borders, thereby realizing the function of assisted driving.

[0056] Furthermore, the 3D model in the 3D visual platform is constructed in 3D by Cinema 4D software, and then rendered by texture. The model is exported in fbx format, loaded into the Unity engine, and set as a prefab to reduce resource utilization. The scene construction is extended with the real terrain as the base map. In the 3D visual platform, the process of generating the 3D visual is as follows:

[0057] Intercept the terrain range that needs to be generated in the satellite map;

[0058] Get its elevation map and convert it into raw data format;

[0059] Create Terrain, merge it with the heightmap through HeightMapper, and set the matching ratio;

[0060] The terrain is rendered with textures, and resources are obtained from satellite images; the details of the grass are constructed and texture mapping is performed; the ocean water effect is achieved by stealing resource packages and adding shaders; the weather effects of rain, snow and fog are achieved using a particle system.

[0061] Furthermore, the virtual-reality fusion display platform constructs a model database in the virtual vision, constructs a three-dimensional model database according to the AIS ship and cargo types, instantiates the data results of the AIS and radar fusion, parses the AIS message and matches it with the three-dimensional model database, and realizes perception instantiation in the virtual vision; specifically as follows:

[0062] In order to experience the safety of unmanned ship navigation from a three-dimensional perspective, the unmanned ship danger warning function is realized in a graphical way, the navigation information of the ship and the status information of the target ship are comprehensively analyzed, and a ship collision risk model and trajectory calculation based on physical information are generated; the three-dimensional visualization presentation allows the operator to more intuitively observe the operation safety of the unmanned ship; the ship collision risk model mainly considers TCPA and DCPA, and its risk level is visualized;

[0063] Calculate the minimum encounter distance, minimum encounter time and distance between two ships, set three levels of risk, and render corresponding colors according to different risk levels to make it more intuitive in the virtual vision;

[0064] Design the UI interface and complete the interactive operation, mainly including multi-angle observation of the ship's motion status, divided into front, back, left, right and third-person perspectives, let Unity build in several cameras, each camera corresponds to the corresponding angle, set the camera's depth, select perspective projection as the projection method, and adjust the field of view parameters to adjust the camera's viewing angle; the main perspective camera is a follow camera, which can follow the target movement and can achieve close-up, far-reaching, and 360-degree observation;

[0065] The electronic chart display platform is used as the data supplement interface of the virtual vision display platform. The data of the electronic chart is extracted and sent to the virtual vision server to supplement the static navigation information of the virtual vision, including isobaths and channel information.

[0066] Furthermore, the virtual-reality fusion display platform includes three submodules: visual guidance of unmanned ships, visual guidance of routes, and ship recognition of navigation images; wherein:

[0067] Visual guidance of unmanned ships is to integrate the information of virtual guidance of unmanned ships with the real world, and superimpose the real navigation environment and the virtual guidance of unmanned ships on the same screen or space in real time. The visualization system provides information that is different from what humans can perceive under normal circumstances. The visualization display system will integrate all ship navigation information to guide the physical unmanned ships to navigate safely. Specifically, the images collected by the onboard camera of the unmanned ship are transmitted to the shore command center and projected onto UI-image. The motion status of the physical unmanned ship, including speed, heading, and pitch angle, is collected through multiple sensors such as inertial navigation. The built-in camera of Unity is used to control its viewing angle and status through scripts to simulate the real shipborne camera.

[0068] Furthermore, the specific process of the visually guided unmanned ship guiding the physical unmanned ship to navigate safely is as follows:

[0069] The visualization guiding unmanned ship rendering is based on the Unity engine. The optimal decision path for this task is extracted from the database and converted into world coordinates. In order to distinguish the static planning route from the guiding unmanned ship route, the static route is set as three lineRenderer components, and its width is increased. The spacing between each component is 0.5m, and the color is set to red. The path of the guiding unmanned ship is set to green, and the guiding unmanned ship is instantiated as purple. Bezier optimization is performed on its path. The linear formula of the Bezier curve is as follows:

[0070] B(t)=P0+(P1-P0)t=(1-t)P0+tP1,t∈[0,1]

[0071] Use scripts to modify the properties of the LineRenderer component and set the interpolation points to draw a continuous smooth curve. In order to increase visibility, set the local path to dotted green. First, prepare a transparent background dotted line png, with a transparent background instead of a white background, import it into the Unity project, create a new material, modify its shader to the newly imported png, adjust the Tiling and Offset parameters to make the dotted line suitable, and set the LineRenderer component material to the newly created material. At this time, you can display the real video stream in Unity and display the route of the unmanned ship at the corresponding position;

[0072] In order to achieve the effect of safe navigation, it is necessary to display the predicted position of the guide ship in the real video stream. The position of the guiding unmanned ship is set to the position predicted for the optimal decision path of the unmanned ship for thirty seconds. The position and tangent angle of a certain point on the Bezier curve are calculated. n points are taken on the Bezier curve, 50 points are taken by default, the length of the straight line between the points is calculated, and the sum is added to obtain the approximate length of a curve. The more points are taken, the more accurate the length tends to be. When the Bezier curve expression is known, in order to obtain the tangent of a certain point, the curve is derived, and the following can be obtained:

[0073] B(t)=P0*3(1-t) 2 *(-1)+3P1*((1-t) 2 +t*2(1-t)*(-1))+3P2*(2t(1-t)+t 2 *(-1))+P3*3t 2

[0074] Rearranging the above formula, we can get:

[0075] B(t)=-3P0(1-t) 2 +3P1((1-t) 2 -2t(1-t))+3P2(2t(1-t)-t 2 )+3P3t 2

[0076] The angle between the two vectors of the parameters thirty seconds and thirty-one seconds is the angle of the target ship at this location. The Unity implementation function is as follows:

[0077] Mathf.Atan(ax / bz)*Mathf.Rad2Deg+del tan Angle

[0078] Among them, a is the thirty-second position point vector of the guiding unmanned ship, and b is the position of the guiding unmanned ship at the next moment, that is, the heading of the guiding unmanned ship at this moment is obtained, and it can be converted into a four-tuple to instantiate the guiding unmanned ship.

[0079] Compared with the prior art, the present invention has the following advantages:

[0080] 1. The unmanned ship virtual-reality fusion multi-twin intelligent perception enhancement method provided by the present invention can improve the visual perception navigation aid system of the unmanned ship, so that the operator can observe the navigation status of the unmanned ship more intuitively and in real time from the shore.

[0081] 2. The unmanned ship virtual-reality fusion multi-twin intelligent perception enhancement method provided by the present invention takes the electronic nautical chart as the basic navigation aid system and conducts secondary development on it. It can observe the ship's operating track and the surrounding marine environment terrain and the dynamic and static information of ships sailing in nearby waters from a two-dimensional digital perspective, so that the operator can observe the navigation safety of the unmanned ship from a global perspective, and extract the data of the electronic nautical chart and transmit it to the virtual vision platform to form data supplement.

[0082] 3. The unmanned ship virtual-reality fusion multi-twin intelligent perception enhancement method provided by the present invention, its virtual vision reflects the navigation status of the unmanned ship from a three-dimensional perspective, and introduces the risk of ship collision, so as to intuitively feel the safety situation of the unmanned ship's navigation in a graphical way, and fuse its AIS and radar data, match the model library, and instantiate it into the virtual vision, which can enhance the visualization of intelligent perception.

[0083] 4. The unmanned ship virtual-real fusion multi-twin intelligent perception enhancement method provided by the present invention completes rendering on the ship-borne video stream, superimposes the guide ship and the guide route, and performs ship detection, achieving the effect of integrated assisted driving.

[0084] Based on the above reasons, the present invention can be widely promoted in the fields of safe navigation assistance for unmanned ships, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0086] Figure 1 This is the overall framework diagram of the system of the present invention.

[0087] Figure 2 A schematic diagram of radar and AIS complementarity provided by an embodiment of the present invention.

[0088] Figure 3 A statistical weighted model provided by an embodiment of the present invention.

[0089] Figure 4 A data fusion diagram provided by an embodiment of the present invention.

[0090] Figure 5 A flow chart of three-dimensional scene generation provided by an embodiment of the present invention.

[0091] Figure 6 A flow chart of the ship collision risk provided by an embodiment of the present invention.

[0092] Figure 7 This is a block diagram of an enhanced display provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0093] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0094] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0095] like Figure 1 As shown, the present invention provides a method for enhancing the virtual-real fusion multi-twin intelligent perception of an unmanned ship, comprising:

[0096] S1, (data acquisition module) obtains data information;

[0097] S2, (data processing module) performs data fusion processing on the acquired data information, stores the fusion results in a database and constructs a model database;

[0098] S3, (assisted driving module) transmits the data in the model database to the electronic chart display platform, the three-dimensional vision display platform and the virtual-reality fusion display platform to realize assisted driving of the unmanned ship.

[0099] In specific implementation, as a preferred embodiment of the present invention, in step S1, the acquired data information includes environmental information, own ship information and other ship information, wherein:

[0100] Environmental information includes weather information, ocean current information, waterway information, and buoy information; a variety of sensors and equipment jointly obtain environmental information for unmanned ship navigation. Sensor equipment for obtaining environmental information includes depth sounders for measuring water depth, wind speed and direction meters for measuring wind speed and direction, current meters for measuring water flow speed and direction, wave meters for measuring sea wave data, and electronic nautical charts for providing waterway information;

[0101] The ship's information includes data information generated when the ship is sailing, which is used to reflect the ship's navigation status; the equipment for obtaining the ship's information includes a gyrocompass for providing the unmanned ship's orientation, a GPS for positioning the ship and providing accurate longitude and latitude information, a speedometer for measuring the voyage and performing track calculation, and a rudder angle feedback device for obtaining feedback information from the unmanned ship's automatic rudder for optimization and adjustment;

[0102] Other ship information includes data information of target ships detected within a certain range, and the equipment for obtaining target ship data information includes radar and AIS.

[0103] The data collected by the shipborne sensing equipment is processed, mainly for data fusion of AIS and radar information. In weather conditions with large waves, the radar is prone to form false echoes, resulting in loss errors in tracking the target. AIS does not have the above problems, but AIS can only detect targets equipped with AIS equipment. Therefore, in summary, the fusion of radar information and AIS information can not only better obtain the target track, but also suppress the interference of clutter to improve the reliability of perception. This paper mainly performs data fusion on the redundant parts of AIS and radar data. Figure 2 Schematic diagram of redundant complementation of AIS and radar data. In specific implementation, as a preferred embodiment of the present invention, in step S2, data fusion processing is performed on the acquired data information, mainly data fusion is performed on the redundant parts of the target ship data information acquired by radar and AIS, specifically as follows:

[0104] Since radar uses polar coordinates for sensing the location information of the target and AIS uses geodetic coordinates, spatial unification is required. And because radar and AIS have different sampling periods for the target, temporal unification is also required on the basis of spatial unification.

[0105] Space Unification:

[0106] The Mercator projection is used to convert the AIS information coordinates. The Mercator projection formula is as follows:

[0107]

[0108]

[0109] Among them, a represents the major axis of the ellipsoid, and the value of a is 6378137m; b represents the minor semi-axis of the spheroid, and its value is approximately 6356752.3142m; represents the base latitude; λ represents the longitude, in radians; represents latitude, the unit is unit radian; λ0 represents the central meridian and the central longitude is 0; represents the radius of curvature of the yoke; e represents the first eccentricity and e · represents the second eccentricity and

[0110] The radar information format is converted from the polar coordinate system to the rectangular coordinate system. The conversion formula is as follows:

[0111]

[0112] Where R represents the distance of the radar target at time k; θ(k) represents the azimuth of the radar target at time k; x(k) represents the x-axis component of the radar target at time k in the rectangular coordinate system; y(k) represents the y-axis component of the radar target at time k in the rectangular coordinate system;

[0113] Time Unification:

[0114] Since the radar and AIS have different sampling periods, the data collected by them are interpolated to achieve time alignment. Since the motion of targets at sea is mostly approximate to linear motion, the error accuracy of different interpolation algorithms is analyzed, so this embodiment uses a linear interpolation algorithm to achieve time alignment, as follows:

[0115] On the interval [a,b], given n+1 nodes: a=t0<t1<…<t n-1 <t n =b and corresponding to y0,y1,…,y n , let h i =t i+1 -t i , The interpolation function is S h (t), which is a broken line function. In any small interval [t i ,t i+1 ] can be expressed as t i <t<t i+1 ;

[0116] The basis function method is used to construct a piecewise linear function. h (t) is Among them l j (t) is the Langrange interpolation basis function, and the formula of the Langrange interpolation basis function is as follows:

[0117]

[0118] In order to speed up the data fusion efficiency, since the information of the ship is only correlated with the ships within a certain time and distance range from the ship, the track association algorithm first performs coarse time association and coarse space association to narrow the association range and improve the efficiency of the algorithm. Coarse time association mainly refers to the AIS information within the set time threshold range based on the radar's perception time. Coarse space association refers to taking the radar center as the origin and only making association judgments on AIS targets within the radar range to reduce the traversal range. Fine association is performed on targets that meet the coarse association requirements. This embodiment adopts a distance-speed comprehensive fine association algorithm to perform association based on the same motion characteristics of the same target within a certain period of time, as follows:

[0119] Select a track within a certain sampling time, and calculate the difference in position and speed between the corresponding points of the radar and AIS in the x and y directions. The target with the smallest difference is considered to be the associated target. The measurement matrix of the radar to the target is known to be: The measurement matrix of AIS to the target is: The calculation formula is as follows:

[0120]

[0121] in, Where Dx ij ,Dy ij Respectively represent the average speed difference between radar target i and AIS target j in the x and y directions at N measurement points, The radar tracking target i and AIS tracking target j that meet the minimum requirement are determined to be associated; the track fusion algorithm is based on the statistical weighted criterion, and the extended Kalman filter is used to process the fusion result to make the track closer to the real track;

[0122] The accuracy of weight distribution for multi-sensor data fusion determines the accuracy of data fusion. Suppose there are n sensing devices, and the fusion diagram is as follows: Figure 3 As shown: Among them, z i (i=1,2,3,...,n) is the measured value, is the weight, and the measurement value of the i-th sensor among n sensors is z i =x i +e i , where z i is the measured value, x i is the true value, e i is the error value. Since there is no interference between the sensors and they are independent of each other, they all obey the normal distribution, as shown in the following formula:

[0123]

[0124] Statistical weighting is to multiply the weighed and measured values ​​and add them up to get an estimate To achieve Unbiased estimation needs to satisfy get

[0125] The minimum mean square error is used to allocate weights. The formula is as follows:

[0126]

[0127] Convert it into a linear programming problem as follows:

[0128]

[0129]

[0130] For linear programming, the Lagrange multiplier method is used and an auxiliary function is constructed to solve the problem. The auxiliary function is set as The optimal weight and minimum variance are calculated as follows:

[0131]

[0132] Data Fusion:

[0133] The data of distance, direction, speed and heading are fused, and the fusion formula is as follows:

[0134]

[0135] in, is the radar and AIS distance variance, is the azimuth variance, They are speed variance and heading variance respectively;

[0136] In order to improve the fusion accuracy, the fusion result is processed by extended Kalman filtering. The extended Kalman filtering formula is as follows:

[0137]

[0138] in, is the state prediction mean, P k|k-1 is the state prediction variance, K k is the Kalman gain, is the mean of the target posterior, P k is the variance of the target posterior, and the state vector of the ship motion model is x = [X, Y, SOG, COG] T , X, Y, SOG, COG are the horizontal and vertical coordinates, speed, and heading respectively, and the ship motion system equation is as follows:

[0139]

[0140] Among them, the observation matrix is ​​z = [D, B, SOG, COG] T , D, B, SOG, COG are distance, azimuth, speed, and heading respectively. The observation equation is as follows:

[0141]

[0142] Because the system equation is nonlinear, the extended Kalman filter is introduced to make it closer to the true value. The fusion result is as follows Figure 4 shown.

[0143] In specific implementation, as a preferred embodiment of the present invention, in step S3:

[0144] The electronic chart display platform is used as a basic navigation aid display system to realize chart operation and route correction, and to extract the channel and buoy information of the electronic chart in real time, and send the real-time extracted information to the three-dimensional visual platform to complete data supplement;

[0145] The 3D visual platform uses Unity as a development engine to build a 3D virtual scene, and implements perception instantiation, weather system and risk warning functions to achieve the effect of intelligent perception 3D visualization;

[0146] The virtual-reality fusion display platform renders the navigation images captured by the ship-borne camera, recognizes the ships within the observation range through the YOLOv3 algorithm, and renders the guiding ships and guiding routes, so that the shore operator can judge the navigation safety of the unmanned ship through intuitive guidance information and identification borders, thereby realizing the function of assisted driving.

[0147] In specific implementation, as a preferred embodiment of the present invention, the 3D model in the 3D visual platform is constructed in 3D by Cinema 4D software, and then rendered by texture, and the model is exported in fbx format, loaded into the Unity engine, and set as a prefab to reduce resource utilization. The scene construction is extended with the real terrain as the base map; Figure 5 As shown, in the three-dimensional visual platform, the process of generating a three-dimensional visual is as follows:

[0148] Intercept the terrain range that needs to be generated in the satellite map;

[0149] Get its elevation map and convert it into raw data format;

[0150] Create Terrain, merge it with the heightmap through HeightMapper, and set the matching ratio;

[0151] The terrain is rendered with textures, and resources are obtained from satellite images; the details of the grass are constructed and textured. The ocean water effect is realized by stealing resource packages and adding shaders; the weather effects of rain, snow and fog are realized by particle systems.

[0152] A model database is constructed in the virtual vision, a three-dimensional model database is constructed according to the AIS ship and cargo types, the data results of the AIS and radar fusion are instantiated, the AIS message is parsed and matched with the three-dimensional model database, and the perception instantiation is realized in the virtual vision. For the presence of multiple targets in complex sea areas, an interface is provided for the operator to realize the situation monitoring of important targets. In the specific implementation, as a preferred embodiment of the present invention, the virtual-reality fusion display platform constructs a model database in the virtual vision, a three-dimensional model database is constructed according to the AIS ship and cargo types, the data results of the AIS and radar fusion are instantiated, the AIS message is parsed and matched with the three-dimensional model database, and the perception instantiation is realized in the virtual vision; the details are as follows:

[0153] In order to experience the navigation safety of unmanned ships from a three-dimensional perspective, the unmanned ship danger warning function is realized in a graphical way. The navigation information of the ship and the status information of the target ship are comprehensively analyzed to generate a ship collision risk model and trajectory calculation based on physical information. The three-dimensional visualization presentation allows the operator to more intuitively experience the operation safety of the unmanned ship. The ship collision risk model mainly considers TCPA and DCPA, and visualizes their risk levels. The calculation process is as follows: Figure 6 As shown:

[0154] Calculate the minimum encounter distance, minimum encounter time and distance between two ships, set three levels of risk, and render corresponding colors according to different risk levels to make it more intuitive in the virtual vision;

[0155] Design the UI interface and complete the interactive operation, mainly including multi-angle observation of the ship's motion status, divided into front, back, left, right and third-person perspectives, let Unity build in several cameras, each camera corresponds to the corresponding angle, set the camera's depth, select perspective projection as the projection method, and adjust the field of view parameters to adjust the camera's viewing angle; the main perspective camera is a follow camera, which can follow the target movement and can achieve close-up, far-reaching, and 360-degree observation;

[0156] The electronic chart display platform is used as the data supplement interface of the virtual vision display platform. The data of the electronic chart is extracted and sent to the virtual vision server to supplement the static navigation information of the virtual vision, including isobaths and channel information.

[0157] In specific implementation, as a preferred embodiment of the present invention, the virtual-reality fusion display platform includes three submodules: visual guidance of unmanned ships, visual guidance of routes, and ship recognition of navigation images; wherein:

[0158] Visual guidance of unmanned ships is to integrate the information of virtual guidance of unmanned ships with the real world, and superimpose the real navigation environment and virtual guidance of unmanned ships in the same screen or space in real time. The visualization system provides information that is different from what humans can perceive under normal circumstances. The visualization display system will integrate all ship navigation information and guide the physical unmanned ship to sail safely; the generated guidance of unmanned ships will be superimposed on its real navigation video, so that the operator can intuitively feel the navigation status of the physical unmanned ship through virtual and real interaction. The visualization guidance system not only displays the information of the real world, but also displays the virtual guidance information at the same time. The two kinds of information complement and superimpose each other to achieve the guidance effect. The guidance of unmanned ships continuously guides the physical unmanned ships to move and ensure navigation safety. This system transmits the images collected by the unmanned ship's onboard camera to the shore command center, projects it to UI-image, collects the motion status of the physical unmanned ship through multiple sensors such as inertial navigation, including speed, heading, and pitch angle, and uses Unity's built-in camera to control its perspective and status through scripts to simulate the real ship-borne camera.

[0159] In specific implementation, as a preferred embodiment of the present invention, the specific process of the visually guided unmanned ship guiding the physical unmanned ship to navigate safely is as follows:

[0160] The rendering process is as follows Figure 7As shown, the visualization guiding unmanned ship rendering is based on the Unity engine. The optimal decision path for this task is extracted from the database and converted into world coordinates. In order to distinguish the static planning route from the guiding unmanned ship route, the static route is set as three lineRenderer components, and its width is increased. The spacing between each component is 0.5m, and the color is set to red. The path of the guiding unmanned ship is set to green, and the guiding unmanned ship is instantiated as purple. Bezier optimization is performed on its path. The linear formula of the Bezier curve is as follows:

[0161] B(t)=P0+(P1-P0)t=(1-t)P0+tP1,t∈[0,1]

[0162] Use scripts to modify the properties of the LineRenderer component and set the interpolation points to draw a continuous smooth curve. In order to increase visibility, set the local path to dotted green. First, prepare a transparent background dotted line png, with a transparent background instead of a white background, import it into the Unity project, create a new material, modify its shader to the newly imported png, adjust the Tiling and Offset parameters to make the dotted line suitable, and set the LineRenderer component material to the newly created material. At this time, you can display the real video stream in Unity and display the route of the unmanned ship at the corresponding position;

[0163] In order to achieve the effect of safe navigation, it is necessary to display the predicted position of the guide ship in the real video stream. The position of the guiding unmanned ship is set to the position predicted for the optimal decision path of the unmanned ship for thirty seconds. The position and tangent angle of a certain point on the Bezier curve are calculated. n points are taken on the Bezier curve, 50 points are taken by default, the length of the straight line between the points is calculated, and the sum is added to obtain the approximate length of a curve. The more points are taken, the more accurate the length tends to be. When the Bezier curve expression is known, in order to obtain the tangent of a certain point, the curve is derived, and the following can be obtained:

[0164] B(t)=P0*3(1-t) 2 *(-1)+3P1*((1-t) 2 +t*2(1-t)*(-1))+3P2*(2t(1-t)+t 2 *(-1))+P3*3t 2

[0165] Rearranging the above formula, we can get:

[0166] B(t)=-3P0(1-t) 2 +3P1((1-t) 2 -2t(1-t))+3P2(2t(1-t)-t 2 )+3P3t2

[0167] The angle between the two vectors of the parameters thirty seconds and thirty-one seconds is the angle of the target ship at this location. The Unity implementation function is as follows:

[0168] Mathf.Atan(ax / bz)*Mathf.Rad2Deg+del tan Angle

[0169] Among them, a is the thirty-second position point vector of the guiding unmanned ship, and b is the position of the guiding unmanned ship at the next moment, that is, the heading of the guiding unmanned ship at this moment is obtained, and it can be converted into a four-tuple to instantiate the guiding unmanned ship.

[0170] In specific implementation, as a preferred embodiment of the present invention, the navigation images collected by the ship-borne camera are analyzed to identify the target ship, which has a certain safety warning effect. This system uses the Darknet network and combines it with YOLOV3 to identify it. In order to reduce the collection of data sets, this system uses the three-dimensional model in the virtual scene as a data set to greatly improve work efficiency. It is trained to build a network model, and a convolution operation is performed to extract ship features. Each frame of the video from the ship-borne camera is processed to achieve ship detection.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for enhancing the multi-twin intelligent perception of unmanned ships by fusion of virtual and real, characterized in that: include: S1. Obtain data information; In step S1, the acquired data information includes environmental information, own ship information and other ship information, wherein: Environmental information includes weather information, ocean current information, waterway information, and buoy information; a variety of sensors and equipment jointly obtain environmental information for unmanned ship navigation. Sensor equipment for obtaining environmental information includes depth sounders for measuring water depth, wind speed and direction meters for measuring wind speed and direction, current meters for measuring water flow speed and direction, wave meters for measuring sea wave data, and electronic nautical charts for providing waterway information; The ship's information includes data information generated when the ship is sailing, which is used to reflect the ship's navigation status; the equipment for obtaining the ship's information includes a gyrocompass for providing the unmanned ship's orientation, a GPS for positioning the ship and providing accurate longitude and latitude information, a speedometer for measuring the voyage and performing track calculation, and a rudder angle feedback device for obtaining feedback information from the unmanned ship's automatic rudder for optimization and adjustment; Other ship information includes data information of target ships detected within a certain range. The equipment for obtaining data information of target ships includes radar and AIS; S2, performing data fusion processing on the acquired data information, storing the fusion results in a database and constructing a model database; In step S2, data fusion processing is performed on the acquired data information, mainly data fusion is performed on the redundant parts of the target ship data information acquired by radar and AIS, specifically as follows: Space Unification: The Mercator projection is used to transform the AIS information coordinates. The Mercator projection formula is as follows: Among them, a represents the major axis of the ellipsoid, and the value of a is 6378137m; b represents the minor semi-axis of the spheroid, and its value is 6356752.3142m; represents the base latitude; λ represents the longitude, in radians; represents latitude, the unit is unit radian; λ0 represents the central meridian and the central longitude is 0; represents the radius of curvature of the yoke; e represents the first eccentricity and e. represents the second eccentricity and The radar information format is converted from the polar coordinate system to the rectangular coordinate system. The conversion formula is as follows: Where R represents the distance of the radar target at time k; θ(k) represents the azimuth of the radar target at time k; x(k) represents the x-axis component of the radar target at time k in the rectangular coordinate system; y(k) represents the y-axis component of the radar target at time k in the rectangular coordinate system; Time Unification: The linear interpolation algorithm is used to achieve time alignment, as follows: On the interval [a,b], given n+1 nodes: a=t0<t1<…<t n-1 <t n =b and corresponding to y0,y1,…,y n , let h i =t i+1 -t i , The interpolation function is S h (t), which is a broken line function. In any small interval [t i ,t i+1 ] is expressed as t i <t<t i+1 ; The basis function method is used to construct a piecewise linear function. h (t) is Among them l j (t) is the Langrange interpolation basis function, and the formula of the Langrange interpolation basis function is as follows: The distance-speed comprehensive fine association algorithm is used to associate the same target based on its same motion characteristics within a certain period of time, as follows: Select a track within a certain sampling time, and calculate the difference in position and velocity values ​​between the corresponding points of the radar and AIS in the x and y directions. The target with the smallest difference is considered to be the associated target. The measurement matrix of the radar to the target is known to be: The measurement matrix of AIS to the target is: The calculation formula is as follows: in, Where Dx ij ,Dy ij Respectively represent the average speed difference between radar target i and AIS target j in the x and y directions at N measurement points, The minimum radar tracking target i and AIS tracking target j are determined to be associated; the track fusion algorithm is based on the statistical weighted criterion, and the extended Kalman filter is used to process the fusion result to make the track closer to the real track; The accuracy of weight distribution for multi-sensor data fusion determines the accuracy of data fusion. Suppose there are n sensing devices, z i (i=1,2,3,...,n) is the measured value, is the weight, and the measurement value of the i-th sensor among n sensors is z i =x i +e i , where z i is the measured value, x i is the true value, e i is the error value. Since there is no interference between the sensors and they are independent of each other, they all obey the normal distribution, as shown in the following formula: Statistical weighting is to multiply the weighed and measured values ​​and add them up to get an estimate To achieve Unbiased estimation needs to satisfy get The minimum mean square error is used to allocate weights. The formula is as follows: Convert it into a linear programming problem as follows: For linear programming, the Lagrange multiplier method is used and an auxiliary function is constructed to solve the problem. The auxiliary function is set as The optimal weight and minimum variance are calculated as follows: Data Fusion: The data of distance, direction, speed and heading are fused, and the fusion formula is as follows: in, is the radar and AIS distance variance, is the azimuth variance, They are speed variance and heading variance respectively; In order to improve the fusion accuracy, the fusion result is processed by extended Kalman filter. The extended Kalman filter formula is as follows: in, is the state prediction mean, P k|k-1 is the state prediction variance, K k is the Kalman gain, is the mean of the target posterior, P k is the variance of the target posterior, and the state vector of the ship motion model is x = [X, Y, SOG, COG] T , X, Y, SOG, COG are the horizontal and vertical coordinates, speed, and heading respectively, and the ship motion system equation is as follows: Among them, the observation matrix is ​​z = [D, B, SOG, COG] T , D, B, SOG, COG are distance, azimuth, speed, and heading respectively. The observation equation is as follows: S3, transmitting the data in the model database to the electronic chart display platform, the three-dimensional scene display platform and the virtual-reality fusion display platform to realize the assisted driving of the unmanned ship; In step S3: The electronic chart display platform is used as a basic navigation aid display system to realize chart operation and route correction, and to extract the channel and buoy information of the electronic chart in real time, and send the real-time extracted information to the three-dimensional visual display platform to complete data supplement; The three-dimensional visual display platform uses Unity as a development engine to build a three-dimensional virtual scene, and implements perception instantiation, weather system and risk warning functions to achieve the effect of intelligent perception three-dimensional visualization; The 3D model in the 3D visual display platform is constructed in 3D by Cinema 4D software, and then rendered by texture. The model is exported in fbx format, loaded into the Unity engine, and set as a prefab to reduce resource utilization. The scene construction is extended with the real terrain as the base map. In the 3D visual display platform, the process of generating 3D visual is as follows: Intercept the terrain range that needs to be generated in the satellite map; Get its elevation map and convert it into raw data format; Create Terrain, merge it with the heightmap through HeightMapper, and set the matching ratio; The terrain is rendered with textures, and resources are obtained from satellite images. The details of the grass are constructed and textured. The ocean water effect is realized by stealing resource packages and adding shaders. The weather effects of rain, snow and fog are realized by using particle systems. The virtual-reality fusion display platform renders the navigation images collected by the ship-borne camera, recognizes the ships within the observation range through the yolov3 algorithm, and renders the guided ships and the guided routes, so that the shore operator can judge the navigation safety of the unmanned ship through intuitive guidance information and identification borders, thus realizing the function of assisted driving; The virtual-reality fusion display platform constructs a model database in the virtual scene, constructs a three-dimensional model database according to the AIS ship and cargo type, instantiates the data results of AIS and radar fusion, parses the AIS message and matches it with the three-dimensional model database, and realizes perception instantiation in the virtual scene; the details are as follows: In order to experience the safety of unmanned ship navigation from a three-dimensional perspective, the unmanned ship danger warning function is realized in a graphical way, the navigation information of the ship and the status information of the target ship are comprehensively analyzed, and a ship collision risk model and trajectory calculation based on physical information are generated; the three-dimensional visualization presentation allows the operator to more intuitively observe the operation safety of the unmanned ship; the ship collision risk model mainly considers TCPA and DCPA, and its risk level is visualized; Calculate the minimum encounter distance, minimum encounter time and distance between two ships, set three levels of risk, and render corresponding colors according to different risk levels to make it more intuitive in the virtual vision; Design the UI interface and complete the interactive operation, mainly including multi-angle observation of the ship's motion status, divided into front, back, left, right and third-person perspectives, let Unity build in several cameras, each camera corresponds to the corresponding angle, set the camera's depth, select perspective projection as the projection method, and adjust the field of view parameters to adjust the camera's viewing angle; the main perspective camera is a follow camera, which can follow the target movement and can achieve close-up, far-reaching, and 360-degree observation; The electronic chart display platform is used as a data supplement interface for the virtual vision display platform. The data of the electronic chart is extracted and sent to the virtual vision server to supplement the static navigation information of the virtual vision, including isobaths and channel information. The virtual-reality fusion display platform includes three submodules: visual guidance of unmanned ships, visual guidance of routes, and ship recognition of navigation images; wherein: Visual guidance of unmanned ships is to integrate the information of virtual guidance of unmanned ships with the real world, and superimpose the real navigation environment and the virtual guidance of unmanned ships in the same screen or space in real time. The visualization system provides information that is different from human perception in general. The visualization display system will integrate all ship navigation information and guide the physical unmanned ships to sail safely. Specifically, the images collected by the onboard camera of the unmanned ship are transmitted to the shore-side command and control center, projected onto UI-image, and the motion state of the physical unmanned ship, including speed, heading, and pitch angle, is collected through the inertial navigation multi-sensor. The built-in camera of Unity is used to control its viewing angle and state through scripts to simulate the real ship-borne camera. The specific process of visually guiding the unmanned ship to guide the physical unmanned ship to navigate safely is as follows: The visualization guiding unmanned ship rendering is based on the Unity engine. The optimal decision path for this task is extracted from the database and converted into world coordinates. In order to distinguish the static planning route from the guiding unmanned ship route, the static route is set as three lineRenderer components, and its width is increased. The spacing between each component is 0.5m, and the color is set to red. The path of the guiding unmanned ship is set to green, and the guiding unmanned ship is instantiated as purple. Bezier optimization is performed on its path. The linear formula of the Bezier curve is as follows: B(t)=P0+(P1-P0)t=(1-t)P0+tP1,t∈[0,1] Use scripts to modify the properties of the LineRenderer component, set the interpolation points, and draw continuous smooth curves. In order to increase visibility, set the local path to dotted green. First, prepare a transparent background dotted line png, with a transparent background instead of a white background, import it into the Unity project, create a new material, modify its shader to the newly imported png, adjust the Tiling and Offset parameters to make the dotted line suitable, and set the LineRenderer component material to the newly created material. At this time, both the real video stream and the guiding route of the unmanned ship are displayed in Unity at the corresponding position; In order to achieve the effect of safe navigation, it is necessary to display the predicted position of the guide ship in the real video stream. The position of the guiding unmanned ship is set to the position predicted for the optimal decision path of the unmanned ship for thirty seconds. The position and tangent angle of a certain point on the Bezier curve are calculated. n points are taken on the Bezier curve, 50 points are taken by default, the length of the straight line between the points is calculated, and the sum is added to obtain the approximate length of a curve. The more points are taken, the more accurate the length tends to be. When the Bezier curve expression is known, in order to obtain the tangent of a certain point, the curve is derived as follows: B(t)=P0*3(1-t) 2 *(-1)+3P1*((1-t) 2 +t*2(1-t)*(-1)) +3P2*(2t(1-t)+t 2 *(-1))+P3*3t 2 Rearranging the above formula, we get: B(t)=-3P0(1-t) 2 +3P1((1-t) 2 -2t(1-t)) +3P2(2t(1-t)-t 2 )+3P3t 2 The angle between the two vectors of the parameters thirty seconds and thirty-one seconds is the angle of the target ship at this location. The Unity implementation function is as follows: Mathf.Atan(ax / bz)*Mathf.Rad2Deg +deltanAngle Among them, a is the thirty-second position point vector of the guiding unmanned ship, and b is the position of the guiding unmanned ship at the next moment, that is, the heading of the guiding unmanned ship at this moment is obtained and converted into a four-tuple to instantiate the guiding unmanned ship.

Images