A ship motion visualization method based on digital twinning

Abstract

The application relates to a ship motion visualization method based on digital twinning, comprising the following steps: acquiring ship information in reality from multiple data sources, and acquiring ship state data by analysis; based on the ship state data, loading a preset ship model and an environment model matched with the ship position; based on the current ship state data, updating the motion state of a digital twin ship through Kalman filtering, and dynamically adjusting the motion state update frequency of the digital twin ship in the updating process; acquiring observation viewpoint information, and outputting a visualization signal to a visualization terminal based on the ship model, the environment model, viewpoint observation selection information and the motion state of the digital twin ship. Compared with the prior art, the application can smoothly transition when updating the state of the digital twin ship, and is low in cost and convenient to maintain.

Description

TECHNICAL FIELD

[0001] The application relates to the field of water transportation, and in particular to a ship motion visualization method based on digital twinning. BACKGROUND

[0002] The application of digital twinning technology in ship navigation and operation has attracted widespread attention in the field of water transportation. A method and device for constructing an inland waterway digital twinning scene are disclosed in a Chinese patent document CN113223162, which discloses a method for law enforcement management by constructing a three-dimensional waterway. The focus is on the three-dimensional construction of the waterway, and the management of waterway congestion is not considered. A digital twinning waterway construction method and system are disclosed in a Chinese patent document CN114529680A, which discloses operation management in a waterway digital twinning scene after accessing waterway ubiquitous perception data. The focus is on data access, and the redundancy and screening of multiple data are not fully considered.

[0003] Chinese patent application CN202111190105.4 provides a digital twinning ship driving method and device, which relates to the field of water transportation. The method includes using a data stream engine to clean data messages, removing abnormal data messages in the data messages based on the vector data of the waterway center line and the waterway shoreline, and the abnormal data messages are the data messages indicating that the positioning of the ship is outside the waterway shoreline. The initialization request sent from the client is received, the data messages are encapsulated based on the initialization request and returned to the client, so that the client updates the position of each ship in the digital twinning scene based on the data messages. The application discloses cleaning the ship data messages and removing the abnormal data of the ship, but does not consider the consistency of the sampling period of the data and the frame update frequency in virtual reality technology. The track smoothing algorithm does not consider the stability of ship motion in virtual reality, which can easily cause the lag of ship motion picture.

[0004] In summary, there is currently a lack of a ship motion state monitoring method to solve the problem that the existing method does not consider the stability of ship motion in virtual reality, which can easily cause the lag of ship motion picture. SUMMARY

[0005] The purpose of the present application is to overcome the defects of the prior art and provide a ship motion visualization method based on digital twinning. The ship information in reality is obtained from multiple data sources and analyzed. The motion state of the digital twinning ship is updated by Kalman filtering, and the motion state update frequency of the twinning ship is dynamically adjusted to ensure smooth transition of the motion state of the twinning ship during motion state update, reduce data rollback and jump.

[0006] The object of the present application can be realized by the following technical solutions.

[0007] The present application provides a ship motion visualization method based on digital twinning, comprising the following steps:

[0008] Obtain ship information in reality from multiple data sources, and obtain ship state data by analysis;

[0009] Based on the ship state data, load a preset ship model and an environment model matching the ship position;

[0010] Based on the current ship state data, update the motion state of the digital twin ship through Kalman filtering, and dynamically adjust the motion state update frequency of the twin ship during the updating process;

[0011] Obtain observation point information, and output a visualization signal to a visualization terminal based on the ship model, the environment model, the observation point information, and the motion state of the digital twin ship.

[0012] As a preferred technical solution, the implementation process of dynamically adjusting the motion state update frequency of the twin ship comprises the following steps:

[0013] Calculate the interval time T between the current obtained ship information and the last obtained ship information, select an appropriate twin ship motion state update frequency f based on the interval time T, so that the state update interval t of the digital twin ship is the same as the interval time T, wherein n is a positive integer.

[0014] As a preferred technical solution, the process of updating the motion state of the digital twin ship through Kalman filtering comprises the following steps:

[0015] Based on the ship state data in reality, obtain the initial state value of the digital twin ship;

[0016] Based on the initial state value, obtain the predicted state value of the digital twin ship through Kalman filtering to realize the update of the motion state of the digital twin ship.

[0017] As a preferred technical solution, the initial state value of the digital twin ship is obtained by the following formula:

[0018] X0=CS0

[0019] In the formula, X0 is the initial state value of the digital twin ship, C is a dimension conversion matrix, S0 is the ship state data in reality, and the ship state data in reality includes ship position coordinates and ship attitude coordinates.

[0020] As a preferred technical solution, the prediction equation of the Kalman filter is:

[0021]

[0022]

[0023] The update equation for the Kalman filter is as follows:

[0024]

[0025]

[0026]

[0027]

[0028] in, Indicates t k The estimated value of time, Indicates by t k-1 Time estimate The obtained t k The predicted value at time t is given by F, which represents the state transition matrix, and H, which represents the measurement matrix. The subscripts k-1 and k indicate the change at time t. The broken line symbol above the letter indicates that the current letter is an estimated value. Indicates t k-1 Time to t k The predicted value of the state covariance at time t, Q k-1,k-1 For t k-1 The process noise covariance at time step, R is the variance of Gaussian measured white noise, Z is... k For t k The target observation value at time t, I is the identity matrix, y represents the difference between the predicted and measured values, and K is generally called the Kalman gain coefficient.

[0029] As a preferred technical solution, the visualization signal also includes an electronic nautical chart that matches the location of the observation viewpoint.

[0030] As a preferred technical solution, the following are also included:

[0031] Determine whether the visualization terminal has multiple displays. If so, expand the multiple displays and set multiple viewports. By modifying the observation projection matrix of each viewport, send visualization signals from different observation viewpoints to the visualization terminal.

[0032] As a preferred technical solution, the environmental model includes at least one of the following: sky model, water area model, port terrain and building model, and waterway beacon model.

[0033] As a preferred technical solution, the ship state data comprises ship identification data and ship position data, wherein the ship identification data comprises at least one of a ship identification code, a ship length, and a ship width, and the ship position data comprises at least one of latitude and longitude, a heading, and a speed.

[0034] As a preferred technical solution, the plurality of data sources comprise a real-time transmission data source and an offline backup data source, wherein the data transmitted by the real-time transmission data source comprises one or more of AIS information, a log information, GPS information, wind direction and speed information, and a depth sounder information, and the offline backup data source is a VDR device and / or a receiving base station of the ship.

[0035] Compared with the prior art, the present application has the following advantages:

[0036] (1) The digital twin ship state can be smoothly transitioned: the ship information in reality is obtained from a plurality of data sources and is analyzed, the motion state of the digital twin ship is updated through Kalman filtering, and the motion state update frequency of the digital twin ship is dynamically adjusted, combined with the observation viewpoint information, and the final visual signal is output. The present method updates the state through Kalman filtering to obtain the predicted state of the digital twin ship, and dynamically adjusts the state update frequency to ensure smooth transition of the motion state of the digital twin ship during motion state update, thereby reducing data rollback and jump.

[0037] (2) Low cost and convenient maintenance: when a traditional method uses a camera and other sensors to construct a remote monitoring system, camera hardware devices need to be added at multiple positions of the ship. A large number of camera sensor hardware devices of a ship need to be connected through a data transmission line to construct a network system, the network system needs to be sent to the nearest network node through a signal sending terminal, and multiple network sending nodes of the ships need to construct a stable large-scale data transmission system. All of these require a large amount of time cost and economic cost. Compared with directly using a camera and other sensors for remote monitoring, the present method uses existing hardware devices on the ship to obtain the real-time three-dimensional motion state of the ship by using AIS data, thereby reducing the investment of economic cost and time cost. The present application can be used as a transition for real remote monitoring of unmanned ships and intelligent ships. The three-dimensional ship motion state generated by the backup VDR data of the ship can be used as a basis for post-hoc navigation analysis of the ship. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The flowchart of the ship motion visualization method based on digital twin in Example 1 is shown in the figure.

[0039] Figure 2 The schematic diagram of the data source and the terminal is shown in the figure.

[0040] Figure 3 Fig. 1 is a schematic diagram of a viewing angle in a constructed virtual reality three-dimensional scene;

[0041] Figure 4 Fig. 2 is a schematic diagram of a virtual scene presented on a small mobile terminal device;

[0042] Figure 5 Fig. 3 is a schematic diagram of a virtual scene presented on a large terminal device;

[0043] Figure 6 Fig. 4 is a schematic diagram of a digital twin ship motion state synchronization algorithm. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor should fall within the scope of protection of the present application.

[0045] Embodiment 1

[0046] At the present stage, a ship can be monitored in two dimensions in real time by using an automatic identification system (AIS) in combination with an electronic chart and a marine radar, and the navigation situation around the ship can be monitored by using a camera sensor device. The present application can construct the motion state of a twin ship by accessing real-time and backup ship navigation data, can assist a shipping company in remotely monitoring the running state of the ship, can help a traffic department in supervising the traffic flow in the waters within the jurisdiction, and can serve as an analysis basis for a maritime research department to analyze the ship traffic situation, such as Figure 1 The present embodiment provides a ship motion visualization method based on digital twinning, which comprises the following steps:

[0047] Step S1, a software program starts running, and a data source is selected as needed, the data source being composed of real-time transmission data and offline record backup data, the real-time data being used for remote monitoring, and the backup data being used for post-navigation analysis. The real-time transmission data usually includes AIS information, a log information, GPS information, wind direction and speed information, and a depth sounder information. The offline record backup data is stored in a VDR device end of a navigation ship and a hard disk storage device of each receiving base station, and the receiving base stations are usually distributed near each port or along a waterway.

[0048] Step S2, due to the different internal protocols of each data source, the selected data source needs to be parsed, and the identification information of the ship (ship identification code, ship length, ship width) and the key position information (latitude, longitude, heading, speed) are the basic data that must be parsed.

[0049] Step S3, the dynamic loading model operation refers to the operation of dynamically loading the sky model, water area model, port terrain building model, channel beacon model and ship model to the model display engine according to the parsed data, the sky model and water area model are generated through simulation data, and the port terrain building model, beacon model and ship model need to be drawn through a three-dimensional modeling tool software, at least two ship models of bulk cargo ship and container ship need to be drawn, a large number of three-dimensional virtual ship models can be generated by scaling the size of the two models, and a three-dimensional model display engine can be selected from a commercial or free open source engine development package for secondary development. The twin ship operation is the key of the present application, the state of the twin ship needs to be calculated according to the content of the data source, the initial real state of the ship and the real updated state of the ship are parsed from the data source, and the motion state of the twin ship is analyzed and calculated by using an optimization algorithm.

[0050] Figure 6 The figure shows a digital twin ship motion state synchronization algorithm diagram, the combination of T axis and Data axis represents the transmission of real ship state data in time sequence, the combination of t axis and Data axis represents the twin of virtual ship state data in time sequence, Delay represents the delay between the real world and the virtual simulation, and the delay time is represented by t d , T1 represents the interval time between time A and time B when the ship state data is sent, that is, T1 can represent the transmission frequency of the ship state data, t f represents the frame update frequency of the virtual simulation engine. Due to the limitation of network transmission resources and record backup file space, T1 in the real world is in seconds, and the AIS receiver can receive AIS data every 2-3 seconds, and the time interval of the data stored by the maritime server is longer. In virtual reality technology, in order to meet the continuity of naked eye observation, at least 25 frames of video animation need to be played per second, and the playing time of each frame does not exceed 0.04 seconds. In order to improve the running efficiency of CPU and GPU, the virtual engine on the market usually adopts a non-fixed rate to update the frame picture to maintain the update of the scene. t1 represents the interval time between time a and time b when the data is received in the twin system, and in the stable state, T1=t1. In the twin virtual system, in order to ensure the smooth transition of the motion state of the twin ship in the t1 time period, the received data needs to be smoothed in a reasonable way, and the processed data needs to be smoothly transitioned with the data received at time b to prevent data rollback and jump. For the data in the read record backup file, the present application adjusts the frame update frequency t fThe magnitude of the value determines the frequency of updating the twin ship's motion state. This invention uses Kalman filtering to calculate the received actual ship state data to update the twin ship's motion state.

[0051] The basic equation of Kalman filtering is:

[0052] predict

[0053]

[0054]

[0055] renew

[0056]

[0057]

[0058]

[0059]

[0060] in, Indicates t k The estimated value of time, Indicates by t k-1 Time estimate The obtained t k The predicted value at time t is given by F, which represents the state transition matrix, and H, which represents the measurement matrix. The subscripts k-1 and k indicate the change at time t. The broken line symbol above the letter indicates that the current letter is an estimated value. Indicates t k-1 Time to t k The predicted value of the state covariance at time t, Q k-1,k-1 For t k-1 The process noise covariance at time step, R is the variance of Gaussian measured white noise, Z is... k For t k The target observation value at time t, where I is the identity matrix. According to the Kalman filter equation, given the initial values... In the case of P0, as time t k Measurement Z k The derivation of t can be obtained through recursive calculation. k State estimation at time 1

[0061] Data represents the ship's motion state parameters S = [Lon, Lat, Height, HPR, GV, ROT, T], where (Lon, Lat, Height) represent the ship's position coordinates, and (H, P, R) represent the ship's attitude coordinates (Heading, Pitch, Roll). During actual navigation, the ship's rolling attitude is affected by natural environmental factors such as wind, current, and waves. The rolling motion of the twin ship in the virtual visual system can be simplified. The simplified ship motion state data of this invention is as follows: Ship motion measurement data is Z = [Lon Lat Heading] T Measurement matrix State transition matrix The initial state values ​​of the twin ship are obtained from the state data S0 = [Lon0 Lat0 Height0 H0 P0 R0 GV0 ROT0 T0] of the real ship obtained by parsing the AIS data source at time t0. The calculation process is X0 = CS0, where This is the dimension transformation matrix. In t... k-1 ~t k At a certain moment t in the time period i The predicted value obtained using the state transition matrix F As the motion state of the twin ship, t k The motion state of the twin ships is obtained using Kalman filtering.

[0062] Step S4, the viewpoint observation operation, refers to adjusting the position of the observation viewpoint in the virtual imaging software. In order to obtain a better observation angle, the observation viewpoint can be adjusted and moved in six degrees of freedom: up and down, left and right, forward and backward, and rotation.

[0063] like Figure 2The schematic diagram of the hardware in the present method is shown. The receiving antenna in the data source represents an AIS signal receiver hardware device or AIS data sent through a network and a serial port, and the hard disk storage device represents recorded data copied directly from a shipborne VDR device, an AIS server data backup device, or a receiving base station. The written virtual imaging software can run on large terminals and small terminals. The small terminals can be mobile communication devices such as mobile phones, tablets, and PDAs with display screens. The large terminals are computers equipped with multiple displays according to different display angle ranges, and three displays are shown in the figure. Different configurations of hardware graphics cards can be equipped with different numbers of displays, and each display can display virtual images at different viewpoint position angles. All these displays present virtual images at a large number of different viewpoint position angles. In three-dimensional virtual reality display technology, there are multiple ways to display multiple viewpoint positions using multiple displays. The present application identifies multiple displays as one whole GraphicsContext after using an extended display mode, sets multiple viewports on the whole GraphicsContext, and modifies the observation projection matrix of each viewport to achieve different observation angles for each viewport.

[0064] Figure 3 The schematic diagram of an observation angle is shown. The observation angle can be located at a position a certain distance away from the observed ship, and can be deflected at multiple angle positions to achieve the observation effect of top view, side view, and front view positions. Traditionally, to obtain real-time video images of multiple position angles of a ship, image sensors need to be pre-installed at corresponding positions of the ship and a transmission network needs to be established, or a mode of unmanned aerial vehicle aerial photography needs to be adopted. At the present stage, if real-time images of numerous ships in a navigation area are to be achieved, a large amount of hardware and software devices need to be invested, and the cost is huge. The present application uses the existing maritime AIS system to analyze real-time AIS data or backup ship VDR data to obtain the basic position and motion state of the ship, and constructs numerous digital twin ships in a virtual reality scene. E in the figure represents the observation lens position, the three coordinate axes (Side, Front, Up) represent the observation angle, frame I is the display of the ship motion state and basic information, and the virtual ships within the observation lens range can display their related information as needed.

[0065] Figure 4The figure shows a virtual scene display interface, which can run on a small mobile terminal display device. The small terminal display device has a touch screen function, and the 6 degrees of freedom movement and rotation of the observation point position (left and right, front and back, up and down) are realized by using multi-point touch. The eagle eye diagram represents the overhead view of the observation point position or the electronic chart of the area near the current position, which displays the two-dimensional motion state and trajectory of the ship. The AIS information display interface is an optional display item. When a target ship appears within the observation point range, the static and dynamic information of the target ship will be displayed on the information display interface, and the information display interface can be canceled.

[0066] Figure 5 The figure shows a virtual scene display interface, which can run on a large terminal display device. There are many ways to realize the large display terminal by using multiple displays. Under the premise of considering the cost of hardware procurement, the invention uses a mode of installing a multi-output graphics card on multiple displays with a host computer. In the operating system of the host computer, multiple displays are set to an extended mode, multiple viewports are set on a whole GraphicsContext, and different observation angles of each viewport are realized by modifying the observation projection matrix of each viewport. Each display displays a viewport, and the observation point position and angle on each display are moved by operating the mouse and keyboard combination.

[0067] The invention can be used for remote ship motion state monitoring. Compared with the two-dimensional ship state data displayed on the electronic chart, the three-dimensional visualization data is more intuitive and convenient.

[0068] Compared with directly using cameras and other sensors for remote monitoring, the invention has low cost and convenient maintenance. When using cameras and other sensors to build a remote monitoring system, cameras need to be added to multiple positions on the ship. A large number of camera sensor hardware devices on a ship need to be connected through a data transmission line to build a network system, which needs to be sent to the nearest network node through a signal sending terminal. Multiple network sending nodes of multiple ships need to build a stable large data transmission system, which requires the joint efforts of many shipping companies and countries around the world for many years. The invention directly uses AIS data to generate real-time three-dimensional motion state video of the ship under the condition of using existing hardware devices on the ship, reducing the economic cost and time cost. The invention can be used as a transition for real-time remote monitoring of unmanned ships and intelligent ships.

[0069] The three-dimensional ship motion state generated by the backup ship VDR data can be used as a basis for post-navigation analysis of the ship, and provides a three-dimensional visual tool for the traffic management department to supervise the traffic flow in the jurisdiction. In the case that the aerial photography technical means are limited and the monitoring network coverage is insufficient in the jurisdiction, the setting device provided by the present application can display the channel, water area, weather, ship motion and other states, which is a kind of low-cost transitional option.

[0070] Embodiment 2

[0071] The embodiment provides an electronic device, including: one or more processors and a memory, the memory has one or more programs stored therein, and the one or more programs include instructions for executing the ship motion visualization method based on digital twinning as described in embodiment 1.

[0072] Embodiment 3

[0073] The embodiment provides a computer-readable storage medium, including one or more programs for one or more processors of an electronic device to execute, and the one or more programs include instructions for executing the ship motion visualization method based on digital twinning as described in embodiment 1.

[0074] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for ship motion visualization based on digital twinning, characterized in that, It comprises the following steps: Obtain real ship information from multiple data sources, and obtain ship state data by analysis; Based on the ship state data, load the preset ship model and the environment model matched with the ship position; Based on the current ship state data, update the motion state of the digital twin ship through Kalman filtering, and dynamically adjust the motion state update frequency of the twin ship during the updating process; Obtain observation point information, and output visualization signals to the visualization terminal based on the ship model, the environment model, the observation point information, and the motion state of the digital twin ship, The implementation process of dynamically adjusting the motion state update frequency of the twin ship comprises the following steps: An interval time T between the current acquired ship information and the last acquired ship information is calculated, and based on the interval time T, an appropriate twin ship motion state update frequency f is selected so that the state of the digital twin ship is updated at an interval same as the interval time T, wherein n is a positive integer, The process of updating the motion state of the digital twin ship through Kalman filtering comprises the following steps: Based on the ship state data in reality, obtain the initial state value of the digital twin ship; Based on the initial state value, obtain the predicted state value of the digital twin ship through Kalman filtering to update the motion state of the digital twin ship.

2. The ship motion visualization method based on digital twinning according to claim 1, characterized in that, The initial state value of the digital twin ship is obtained by the following formula: In the formula, is an initial state value of the digital twin ship, is a dimension conversion matrix, is a real ship state data, and the real ship state data includes a ship position coordinate and a ship attitude coordinate.

3. The ship motion visualization method based on digital twinning according to claim 1, characterized in that, The prediction equation of the Kalman filter is: The update equation of the Kalman filter is: in, express The estimated value of time, Indicates by Time estimate Received The predicted value at time, F represents the state transition matrix, H represents the measurement matrix, and the subscripts are... and This represents the change at time t. The broken line symbol above the letter indicates that the current letter is an estimated value. express Time's up The predicted value of the state covariance at time t. for Process noise covariance at time step To measure the variance of white noise using Gaussian methods. for The target observation value at time t. It is the identity matrix. This represents the difference between the predicted value and the measured value. is the Kalman gain coefficient.

4. The ship motion visualization method based on digital twinning according to claim 1, characterized in that, The visualization signal also includes an electronic chart matched with the position of the current observation point.

5. The ship motion visualization method based on digital twinning according to claim 1, characterized in that, It also includes: Determine whether the visualization terminal has multiple displays, and if so, expand the multiple displays and set multiple viewports, modify the observation projection matrix of each viewport, and send visualization signals of different observation points to the visualization terminal.

6. The ship motion visualization method based on digital twinning according to claim 1, characterized in that, The environment model includes at least one of a sky model, a water area model, a port terrain building model, and a channel light mark model.

7. The ship motion visualization method based on digital twinning according to claim 1, characterized in that, The ship state data includes ship identification data and ship position data, wherein the ship identification data includes at least one of a ship identification code, a ship length, and a ship width, and the ship position data includes at least one of latitude, longitude, heading, and speed.

8. The ship motion visualization method based on digital twinning according to claim 1, characterized in that, The multiple data sources include real-time transmission data sources and offline backup data sources, wherein the data transmitted by the real-time transmission data sources includes one or more of AIS information, log information, GPS information, wind direction and speed information, and depth sounder information, and the offline backup data sources are VDR devices and / or receiving base stations of the ship.

Images