A method of ship navigation situation awareness based on augmented reality
Through the ship navigation situation awareness method based on augmented reality technology, using machine learning and virtual and real coordinate transformation, the gap in information processing and display in ship navigation is solved, and the intuitive display of multiple information and the integration of virtual and real data is realized, which improves the convenience of data observation sex.
Patent Information
- Application Number
- CN202011434833.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-10
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-12-10
AI Technical Summary
There is still a lack of mature solutions in the field of real-life enhancement technology for domestic ship navigation, which leads to ship drivers having to manually analyze and process a large amount of information, and lacks intuitive information display and virtual and real data fusion capabilities.
Using augmented reality-based ship navigation situational awareness method, video images are captured in real time through a webcam, combined with radar fusion identification information and sea chart marking information, machine learning technology is used to extract the virtual three-dimensional coordinates of the target ship and compare it with the radar information to achieve Virtual and real coordinate transformation, superimposed display of three-dimensional models, and enhanced information display.
It realizes multiple display and intuitive information in ship navigation, improves data observation convenience and the fusion effect of virtual and real, and reduces the driver's information processing burden.
Smart Images

Figure CN112665588B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of ship navigation augmented reality, and in particular to a ship navigation situation perception method based on augmented reality. Background Art
[0002] Augmented Reality (AR) is a technology that calculates the position and angle of camera images in real time and adds corresponding images, thereby "seamlessly" integrating real-world information and virtual-world information. The goal of this technology is to integrate the virtual world and the real world on the display screen. It has three outstanding features: (1) information integration between the real world and the virtual world; (2) real-time interactivity; and (3) the ability to add and locate virtual objects in three-dimensional space. At present, augmented reality technology has been widely used in medical research and anatomical training, precision instrument manufacturing and maintenance, military aircraft navigation, engineering design, and remote robot control.
[0003] In the field of ship navigation, the augmented reality navigation system can display information about other ships on a ship's planned route, as well as information about surrounding waters and other sea conditions (such as shallow waters) on a tablet or screen. It integrates information from the ship's automatic identification system (AIS), radar information, and real-time video images from the camera on the bridge. As a major shipping country, at present, there is still a gap in the research and development of augmented reality for ship navigation in China. The present invention independently develops an augmented reality navigation system for ships with complete intellectual property rights. Summary of the invention
[0004] In view of this, the present invention provides a ship navigation situation awareness method based on augmented reality to solve the problems existing in the above-mentioned background technology.
[0005] A method for ship navigation situation awareness based on augmented reality comprises the following steps:
[0006] S1, the network camera captures the video image in front of the ship's navigation route in real time and transmits the video image to the computer through the network;
[0007] S2, multicast / broadcast to obtain radar fusion identification information and chart mark information and transmit them to the computer through the network;
[0008] S3, the video processing module of the computer analyzes the received video image, extracts the virtual three-dimensional coordinates of the target ship from the analyzed video image using machine learning technology, compares the virtual three-dimensional coordinates of the target ship with the actual three-dimensional coordinates in the radar fusion recognition information, and obtains the mapping relationship of the virtual and real coordinates conversion of the target ship;
[0009] S4, mapping the target ship and superimposing it on the video image according to the mapping relationship of the virtual and real coordinates conversion of the target ship, and at the same time, the nautical chart marking information is also superimposed on the video image.
[0010] Preferably, the specific steps of extracting the virtual three-dimensional coordinates of the target vessel from the parsed video image using machine learning technology in step S3 are:
[0011] First, the trained convolutional neural network is used to extract high-dimensional features from video images to obtain feature maps of multiple scales.
[0012] Then, a multi-scale branch network is used to perform feature aggregation processing on the feature maps at the corresponding scales and pass the processed image features to the detection classifier;
[0013] The detection classifier predicts the features of the received image, determines the position of the bounding box and the type of ship within the bounding box, and obtains the virtual three-dimensional coordinates of the target ship.
[0014] Preferably, the radar fusion identification information includes the distance of the target ship from the ship, the speed of the target ship over the ground, the minimum encounter distance between the target ship and the ship, the time when the target ship reaches the expected closest encounter point, the distance of the target ship passing the bow of the ship, the time when the target ship passes the bow of the ship, the latitude and longitude of the target ship, and the AIS information of the target ship.
[0015] Preferably, the AIS information of the target ship includes the ship type, MMSI, call sign and ship name, length and width of the target ship.
[0016] Preferably, step S5 is also included, when the target ship in the video image enters the warning area of the own ship, or the time for the target ship to reach the expected closest encounter point is less than a set value, or the minimum encounter distance between the target ship and the own ship is less than a set value, the computer issues an alarm command.
[0017] Preferably, the nautical chart marking information includes the water depth of the current sea area, waypoints, non-navigable areas, navigation stations, buoys, safety boundaries, coastlines, and landmarks.
[0018] The beneficial effects of the present invention are:
[0019] The method of the present invention fills the research and development gap in the field of augmented reality technology for domestic ship navigation. By using AR technology in ship navigation, a large amount of information that was originally analyzed and processed by the ship driver is processed through the AR analysis and synthesis system, and then integrated with the basic background parameters, the processed results are enhanced and displayed on the screen in front of the driver, so as to realize simultaneous and intuitive enhanced display of multiple information.
[0020] At the same time, the method of the present invention realizes the superposition of navigation data and real scenes based on hybrid three-dimensional registration technology, and accesses the data of target recognition and intelligent collision, so as to achieve the purpose of virtual-real fusion and improve the convenience and intuitiveness of data observation. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0022] Figure 1 It is a diagram of the composition of the ship's navigation situation awareness system.
[0023] Figure 2 It is a hierarchical diagram of the data processing system within a computer.
[0024] Figure 3 It is a schematic diagram of extracting virtual three-dimensional coordinates from video images using machine learning technology. DETAILED DESCRIPTION
[0025] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0026] It should be clear that the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0027] The present application is further described in detail below through specific embodiments and in conjunction with the accompanying drawings.
[0028] The present invention provides a method for ship navigation situation awareness based on augmented reality, which fills the research and development gap in the field of domestic ship navigation reality augmented technology. AR technology is used in ship navigation, and a large amount of information originally analyzed and processed by the ship driver is processed by the AR analysis and synthesis system, and then integrated with the basic background parameters, and the processed results are enhanced and displayed on the screen in front of the driver, so as to realize simultaneous and intuitive enhanced display of multiple information.
[0029] Specifically, the ship navigation situation awareness system consists of a network high-definition camera and one or more AR display consoles (i.e., computers). The network high-definition camera and the AR display console communicate through an Ethernet switch, and the AR display console also communicates with an external data service unit through the Ethernet switch.
[0030] External data service units mainly include four categories, namely target fusion identification information unit, electronic chart marking unit, own ship sensor information unit, and collision avoidance information unit. Each unit transmits information in a fixed format; except for the strict agreement on the network communication protocol format, there is no requirement for the physical form of each information input unit, as long as the unit can send data to the local area network in the form of network multicast or broadcast. All data are exchanged and transmitted through the network switch. The switch acts as a data hub and is the collection node of the data flow, information flow and control flow of the entire ship navigation situation awareness system. The AR display console has a one-way communication relationship with the camera and the five types of information input units. The console is the receiver of the information, and the other units are the senders of the information.
[0031] The network camera adopts a fixed focal length / fixed installation method to obtain real-time video images in front of the ship's navigation route. In this embodiment, the network camera uses a Hikvision POE powered network camera.
[0032] The AR display console (i.e., computer) is responsible for superimposed display and human-computer interaction. It is equipped with a large display screen, which superimposes and displays the video images transmitted in real time by the network camera and the information sent by various external data service units. It is also equipped with voice input and output devices to support user voice command control and alarm reminder functions. If multiple sets of AR display consoles are deployed, each AR display console is independent of the other, without master-slave distinction.
[0033] The AR display console mainly consists of the following parts:
[0034] 1) Video processing module: communicates with the network camera to receive and analyze the video stream.
[0035] 2) Network information processing module: obtain radar fusion recognition results, chart marking information, ship sensor information, etc. through multicast / broadcast, and classify, decode and store them in accordance with the agreed format to provide data support for subsequent augmented reality overlay of camera video and navigation elements.
[0036] 3) Augmented reality fusion overlay module: It is the core of the system. Based on the processed video, it calibrates the three-dimensional coordinate system, uses hybrid tracking and registration technology, combines the ship posture information of the sensor and the results of target recognition, integrates and complements the data, and obtains more complete data. Then the video image is displayed in three-dimensional space, showing various navigation elements such as collision avoidance information, own ship sensor information, chart mark information, target fusion recognition processing information, etc.
[0037] 4) Voice control module: command dictionary creation, recognition model training, control command translation and execution.
[0038] 5) Auxiliary modules: setting options, alarm prompts, and other auxiliary components.
[0039] The augmented reality fusion module is responsible for organically integrating the camera video stream and information such as targets, navigation chart elements, collision avoidance and local sensors in the three-dimensional space coordinate system. Based on this core function, the system can be roughly divided into two parts: the operator end and the data end.
[0040] The ship navigation situation awareness method based on augmented reality of the present invention comprises the following steps:
[0041] S1, the network camera captures the video image in front of the ship's sailing route in real time and transmits the video image to the computer through the network.
[0042] S2, multicast / broadcast obtains radar fusion identification information and chart mark information and transmits it to the computer through the network.
[0043] The radar fusion identification information includes the distance (RNG) between the target ship and the own ship, the speed of the target ship over the ground (SOG), the minimum encounter distance (CPA) between the target ship and the own ship, the time when the target ship reaches the expected closest encounter point (TCPA), the distance (BCR) at which the target ship passes the bow of the own ship, the time (BCT) at which the target ship passes the bow of the own ship, the latitude and longitude of the target ship, and the AIS information of the target ship.
[0044] The AIS information of the target ship includes the ship type, MMSI, call sign and ship name, length and width of the target ship.
[0045] The nautical chart marking information includes the water depth of the current sea area, waypoints, non-navigable areas, navigation stations, buoys, safety boundaries, coastlines, and landmarks.
[0046] In this embodiment, except for the video data captured by the camera, the rest of the data is transmitted in the local area network in the form of network multicast or broadcast. Therefore, the efficiency of network packet reception is worth considering. The present invention uses ACE open source network components, whose main application scenarios are high-performance and real-time communication services and applications, such as the development of high-performance servers. The use of ACE can effectively simplify the development of event demultiplexing, explicit dynamic linking and concurrent OO network applications and services in this project. Specifically, the ACE_Proactor framework is used to receive network packets, and the UDP data packet is read into the ACE_MessageBlock message block structure in the event object Event_Handler, and then the received message block is processed by multi-threading through the thread pool structure TP_UpdateVideo (control class), and the data is placed in the derived class of Package Descriptor according to the different protocol messages, and sorted and stored in the display buffer Update Buffer in the form of Package Descriptor pointers.
[0047] S3, the video processing module of the computer analyzes the received video image and extracts the virtual three-dimensional coordinates of the target ship from the analyzed video image using machine learning technology.
[0048] The specific steps of using machine learning technology to extract the virtual three-dimensional coordinates of the target ship from the parsed video image are as follows:
[0049] First, the trained convolutional neural network is used to extract high-dimensional features from video images to obtain feature maps of multiple scales.
[0050] Then, a multi-scale branch network is used to perform feature aggregation processing on the feature maps at the corresponding scales and pass the processed image features to the detection classifier;
[0051] The detection classifier predicts the features of the received image, determines the position of the bounding box and the type of ship within the bounding box, and obtains the virtual three-dimensional coordinates of the target ship.
[0052] The virtual three-dimensional coordinates of the target ship are compared with the actual three-dimensional coordinates in the radar fusion identification information to obtain the mapping relationship of the virtual and real coordinate conversion of the target ship.
[0053] S4, mapping the three-dimensional model of the target ship and superimposing it on the video image according to the mapping relationship of the virtual-real coordinate conversion of the target ship, and at the same time, the nautical chart marking information is also superimposed on the video image.
[0054] The three-dimensional model of the virtual image is modeled with the help of the cross-platform three-dimensional open source scene graph system application development interface (API) of Open Scene Graph (OSG, http: / / www.openscenegraph.org / ). OSGEarth provides a complete set of geographic spatial reference systems, including geographic coordinate systems, projection transformations, etc. You can also customize the coordinate system and projection method. In OSG / OSGEarth, there are two core concepts in the organization of three-dimensional scene data: nodes and scene trees. The basic unit of the three-dimensional scene in OSG is the node, including group nodes and leaf nodes. The leaf node manages the information of one or more drawable bodies, and the information of the drawable bodies can be queried through its interface function. The OSG rendering engine uses a top-down, hierarchical tree structure (scene tree) to implement the organization of spatial data, which greatly improves the rendering efficiency of the scene.
[0055] The ship navigation situation awareness method based on augmented reality of the present invention also includes step S5, when the target ship in the video image enters the warning area of the own ship, or the time for the target ship to reach the expected closest encounter point is less than a set value, or the minimum encounter distance between the target ship and the own ship is less than a set value, the computer issues an alarm instruction.
[0056] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for ship navigation situation awareness based on augmented reality, characterized in that: The following steps are involved: S1, the network camera captures the video image in front of the ship's navigation route in real time and transmits the video image to the computer through the network; S2, multicast / broadcast to obtain radar fusion identification information and chart mark information and transmit them to the computer through the network; S3, the video processing module of the computer analyzes the received video image, extracts the virtual three-dimensional coordinates of the target ship from the analyzed video image using machine learning technology, compares the virtual three-dimensional coordinates of the target ship with the actual three-dimensional coordinates in the radar fusion recognition information, and obtains the mapping relationship of the virtual and real coordinates conversion of the target ship; The specific steps of extracting the virtual three-dimensional coordinates of the target ship from the parsed video image using machine learning technology are as follows: first, the trained convolutional neural network is used to extract high-dimensional features of the video image to obtain multiple feature maps of different scales; Then, a multi-scale branch network is used to perform feature aggregation processing on the feature maps at the corresponding scales and pass the processed image features to the detection classifier; The detection classifier predicts the features of the received image, determines the position of the bounding box and the type of ship within the bounding box, and obtains the virtual three-dimensional coordinates of the target ship; S4, superimposing the three-dimensional virtual model of the target ship on the video image according to the mapping relationship of the virtual-real coordinate conversion of the target ship, and the nautical chart marking information is also superimposed on the video image; The three-dimensional model of the virtual image is modeled with the help of Open Scene Graph, a cross-platform three-dimensional open source scene graph system application development interface API, using OSGEarth to provide a complete geographic spatial reference system. The basic unit of the three-dimensional scene in OSG is the node, including group nodes and leaf nodes. The leaf node manages the information of one or more drawable bodies and can query the information of the drawable body through its interface function. The OSG rendering engine adopts a top-down, hierarchical tree structure to implement the organization of spatial data.
2. The method for ship navigation situation awareness based on augmented reality according to claim 1, characterized in that: The radar fusion identification information includes the distance of the target ship from the ship, the speed of the target ship over the ground, the minimum encounter distance between the target ship and the ship, the time when the target ship reaches the expected closest encounter point, the distance of the target ship passing the bow of the ship, the time when the target ship passes the bow of the ship, the latitude and longitude of the target ship, and the AIS information of the target ship.
3. The method for ship navigation situation awareness based on augmented reality according to claim 2, characterized in that: The AIS information of the target ship includes the ship type, MMSI, call sign and ship name, length and width of the target ship.
4. The method for ship navigation situation awareness based on augmented reality according to claim 2, characterized in that: The method further includes step S5, in which the computer issues an alarm command when the target vessel in the video image enters the warning zone of the own vessel, or the time for the target vessel to reach the expected closest encounter point is less than a set value, or the minimum encounter distance between the target vessel and the own vessel is less than a set value.
5. The method for ship navigation situation awareness based on augmented reality according to claim 1, characterized in that: The nautical chart marking information includes the water depth of the current sea area, waypoints, non-navigable areas, navigation stations, buoys, safety boundaries, coastlines, and landmarks.
