A mobile augmented reality-based maritime key target information display method
By fusing ship sensor information with mobile augmented reality technology and deep learning algorithms, real-time identification and display of key targets at sea can be achieved, solving the problem of information dispersion in traditional methods and improving navigation safety.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- DALIAN MARITIME UNIVERSITY
- Filing Date
- 2026-03-17
- Publication Date
- 2026-07-14
AI Technical Summary
In existing technologies, ship operators need to manually integrate data from multiple sensors, resulting in fragmented information, which affects decision-making efficiency. Furthermore, traditional methods fail to effectively identify and display information about key targets at sea.
By employing mobile augmented reality technology and fusing deep learning target detection algorithms with ship sensor information, the system enables visual detection and information fusion display of key maritime targets. It combines multi-sensor data for real-time identification and tracking, and provides risk warnings through a layered overlay information display method.
It provides centralized and intuitive visual perception assistance, reducing the risk of collisions and improving navigation safety, especially in low visibility or nighttime navigation scenarios, effectively supporting pilot decision-making.
Smart Images
Figure CN122391816A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of maritime transportation technology, and in particular to a method for displaying key maritime target information based on mobile augmented reality. Background Technology
[0002] During navigation, ship operators need to constantly monitor dynamic and static information of maritime targets from navigation sensors (such as radar, AIS, and GPS). The fixed installation locations of these information display devices hinder the ease of information access.
[0003] Patent application CN108550281A discloses a visual AR-based ship assisted driving system. Its core solution includes: converting image pixels into a geodetic coordinate system via a video calibration module; fusing AIS, radar, and GPS data to obtain the target ship's position information; overlaying AR display and providing collision warnings based on relative position / speed (using green / yellow / red color markers). The drawback of this method is that it only proposes the implementation flow without detailing the specific implementation method.
[0004] Therefore, there is a need to provide a method that can effectively identify key targets around a ship and display information. Summary of the Invention
[0005] To address the aforementioned technical problems, this invention provides a method for displaying key maritime target information based on mobile augmented reality. This invention employs mobile augmented reality technology to achieve fusion display of multi-sensor information on maritime targets, realizing a WYSIWYG target display. By fusing deep learning target detection algorithms with ship sensor information, visual detection and information fusion display of key targets around the ship are achieved.
[0006] The technical means employed in this invention are as follows:
[0007] A method for displaying key maritime target information based on mobile augmented reality includes: acquiring data information of the ship and target ships by decoding AIS information, and acquiring IMU data using an IMU sensor; predicting the motion state of the target ship using the decoded AIS information of the target ship, and filling in gaps in the AIS data; converting the geodetic coordinate system in the supplemented AIS data into a pixel coordinate system used by the mobile terminal; constructing a target detection model based on YOLOv11n; performing real-time identification of key maritime targets based on the target detection model, and achieving multi-target tracking by combining ByteTrack and ReID features; matching AIS targets with visual targets, and using a real-time inter-frame matching method for newly appearing targets or targets that reappear after a brief loss; and using a layered overlay information display method to divide the AIS information required for navigation into a basic layer and a detailed layer to display key maritime target information.
[0008] Furthermore, the data information includes: ship position, heading, speed, MMSI, call sign, and ship draft information; the IMU sensors include accelerometers, gyroscopes, and magnetometers; the IMU data includes three-axis acceleration, three-axis angular velocity, and three-axis orientation.
[0009] Furthermore, the prediction of the target vessel's motion state using the decoded AIS information of the target vessel specifically includes: Linear interpolation algorithm is used to complete the trajectory of the vessel, assuming the vessel trajectory is... ,in , , Representing longitude and latitude respectively; two adjacent trajectory points , The time interval is obtained through linear function interpolation. ship motion status :
[0010] For ships in the turning phase, the trajectory is approximated as an ideal circular arc and completed using a cubic spline interpolation algorithm; boundary points of the navigation trajectory are defined. , During the time period Inside, ship latitude is set. A cubic function of time:
[0011] in, Indicates the time period Inner, ship latitude Over time A changing cubic spline interpolation function, , , , This represents the coefficients to be solved for; ship latitude The first and second derivatives are expressed as:
[0012]
[0013] Based on boundary conditions and boundary points , Through derivation, we obtain:
[0014] in, Indicates at the boundary point The latitude of the ship at that location.
[0015] Furthermore, the process of converting the geodetic coordinate system in the completed AIS data into the pixel coordinate system used by the mobile device specifically includes: The target ship's geodetic coordinate system provided by AIS ( , , Convert to geocentric rectangular coordinate system ( , , ):
[0016] Among them, the radius of curvature of the Mao-You circle First eccentricity square ; Convert the geocentric rectangular coordinate system to the northeast-northeast coordinate system (ENU), and represent the geodetic coordinates of the equipment location as ( , , The geocentric coordinates are represented as ( , , ), calculate the northeast celestial coordinates ( , , ):
[0017] Convert the northeast celestial coordinate system to the camera coordinate system. , , ):
[0018] in, T It is a translation vector. Let the rotation matrix be y = y', pitch, and roll angles respectively. , , ,but:
[0019] in, Set the equipment to factory calibration parameters; convert the camera coordinate system to the screen pixel coordinate system. :
[0020] in, , For camera focal length, , The coordinates of the main point.
[0021] Furthermore, the improvement of the target detection model based on YOLOv11n specifically includes: GhostConv replaces ordinary convolutional blocks, and the exponential average shift idea of the EMA attention mechanism is combined with the inverted residual shift block of the iRMB attention mechanism to construct the iEMA module. The iEMA module performs local feature extraction and global dependency modeling through iRMB, and uses the EMA mechanism to smooth features. UIoU is used instead of the bounding box loss function, and model pruning techniques are employed to remove connections or neurons with weights less than a preset threshold, reducing the number of model parameters. Knowledge distillation is used, with YOLOv11x as the teacher model to guide the student model YOLOv11n, compensating for the decrease in recognition accuracy caused by the reduction in the number of parameters. The UIoU loss function is expressed as:
[0022] in, As a weighting factor, This represents the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. The center point distance penalty term, The calculation formula is as follows:
[0023] in, and These represent the proportions of the predicted bounding box and the ground truth bounding box in the height and width directions, respectively. and These are adjustable hyperparameters.
[0024] Furthermore, the real-time identification of key maritime targets based on the target detection model, combined with ByteTrack and ReID features to achieve multi-target tracking, specifically includes: Before tracking, a tracking trajectory is created for each target, and Kalman filtering is used to predict the bounding box of each trajectory in the next frame. Candidate target boxes are output by the detector and divided into high-scoring and low-scoring boxes based on their confidence scores. ByteTrack employs a phased matching strategy, associating high-confidence detection results with existing trajectories and matching unmatched trajectories with low-confidence detection boxes. ReID appearance features are introduced based on Kalman filtering prediction to achieve effective association between detection boxes and trajectories, jointly constructing a cost matrix. A motion cost matrix is then constructed. :
[0025] in, For the first j Predicted bounding boxes for each trajectory, For the first Detection boxes; construct the ReID feature matrix. :
[0026] in, For ship trajectory j ReID feature vectors, For detection box The ReID feature vectors are obtained; the motion cost matrix and the ReID feature matrix are combined into a cost matrix:
[0027] in, Using cost weights, the Hungarian algorithm is used to solve the fusion cost matrix to obtain the optimal detection and trajectory matching pair. For successfully matched trajectories, the Kalman filter state is updated using the associated detection boxes, and the appearance features of the trajectory are updated using the exponential moving average method.
[0028] in, Indicates the updated number The appearance feature vector of a ship tracking trajectory Indicates the number before the update The appearance feature vector of a ship tracking trajectory The feature smoothing factor is used; for unmatched trajectories, they are retained by prediction, and if they are not matched for multiple consecutive frames, the trajectories are deleted.
[0029] Furthermore, the matching of AIS targets with visual targets, and the use of real-time inter-frame matching for newly appearing targets or targets that have been briefly lost and then reappeared, specifically includes: The nearest neighbor matching method is used to calculate the AIS projection points ( , ) and the center of the visual inspection box ( , Euclidean distance :
[0030] Select Euclidean distance d The smallest visual target is selected as a matching candidate, and the matching criteria are appropriately relaxed based on the horizontal distance to the target. Trajectory verification is performed, and the total displacement vector generated by the two trajectories within the most recent time window is calculated. The AIS displacement vector and the visual displacement vector are expressed as follows:
[0031]
[0032] Calculating the cosine similarity between two vectors measures their directional consistency:
[0033] like A value close to 1 indicates that the AIS target and the visual target are highly consistent in their direction of movement, confirming a match. For continuously tracked stable targets, trajectory contour matching is used to extract the complete projection of the AIS target over a past period. and the tracking trajectories of candidate visual targets within the same time period. The similarity between the two trajectories is calculated using the Dynamic Time Warping (DTW) algorithm. Construct a matrix D , where matrix elements express and The goal of the dynamic time warping algorithm is to find a path from the midpoint to the point using the Euclidean distance between the midpoints. arrive Regular path Each of them This path minimizes the cumulative distance cost, and the path must satisfy boundary conditions, monotonicity conditions, and continuity constraints. The boundary conditions are as follows: and ; The monotonic condition is: if Then for the next point on the path It needs to meet the following requirements. and ,in, , This indicates the index position of the previous point on the path in both time series. , This indicates the index position of the currently considered point on the path in both time series.
[0034] The continuity constraint is: if Then for the next point on the path It needs to meet the following requirements. and ; The minimum cumulative distance corresponding to the optimal normalized path is the DTW distance between the two sequences:
[0035] like If the value is less than the threshold, it is determined that the two trajectories originate from the same target, and the match is confirmed.
[0036] Furthermore, the layered overlay information display method divides the AIS information required for navigation into a basic layer and a detailed layer to display key maritime target information, specifically including: The base layer includes the target ship's name, distance, and bearing, and is always visible on the display interface; the detailed layer includes the target ship's heading, speed, TCPA, DCPA, and MMSI, and is displayed when the driver clicks the target ship detection box on the display interface. A risk warning mechanism is set up in the process of displaying the information of key maritime targets. The safety range of CPA / TCPA is determined according to different water types, and the risk level is determined by calculating CPA / TCPA. The risk level includes low-risk targets, medium-risk targets, and high-risk targets. When displayed, low-risk targets are marked with a green semi-transparent border; medium-risk targets trigger a yellow flashing border and are accompanied by a single vibration alarm; high-risk targets display a red flashing border and mark the danger zone, and at the same time activate a continuous vibration alarm.
[0037] Furthermore, to address low-visibility scenarios and situations where visual perception is limited, the display of key maritime target information introduces a condition-triggered pure AIS display mode as a security mechanism; a safe distance threshold is set as follows: The threshold for the duration of visual perception failure is For any AIS target, the triggering logic is as follows: Calculate the real-time distance between the target ship and the ship. ; The duration for which a target vessel is continuously failed to be identified by the target detection model. ; The pure AIS display mode is activated for the target vessel only if the following conditions are met:
[0038] In the pure AIS display mode, AIS targets are displayed with a red flashing dashed frame, and the AIS-ONLY character is added to the label information to warn the driver that the target ship information originates from AIS and is not visually confirmed.
[0039] Compared with the prior art, the present invention has the following advantages: This invention provides a method for displaying key maritime targets based on mobile augmented reality. It uses images of the ship's surrounding environment captured by a mobile camera and ship sensor data as identification data sources, and employs target detection and tracking algorithms to identify and track key maritime targets. Simultaneously, it utilizes mobile augmented reality technology to fuse multi-source information, employing adaptive algorithms to supplement the AIS data for different motion states of the target ship, and then fuses the AIS information with real-time video images on the mobile device through a four-dimensional coordinate transformation. It provides early warning prompts based on different risk levels calculated in real time, solving the problem of decision-making delays caused by scattered information in traditional bridges and the need for drivers to manually integrate various sensor data. Especially in poor visibility or night navigation scenarios, it provides drivers with more intuitive and centralized visual perception assistance than conventional instruments, effectively reducing collision risks and improving navigation safety. Attached Figure Description
[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0041] Figure 1 This is a flowchart of the method for displaying key maritime target information based on mobile augmented reality in this invention. Detailed Implementation
[0042] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0044] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0045] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps described in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0046] like Figure 1 As shown, this invention provides a method for displaying key maritime target information based on mobile augmented reality, including: obtaining data information of the ship and the target ship by decoding AIS information, and acquiring IMU data using IMU sensors; in a preferred embodiment of this invention, the data information includes: ship position, heading, speed, MMSI, call sign, and ship draft information; the IMU sensors include accelerometers, gyroscopes, and magnetometers; the IMU data includes three-axis acceleration, three-axis angular velocity, and three-axis orientation. In practice, the target ship's AIS information is acquired via a Socket.
[0047] The target vessel's motion state is predicted using its decoded AIS information, filling in gaps in the AIS data. Specifically, in a preferred embodiment of this invention, a linear interpolation algorithm is used to complete the data for the target vessel, assuming the vessel's trajectory is... ,in , , Representing longitude and latitude respectively; two adjacent trajectory points , The time interval is obtained through linear function interpolation. ship motion status :
[0048] For vessels in the turning phase, due to their short AIS reporting frequency (no more than 6 seconds), the trajectory is approximated as an ideal circular arc, and a cubic spline interpolation algorithm is used to complete it; boundary points of the navigation trajectory are set. , During the time period Inside, ship latitude is set. A cubic function of time:
[0049] in, Indicates the time period Inner, ship latitude Over time A changing cubic spline interpolation function, , , , All of these represent the coefficients to be solved.
[0050] ship latitude The first and second derivatives are expressed as:
[0051]
[0052] Based on the boundary conditions (location continuity and continuity of first and second derivatives) and boundary points , Through derivation, we obtain:
[0053] in, Indicates at the boundary point The latitude of the ship at that location can be determined by solving the above expression and then calculating the latitude at the corresponding time interval based on the interpolation time interval.
[0054] The geodetic coordinate system in the supplemented AIS data is converted to the pixel coordinate system used by the mobile device; in a specific implementation, as a preferred embodiment of the present invention, the geodetic coordinate system of the target ship provided by AIS is converted to the pixel coordinate system used by the mobile device. , , Convert to geocentric rectangular coordinate system ( , , ):
[0055] Among them, the radius of curvature of the Mao-You circle First eccentricity square ; Convert the geocentric rectangular coordinate system to the northeast-northeast coordinate system (ENU), and represent the geodetic coordinates of the equipment location as ( , , The geocentric coordinates are represented as ( , , ), calculate the northeast celestial coordinates ( , , ):
[0056] The advantage of introducing the Northeast-Eastern Sky coordinate system lies in the fact that the various attitude data provided by the mobile device's IMU are based on the local horizontal plane. This allows the transformation from the ENU coordinate system to the camera coordinate system to directly use IMU data, simplifying the calculation process, avoiding additional transformation errors, and because the coordinate values of the geocentric rectangular coordinate system are on the order of magnitude, while ENU coordinate values are typically hundreds to thousands of meters, the numerical calculations are more stable and have higher accuracy. (The process of converting the Northeast-Eastern Sky coordinate system to the camera coordinate system is described.) , , ):
[0057] in, T It is a translation vector. The rotation matrix is provided by the device's IMU. Let the yaw, pitch, and roll angles be... , , ,but:
[0058] in, These are the factory-calibrated parameters of the equipment, which can be approximated as follows: Convert the camera coordinate system to the screen pixel coordinate system. :
[0059] in, , For camera focal length, , The coordinates of the main point are obtained through calibration.
[0060] An object detection model is constructed based on YOLOv11n. In a preferred implementation, GhostConv replaces ordinary convolutional blocks to expand the input channels through simple operations (such as linear transformations), generating more feature maps and reducing computational cost while improving efficiency. The exponential average shift idea of the EMA attention mechanism is combined with the inverted residual shift block of the iRMB attention mechanism to construct a lighter and more efficient iEMA module. The iEMA module performs local feature extraction and global dependency modeling through iRMB, and uses the EMA mechanism to smooth features, thus improving the model's stability and generalization ability while maintaining its lightweight nature. UIoU is used instead of the bounding box loss function to improve the model's convergence efficiency. Simultaneously, model pruning techniques are employed to remove connections or neurons with weights less than a preset threshold, reducing the number of model parameters. Knowledge distillation is used, with YOLOv11x serving as the teacher model to guide the student model YOLOv11n, compensating for the decrease in recognition accuracy caused by the reduction in parameters. The UIoU loss function is expressed as:
[0061] in, As a weighting factor, This represents the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. The center point distance penalty term, The calculation formula is as follows:
[0062] in, and These represent the proportions of the predicted bounding box and the ground truth bounding box in the height and width directions, respectively. and These are adjustable hyperparameters.
[0063] Real-time identification of key maritime targets is achieved based on a target detection model, and multi-target tracking is realized by combining ByteTrack and ReID features. In a preferred embodiment, a tracking trajectory is created for each target before tracking, and Kalman filtering is used to predict the bounding box of the next frame for each tracking trajectory. Candidate target boxes are output by a detector and categorized into high-score and low-score boxes based on confidence scores. ByteTrack uses a phased matching strategy to associate high-confidence detection results with existing trajectories, matching unmatched trajectories with low-confidence detection boxes. ReID appearance features are introduced based on Kalman filtering prediction to achieve effective association between detection boxes and trajectories, jointly constructing a cost matrix. A motion cost matrix is then constructed. :
[0064] in, For the first j Predicted bounding boxes for each trajectory, For the first Detection boxes; construct the ReID feature matrix. :
[0065] in, For ship trajectory j The 128-dimensional ReID feature vector, For detection box The ReID feature vectors are obtained; the motion cost matrix and the ReID feature matrix are combined into a cost matrix:
[0066] in, Using cost weights, the Hungarian algorithm is used to solve the fusion cost matrix to obtain the optimal detection and trajectory matching pair. For successfully matched trajectories, the Kalman filter state is updated using the associated detection boxes, and the appearance features of the trajectory are updated using the exponential moving average method.
[0067] in, Indicates the updated number The appearance feature vector of a ship tracking trajectory Indicates the number before the update The appearance feature vector of a ship tracking trajectory The feature smoothing factor is used; for unmatched trajectories, they are retained by prediction, and if they are not matched for multiple consecutive frames, the trajectories are deleted.
[0068] Matching AIS targets with visual targets, and using a real-time inter-frame matching method for newly appearing targets or targets that reappear after a brief loss; in a preferred embodiment of the present invention, matching AIS targets with visual targets, and using a real-time inter-frame matching method for newly appearing targets or targets that reappear after a brief loss, specifically includes: The nearest neighbor matching method is used to calculate the AIS projection points ( , ) and the center of the visual inspection box ( , Euclidean distance :
[0069] Select Euclidean distance dThe smallest visual target is selected as a matching candidate, and the matching criteria are appropriately relaxed based on the horizontal distance to the target. Trajectory verification is performed, and the total displacement vector generated by the two trajectories within the most recent time window is calculated. The AIS displacement vector and the visual displacement vector are expressed as follows:
[0070]
[0071] Calculating the cosine similarity between two vectors measures their directional consistency:
[0072] like A value close to 1 indicates that the AIS target and the visual target are highly consistent in their direction of movement, confirming a match. For continuously tracked stable targets, trajectory contour matching is used to extract the complete projection of the AIS target over a past period. and the tracking trajectories of candidate visual targets within the same time period. The Dynamic Time Warping (DTW) algorithm is used to calculate the similarity between two trajectories. DTW is an algorithm used to measure the similarity between two time series of different lengths and out of time. It can find the best point-to-point mapping relationship between the two series, thereby overcoming the problem of trajectory data misalignment caused by signal delays and brief obstructions of maritime targets.
[0073] Construct a matrix D , where matrix elements express and The goal of the dynamic time warping algorithm is to find a path from the midpoint to the point using the Euclidean distance between the midpoints. arrive Regular path Each of them This path minimizes the cumulative distance cost, and the path must satisfy boundary conditions, monotonicity conditions, and continuity constraints. The boundary conditions are: and ; The monotonic condition is: if Then for the next point on the path It needs to meet the following requirements. and ,in, , This indicates the index position of the previous point on the path in both time series. , This indicates the index position of the currently considered point on the path in both time series.
[0074] The continuity constraint is: if Then for the next point on the path It needs to meet the following requirements. and ; The minimum cumulative distance corresponding to the optimal normalized path is the DTW distance between the two sequences:
[0075] like If the value is less than the threshold, it is determined that the two trajectories originate from the same target, and the match is confirmed.
[0076] This invention employs a layered overlay information display method, dividing the AIS information required for navigation into a basic layer and a detailed layer to display key maritime target information. In a preferred embodiment, the AIS data contains over 20 items of dynamic and static information, and the display area on a mobile device is limited. Over-displaying information leads to a cluttered and congested interface, while excessive deletion results in information loss, affecting the pilot's judgment of the navigation environment. Therefore, this paper uses a layered overlay information display method to divide the AIS information required for navigation into a basic layer and a detailed layer. The basic layer includes the target vessel's name, distance, and bearing, and is always visible on the display interface. The detailed layer includes the target vessel's heading, speed, TCPA, DCPA, and MMSI, and is displayed when the pilot clicks on the target vessel detection box on the display interface. A risk warning mechanism was implemented to display information on critical maritime targets. The safety range of CPA / TCPA was determined based on different water types, and the risk level was assessed by calculating the CPA / TCPA. Risk levels included low-risk, medium-risk, and high-risk targets. During display, low-risk targets were marked with a green semi-transparent border; medium-risk targets triggered a yellow flashing border and a single vibration alarm; high-risk targets displayed a red flashing border and marked danger zones, while simultaneously activating a continuous vibration alarm. The CPA / TCPA safety ranges for different water types are shown in Table 1, and the risk level determination criteria are shown in Table 2.
[0077] Table 1. CPA / TCPA Safety Ranges for Different Water Types
[0078] Table 2 Risk Level Determination Criteria
[0079] In specific implementation, as a preferred embodiment of the present invention, the display of key maritime target information, in response to low visibility scenarios and limited visual perception, introduces a conditionally triggered pure AIS display mode as a safety mechanism; the safety distance threshold is set as follows: The threshold for the duration of visual perception failure is For any AIS target, the triggering logic is as follows: Calculate the real-time distance between the target ship and the ship. ; The duration for which a target vessel is continuously failed to be identified by the target detection model. ; The pure AIS display mode is activated for the target vessel only if the following conditions are met:
[0080] In pure AIS display mode, AIS targets are displayed with a flashing red dashed frame, and the words "AIS-ONLY" are added to the label information to warn the driver that the target vessel information is from AIS and there is no visual confirmation.
[0081] Example This invention obtains information such as latitude and longitude, heading, speed, MMSI, ship number, and tonnage of surrounding vessels by decoding AIS information, and simultaneously receives relative distance and bearing information of surrounding targets from shipborne radar. Mobile devices acquire the ship's position and IMU data via GPS and sensors.
[0082] The decoded AIS information and video information are spatiotemporally unified, and a differentiated interpolation strategy is implemented according to the ship's motion state: linear motion model is used to complete the ship's dynamic information data for ships sailing straight, while cubic spline interpolation is used to complete the ship's dynamic information data for ships in the turning phase.
[0083] The completed AIS information undergoes coordinate transformation, which involves a four-step transformation process: converting the geodetic coordinate system data provided by AIS to the geocentric rectangular coordinate system, then to the local northeast-sky coordinate system, converting it to the camera coordinate system through the mobile device's AR framework, and finally projecting it onto the screen's pixel coordinate system.
[0084] The target detection module is improved and optimized by using network improvement, model pruning and knowledge distillation to process the target detection model, which significantly improves the running efficiency on mobile devices while maintaining detection accuracy.
[0085] Real-time dynamic video streams around the vessel are acquired via mobile devices, and key targets such as vessels, buoys, drilling platforms, and wind turbines are identified using an optimized detection model. An enhanced tracking scheme is implemented for moving targets, employing the ByteTrack algorithm in conjunction with a ReID enhancement module to address the issue of vessel appearance similarity.
[0086] The target matching adopts a hierarchical strategy: real-time inter-frame matching based on motion consistency is implemented for newly appearing targets or targets that disappear briefly and then reappear, while more accurate trajectory contour matching is used as the core verification method for continuously tracked stable targets, so as to achieve a balance between matching efficiency and accuracy.
[0087] The augmented reality information overlay stage employs a layered information overlay strategy and a multimodal alarm mechanism. Differentiated visual alarms are provided based on collision risk levels: low-risk targets are marked with a green semi-transparent border; medium-risk targets trigger a yellow flashing border accompanied by a single vibration alarm; high-risk targets display a red flashing border and mark the danger zone, while simultaneously initiating a continuous vibration alarm. Risk levels are calculated in real-time using CPA / TCPA algorithms. Furthermore, a conditionally triggered pure AIS display mode is designed as a safety mechanism for low-visibility scenarios such as night flight and heavy fog, as well as situations where visual perception is limited due to target occlusion.
[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for displaying key maritime target information based on mobile augmented reality, characterized in that, include: Data information of the ship and the target ship is obtained by decoding AIS information, and IMU data is obtained by using IMU sensors. The motion status of the target vessel is predicted by using the decoded AIS information of the target vessel, thus filling in the gaps in the AIS data. Convert the geodetic coordinate system in the completed AIS data to the pixel coordinate system used by the mobile device; An object detection model was constructed based on improvements to YOLOv11n. Based on the target detection model, key maritime targets are identified in real time, and multi-target tracking is achieved by combining ByteTrack and ReID features; Match AIS targets with visual targets, and use real-time inter-frame matching for newly appearing targets or targets that reappear after being briefly lost. By using a layered overlay information display method, the AIS information required for navigation is divided into a basic layer and a detailed layer, enabling the display of information on key maritime targets.
2. The method for displaying key maritime target information based on mobile augmented reality according to claim 1, characterized in that, The data information includes: ship position, heading, speed, MMSI, call sign, and ship draft information; the IMU sensors include accelerometers, gyroscopes, and magnetometers; the IMU data includes three-axis acceleration, three-axis angular velocity, and three-axis orientation.
3. The method for displaying key maritime target information based on mobile augmented reality according to claim 1, characterized in that, The method of predicting the motion state of the target vessel using its decoded AIS information specifically includes: Linear interpolation algorithm is used to complete the trajectory of the vessel, assuming the vessel trajectory is... ,in , , Representing longitude and latitude respectively; two adjacent trajectory points , The time interval is obtained through linear function interpolation. ship motion status : For ships in the turning phase, the trajectory is approximated as an ideal circular arc and completed using a cubic spline interpolation algorithm; boundary points of the navigation trajectory are defined. , During the time period Inside, ship latitude is set. A cubic function of time: in, Indicates the time period Inner, ship latitude Over time A changing cubic spline interpolation function, , , , This represents the coefficients to be solved for; ship latitude The first and second derivatives are expressed as: Based on boundary conditions and boundary points , Through derivation, we obtain: in, Indicates at the boundary point The latitude of the ship at that location.
4. The method for displaying key maritime target information based on mobile augmented reality according to claim 1, characterized in that, The process of converting the geodetic coordinate system in the completed AIS data into the pixel coordinate system used by the mobile device specifically includes: The target ship's geodetic coordinate system provided by AIS ( , , Convert to geocentric rectangular coordinate system ( , , ): Among them, the radius of curvature of the Mao-You circle First eccentricity square ; Convert the geocentric rectangular coordinate system to the northeast-northeast coordinate system (ENU), and express the geodetic coordinates of the equipment location as ( , , The geocentric coordinates are represented as ( , , ), calculate the northeast celestial coordinates ( , , ): Convert the northeast celestial coordinate system to the camera coordinate system. , , ): in, T It is a translation vector. Let the rotation matrix be y = y', pitch, and roll angles respectively. , , ,but: in, Set the equipment to factory calibration parameters; convert the camera coordinate system to the screen pixel coordinate system. : in, , For camera focal length, , The coordinates of the main point.
5. The method for displaying key maritime target information based on mobile augmented reality according to claim 1, characterized in that, The improvement upon YOLOv11n to construct a target detection model specifically includes: GhostConv replaces ordinary convolutional blocks, and the exponential average shift idea of the EMA attention mechanism is combined with the inverted residual shift block of the iRMB attention mechanism to construct the iEMA module. The iEMA module performs local feature extraction and global dependency modeling through iRMB, and uses the EMA mechanism to smooth features. UIoU is used instead of the bounding box loss function, and model pruning techniques are employed to remove connections or neurons with weights less than a preset threshold, reducing the number of model parameters. Knowledge distillation is used, with YOLOv11x as the teacher model to guide the student model YOLOv11n, compensating for the decrease in recognition accuracy caused by the reduction in the number of parameters. The UIoU loss function is expressed as: in, As a weighting factor, This represents the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. The center point distance penalty term, The calculation formula is as follows: in, and These represent the proportions of the predicted bounding box and the ground truth bounding box in the height and width directions, respectively. and These are adjustable hyperparameters.
6. The method for displaying key maritime target information based on mobile augmented reality according to claim 1, characterized in that, The method for real-time identification of key maritime targets based on a target detection model, combined with ByteTrack and ReID features to achieve multi-target tracking, specifically includes: Before tracking, a tracking trajectory is created for each target, and Kalman filtering is used to predict the bounding box of each trajectory in the next frame. Candidate target boxes are output by the detector and divided into high-scoring and low-scoring boxes based on their confidence scores. ByteTrack employs a phased matching strategy, associating high-confidence detection results with existing trajectories and matching unmatched trajectories with low-confidence detection boxes. ReID appearance features are introduced based on Kalman filtering prediction to achieve effective association between detection boxes and trajectories, jointly constructing a cost matrix. A motion cost matrix is then constructed. : in, For the first j Predicted bounding boxes for each trajectory, For the first Detection boxes; construct the ReID feature matrix. : in, For ship trajectory j ReID feature vectors, For detection box The ReID feature vectors are obtained; the motion cost matrix and the ReID feature matrix are combined into a cost matrix: in, Using cost weights, the Hungarian algorithm is used to solve the fusion cost matrix to obtain the optimal detection and trajectory matching pair. For successfully matched trajectories, the Kalman filter state is updated using the associated detection boxes, and the appearance features of the trajectory are updated using the exponential moving average method. in, Indicates the updated number The appearance feature vector of a ship tracking trajectory Indicates the number before the update The appearance feature vector of a ship tracking trajectory The feature smoothing factor is used; for unmatched trajectories, they are retained by prediction, and if they are not matched for multiple consecutive frames, the trajectories are deleted.
7. The method for displaying key maritime target information based on mobile augmented reality according to claim 1, characterized in that, The matching of AIS targets with visual targets, and the use of real-time inter-frame matching for newly appearing targets or targets that have been briefly lost and then reappeared, specifically includes: The nearest neighbor matching method is used to calculate the AIS projection points ( , ) and the center of the visual inspection box ( , Euclidean distance : Select Euclidean distance d The smallest visual target is selected as a matching candidate, and the matching criteria are appropriately relaxed based on the horizontal distance to the target. Trajectory verification is performed, and the total displacement vector generated by the two trajectories within the most recent time window is calculated. The AIS displacement vector and the visual displacement vector are expressed as follows: Calculating the cosine similarity between two vectors measures their directional consistency: like A value close to 1 indicates that the AIS target and the visual target are highly consistent in their direction of movement, confirming a match. For continuously tracked stable targets, trajectory contour matching is used to extract the complete projection of the AIS target over a past period. and the tracking trajectories of candidate visual targets within the same time period. The similarity between the two trajectories is calculated using the Dynamic Time Warping (DTW) algorithm. Construct a matrix D , where matrix elements express and The goal of the dynamic time warping algorithm is to find a path from the midpoint to the point using the Euclidean distance between the midpoints. arrive Regular path Each of them This path minimizes the cumulative distance cost, and the path must satisfy boundary conditions, monotonicity conditions, and continuity constraints. The boundary conditions are as follows: and ; The monotonic condition is: if Then for the next point on the path It needs to meet the following requirements. and ,in, , This indicates the index position of the previous point on the path in both time series. , This indicates the index position of the currently considered point on the path in both time series. The continuity constraint is: if Then for the next point on the path It needs to meet the following requirements. and ; The minimum cumulative distance corresponding to the optimal normalized path is the DTW distance between the two sequences: like If the value is less than the threshold, it is determined that the two trajectories originate from the same target, and the match is confirmed.
8. The method for displaying key maritime target information based on mobile augmented reality according to claim 1, characterized in that, The method of using layered overlay information display divides the AIS information required for navigation into a basic layer and a detailed layer to display information on key maritime targets, specifically including: The base layer includes the target ship's name, distance, and bearing, and is always visible on the display interface; the detailed layer includes the target ship's heading, speed, TCPA, DCPA, and MMSI, and is displayed when the driver clicks the target ship detection box on the display interface. A risk warning mechanism is set up in the process of displaying the information of key maritime targets. The safety range of CPA / TCPA is determined according to different water types, and the risk level is determined by calculating CPA / TCPA. The risk level includes low-risk targets, medium-risk targets, and high-risk targets. When displayed, low-risk targets are marked with a green semi-transparent border; medium-risk targets trigger a yellow flashing border and are accompanied by a single vibration alarm; high-risk targets display a red flashing border and mark the danger zone, and at the same time activate a continuous vibration alarm.
9. The method for displaying key maritime target information based on mobile augmented reality according to claim 1, characterized in that, The display of key maritime target information, designed for low-visibility scenarios and situations with limited visual perception, introduces a condition-triggered pure AIS display mode as a safety mechanism; the safety distance threshold is set as follows: The threshold for the duration of visual perception failure is For any AIS target, the triggering logic is as follows: Calculate the real-time distance between the target ship and the ship. ; The duration for which a target vessel is continuously failed to be identified by the target detection model. ; The pure AIS display mode is activated for the target vessel only if the following conditions are met: In the pure AIS display mode, AIS targets are displayed with a red flashing dashed frame, and the AIS-ONLY character is added to the label information to warn the driver that the target ship information originates from AIS and is not visually confirmed.