An augmented reality interaction system suitable for unmanned ship guided tours

Abstract

The application provides an augmented reality interaction system suitable for unmanned ship guiding, and relates to the fields of augmented reality technology and intelligent ship interaction technology. The system integrates high-precision Beidou / GPS positioning and nine-axis IMU sensors to obtain real-time ship position, heading, roll and pitch attitude information, and constructs a porthole direction vector set and a three-dimensional orientation reference to solve the window view misplacement problem in dynamic navigation. The system divides the guiding area into polygon explanation node areas, combines a position-pose dual-drive trigger mechanism to realize multi-modal synchronization (error < 50 ms) of visual, audio and graphic content, and constructs a heat model through user interaction data to dynamically optimize content pushing strategies. The scheme effectively solves the problems of dynamic environment adaptation, multi-terminal cooperation and personalized scheduling in the closed water area low-speed unmanned ship scene, and provides an AR solution with high immersion and strong stability for travel guiding.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of augmented reality technology and intelligent ship interaction, in particular to an augmented reality interaction system suitable for unmanned ship tour. BACKGROUND

[0002] With the application of augmented reality (AR) technology in the field of travel and tourism, the AR system for water transportation scenarios, especially low-speed unmanned ships (Unmanned Ship, USV), which is a full-automatic water robot that does not require manual remote control, relies on precise satellite positioning, sensors, and autonomous control systems to navigate on the water surface according to pre-set tasks, and the AR system for low-speed unmanned ships with a speed of ≤5 knots faces the following technical challenges:

[0003] 1. Insufficient dynamic adaptation of the environment: When the ship sails in the water area, the heading, roll, pitch, and other attitudes change in real time. The traditional manual broadcasting or time-triggered AR content cannot accurately match the dynamic scenery outside the window, resulting in window-scenery misalignment problems;

[0004] 2. Poor synchronization of multi-modal: The window visual content and voice commentary lack a linkage mechanism, and the time difference is generally more than 200ms, affecting the immersive experience;

[0005] 3. Lack of interaction feedback: The existing system lacks user behavior data collection and analysis functions, making it difficult to dynamically adjust the tour content according to the interests of tourists.

[0006] Therefore, there is an urgent need for an AR tour system that integrates ship state perception, multi-terminal collaboration, and intelligent interaction to meet the detailed needs of low-speed unmanned ships for red education, cultural interpretation, and other travel and tourism scenarios in closed / semi-open water areas. SUMMARY

[0007] To solve the technical problems of content triggering lag, audio-visual synchronization, and single interaction experience in existing AR tours, the present application proposes an augmented reality interaction system suitable for unmanned ship tour, which can accurately realize the immersive effect of "ship sailing and scene appearing" and "window moving and commentary giving", and is suitable for red education, cultural interpretation, and other travel and tourism scenarios in closed water areas such as lakes and inland rivers.

[0008] To achieve the above purpose, the technical solution adopted by the present application is:

[0009] An augmented reality interaction system suitable for unmanned ship tour, the system comprises:

[0010] A ship state perception module, by integrating Beidou or GPS positioning and nine-axis IMU sensors, real-time perception of ship position, heading, roll, and pitch attitude changes, and construction of a set of side window direction vectors to form a three-dimensional orientation reference;

[0011] The window view matching trigger module adopts a position-attitude dual-drive trigger mechanism and a polygon region matching algorithm, combined with AR image registration and parallax compensation technology, to compress the synchronization error of visual content, voice narration and graphic layers to within 50ms;

[0012] The multi-terminal synchronization module achieves content consistency synchronization between porthole displays, shipboard speakers, and mobile terminals through an event-driven mechanism and a timestamp compensation model.

[0013] The user interaction optimization module builds a popularity model and a reinforcement learning scheduling function based on user interaction data to dynamically optimize the content push strategy.

[0014] Specifically, the window view matching trigger module includes: dividing the tour area into several preset polygonal explanation node areas. ={ , ,..., }, each explanation node area They are all associated with the corresponding heading angle range. Where i is an integer greater than 0; the ray method is used to determine the point's position within the polygon, combined with the shortest distance from the point to the polygon's boundary. Optimize to enable the identification of polygonal explanation node regions;

[0015] As a ship navigates, its position and heading constantly change. The system monitors and analyzes the ship's status in real time. Once the ship's position P and heading angle... Meet the preset conditions and At that time, the system triggers the binding to the explanation node area. Content Index V i The content index V i The content index V corresponds to the explanation node area in the content library. i Includes 3D model M i Audio commentary track A i Image and text layer T i .

[0016] Compared with the prior art, the beneficial effects of the present invention are:

[0017] 1. Environmental dynamic adaptation capability: By integrating high-precision Beidou / GPS positioning (accuracy ±20cm) and a nine-axis IMU sensor (heading angle accuracy ±5°), the system can perceive the changes in the ship's position, heading, roll and pitch attitude in real time, and construct a window direction vector set to realize a three-dimensional orientation reference. This solves the problem of window view misalignment caused by the dynamic navigation of ships in traditional AR navigation, and achieves a seamless integration effect of "window facing the view".

[0018] 2. Multi-modal synchronization accuracy: Using position-pose dual driving trigger mechanism and polygon area matching algorithm, combined with AR image registration and parallax compensation technology, the synchronization error of visual content, voice commentary and graphic text layers is compressed to within 50ms (much lower than the 200ms perception threshold of humans), effectively eliminating the problems of audio-visual misplacement, voice delay and other factors affecting immersion in traditional systems.

[0019] 3. Multi-terminal collaborative stability: Through event-driven mechanism and timestamp compensation model, content consistency synchronization between multiple devices such as porthole display, in-ship loudspeaker, mobile terminal, etc. is realized, even in network fluctuations, it can rely on local cache to maintain short-time synchronization, solving the problem of out-of-control end delay (voice delay) caused by hardware synchronization signal dependence in traditional systems.

[0020] 4. Personalized interactive experience: Based on user interaction data (likes, answers, ratings), a heat model and reinforcement learning scheduling function are constructed to dynamically optimize content push strategies, significantly improving the accuracy of personalized recommendations for red education, cultural interpretation and other guided content, solving the problem of traditional system "one-size-fits-all" content push mode and visitor preference disconnection.

[0021] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation part. BRIEF DESCRIPTION OF DRAWINGS

[0022] Fig. 1 is a structure schematic diagram of an augmented reality interactive system for unmanned ship guided tours according to the present application;

[0023] Fig. 2 is a schematic diagram of the internal structure of an unmanned ship according to the present application;

[0024] Fig. 3 is a schematic diagram of the window view synchronization of an unmanned ship according to the present application. DETAILED DESCRIPTION

[0025] In order for those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application, so that the purpose, characteristics and advantages of the present application can be better understood. It should be understood that the embodiments shown in the drawings are not a limitation on the scope of the present application, but are only to illustrate the essential spirit of the technical solutions of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0026] Unless the context clearly requires otherwise, throughout the description and the claims, the words "comprise," "comprising," and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is to say, in the sense of "including, but not limited to." Words using the singular or plurality number of words include that and plural of said words unless the context clearly dictates otherwise.

[0027] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0028] As used in this description and the following claims, the singular "a," "an," and "the" include plural references unless the context clearly dictates otherwise. It is further noted that the term "or" as used in this description is generally used in the sense of "and / or," unless the context clearly dictates otherwise.

[0029] In the following description, for purposes of clarity, directional terms are used to describe the orientation of the various structures and features of the present application. It will be understood, however, that the language should not be construed to limit the scope of the application in this manner. For example, without limitation, "front," "back," "left," "right," "up," "down," "top," "bottom," "outward," "inward," and the like are not absolute terms of reference but are used for convenience only.

[0030] The following detailed description of implementations of the application will refer to specific implementations thereof. However, the description presented hereinafter is made merely by way of example and is not intended to limit the application as defined in the claims.

[0031] With the acceleration of digital transformation in the global tourism industry, tourists' demand for immersive and personalized travel experiences is growing. Augmented Reality (AR) technology, which seamlessly blends virtual content with real-world scenarios, has become a key tool for enhancing the interactivity and interest of tourism guides. However, in water transportation scenarios, especially for low-speed unmanned boats, the application of AR systems still faces multiple technical challenges, making existing solutions difficult to meet the needs of detailed guides in complex water environments.

[0032] 1. Insufficient dynamic adaptation of the environment, leading to prominent window scene misalignment issues: When a ship navigates in water, its heading, roll, pitch, and other attitudes change in real-time due to factors such as wind, waves, and currents. Traditional AR systems rely on static position triggers or manual on-demand, and cannot dynamically match the scenes outside the window. For example, when the ship turns or rolls, the virtual explanation content may be misaligned with the actual window scene, resulting in a fragmented experience of "scene not matching content." In existing technologies, some systems trigger content through pre-set fixed points, but they cannot adapt to the continuity of dynamic ship navigation, especially in closed water areas such as lakes and rivers, where this problem is more pronounced.

[0033] 2. Poor multi-modal synchronization, leading to audio-visual misalignment and affecting immersion: The core of AR guide lies in the coordination of multiple senses such as vision and hearing, but the existing system's window visual content and voice commentary generally have a time difference. For example, voice commentary may lag behind virtual model display, or text information and live video are out of sync, causing tourists to lose focus. Test data shows that the audio-visual synchronization error of traditional systems often exceeds 200ms, while the human perception threshold is only within 50ms, seriously damaging the immersive experience.

[0034] 3. Lack of interaction feedback, leading to lack of personalized content scheduling: Existing AR guide systems mostly use a "one-size-fits-all" content push mode, lacking real-time collection and analysis of tourist behavior data. For example, tourists may be more interested in certain historical sites than natural scenery, but the system cannot dynamically adjust the content order based on interaction data such as likes and answers, causing the recommended content to deviate from the tourists' preferences. In addition, traditional systems are difficult to optimize scheduling strategies through reinforcement learning and other algorithms, and the accuracy of personalized recommendations is insufficient, limiting the tourists' stay.

[0035] 4. Special nature of closed water area scenarios, causing adaptation difficulties for low-speed unmanned ships: The hydrological conditions of lakes, inland rivers and other closed water areas are complex (such as shoals and undercurrents), and low-speed unmanned ships (speed ≤ 5 knots) need to balance stability and guide functions. However, existing AR technology is mostly designed for land or high-speed ships, and does not fully consider the attitude perception accuracy of unmanned ships (such as heading angle recognition error needs to be ≤ ± 5°), multi-terminal collaboration (such as porthole display screens, speakers, and mobile terminals synchronization) and rapid deployment requirements. For example, some systems require high-precision map pre-labeling, but the electronic map coverage of closed water areas is insufficient, resulting in high deployment costs and long deployment cycles.

[0036] 5. Limitations of traditional solutions: In existing technologies, some systems trigger content through GPS positioning, but the positioning accuracy is insufficient (usually ≥ 1 meter), which cannot meet the meter-level positioning requirements; some systems use inertial measurement units (IMU) to compensate for attitude, but do not combine polygon area matching rules, leading to content triggering lag. In addition, multi-terminal synchronization mostly relies on hardware synchronization signals, lacking timestamp compensation models at the software layer, making it difficult to control the delay between terminals.

[0037] To solve the above problems, the application proposes an augmented reality interactive system suitable for unmanned ship tour guide combined with augmented reality (AR) and positioning awareness technology. Through the "position-pose-content" dynamic matching mechanism, the immersive tour experience of "talking while sailing, scene matching through the window" is realized. It is suitable for low-speed unmanned ship tour guide scene in closed water areas such as lakes and inland rivers, supports rapid deployment in multiple scenes such as red education and cultural interpretation, and significantly improves the immersion and stay time of tourists.

[0038] Specifically, as shown in Figs. 1-3 The augmented reality interactive system suitable for unmanned ship tour guide includes the following four core functional modules, and revolves around the four key functions of "accurate scene recognition, real-time triggering, multi-terminal synchronization, and interaction optimization":

[0039] (1) Ship state perception module, used for real-time positioning and attitude recognition.

[0040] The ship state perception module, as a key starting link in the augmented reality interactive system suitable for unmanned ship tour guide, undertakes the important task of real-time positioning and accurate attitude recognition. Its core purpose is to comprehensively and accurately obtain the dynamic spatial state information of the ship during navigation, laying a foundation for the subsequent whole system to realize the immersive tour experience of "talking while sailing, scene matching through the window".

[0041] In actual operation, this module makes full use of advanced positioning technology and sensor equipment to realize accurate data collection. On the one hand, by integrating the global satellite navigation system GNSS module, we can choose the mature and widely used Beidou system or GPS system. This module continuously and real-time collects the geographic coordinate information of the ship at a stable frequency of 1Hz, and accurately represents it as P(x,y). This high-frequency positioning collection method ensures that the subtle changes in the position of the ship can be captured in time, and the positioning accuracy of up to ±20cm provides reliable guarantee for the accurate positioning of the ship. Whether in complex lake water areas or winding inland river channels, the position of the ship can be accurately known.

[0042] On the other hand, a high-precision nine-axis IMU module is integrated, which can obtain key attitude parameters such as the heading angle θ, the roll angle φ and the pitch angle ψ of the ship in real time and accurately. The identification accuracy of the heading angle θ can be controlled within ±5°. The heading angle θ reflects the forward direction of the ship in the horizontal plane, which is crucial for determining the ship's navigation trajectory and direction guidance; the roll angle φ and the pitch angle ψ describe the attitude changes of the ship in the left and right tilting and the front and back pitching, respectively. These data can help the system accurately perceive the spatial attitude of the ship in different navigation states, such as the tilting of the ship when encountering water flow impact or turning.

[0043] In addition, in order to more comprehensively describe the visual angle information of the ship, based on the ship structure model, the module also constructs a set of side window direction vectors D = {d1, d2,..., d i}, which respectively calibrate the spatial orientation of each side window relative to the ship body (such as port, starboard, bow, stern), and combine the IMU attitude angles (including the heading angle θ, the roll angle φ and the pitch angle ψ) to construct the world coordinate system representation of the current orientation of each side window, forming a three-dimensional orientation reference, which provides an indispensable attitude reference for subsequent visual angle matching and image registration work. By specifying the specific direction of each direction of the ship, the system can more accurately match virtual content with the actual visual angle of the ship, ensuring that the virtual scene seen by the tourists through the side window is perfectly integrated with the real environment, thereby realizing the realistic effect of "window-to-scene", greatly improving the immersion of tourists during the unmanned ship travel guide process.

[0044] In some embodiments, the state vector is defined as: wherein, is the position (ECEF / planar projection), is the velocity, is the attitude represented by a quaternion.

[0045] Three types of factors are constructed:

[0046] 1) IMU pre-integration factor:

[0047]

[0048] wherein, IMU is the Inertial Measurement Unit, the IMU pre-integration factor converts the IMU measurement values between adjacent key frames into relative pose constraints, providing motion prior information for the optimization problem; is the IMU motion constraint residual; is the actual measurement value of the IMU module; (·) is the model prediction value based on the state variable.

[0049] 2) GNSS observation factor:

[0050]

[0051] where, is the residual of GNSS observation factor, reflecting the difference between the predicted value and the measured value of GNSS positioning; is the state vector at the kth moment; is the state vector contains the ship position (coordinates in ECEF coordinate system or planar projection coordinate system) in is the measured coordinate data of the GNSS module at the kth moment.

[0052] 3) Visual odometry factor:

[0053]

[0054] where, is the visual observation relative pose; is the predicted pose; i, j are adjacent frame indices; is the residual in visual odometry.

[0055] Next, a sliding window mechanism is used to achieve the overall optimization objective, which improves the optimization efficiency by limiting the calculation range. The optimization objective function is as follows:

[0056]

[0057] where, is the optimal state vector set; is the residual of the fth factor, measuring the deviation between the predicted value and the measured value under this factor; is the inverse of the covariance matrix of the fth factor residual, used to assign weights to the residual (the smaller the covariance, the more accurate the observation, the greater the weight, and the stronger the influence on the optimization result).

[0058] The system constructs a factor graph model based on IMU pre-integration factors, GNSS observation factors and visual odometry factors. The ship state perception module constructs a least squares target containing IMU pre-integration factors, GNSS observation factors and visual odometry factors based on the factor graph model, and solves the optimal state in a sliding window by using an incremental sparse solver (such as iSAM2), so as to maintain the attitude accuracy under the condition of temporary loss of GNSS or nonlinear disturbance. The incremental sparse solver is an algorithm tool specially used for efficiently solving large-scale sparse nonlinear optimization problems, and is widely used in the fields of SLAM (simultaneous localization and mapping) and robot navigation. Its core feature is to utilize the sparse structure of the problem and reduce repeated calculations through an incremental updating strategy, thereby significantly improving the solving efficiency while ensuring accuracy.

[0059] (2) A window matching trigger module for realizing intelligent interpretation driven by "position + attitude".

[0060] The window matching trigger module is the core driving module for realizing the intelligent interpretation function in the immersive navigation experience of "sailing and speaking, window matching and scene". It mainly undertakes the task of accurately and timely activating the interpretation content corresponding to the window based on the real-time position and heading state of the ship, so as to ensure that the tourists can obtain interpretation information that is highly consistent with the real scene in front of them and lively and rich during the sailing process.

[0061] Specifically, the window matching trigger module adopts a set of fine and scientific mechanism to realize its function. First, in order to accurately push the interpretation content according to the different positions and heading states of the ship, the module divides the navigation area into a plurality of preset polygonal interpretation node area sets ={ , ,..., } to divide the entire navigation area into a plurality of meaningful interpretation areas, and each interpretation node area is associated with a corresponding heading angle range . In the augmented reality interactive system suitable for unmanned ship navigation, the polygonal interpretation node area refers to a plurality of geometric areas with clear boundaries divided according to geographical features, point distribution or navigation path, and each area is defined by the vertex coordinates of the polygon (such as triangle, quadrilateral, etc.) to define its spatial range, and is associated with a specific heading angle range and interpretation content index. Such design enables the system to quickly determine the interpretation area where the ship is located according to the current heading of the ship, thereby providing accurate positioning basis for subsequent interpretation content triggering.

[0062] In some embodiments, the ray method of point in polygon can be used for judgment, and the nearest distance of point to polygon boundary Optimization is performed to realize polygon explanation node area judgment.

[0063] When the ship is in the process of sailing, its position and heading state are constantly changing. The system will monitor and analyze the state of the ship in real time. Once the position P(x, y) and the heading angle of the ship meet the preset conditions , and , the system will quickly and accurately trigger the content index V i bound to the explanation node area; wherein the content index V i has a corresponding relationship with the explanation node area in the content library. For the judgment of , the polygon associated heading preset interval is used, and only when , it is considered that the heading matches; in the formula, is the deviation of the actual heading angle and the preset interval; is the tolerance threshold of the heading matching. When the calculated minimum heading deviation is less than or equal to the threshold, it is considered that the ship heading meets the matching condition.

[0064] In some embodiments, the view angle confidence C is used to comprehensively judge whether the subsequent explanation content trigger provides accurate positioning basis, and the expression of C is as follows:

[0065]

[0066] In the formula, R is the speed or roll-pitch stability index (such as roll angle variance within nearly 2s), and the weight is set to ; the view angle confidence .

[0067] wherein, is the distance factor weight, corresponding to the nearest distance of the ship to the boundary of the preset explanation area (polygon) ; this coefficient weight is the highest, meaning that "whether the ship is in the target area" is the core basis for content triggering. is the heading factor weight, corresponding to the deviation of the actual heading angle of the ship and the preset interval , the weight value is only second to the distance factor weight value, ensuring that the AR content and the window view outside the ship are accurately matched, avoiding "window view misplacement". is the stability factor weight, corresponding to the speed or roll-pitch stability index R, the weight value is the lowest, used to filter the false triggering caused by short-time shaking of the ship body, and to ensure the triggering stability; is the standard deviation of the distance factor, used to adjust the influence degree of the distance deviation on the confidence C; is the standard deviation of the heading factor, used to adjust the influence degree of the heading deviation on the confidence C; The nearest distance from the current position of the ship to the boundary of the preset polygon explanation area is a key parameter for judging whether the ship is in the target explanation area.

[0068] Further, each content index V i is not simply a single information, but contains a three-dimensional model M i , a voice commentary track A i , and a text and image layer T i ; wherein the three-dimensional model M i can present objects or scenes related to the explanation content in a realistic way, making tourists feel as if they can see the real object in front of them; the voice commentary track A i provides lively and detailed voice explanations, telling tourists about the historical stories, cultural background, and other information related to the current real scene, so that tourists can not only enjoy visually, but also have a rich auditory experience; the text and image layer T i further supplements and explains the explanation content in the form of intuitive pictures and text, providing more comprehensive and easy-to-understand information for tourists. These three elements complement each other and together form the complete AR material of the explanation node, creating an immersive explanation scene for tourists, and enabling tourists to better understand the cultural connotations of the attractions they pass through during the unmanned ship travel guide, greatly improving the tourists' sense of participation and immersion.

[0069] In some embodiments, the three-dimensional model M i can be subjected to AR image registration and parallax compensation to ensure that the virtual content is perfectly integrated with the real window view.

[0070] Specifically, in the augmented reality interaction system applicable to unmanned ship guide, the key to achieving the realistic effect of "window-to-scene" is to eliminate the virtual image misalignment problem caused by changes in the ship's attitude, and to ensure that the virtual content is perfectly integrated with the real window view. To this end, the system, after triggering the explanation content, constructs an AR image registration and parallax compensation mechanism. This mechanism takes the real-time attitude angle data provided by the nine-axis IMU module as the core basis, fully utilizes key parameters such as roll angle φ and pitch angle ψ, and combines the pre-constructed direction vector set D = {d1, d2,..., d i} to perform all-around coordinate and projection transformation on the three-dimensional model M i . In the specific implementation process, the system first constructs an affine transformation matrix using the direction vector d i and the attitude angle, which can accurately describe the spatial transformation relationship of the virtual content relative to the real world under the current attitude of the ship. Subsequently, based on the affine transformation matrix, the system performs coordinate rotation and translation compensation on the three-dimensional model M i to ensure that the three-dimensional model M iThe position and angle can be adjusted in real time with the change of the ship's attitude, and the relative position relationship with the real window view is always maintained. In addition, to solve the parallax problem in the AR view, the system further corrects the parallax of the AR view, adjusts the angle between the three-dimensional model M i and the observer's line of sight, so that the three-dimensional model M i and the real window view are visually "parallelly aligned", thereby eliminating the image misplacement phenomenon caused by the difference in viewing angle. Finally, through image rendering, the corrected three-dimensional model M i is frame-synchronized and synthesized with the real window view taken by the real-time camera, and is displayed through the porthole transparent OLED display screen or glass projection area. In this process, the system strictly maintains the frame rate above 30fps, and the porthole transmittance is not less than 70%, ensuring the smoothness and naturalness of the visual effect, and bringing an immersive tour experience to the tourists.

[0071] (3) Multi-terminal synchronization module, for realizing multi-modal consistency display through event-driven and time compensation.

[0072] In the augmented reality interaction system suitable for unmanned ship tour, in order to ensure that the tourists obtain seamless, smooth and highly immersive tour experience, and prevent the "sound and picture misplacement" or "voice delay" between the window view and the voice, text and picture content from affecting the experience, the multi-terminal synchronization module realizes multi-modal consistency display through the event-driven mechanism and the timestamp compensation algorithm.

[0073] In the unmanned ship tour scene, the tourists receive tour information through various terminal devices, including the porthole transparent OLED display screen or glass projection area, which is responsible for presenting the AR superimposed content, fusing the virtual landscape, information and real window view to bring better visual effect to the tourists; the in-ship loudspeaker synchronously plays the voice commentary track, telling the tourists the historical stories and cultural connotations behind the scenic spots with lively and professional voice, so that the tourists can also understand in depth in hearing; the on-ship tablet and the tourists' mobile phones present the text and picture auxiliary information and interactive answering modules, through which the tourists can view more detailed text and picture materials and participate in interactive answering, increasing the interest and participation of the tour.

[0074] In some embodiments, the porthole transparent OLED display screen or glass projection area adopts bidirectional time synchronization to estimate clock offset O and round-trip time RTT, so as to ensure that the AR content of the porthole transparent OLED display screen or glass projection area is accurately synchronized with the real scene and other devices (such as voice and text), and avoid visual misplacement or information delay caused by time deviation.

[0075] The master control (on-ship server) sends a timestamp t 1, the terminal receives the time t2, the terminal returns a reply with its own time t 3, the master receives the reply time t 4, the expression is calculated as follows:

[0076]

[0077] The real-time estimation of dynamic propagation delay δ uses an exponential weighted moving average:

[0078]

[0079] where, δkis the dynamic propagation delay estimation value at the current time k (unit: ms), used for synchronization compensation; RTT is the round-trip time, i.e., the time required for the master to return to the terminal (ms); δk-1is the delay estimation value calculated at the previous time k-1 (ms); α is the smoothing factor (range 0.1-0.3), controlling the weight of the current measurement .

[0080] Due to the differences in processing speed, signal transmission, and other factors of different terminal devices, it is easy to have different content synchronization. To solve this problem, the system uses an event-driven mechanism and a timestamp compensation algorithm. The event-driven mechanism sends synchronization instructions to each terminal device in a timely manner according to various events occurring in the system, such as the arrival of a ship at a specific location, the triggering of specific content, etc., to ensure that they perform the corresponding operations at the correct time.

[0081] The timestamp compensation algorithm is the core technology to ensure synchronization accuracy. The synchronization mechanism of the multi-terminal synchronization module uses a timestamp compensation model, whose expression is:

[0082] T = max(T_local, T_content) + δ

[0083] where δ is the dynamically estimated network propagation delay; T_local is the current time of the local terminal, representing the local system clock time of the receiving end device (such as the porthole display screen, the in-ship loudspeaker, the tourist's mobile phone, etc.), which serves as the reference time for measuring the real-time state of the content when it arrives at the terminal; T_content is the generation or sending time of the content, representing the timestamp of the AR superimposed content, voice commentary, or graphic information generated or marked at the sending end (such as the control server), which is used to track the whole process from the generation to the transmission of the content.

[0084] The system adopts an event-driven mechanism for content triggering and distribution, and a uniform play event is issued by the shipborne master control unit, and each terminal realizes synchronization according to a local clock and a timestamp compensation model. The master control end calculates the clock offset by sending the timestamp and the round-trip time delay (RTT), and dynamically estimates the propagation delay δ by using the exponential weighted moving average algorithm, so as to realize high-precision time alignment. The system can control the synchronization error of the porthole picture, audio and terminal display within 50ms, ensuring consistent sound and picture, natural interaction and immersive experience.

[0085] Firstly, the timestamp compensation model can dynamically adjust the play time of the content according to the signal transmission delay between different terminal devices. For example, when the voice commentary needs to be synchronized with a certain picture in the window view, the system will calculate the appropriate δ value according to the propagation delay of the voice signal from the loudspeaker to the ear of the tourist, and the display delay of the window view picture on the display screen, and accurately compensate the play time of the voice and picture. Through this fine adjustment, the system can control the inter-terminal synchronization error within 50ms, which is almost imperceptible to the tourists, thereby ensuring that the tourists can enjoy seamless and highly consistent experience throughout the tour.

[0086] Secondly, for the text and picture auxiliary content displayed synchronously by the mobile terminal, the system can ensure its consistency. Whether the tourist uses the shipborne tablet or his own mobile phone, the text and picture information he sees is exactly the same and synchronized with the porthole AR content and voice commentary. In this way, the tourist can obtain detailed text and picture materials through various devices, further deepen his understanding of the scenic spot, and there will be no inconsistent or asynchronous information due to device differences. For example, when the porthole displays a certain natural landscape, the mobile terminal will synchronously display the formation reason, ecological characteristics and other related text and picture information of the landscape, helping the tourist to understand the scenic spot more comprehensively.

[0087] Finally, considering the complexity of the network environment, the system has the ability to cope with unstable network. When the network fluctuates or is interrupted, the system can maintain short-time synchronization through local cache data. Some tour content is pre-stored in the local cache, and in the case of unstable network, the system can call these cache data to ensure that the porthole AR content, voice commentary and text and picture auxiliary information can still be synchronized and displayed within a certain time, avoiding the interruption of the tour or the confusion of the content due to network problems, thereby providing stable and reliable tour service for tourists.

[0088] (4) User interaction optimization module, for intelligent scheduling based on behavior feedback content.

[0089] In the augmented reality interactive scene of unmanned ship guiding, in order to meet the personalized needs of different tourists and realize the precision and dynamic adaptation of content scheduling, the core of the user interaction optimization module is to learn the user interaction behavior in real time, and to scientifically and reasonably sort and optimize the content according to the learning results, so as to provide the tourists with more targeted and more interest-preferred guiding content.

[0090] Specifically, the interactive data collection is the basic link of the whole user interaction optimization process. In the process of using the terminal device for guiding experience, the tourists can interact with the system in various ways, for example, like the content of interest, express approval and love for the content; participate in interactive answering link, which not only increases the interest of guiding, but also reflects the mastery degree and interest point of the tourists on the specific knowledge; score the guiding content, and intuitively give their evaluation on the content quality, explanation effect, etc. These interactive data can understand the needs and interests of tourists, and provide basis for subsequent content optimization.

[0091] After obtaining the interactive data, the heat modeling and scheduling function are used to realize the intelligent scheduling of content. The scheduling function expression is:

[0092] S_next=argmax i (α i ×H i (t)+β i ×R i )

[0093] Wherein, the heat value H i (t) of the content S_next suitable for pushing to the tourists is determined by heat modeling, H i (t) represents the heat value of the i-th content at the current time, which is not fixed, but is updated in real time and dynamically according to the user behavior. When a large number of users show strong interest in a content, such as frequent likes and active participation in related interactions, the heat value of the content will rise accordingly, reflecting the degree of attention of the tourists at the current time. And R i is the system weight or quality score of the content, which considers the value of the content from the system level, such as the accuracy, richness, professionalism, etc. of the content. The system weight or quality score is an objective evaluation standard, which ensures that the pushed content not only meets the interests of the tourists, but also has a high quality level. α i and β i are the weight coefficients of the reinforcement learning model dynamic optimization, which plays a key role in balancing and adjusting in the whole scheduling function, and can dynamically adjust the relationship between content interest priority, quality orientation and task coverage according to the actual situation. For example, in some cases, in order to meet the immediate needs of tourists for popular content, α iThe value of β may be relatively large, making the popularity value play a more important role in content selection; however, when focusing on the overall quality of the navigation content and the completion rate of system tasks, β... i The value of can be appropriately increased to ensure that the pushed content not only matches the interests of tourists but also guarantees the systematicity and completeness of the tour guide content. Through this dynamic optimization method, the reinforcement learning model can continuously adapt to changes in tourist behavior and the actual needs of the system, achieving personalized and intelligent content scheduling.

[0094] In practical applications, the user interaction optimization module continuously collects and analyzes tourist interaction data, constantly adjusting various parameters in the heat modeling and scheduling function to ensure that content recommendations more accurately match tourists' interests and needs. For example, when the system detects that a group of tourists shows high interest in historical and cultural content, it will promptly increase the frequency and weight of related content recommendations; conversely, it will reduce recommendations for lower-quality or less popular content, or even optimize or eliminate it altogether. This intelligent content scheduling method based on behavioral feedback not only enhances tourists' satisfaction and engagement during the unmanned boat cultural tourism tour but also effectively improves the utilization efficiency of tour resources, providing tourists with a higher-quality, more personalized, and immersive tour experience.

[0095] In typical application scenarios, this invention is deployed as a low-speed unmanned vessel navigation platform on lakes or inland waterways. Through GNSS and IMU fusion positioning, the system achieves centimeter-level position recognition and high-precision heading detection, with content triggering latency of less than 200ms, audio-visual synchronization error of less than 50ms, and stable and smooth virtual overlay images. The system supports rapid configuration of explanation areas and content nodes via electronic maps, and can be flexibly applied to various navigation scenarios such as red tourism education, cultural explanations, and immersive cultural tourism displays. The overall system operates stably, with natural visuals and a seamless experience, effectively solving the problems of delayed content triggering, audio-visual asynchrony, and limited interaction in traditional AR navigation, significantly improving the intelligence and immersiveness of ship-based cultural tourism navigation.

[0096] The application provides an augmented reality interaction system specially designed for unmanned ship guiding, which innovatively combines four core technical modules of ship state perception, window view accurate matching triggering, multi-terminal event-driven synchronization and user behavior feedback optimization. Through high-precision positioning and real-time capture of ship dynamics by nine-axis IMU sensors, the system can accurately identify the position and attitude changes of the ship body, realize the seamless integration of AR content and the real scene outside the window, and trigger the explanation content by using the position and attitude double driving mechanism to ensure the multi-modal synchronous display of visual, auditory and graphic information, with the error controlled within the human perception threshold. Further, the event-driven and timestamp compensation algorithm ensures the content consistency among multiple terminals (porthole display, voice commentary, mobile device), and even in the case of network fluctuations, it can maintain short-time synchronization. Finally, with the help of user interaction data (likes, answers, scores), the content push strategy is dynamically adjusted to improve the guiding personalization and visitor participation. The system effectively overcomes the technical bottlenecks of insufficient environmental adaptation, audio-visual dislocation and single content of traditional AR guiding in dynamic water areas, significantly enhances the immersion and intelligent level of closed / semi-open water area low-speed unmanned ship tourism scenes, and provides an efficient and flexible solution for red education, cultural interpretation and other applications.

[0097] Although the application has been described in detail with reference to the preferred embodiments, the application is not limited thereto. Those skilled in the art can make various equivalent modifications or replacements to the embodiments of the application without departing from the spirit and essence of the application, and these modifications or replacements shall be within the scope of the application or any skilled person in the art can easily think of changes or replacements within the technical range disclosed by the application, which shall be covered within the protection scope of the application. Therefore, the protection scope of the application shall be subject to the protection scope of the claims.

Claims

1. An augmented reality interaction system suitable for unmanned boat tour guiding, characterized in that, The system comprises: A ship body state sensing module, which senses the ship body position, heading, roll and pitch attitude changes in real time by integrating Beidou or GPS positioning and nine-axis IMU sensors, and constructs a three-dimensional direction reference by forming a set of side window direction vectors; A window view matching trigger module, which adopts a position-attitude dual-drive trigger mechanism and a polygon region matching algorithm, combines AR image registration and parallax compensation technology, and compresses the synchronization error of visual content, voice commentary and graphic text layers to within 50 ms; A multi-terminal synchronization module, which realizes the content consistency synchronization among the side window display, the in-ship loudspeaker and the mobile terminal through an event-driven mechanism and a timestamp compensation model; A user interaction optimization module, which constructs a heat model and a reinforcement learning scheduling function based on user interaction data, and dynamically optimizes the content push strategy; The window view matching triggering module specifically comprises: dividing a guide area into a plurality of preset polygon explanation node area sets ={ , ,..., } and each explanation node area is associated with a corresponding heading angle range , i is an integer greater than 0; the polygon explanation node area is judged by using a ray method of a point in a polygon and combining the nearest distance of a point to a polygon boundary for optimization. As a ship navigates, its position and heading constantly change. The system monitors and analyzes the ship's status in real time. Once the ship's position P and heading angle... Meet the preset conditions and At that time, the system triggers the binding to the explanation node area. Content Index V i The content index V i The content index V corresponds to the explanation node area in the content library. i Includes 3D model M i Audio commentary track A i Image and text layer T i .

2. The system of claim 1, wherein, For the judgment of the polygon associated heading preset interval is adopted the heading is considered to match only if ; in the formula, is the deviation of the actual heading angle from the preset interval; is the tolerance threshold of the heading match, and the ship heading is considered to meet the matching condition when the calculated minimum heading deviation is less than or equal to the tolerance threshold.

3. The system of claim 2, wherein, The ship body state sensing module specifically comprises: an integrated nine-axis IMU module for obtaining attitude parameters of the ship, wherein the attitude parameters include a heading angle θ, a roll angle φ and a pitch angle ψ, and the heading angle θ recognition accuracy is controlled within ±5°.

4. The system of claim 3, wherein, Based on the ship structure model, the ship body state perception module also constructs a porthole direction vector set D = {d1, d2,..., d i}, which respectively calibrates the spatial orientation of each porthole relative to the ship body, including port, starboard, bow, and stern; in combination with the attitude parameter, the ship body state perception module constructs a world coordinate system representation of the current orientation of each porthole, forming a three-dimensional orientation reference.

5. The system of claim 1, wherein, The ship body state sensing module further comprises: Definition of state vector , the expression is: wherein, is the position, is the velocity, is the attitude represented by a quaternion; Three types of factors are constructed: 1) IMU pre-integration factor: where the IMU pre-integration factor provides motion prior information for the optimization problem by converting IMU measurements between adjacent keyframes into relative pose constraints; is the IMU motion constraint residual; is the actual measurement of the IMU module; (·) is the model prediction based on the state variable; 2) GNSS observation factor: wherein, is a residual of GNSS observation factor, reflecting the difference between the predicted value and the measured value of GNSS positioning; is a state vector at the kth moment; is a state vector contains the position of the ship in the middle; is the measured coordinate data of the GNSS module at the kth moment; 3) Visual odometry factor: wherein, visu al observation relative pose; (·) is a predicted pose; i,j are adjacent frame indices; is a residual in visual odometry; Next, a sliding window mechanism is used to achieve the overall optimization goal, and the optimization objective function is as follows: wherein, is the optimal state vector set; is the residual of the fth factor; is the inverse of the covariance matrix of the fth factor residual, used to weight the residual. The system constructs a factor graph model based on the IMU pre-integration factor, the GNSS observation factor and the visual odometry factor, the ship body state sensing module constructs a least squares target containing the IMU pre-integration factor, the GNSS observation factor and the visual odometry factor based on the factor graph model, and uses an incremental sparse solution algorithm to solve the optimal state within the sliding window.

6. The system of claim 5, wherein, The window view matching trigger module further comprises: a view confidence C is used to comprehensively judge whether to provide accurate positioning basis for subsequent explanation content triggering, and the expression of the view confidence C is as follows: where R is the speed or roll-pitch stability indicator; the weight is set to ; the view angle confidence ; in, The distance factor weight corresponds to the shortest distance from the ship to the boundary of the preset polygonal explanation area. ; The heading factor weight corresponds to the deviation between the ship's actual heading angle and the preset range. ; For the stability factor weight, corresponding to the speed or roll / pitch stability index R; The standard deviation of the distance factor is used to adjust for distance deviation. The degree of influence of confidence level C; The standard deviation of the heading factor is used to adjust the heading deviation. The degree of influence of confidence level C.

7. The system of claim 6, wherein, performing AR image registration and parallax compensation on the three-dimensional model M i performing AR image registration and parallax compensation, specifically including: First, a direction vector d in the set of direction vectors D of the porthole is used i An affine transformation matrix is constructed with the attitude parameters, which is used to describe the spatial transformation relationship of the virtual content relative to the real world in the current attitude of the ship. Subsequently, based on the affine transformation matrix, the system performs coordinate rotation and translation compensation on the three-dimensional model M i , ensuring that the three-dimensional model M i can adjust its position and angle in real time as the ship's attitude changes, maintaining the relative position relationship with the real outdoor scenery. In addition, in view of parallax appearing in the AR view, the system corrects the parallax of the AR view by adjusting the three-dimensional model M i and the angle between the line of sight of the observer, so that the three-dimensional model M i and the real outdoor scene are visually parallel and aligned, thereby eliminating the image misplacement phenomenon caused by the difference in view angle. Finally, the modified three-dimensional model M i Frame synchronization synthesis with the real scene outside the window photographed by the real-time camera, and display through the porthole transparent OLED display screen or glass projection area.

8. The system of claim 7, wherein, The multi-terminal synchronization module specifically comprises: a clock offset is estimated by using bidirectional time synchronization in a porthole display area and a round-trip time , ensuring that AR content displayed in the porthole is synchronized with the real scene and the mobile terminal. Master controller sends timestamp t 1. Terminal receives timestamp t 2. Terminal returns reply with its own time t 3. Master controller receives reply time t 4. Calculate expression as follows: The dynamic propagation delay δ is estimated in real time, and an exponential weighted moving average is used: wherein, is the dynamic propagation delay estimate value for the current time instant k, used for synchronization compensation; is the round-trip delay, representing the time needed for the master controller end to the end terminal and back; is the delay estimate value calculated for the previous time instant k-1 ; a is the smoothing factor, ranging from 0.1 - 0.3, controlling the weight of the current measurement .

9. The system of claim 8, wherein, The multi-terminal synchronization module uses a timestamp compensation model to realize the content consistency synchronization among terminals; wherein the expression of the timestamp compensation model is: T=max(T_local,T_content)+δ In the formula, δ is the dynamic propagation delay; T_local is the current time of the local terminal, which represents the local system clock time of the receiving end device as the reference time, used to measure the real-time state when the content arrives at the terminal; T_content is the generation or sending time of the content, which represents the timestamp of the AR superimposed content, voice commentary or graphic information generated or marked at the sending end, used to track the whole process from generation to transmission of the content; The timestamp compensation model can calculate the appropriate δ value according to the signal transmission delay between different terminal devices, and compensate the playing time of the content.

10. The system of claim 9, wherein, The user interaction optimization module specifically comprises: Interaction data is collected, including: like operation data on the content of interest; data in the interactive answering process, and score values for scoring the guide content; After the interaction data is acquired, a heat modeling and reinforcement learning scheduling function is used to realize intelligent scheduling of content, and an expression of the scheduling function is as follows: S_next = argmax i (α i ×H i (t)+β i ×R i ) Wherein, the heat value H of the content S_next suitable for pushing to the tourists is determined through the heat modeling i (t), H i (t) represents the heat value of the i-th content at the current time; R i is the system weight or quality score of the content, considering the value of the content from the system level; α i and β i is a weight coefficient dynamically optimized by the reinforcement learning model, dynamically adjusting the relationship between content interest priority, quality orientation and task coverage according to actual conditions.

Images