Multi-modal data fusion method and system for indoor and outdoor seamless positioning of unmanned ship

Abstract

The invention discloses a multi-modal data fusion method and system for indoor and outdoor seamless positioning of an unmanned ship, and relates to the technical field of measurement, and the method comprises the steps: building a multi-modal positioning architecture comprising a visual camera, an RTK module, an IMU unit and an SLAM module, and collecting an environment image, RTK positioning, IMU motion and SLAM composition data of the unmanned ship; switching a positioning sensor according to the operation scene and the signal intensity, generating a positioning source switching mechanism, activating a corresponding data source, and obtaining multi-modal positioning data; and constructing a fusion channel, including extended Kalman filtering and a deep neural network, performing weighted fusion on the multi-modal data, and generating an unmanned ship dynamic positioning map. Therefore, the technical effects of diversification of positioning modes, improvement of data fusion efficiency and enhancement of positioning precision are achieved.

Description

Technical Field

[0001] This invention relates to the field of measurement technology, and in particular to a multimodal data fusion method and system for seamless indoor and outdoor positioning of unmanned vessels. Background Technology

[0002] With the continuous development of unmanned surface vessel (USV) technology, its application in various scenarios is becoming increasingly widespread. However, traditional USV positioning methods have certain limitations in indoor and outdoor environments. Outdoors, satellite positioning systems (such as GPS) can provide relatively accurate positioning information, but indoors or in areas where satellite signals are blocked (such as locks, docks, etc.), their positioning accuracy will drop significantly or even become ineffective.

[0003] Existing multimodal positioning technologies have several shortcomings when applied to seamless indoor and outdoor positioning of unmanned surface vessels. On the one hand, most methods do not fully consider the impact of changes in indoor and outdoor environments on the signal strength of different sensors, failing to achieve intelligent sensor switching and resulting in a sharp drop in positioning accuracy when the environment changes. On the other hand, data fusion algorithms are relatively simple, making it difficult to effectively integrate multiple types of data and failing to fully leverage the advantages of each sensor to construct a high-precision positioning map. Summary of the Invention

[0004] This invention provides a multimodal data fusion method and system for seamless indoor and outdoor positioning of unmanned vessels, in order to solve the technical problems of single positioning mode, poor positioning data fusion efficiency, and impact on positioning accuracy in the prior art, and achieve the technical effects of diversified positioning modes, improved data fusion efficiency, and enhanced positioning accuracy.

[0005] In a first aspect, the present invention provides a multimodal data fusion method for seamless indoor and outdoor positioning of unmanned surface vessels, wherein the multimodal data fusion method for seamless indoor and outdoor positioning of unmanned surface vessels includes: A multimodal positioning architecture is constructed, which includes a visual camera, an RTK module, an IMU inertial unit, and a SLAM mapping module. The target unmanned surface vessel's environmental image data, RTK positioning data, IMU motion data, and SLAM mapping data are acquired through the multimodal positioning architecture.

[0006] Based on the indoor and outdoor operating scenarios of the unmanned vessel and the sensor signal strength, the switching analysis of each positioning sensor in the multimodal positioning architecture is performed to generate a positioning source switching mechanism.

[0007] The aforementioned positioning source switching mechanism is introduced to switch and activate the environmental image data, RTK positioning data, IMU motion data, and SLAM mapping data to obtain target multimodal positioning data.

[0008] A multimodal data fusion channel is constructed, which includes an extended Kalman filter channel and a deep neural network channel.

[0009] The multimodal data fusion channel is used to perform weighted fusion of the target multimodal positioning data to construct a dynamic positioning map of the unmanned vessel.

[0010] In one feasible implementation, the location source switching mechanism includes: Element identification is performed on indoor and outdoor operating scenes of unmanned vessels to obtain a set of operating scene elements.

[0011] Based on the set of operational scenario elements, the indoor and outdoor operational scenarios of the unmanned vessel are clustered to obtain a set of key operational scenarios.

[0012] Positioning interference analysis is performed on each of the key operation scenarios in the set of key operation scenarios in turn to obtain a set of positioning interference parameters for each operation scenario.

[0013] Based on the set of positioning interference parameters for the operating scenario and the sensor signal strength, the switching analysis of each positioning sensor in the multimodal positioning architecture is performed to construct a positioning source switching mechanism.

[0014] In one feasible implementation, the establishment of the location source switching mechanism includes: Based on the set of positioning interference parameters for the operating scenario, the positioning requirements of each positioning sensor in the multimodal positioning architecture are analyzed to obtain the positioning requirement parameters for the operating scenario.

[0015] Based on the sensor signal strength, the available thresholds for each positioning sensor in the multimodal positioning architecture are divided to obtain the available thresholds for positioning signal strength.

[0016] Based on the positioning requirement parameters of the operating scenario and the available threshold of the positioning signal strength, the sensor switching of the multimodal positioning architecture is analyzed to determine the positioning sensor switching conditions and the positioning sensor switching priority.

[0017] Based on the location sensor switching conditions and location sensor switching priorities, the location source switching mechanism is constructed.

[0018] In one feasible implementation, obtaining the target multimodal localization data includes: Based on the environmental image data, RTK positioning data, IMU motion data, and SLAM mapping data, scene recognition and signal strength assessment are performed to determine the target operating scene and the real-time signal strength of the sensors.

[0019] The aforementioned positioning source switching mechanism is introduced to switch and match the target operating scene and the real-time signal strength of the sensor to obtain the target positioning sensor source.

[0020] The target multimodal positioning data is obtained by activating and calling the environmental image data, RTK positioning data, IMU motion data and SLAM mapping data according to the source of the target positioning sensor.

[0021] In one feasible implementation, the construction of the multimodal data fusion channel includes: Based on the data characteristics of the multimodal localization architecture, a Kalman filter is initialized and then optimized and updated to obtain an extended Kalman filter channel.

[0022] Based on the data characteristics of the multimodal localization architecture, a deep neural network channel is constructed. The deep neural network channel includes an input layer, a hidden layer, and an output layer. The hidden layer includes a convolutional layer, a fully connected layer, and a recurrent layer.

[0023] The extended Kalman filter channel and the deep neural network channel are connected in parallel and integrated to construct the multimodal data fusion channel.

[0024] In one feasible implementation, constructing the dynamic positioning map of the unmanned vessel includes: The target multimodal localization data is aligned according to timestamps to obtain multimodal localization sequence data.

[0025] The multimodal positioning sequence data is normalized and noise filtered to obtain multimodal standard positioning sequence data.

[0026] A global static map is constructed based on the historical modal data of the unmanned vessel.

[0027] Based on the global static map, the multimodal data fusion channel is used to perform weighted fusion and dynamic map updates on the multimodal standard positioning sequence data to obtain a dynamic positioning map of the unmanned vessel.

[0028] In one feasible implementation, obtaining the dynamic positioning map of the unmanned vessel includes: The positioning estimation information of the unmanned vessel is obtained by performing positioning estimation on the multimodal standard positioning sequence data based on the extended Kalman filter channel.

[0029] The deep neural network channel is used to perform positioning prediction on the multimodal standard positioning sequence data to obtain unmanned vessel positioning prediction information.

[0030] Based on the target operating scenario and the confidence levels of the extended Kalman filter channel and the deep neural network channel, the channel fusion weight factor is determined.

[0031] Based on the global static map, the unmanned vessel positioning estimation information and the unmanned vessel positioning prediction information are weighted and fused according to the channel fusion weight factor, and the map is locally updated to obtain the unmanned vessel dynamic positioning map.

[0032] In one feasible implementation, obtaining the unmanned vessel positioning estimation information includes: Based on the extended Kalman filter channel, the state initialization and state prediction of the multimodal standard positioning sequence data are performed to obtain the unmanned vessel state prediction value.

[0033] By combining the multimodal standard positioning sequence data and the unmanned vessel state prediction value, a state update estimation is performed to obtain the unmanned vessel positioning estimation information.

[0034] In one feasible implementation, obtaining the unmanned vessel positioning prediction information includes: The deep neural network channel is used to extract features from the multimodal standard positioning sequence data to obtain the unmanned vessel positioning associated feature set.

[0035] Based on the unmanned vessel's positioning association feature set, positioning fusion prediction is performed, and the unmanned vessel's positioning prediction information is output.

[0036] Secondly, the present invention also provides a multimodal data fusion system for seamless indoor and outdoor positioning of unmanned surface vessels, wherein the multimodal data fusion system for seamless indoor and outdoor positioning of unmanned surface vessels includes: A multimodal acquisition capability building component is used to build a multimodal positioning architecture. The multimodal positioning architecture includes a visual camera, an RTK module, an IMU inertial unit, and a SLAM mapping module. The multimodal positioning architecture is used to acquire environmental image data, RTK positioning data, IMU motion data, and SLAM mapping data of the target unmanned surface vessel.

[0037] The switching mechanism configuration component is used to analyze the switching of each positioning sensor in the multimodal positioning architecture based on the indoor and outdoor operating scenarios of the unmanned vessel and the sensor signal strength, and generate a positioning source switching mechanism.

[0038] The positioning data acquisition component is used to introduce the positioning source switching mechanism to switch and activate the environmental image data, RTK positioning data, IMU motion data and SLAM mapping data to obtain target multimodal positioning data.

[0039] A data fusion capability component is used to construct a multimodal data fusion channel, which includes an extended Kalman filter channel and a deep neural network channel.

[0040] The fusion and map forming component is used to perform weighted fusion of the target multimodal positioning data using the multimodal data fusion channel to construct a dynamic positioning map of the unmanned vessel.

[0041] This invention discloses a multimodal data fusion method and system for seamless indoor and outdoor positioning of unmanned surface vessels (USVs), comprising: constructing a multimodal positioning system architecture that integrates a visual camera, an RTK positioning module, an IMU inertial measurement unit (IMU), and a SLAM mapping module, for simultaneously acquiring image information of the target USV's environment, RTK position information, IMU attitude and motion data, and SLAM mapping information; dynamically analyzing and switching positioning data sources based on the characteristics of the USV's operation in indoor and outdoor environments and the signal quality of various sensors, establishing an adaptive positioning source switching mechanism; and performing orderly fusion of environmental images, RTK positioning, IMU motion, and SLAM mapping data through this switching mechanism. By selecting and integrating various data sources, a unified multimodal positioning input dataset is formed. A multimodal fusion processing channel is designed, including a filtering channel based on the extended Kalman filter algorithm and a learning channel based on a deep neural network, to enhance the fusion accuracy of multi-source data. The selected multimodal positioning data is weighted and integrated through the fusion channel to finally construct a dynamic high-precision positioning map reflecting the position and status of the target unmanned vessel. The multimodal data fusion method and system for seamless indoor and outdoor positioning of unmanned vessels disclosed in this invention solves the technical problems of single positioning mode, poor positioning data fusion efficiency, and impact on positioning accuracy, and achieves the technical effects of diversified positioning modes, improved data fusion efficiency, and enhanced positioning accuracy. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating the multimodal data fusion method for seamless indoor and outdoor positioning of unmanned vessels according to the present invention. Figure 2 This is a schematic diagram of the multimodal data fusion system for seamless indoor and outdoor positioning of unmanned vessels according to the present invention.

[0043] Figure labeling: Multimodal acquisition capability building component 11, switching mechanism configuration component 12, positioning data acquisition component 13, data fusion capability component 14, fusion and map forming component 15. Detailed Implementation

[0044] The above technical solutions will now be described in detail with reference to the accompanying drawings and specific embodiments to provide a better understanding of them. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. It should be understood that the present invention is not limited to the exemplary embodiments used only to explain the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. Furthermore, it should be noted that, for ease of description, only the parts related to the present invention are shown in the drawings, not all of them.

[0045] Example 1, as Figure 1 This is a flowchart illustrating the multimodal data fusion method for seamless indoor and outdoor positioning of unmanned surface vessels (USVs) according to the present invention. The multimodal data fusion method for seamless indoor and outdoor positioning of USVs includes: S100: Establish a multimodal positioning architecture, which includes a visual camera, an RTK module, an IMU inertial unit, and a SLAM mapping module. The multimodal positioning architecture is used to acquire environmental image data, RTK positioning data, IMU motion data, and SLAM mapping data of the target unmanned vessel.

[0046] Specifically, a multimodal positioning architecture refers to a system structure that integrates information from multiple sensors to achieve high-precision positioning. Preferably, it includes a visual camera, an RTK module, an IMU inertial measurement unit, and a SLAM mapping module.

[0047] The system includes a visual camera to acquire images of the unmanned surface vessel's (USV) surroundings, supporting target recognition and visual odometry. The RTK (Real-Time Kinematic) module is a high-precision satellite positioning system based on carrier phase differential technology, providing centimeter-level position information. The IMU (Inertial Measurement Unit) uses accelerometers and gyroscopes to acquire motion data such as acceleration and angular velocity for attitude estimation and short-term position calculation. The SLAM (Simultaneous Localization and Mapping) module is used to build maps in unknown environments and locate the USV in real time, suitable for indoor environments with weak or unavailable GNSS signals.

[0048] Specifically, through the above architecture, the unmanned surface vessel (USV) simultaneously acquires image, RTK, IMU, and SLAM data while performing its mission, forming multi-source heterogeneous positioning information and providing a foundation for subsequent data fusion. For example, in outdoor environments, RTK can be the primary reference, with IMU used for short-term compensation; indoors or in occluded areas, SLAM and visual data dominate positioning, with IMU providing inertial assistance, thereby achieving continuity and robustness of positioning information.

[0049] The multimodal positioning architecture described above can fully leverage the advantages of different sensors in different environments, effectively addressing the limitations of single sensors in complex environments, such as RTK failure indoors, cumulative errors in IMUs, and susceptibility of vision to lighting conditions.

[0050] S200: Based on the indoor and outdoor operating scenarios of the unmanned vessel and the sensor signal strength, the positioning sensors in the multimodal positioning architecture are switched and analyzed to generate a positioning source switching mechanism.

[0051] Specifically, the above steps mean: based on the environment (indoor or outdoor scene) of the unmanned vessel and the signal strength of each positioning sensor (including visual camera, RTK module, IMU inertial unit and SLAM mapping module), analyze and identify, and then determine the combination of multiple sensors to be used under different conditions, thereby forming a mechanism to dynamically adjust the working state of each sensor to achieve the best positioning effect, which is used to guide the switching operation of the sensors.

[0052] The main benefits of this step are: it diversifies positioning modes, enabling flexible switching between different positioning sensors in different scenarios, thus freeing unmanned vessels from reliance on a single positioning method and enhancing their adaptability to complex environments. Simultaneously, by accurately identifying scene and signal strength changes and switching sensors promptly, more reliable positioning data can be obtained, providing high-quality raw data for subsequent data fusion steps.

[0053] In some embodiments, the location source switching mechanism includes: The unmanned surface vessel (USV) operates in both indoor and outdoor environments. Element identification is performed to obtain a set of operating scene elements. Based on this set, the indoor and outdoor operating scenes are clustered to obtain a set of key operating scenes. Positioning interference analysis is then performed on each operating scene in the key operating scene set to obtain a set of operating scene positioning interference parameters. Based on the set of operating scene positioning interference parameters and sensor signal strength, the switching analysis of each positioning sensor in the multimodal positioning architecture is performed to construct a positioning source switching mechanism.

[0054] Specifically, the operational scenario elements are acquired through sensor perception and environmental perception algorithms, representing key characteristics of the environment in which the unmanned vessel operates. For example, the aforementioned set of operational scenario elements includes lighting conditions, obstacle density, GNSS signal strength, and magnetic field interference levels. The operational scenario positioning interference parameter set is a parameterized representation of the specific interference obtained through analysis, used to quantify the interference intensity under different scenarios.

[0055] Specifically, firstly, environmental data is collected using sensors such as visual cameras, IMUs, and RTK modules. Then, a pre-defined environmental feature extraction algorithm is used to identify key elements of the current operating scene, such as the presence of skylights, metal obstructions, ground texture complexity, and light intensity, forming a set of operating scene elements. Next, clustering algorithms (such as K-means and DBSCAN) are used to perform cluster analysis on the acquired elements, obtaining a representative set of key operating scenes, such as unobstructed outdoor areas, indoor metal factory buildings, and semi-open docks.

[0056] Specifically, the next step is to perform positioning interference analysis for each key scenario. For example, in a closed dock or metal shed, the RTK signal strength is extremely low, but the visual SLAM effect is good. In this case, the RTK interference parameter is recorded as high and the visual interference parameter as low. Statistical analysis is then used to determine the specific parameter range or interval, forming a set of positioning interference parameters for the operating scenario.

[0057] Furthermore, by combining the real-time signal strength of the current sensors (such as RTK signal-to-noise ratio, IMU drift rate, image quality score, etc.), the switching parsing logic is executed to dynamically adjust the dominant sensor for positioning. For example, in areas with strong RTK signals, RTK is used as the dominant sensor, while in areas with weak signals or severe interference, the sensor automatically switches to the visual SLAM+IMU fusion mode, thereby constructing a dynamic and adaptive positioning source switching mechanism.

[0058] Through the above process, the acquired positioning source switching mechanism can automatically identify the most suitable positioning method according to changes in the environment of the unmanned vessel, thereby achieving efficient coordination and seamless switching between different sensors in indoor and outdoor environments. This helps improve the positioning continuity and accuracy of the unmanned vessel in complex environments and avoids positioning interruptions or error accumulation caused by the failure of a single sensor.

[0059] In some implementations, the mechanism for switching the location source includes: Based on the set of positioning interference parameters for the operating scenario, a positioning requirement analysis is performed on each positioning sensor in the multimodal positioning architecture to obtain positioning requirement parameters for the operating scenario; based on the sensor signal strength, an available threshold is divided for each positioning sensor in the multimodal positioning architecture to obtain an available threshold for positioning signal strength; based on the positioning requirement parameters for the operating scenario and the available threshold for positioning signal strength, a sensor switching analysis is performed on the multimodal positioning architecture to determine the positioning sensor switching conditions and the positioning sensor switching priority; based on the positioning sensor switching conditions and the positioning sensor switching priority, the positioning source switching mechanism is constructed.

[0060] Specifically, based on the specific requirements for positioning accuracy, stability, and real-time performance in different operating scenarios, the intensity of positioning tasks to be undertaken by each sensor in the multimodal positioning architecture can be evaluated and quantified into positioning requirement parameters for the operating scenario (i.e., the process of positioning requirement analysis).

[0061] Specifically, the positioning signal strength can be defined by a threshold, which is the minimum acceptable signal quality standard set for each sensor to determine whether the sensor is usable in the current environment. Examples include the signal-to-noise ratio threshold of RTK signals, the drift rate threshold of IMUs, and the sharpness score of visual images.

[0062] Specifically, firstly, based on the set of positioning interference parameters for the operating scenarios obtained in the previous step, the performance requirements for positioning under various scenarios are analyzed. For example, the positioning requirement parameters for the operating scenarios are shown in Table 1 below: Table 1. Exemplary Operational Scenario Location Requirements Parameters

[0063] Specifically, an availability threshold is set for each positioning sensor to quantify the limitations of sensor availability. For example, see Table 2 below: Table 2. Exemplary available thresholds for positioning signal strength

[0064] Furthermore, based on the positioning requirements parameters of the current operating scenario, sensor switching analysis is performed. For example, if the IMU signal is reliable but the visual image is blurry, the IMU+RTK combination is prioritized; if RTK fails but both visual and IMU signals are superior, the system switches to visual SLAM-dominated mode and sets the IMU as an auxiliary reference. Finally, based on the switching conditions and priorities for each scenario, a complete positioning source switching mechanism is constructed to ensure that the selection and switching of the primary positioning source is completed automatically and smoothly during operation.

[0065] Through the above process, precise positioning requirement analysis and signal strength assessment of each positioning sensor were achieved, thereby determining reasonable positioning sensor switching conditions and priorities, and constructing an efficient positioning source switching mechanism. This mechanism can fully consider positioning requirements and sensor performance in different scenarios, ensuring that the most suitable combination of positioning sensors can be selected in various complex environments.

[0066] S300: The positioning source switching mechanism is introduced to switch and activate the environmental image data, RTK positioning data, IMU motion data and SLAM mapping data to obtain target multimodal positioning data.

[0067] Specifically, switching activation refers to dynamically enabling or disabling the data channels of certain sensors in a multimodal positioning system based on a pre-built positioning source switching mechanism, thereby enabling the system to collect and fuse optimal positioning information under different operating scenarios.

[0068] Specifically, target multimodal localization data refers to a high-quality dataset used for localization calculations, output by the selected sensor combination after switching activation in the current operating scenario. This dataset contains multiple modal data sources, such as image frames, GNSS coordinates, inertial measurement values, and map information constructed using SLAM. Through the above process, the most reliable data source can be intelligently selected for fusion based on environmental changes and sensor status, thereby improving the accuracy and stability of the localization system.

[0069] In some embodiments, obtaining the target multimodal localization data includes: Based on the environmental image data, RTK positioning data, IMU motion data, and SLAM mapping data, scene recognition and signal strength assessment are performed to determine the target operating scene and the real-time signal strength of the sensor. The positioning source switching mechanism is introduced to switch and match the target operating scene and the real-time signal strength of the sensor to obtain the target positioning sensor source. The environmental image data, RTK positioning data, IMU motion data, and SLAM mapping data are activated and called according to the target positioning sensor source to obtain the target multimodal positioning data.

[0070] Specifically, firstly, image data captured by cameras is combined with deep learning models (such as ResNet and YOLOv8-Seg) for scene semantic recognition to determine the current environment. For example, if the image contains a large area of ​​sky and water, the RTK signal strength is higher than 45dBHz, and the IMU data is stable, it can be identified as an open water scene. If the image contains many occluded structures, the RTK signal fluctuates drastically, and the SLAM mapping matching rate decreases, it can be identified as a bridge-under-structure scene. Simultaneously, the real-time signal quality of each sensor is evaluated and quantified into index values, such as an IMU angular velocity standard deviation of 0.03° / s, a visual image feature point count of 1500 per frame, and a SLAM relocalization error of 0.12m. Next, the scene category and signal strength are input into a localization source switching mechanism for matching analysis to determine the optimal combination of localization sensors. For example, in an urban canyon, where the RTK signal is unstable but the visual and IMU performance is good, the localization source switching mechanism can select visual + IMU + SLAM as the target localization source.

[0071] Furthermore, the sensor data channels corresponding to the above combination are activated, and image frames, IMU data streams, and SLAM map information are called to generate target multimodal localization data at the current moment.

[0072] Through the above process, joint perception and precise matching of the operating scene and sensor status can be achieved, thereby dynamically generating optimal positioning data output. Simultaneously, it helps optimize system resource allocation and reduce redundant computation and invalid data processing.

[0073] S400: Construct a multimodal data fusion channel, which includes an extended Kalman filter channel and a deep neural network channel.

[0074] Specifically, a multimodal data fusion channel is constructed to integrate different types of positioning data, including an extended Kalman filter channel and a deep neural network channel. The extended Kalman filter channel is a linear estimation-based algorithm that can estimate the system state in real time based on noisy measurement data. The deep neural network channel is a data-driven nonlinear modeling method that can learn the complex relationships between sensors from a large amount of historical data. This channel may include an input layer (receiving multimodal data), hidden layers (such as convolutional layers for image feature extraction and recurrent layers for time series modeling), and an output layer (outputting positioning results or error estimates).

[0075] Optionally, firstly, based on the data type and processing requirements of the unmanned surface vessel (USV) positioning system, determine the model parameters for the extended Kalman filter channel, such as the state transition matrix, observation matrix, and noise covariance matrix. Then, construct the network structure for the deep neural network channel, including the number of neurons in the input layer, hidden layers, and output layer, as well as the selection of activation functions. For example, the number of neurons in the input layer is determined based on the dimension of the data from each sensor, the hidden layer uses the ReLU activation function to introduce nonlinearity, and the number of neurons in the output layer is determined based on the dimension of the fused data.

[0076] In some embodiments, constructing a multimodal data fusion channel includes: Based on the data characteristics of the multimodal localization architecture, a Kalman filter is initialized and optimized to obtain an extended Kalman filter channel. A deep neural network channel is constructed based on the data characteristics of the multimodal localization architecture. The deep neural network channel includes an input layer, hidden layers, and an output layer, wherein the hidden layers include convolutional layers, fully connected layers, and recurrent layers. The extended Kalman filter channel and the deep neural network channel are then integrated in parallel to construct the multimodal data fusion channel.

[0077] Specifically, based on the data characteristics of each sensor in the multimodal positioning architecture (such as IMU being high-frequency and short-term with high accuracy but drift, RTK being low-frequency and high-accuracy but easily obstructed, and visual images being sparse structured data), the state variables, covariance matrix, process noise, and observation noise models of the extended Kalman filter are initialized.

[0078] Simultaneously, a deep neural network channel is constructed, in which the input layer is used to receive multimodal data within the historical time window (such as image frame sequences, IMU acceleration and angular velocity, RTK position data, etc. within the past 5 seconds), the hidden layers include convolutional layers (extracting image spatial features), recurrent layers (such as LSTM or GRU, used to model time dependencies), and fully connected layers (fusion of multimodal features), and the output layer is used to generate the state estimate or error compensation value at the current moment.

[0079] For example, the input image sequence is processed by ResNet-18 to extract spatial features, and the IMU sequence is input into the LSTM network to model motion patterns. After the two are fused, the output is the positional deviation estimate (e.g., ΔX=0.12m, ΔYaw=1.8°).

[0080] Furthermore, the extended Kalman filter channel and the deep neural network channel are integrated in parallel. The extended Kalman filter channel provides state estimation results driven by the physical model, while the deep neural network channel provides error compensation or prediction values ​​based on data. The two complement each other and are integrated to construct a unified multimodal data fusion channel.

[0081] The fusion channel constructed through the above process fully combines the advantages of both model-driven and data-driven approaches. The extended Kalman filter channel provides real-time and stable state estimation, suitable for high-frequency updates and short-term predictions, while the deep neural network channel mines the nonlinear relationships between multimodal data through deep learning, improving the system's adaptability to complex environmental changes. Overall, it enhances the stability and fault tolerance of the data.

[0082] S500: The multimodal data fusion channel is used to perform weighted fusion of the target multimodal positioning data to construct a dynamic positioning map of the unmanned vessel.

[0083] In some embodiments, constructing the dynamic positioning map of the unmanned vessel includes: The target multimodal positioning data is aligned according to timestamps to obtain multimodal positioning sequence data; the multimodal positioning sequence data is normalized and noise filtered to obtain multimodal standard positioning sequence data; a global static map is constructed based on the historical modal data of the unmanned vessel; based on the global static map, the multimodal standard positioning sequence data is weighted and fused using the multimodal data fusion channel, and the map is dynamically updated to obtain a dynamic positioning map of the unmanned vessel.

[0084] Specifically, since the sampling frequency and timestamps of each sensor may be different, timestamp alignment is required, that is, to unify all sensor data to the same time base to ensure the temporal consistency of data fusion.

[0085] Specifically, a global static map is an environmental map built based on positioning data from historical stable environments, reflecting the unchanging geographical or structural features of the scene. A dynamic positioning map, on the other hand, is a dynamically updated map of the area related to the unmanned vessel's position, formed by integrating current real-time positioning information into the global static map. It can respond to and record dynamic environmental changes.

[0086] Specifically, the target multimodal localization data is first aligned using a unified timestamp, aligning each modality to a unified time reference to form a continuous multimodal localization sequence. Subsequently, this sequence is normalized (e.g., Z-score normalization or Min-Max normalization), such as standardizing IMU acceleration to 0 mean and unit variance, normalizing the number of image feature points to the [0,1] interval, and controlling the RTK position error within ±0.1m. Simultaneously, a one-dimensional Kalman filter is used to smooth the position sequence, removing abrupt changes and drift errors to obtain higher-quality standard multimodal localization sequence data.

[0087] Next, a global static map is constructed based on a large amount of stable positioning data collected by the unmanned surface vessel (USV) in historical missions, serving as a static representation of the environment. Then, using this static map as a reference, the aforementioned multimodal data fusion channel is invoked to perform weighted fusion processing on the current standard positioning sequence data. The fused result is compared and updated with the static map, adjusting local areas in the static map related to the USV's current position. For example, if the current SLAM identifies an obstacle position that is 0.8 meters off from the historical map, the map information for that area is updated to the current observation value and marked as a dynamic area or an area to be confirmed, thereby generating a dynamic positioning map for the USV.

[0088] The above process enables the end-to-end construction of a dynamic environmental map from multimodal raw data, which not only improves the temporal consistency and spatial accuracy of positioning data, but also enhances the map's responsiveness to environmental changes. Simultaneously, it allows for continuous map updates, providing a more timely and accurate environmental foundation for subsequent path planning, obstacle avoidance decisions, and task scheduling.

[0089] In some implementations, obtaining the dynamic positioning map of the unmanned vessel includes: The unmanned surface vessel (USV) is positioned by performing positioning estimation on the multimodal standard positioning sequence data using the extended Kalman filter channel to obtain USV positioning estimation information; the USV is then positioned by performing positioning prediction on the multimodal standard positioning sequence data using the deep neural network channel to obtain USV positioning prediction information; a channel fusion weighting factor is determined based on the target operating scenario and the confidence levels of the extended Kalman filter channel and the deep neural network channel; and the USV positioning estimation information and the USV positioning prediction information are then weighted and fused according to the channel fusion weighting factor based on the global static map, and the map is locally updated to obtain the USV dynamic positioning map.

[0090] Specifically, the channel fusion weighting factor is a weighting coefficient set for the fusion output result. It can be determined based on the confidence levels of the extended Kalman filter channel and the deep neural network channel in the current operating scenario. In other words, the channel fusion weighting factor is used to control the relative influence of the two in the final fusion result. Local map updates refer to updating only local areas related to the current unmanned vessel's position based on an existing global static map to improve map real-time performance and computational efficiency, avoiding a full reconstruction of the entire map.

[0091] Specifically, the state estimation of the preprocessed multimodal standard positioning sequence data is first performed using an extended Kalman filter channel. This process involves state prediction (calculating the current state based on the IMU) and observation updates (correcting based on observations from sensors such as RTK and visual SLAM), ultimately outputting the unmanned surface vessel's (USV) positioning estimate. Simultaneously, the same multimodal standard positioning sequence data is input into a deep neural network channel for state prediction, outputting the predicted position and attitude information for the current moment or the short-term future (USV positioning prediction information).

[0092] Furthermore, based on the current target operating scenario (such as open water, narrow port areas, or areas with dense obstacles) and the output confidence levels of the extended Kalman filter channel and the deep neural network channel under the current target operating scenario, the channel fusion weight factor is dynamically determined. The confidence level calculation can be based on residual statistics or uncertainty estimation from the neural network.

[0093] For example, in open waters where RTK signals are stable and IMU drift is small, the extended Kalman filter channel can be assigned a higher weight; while in complex port areas with blurred vision and RTK occlusion, the prediction results of the deep neural network channel are preferred.

[0094] Furthermore, based on the global static map, the positioning results of the two channels (i.e., the unmanned vessel positioning estimation information and the unmanned vessel positioning prediction information) are fused with the aforementioned weighting factors to obtain more accurate fused positioning information. Correspondingly, the local areas in the static map are updated to reflect real-time environmental changes, such as floating objects on the water surface, moving obstacles, and temporary structural changes, thus forming a dynamic positioning map of the unmanned vessel.

[0095] Through the above process, a high-precision dynamic map based on multi-channel information fusion is constructed. Specifically, by dynamically adjusting the fusion weights, the contributions of both channels can be adaptively balanced according to scene changes, thereby improving the accuracy and adaptability of map updates.

[0096] In some implementations, obtaining the unmanned vessel positioning estimation information includes: Based on the extended Kalman filter channel, the state initialization and state prediction of the multimodal standard positioning sequence data are performed to obtain the unmanned vessel state prediction value; combined with the multimodal standard positioning sequence data and the unmanned vessel state prediction value, the state update estimation is performed to obtain the unmanned vessel positioning estimation information.

[0097] Specifically, the unmanned surface vessel (USV) is first initialized, including setting its initial state (such as position, velocity, and attitude) and covariance matrix, serving as the starting point for subsequent predictions and updates. Then, based on acceleration and angular velocity data provided by the IMU, and combined with the USV's kinematic model (such as uniform linear motion or nonholonomic motion), state prediction is performed to form a prior estimate of the current state (i.e., the predicted USV state value). Next, observation information from multimodal standard positioning sequence data is acquired and weighted to form observation values. The residual between the predicted value and the observed value is calculated, and the predicted state is corrected using Kalman gain based on this residual to obtain the posterior estimate (i.e., the USV positioning estimation information). Simultaneously, the state covariance matrix is ​​updated to reflect changes in estimation uncertainty, providing a basis for the next round of filtering.

[0098] The above process utilizes an extended Kalman filter channel to achieve recursive estimation of the unmanned vessel's state. It fully integrates motion model data with multimodal sensor observations, exhibiting good real-time performance and robustness. Through iterative state prediction and updating, continuous estimation of the unmanned vessel's state can be maintained even in the presence of noise, occlusion, or data loss from sensors, thus providing a high-precision, low-latency positioning foundation for dynamic positioning map construction.

[0099] In some implementations, obtaining the unmanned vessel positioning prediction information includes: The deep neural network channel is used to extract features from the multimodal standard positioning sequence data to obtain the unmanned vessel positioning associated feature set; positioning fusion prediction is performed based on the unmanned vessel positioning associated feature set to output the unmanned vessel positioning prediction information.

[0100] Specifically, firstly, high-dimensional feature representations useful for the localization task are extracted from the original multimodal data using the encoding structure of a deep neural network (such as convolutional layers and attention mechanisms). The obtained unmanned surface vessel (USV) localization-related feature set is this set of features that, after modeling, are highly correlated with the USV's position, attitude, and other states. Then, the decoder part of the deep neural network channel (such as fully connected layers or attention mechanisms) performs state prediction based on the above feature set, mapping the USV localization-related feature set to the USV's position and attitude information at the current moment or in the short future, and outputting the USV localization prediction information.

[0101] Through the deep neural network channel described above, the potential nonlinear relationships between multimodal sensor data can be effectively mined, overcoming the limitations of traditional filtering models in modeling complex dynamic environments and providing more stable prediction results.

[0102] In summary, the multimodal data fusion method for seamless indoor and outdoor positioning of unmanned vessels provided by this invention has the following technical effects: By constructing a multimodal positioning system architecture that integrates a visual camera, an RTK positioning module, an IMU inertial measurement unit, and a SLAM mapping module, the system synchronously acquires image information, RTK position information, IMU attitude and motion data, and SLAM mapping information of the target unmanned surface vessel's environment. Based on the characteristics of the unmanned surface vessel's operation in indoor and outdoor environments and the signal quality of various sensors, the system dynamically analyzes and switches positioning data sources, establishing an adaptive positioning source switching mechanism. This switching mechanism sequentially activates and selects environmental images, RTK positioning, IMU motion, and SLAM mapping data to form a unified multimodal positioning input dataset. A multimodal fusion processing channel is designed, including a filtering channel based on the extended Kalman filter algorithm and a learning channel based on a deep neural network, to enhance the fusion accuracy of multi-source data. The selected multimodal positioning data is weighted and integrated through the fusion channel, ultimately constructing a dynamic high-precision positioning map reflecting the target unmanned surface vessel's position and status. This achieves the technical effects of diversified positioning modes, improved data fusion efficiency, and enhanced positioning accuracy.

[0103] Example 2, as Figure 2 This is a schematic diagram of the multimodal data fusion system for seamless indoor and outdoor positioning of unmanned vessels, as per the present invention. For example, Figure 1 The flowchart of the multimodal data fusion method for seamless indoor and outdoor positioning of unmanned vessels of the present invention can be seen as follows: Figure 2 The structure shown is implemented.

[0104] Based on the same concept as the multimodal data fusion method for seamless indoor and outdoor positioning of unmanned vessels described in the embodiments, the present invention also provides a multimodal data fusion system for seamless indoor and outdoor positioning of unmanned vessels, comprising: The multimodal acquisition capability building component 11 is used to build a multimodal positioning architecture. The multimodal positioning architecture includes a visual camera, an RTK module, an IMU inertial unit, and a SLAM mapping module. The multimodal positioning architecture is used to acquire environmental image data, RTK positioning data, IMU motion data, and SLAM mapping data of the target unmanned vessel.

[0105] The switching mechanism configuration component 12 is used to perform switching analysis on each positioning sensor in the multimodal positioning architecture based on the indoor and outdoor operating scenarios of the unmanned vessel and the sensor signal strength, and generate a positioning source switching mechanism.

[0106] The positioning data acquisition component 13 is used to introduce the positioning source switching mechanism to switch and activate the environmental image data, RTK positioning data, IMU motion data and SLAM mapping data to obtain target multimodal positioning data.

[0107] The data fusion capability component 14 is used to construct a multimodal data fusion channel, which includes an extended Kalman filter channel and a deep neural network channel.

[0108] The fusion and map forming component 15 is used to perform weighted fusion of the target multimodal positioning data using the multimodal data fusion channel to construct a dynamic positioning map of the unmanned vessel.

[0109] In some embodiments, the switching mechanism configuration component 12 includes: The operation scene element recognition unit is used to identify elements in the indoor and outdoor operation scenes of the unmanned vessel and obtain a set of operation scene elements.

[0110] The key operation scenario set acquisition unit is used to cluster and classify the indoor and outdoor operation scenarios of the unmanned vessel based on the operation scenario element set to obtain the key operation scenario set.

[0111] The unit for obtaining the set of positioning interference parameters for operating scenarios is used to sequentially perform positioning interference analysis on each operating scenario in the set of key operating scenarios to obtain the set of positioning interference parameters for operating scenarios.

[0112] The positioning source switching mechanism construction unit is used to analyze the switching of each positioning sensor in the multimodal positioning architecture based on the positioning interference parameter set of the operating scenario and the sensor signal strength, and to construct the positioning source switching mechanism.

[0113] In some implementations, the execution steps of the source-location switching mechanism construction unit in the switching mechanism configuration component 12 include: Based on the set of positioning interference parameters for the operating scenario, the positioning requirements of each positioning sensor in the multimodal positioning architecture are analyzed to obtain the positioning requirement parameters for the operating scenario.

[0114] Based on the sensor signal strength, the available thresholds for each positioning sensor in the multimodal positioning architecture are divided to obtain the available thresholds for positioning signal strength.

[0115] Based on the positioning requirement parameters of the operating scenario and the available threshold of the positioning signal strength, the sensor switching of the multimodal positioning architecture is analyzed to determine the positioning sensor switching conditions and the positioning sensor switching priority.

[0116] Based on the location sensor switching conditions and location sensor switching priorities, the location source switching mechanism is constructed.

[0117] In some embodiments, the location data acquisition component 13 includes: The scene recognition and signal strength evaluation unit is used to perform scene recognition and signal strength evaluation based on the environmental image data, RTK positioning data, IMU motion data and SLAM mapping data, and to determine the target operating scene and the real-time signal strength of the sensor.

[0118] The target positioning sensor source matching unit is used to introduce the positioning source switching mechanism to switch and match the target operating scene and the real-time signal strength of the sensor to obtain the target positioning sensor source.

[0119] The target multimodal localization data acquisition unit is used to activate and call the environmental image data, RTK localization data, IMU motion data and SLAM mapping data according to the source of the target localization sensor to obtain the target multimodal localization data.

[0120] In some embodiments, the data fusion capability component 14 includes: An extended Kalman filter channel construction unit is used to initialize a Kalman filter based on the data characteristic information of the multimodal localization architecture, and to optimize and update the Kalman filter to obtain an extended Kalman filter channel.

[0121] A deep neural network channel construction unit is used to construct a deep neural network channel based on the data characteristic information of the multimodal localization architecture. The deep neural network channel includes an input layer, a hidden layer, and an output layer, wherein the hidden layer includes a convolutional layer, a fully connected layer, and a recurrent layer.

[0122] A multimodal data fusion channel construction unit is used to connect and integrate the extended Kalman filter channel and the deep neural network channel in parallel to construct the multimodal data fusion channel.

[0123] In some embodiments, the fusion and map shaping component 15 includes: A multimodal positioning sequence data acquisition unit is used to align the target multimodal positioning data according to timestamps to obtain multimodal positioning sequence data.

[0124] A multimodal standard positioning sequence data generation unit is used to normalize and filter noise in the multimodal positioning sequence data to obtain multimodal standard positioning sequence data.

[0125] The global static map construction unit is used to construct a global static map based on the historical modal data of the unmanned vessel.

[0126] The unmanned vessel dynamic positioning map construction unit is used to obtain an unmanned vessel dynamic positioning map by weighted fusion of the multimodal standard positioning sequence data and dynamic map update using the multimodal data fusion channel based on the global static map.

[0127] In some implementations, the execution steps of the multimodal positioning sequence data acquisition unit in the fusion and map shaping component 15 include: The positioning estimation information of the unmanned vessel is obtained by performing positioning estimation on the multimodal standard positioning sequence data based on the extended Kalman filter channel.

[0128] The deep neural network channel is used to perform positioning prediction on the multimodal standard positioning sequence data to obtain unmanned vessel positioning prediction information.

[0129] Based on the target operating scenario and the confidence levels of the extended Kalman filter channel and the deep neural network channel, the channel fusion weight factor is determined.

[0130] Based on the global static map, the unmanned vessel positioning estimation information and the unmanned vessel positioning prediction information are weighted and fused according to the channel fusion weight factor, and the map is locally updated to obtain the unmanned vessel dynamic positioning map.

[0131] In some implementations, the execution steps of the multimodal positioning sequence data acquisition unit in the fusion and map shaping component 15 also include: Based on the extended Kalman filter channel, the state initialization and state prediction of the multimodal standard positioning sequence data are performed to obtain the unmanned vessel state prediction value.

[0132] By combining the multimodal standard positioning sequence data and the unmanned vessel state prediction value, a state update estimation is performed to obtain the unmanned vessel positioning estimation information.

[0133] In some implementations, the execution steps of the multimodal positioning sequence data acquisition unit in the fusion and map shaping component 15 also include: Based on the extended Kalman filter channel, the state initialization and state prediction of the multimodal standard positioning sequence data are performed to obtain the unmanned vessel state prediction value.

[0134] By combining the multimodal standard positioning sequence data and the unmanned vessel state prediction value, a state update estimation is performed to obtain the unmanned vessel positioning estimation information.

[0135] It should be understood that the focus of the embodiments mentioned in this specification is their difference from other embodiments. The specific embodiments in the aforementioned Embodiment 1 are also applicable to the multimodal data fusion system for seamless indoor and outdoor positioning of unmanned ships described in Embodiment 2. For the sake of brevity, they will not be elaborated further here.

[0136] It should be understood that the embodiments disclosed in this invention and the above description enable those skilled in the art to implement this invention. However, this invention is not limited to the embodiments mentioned above. It should be understood that those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this invention, and should all be included within the protection scope of this invention.

Claims

1. A multi-modal data fusion method for seamless positioning of unmanned ships indoors and outdoors, characterized by, The method comprises the steps of: building a multi-modal positioning architecture comprising a visual camera, an RTK module, an IMU inertial unit and a SLAM mapping module, acquiring environmental image data, RTK positioning data, IMU motion data and SLAM mapping data of a target unmanned ship through the multi-modal positioning architecture; switching and analyzing each positioning sensor in the multi-modal positioning architecture based on indoor and outdoor scenes and sensor signal strength of the unmanned ship, to generate a positioning source switching mechanism; introducing the positioning source switching mechanism to switch and activate the environmental image data, RTK positioning data, IMU motion data and SLAM mapping data, to obtain target multi-modal positioning data; building a multi-modal data fusion channel comprising an extended Kalman filter channel and a deep neural network channel; using the multi-modal data fusion channel to weight and fuse the target multi-modal positioning data, to build a dynamic positioning map of the unmanned ship.

2. The multi-modal data fusion method for indoor-outdoor seamless localization of unmanned ships of claim 1, wherein, The method comprises the steps of: element recognition of indoor and outdoor scenes of the unmanned ship to obtain a set of running scene elements; clustering and dividing the indoor and outdoor scenes of the unmanned ship based on the set of running scene elements to obtain a set of key running scenes; successively analyzing the positioning interference of each running scene in the set of key running scenes to obtain a set of running scene positioning interference parameters; switching and analyzing each positioning sensor in the multi-modal positioning architecture based on the set of running scene positioning interference parameters and sensor signal strength, to build a positioning source switching mechanism.

3. The multi-modal data fusion method for indoor-outdoor seamless localization of unmanned ships of claim 2, wherein, The method comprises the steps of: positioning demand analysis of each positioning sensor in the multi-modal positioning architecture based on the set of running scene positioning interference parameters, to obtain running scene positioning demand parameters; threshold division of each positioning sensor in the multi-modal positioning architecture according to the sensor signal strength, to obtain a positioning signal strength threshold; sensor switching analysis of the multi-modal positioning architecture based on the running scene positioning demand parameters and the positioning signal strength threshold, to determine positioning sensor switching conditions and positioning sensor switching priorities; building the positioning source switching mechanism according to the positioning sensor switching conditions and the positioning sensor switching priorities.

4. The multi-modal data fusion method for indoor-outdoor seamless localization of unmanned ships of claim 1, wherein, The method comprises the steps of: scene recognition and signal strength evaluation based on the environmental image data, RTK positioning data, IMU motion data and SLAM mapping data, to determine a target running scene and real-time sensor signal strength; introducing the positioning source switching mechanism to switch and match the target running scene and real-time sensor signal strength, to obtain a target positioning sensor source; activating and calling the environmental image data, RTK positioning data, IMU motion data and SLAM mapping data according to the target positioning sensor source, to obtain the target multi-modal positioning data.

5. The multi-modal data fusion method for seamless localization of unmanned ship indoor and outdoor as claimed in claim 1 wherein, The method comprises the steps of: Initialize a Kalman filter based on data characteristic information of the multi-modal positioning architecture, and perform optimized updating on the Kalman filter to obtain an extended Kalman filtering channel; According to the data characteristic information of the multi-modal positioning architecture, a deep neural network channel is constructed, which includes an input layer, a hidden layer and an output layer, wherein the hidden layer includes a convolutional layer, a fully connected layer and a recurrent layer; The extended Kalman filtering channel and the deep neural network channel are integrated in parallel to construct the multi-modal data fusion channel.

6. The multi-modal data fusion method for seamless localization of unmanned ship indoor and outdoor as claimed in claim 4 wherein, The method for constructing the dynamic positioning map of the unmanned ship comprises: aligning the target multi-modal positioning data according to the time stamp to obtain multi-modal positioning sequence data; performing normalization processing and noise filtering on the multi-modal positioning sequence data to obtain multi-modal standard positioning sequence data; constructing a global static map according to the historical modal data of the unmanned ship; based on the global static map, using the multi-modal data fusion channel to perform weighted fusion and map dynamic updating on the multi-modal standard positioning sequence data to obtain the dynamic positioning map of the unmanned ship.

7. The multi-modal data fusion method for seamless localization of unmanned ship indoor and outdoor as claimed in claim 6 wherein, The method for obtaining the dynamic positioning map of the unmanned ship comprises: performing positioning estimation on the multi-modal standard positioning sequence data based on the extended Kalman filtering channel to obtain unmanned ship positioning estimation information; using the deep neural network channel to perform positioning prediction on the multi-modal standard positioning sequence data to obtain unmanned ship positioning prediction information; determining a channel fusion weight factor according to the target operating scenario and the confidence of the extended Kalman filtering channel and the deep neural network channel; based on the global static map, performing data weighted fusion and map local updating on the unmanned ship positioning estimation information and the unmanned ship positioning prediction information according to the channel fusion weight factor to obtain the dynamic positioning map of the unmanned ship.

8. The multi-modal data fusion method for indoor-outdoor seamless localization of unmanned ships of claim 7, wherein, The method for obtaining the unmanned ship positioning estimation information comprises: performing state initialization and state prediction on the multi-modal standard positioning sequence data based on the extended Kalman filtering channel to obtain an unmanned ship state prediction value; combining the multi-modal standard positioning sequence data and the unmanned ship state prediction value to perform state update estimation to obtain the unmanned ship positioning estimation information.

9. The multi-modal data fusion method for seamless localization of unmanned ship indoor and outdoor as claimed in claim 7 wherein, The method for obtaining the unmanned ship positioning prediction information comprises: using the deep neural network channel to perform feature extraction on the multi-modal standard positioning sequence data to obtain a set of unmanned ship positioning related features; performing positioning fusion prediction based on the set of unmanned ship positioning related features to output the unmanned ship positioning prediction information.

10. A multi-modal data fusion system for seamless localization of unmanned ships indoors and outdoors, characterized by, A multi-modal data fusion method for seamless indoor and outdoor positioning of an unmanned ship according to any one of claims 1 to 9, comprising: a multi-modal acquisition capability construction component for building a multi-modal positioning architecture, the multi-modal positioning architecture comprising a visual camera, an RTK module, an IMU inertial unit and a SLAM mapping module, and acquiring environmental image data, RTK positioning data, IMU motion data and SLAM mapping data of a target unmanned ship through the multi-modal positioning architecture; The switching mechanism configuration component is configured to switch and analyze each positioning sensor in the multi-modal positioning architecture based on indoor and outdoor scenes of the unmanned ship and sensor signal strength, and generate a positioning source switching mechanism; The positioning data acquisition component is configured to introduce the positioning source switching mechanism to switch and activate the environment image data, RTK positioning data, IMU motion data, and SLAM mapping data, and obtain target multi-modal positioning data; The data fusion capability component is configured to construct a multi-modal data fusion channel, and the multi-modal data fusion channel includes an extended Kalman filter channel and a deep neural network channel; The fusion and map forming component is configured to use the multi-modal data fusion channel to perform weighted fusion on the target multi-modal positioning data, and construct an unmanned ship dynamic positioning map.

Images