Multi-source fusion deep learning visual navigation method, system and equipment and storage medium

Abstract

The invention discloses a multi-source fusion deep learning visual navigation method, system and device and a storage medium, and the method comprises the steps: collecting a visual image sequence and a Beidou positioning signal obtained by a robot in an operation process, carrying out the preprocessing, and generating a fusion input sequence with a quality score; performing two-channel feature coding on the visual image sequence and the Beidou positioning signal, mapping the features to a unified joint representation space, and performing dynamic modeling on the reliability of each mode in the current environment; based on the confidence coefficient of each mode and the cross-mode correlation, generating a double-branch dynamic mask of vision and Beidou, and combining a confidence coefficient modulation vector to fuse the two types of features to obtain a fused feature sequence; and performing context modeling on the fused feature sequence, constructing a joint optimization objective function for cooperative training, and outputting a robot navigation trajectory. According to the invention, the navigation precision, reliability and autonomy of the robot in a complex environment can be improved.

Description

Technical Field

[0001] This invention relates to the technical field of visual navigation, and in particular to a multi-source fusion deep learning visual navigation method, system, device and storage medium. Background Technology

[0002] In the development of autonomous navigation technology for intelligent robots, visual navigation and deep learning methods have become key research and application areas. Existing solutions primarily utilize end-to-end deep neural networks, or combine convolutional neural networks, recurrent neural networks, and reinforcement learning methods to achieve environmental perception, path planning, and obstacle avoidance control, thus reducing reliance on traditional SLAM algorithms and external positioning systems to some extent. Some studies have also introduced inertial measurement units (IMUs) and GPS / BeiDou positioning signals to enhance navigation robustness in complex environments.

[0003] However, the following shortcomings still exist in visual navigation applications combining deep learning: First, insufficient adaptability to complex environments. Most existing deep learning-based visual navigation methods rely on indoor simulation platforms for training and testing. Their models have limited generalization ability in real-world complex environments such as lighting changes, occlusion, and severe weather, leading to a significant decrease in navigation accuracy in outdoor or high-interference scenarios. Second, insufficient utilization of high-precision positioning information such as BeiDou. Although some research and patents attempt to combine GNSS / BeiDou positioning with visual navigation, most methods are still primarily single-modal and lack deep fusion strategies. Existing technologies often only use BeiDou positioning as a rough global coordinate reference, failing to jointly optimize it with visual features, depth estimation, and semantic information, making it difficult to ensure the continuity and reliability of navigation in environments with weak satellite signals or multipath interference. Third, insufficient multi-source data fusion levels. Existing solutions typically adopt a "vision-led, other sensor-assisted" approach in data processing, resulting in limited roles for inertial information, BeiDou positioning data, and environmental semantic information in navigation decisions. Fusion only remains at the data level or result level, lacking deep fusion mechanisms at the feature level and decision level. This makes the system prone to error accumulation and navigation failure in critical tasks. Fourth, the system's collaborative capabilities are limited. Current deep learning visual navigation is mostly used as an independent module, with insufficient collaboration with task planning, path optimization, and safety strategies. For example, when the BeiDou positioning and visual estimation results are inconsistent, the system lacks a rapid conflict detection and self-recovery mechanism, often requiring manual intervention, which reduces autonomy and robustness.

[0004] In summary, although existing technologies have made some progress in visual perception and deep learning navigation, they still generally suffer from problems such as insufficient adaptability to complex environments, inadequate utilization of BeiDou high-precision information, limited multi-source fusion levels, and insufficient system coordination capabilities. They are still unable to meet the navigation requirements of robots in real complex environments for high reliability, high precision, and high autonomy. Summary of the Invention

[0005] In view of the aforementioned existing problems, this invention is proposed. Therefore, this invention provides a multi-source fusion deep learning visual navigation method, system, device, and storage medium to solve the problems mentioned in the background art.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, embodiments of the present invention provide a multi-source fusion deep learning visual navigation method, comprising: acquiring visual image sequences and BeiDou positioning signals obtained by a robot during operation, and preprocessing the visual image sequences and BeiDou positioning signals to generate a fusion input sequence with quality scores; The visual image sequence and the BeiDou positioning signal are subjected to dual-channel feature encoding, and the visual features and BeiDou features are mapped to a unified joint representation space. The reliability of each modality in the current environment is dynamically modeled through a confidence scoring network. Based on the confidence level of each modality and cross-modal correlation, a dual-branch dynamic mask of vision and Beidou is generated. Combined with the confidence modulation vector, the two types of features are fused through a dynamic reconstruction operator to obtain a fused feature sequence. The fused feature sequence is contextually modeled, with global consistency constraints weighted by BeiDou confidence, and combined with trajectory smoothness and modal consistency loss to construct a joint optimization objective function for collaborative training, and output the robot navigation trajectory.

[0007] As a preferred embodiment of the deep learning visual navigation method for multi-source fusion described in this invention, the step of preprocessing the visual image sequence and the BeiDou positioning signal to generate a fused input sequence with quality scores includes: performing field standardization processing on the visual image sequence and the BeiDou positioning signal respectively; calculating the optimal time delay based on the cross-correlation function between the visual signal and the BeiDou positioning signal, and resampling and interpolating the BeiDou positioning signal according to the optimal time delay, and aligning the BeiDou positioning signal and the visual image sequence on a unified time axis; Feature extraction and denoising processing are performed on the standardized and synchronized visual image sequence and the BeiDou positioning signal to obtain visual features and BeiDou features. The visual features are obtained by extracting spatial geometric and temporal dynamic information from continuous image frames through convolutional neural networks and temporal modeling operators. The BeiDou features are obtained by denoising the BeiDou three-dimensional coordinate sequence and concatenating it with quality indicators that characterize signal quality.

[0008] As a preferred embodiment of the deep learning visual navigation method with multi-source fusion described in this invention, the method further includes: performing anomaly detection and missing feature completion on the visual features and BeiDou features respectively; when the residual between the BeiDou coordinates and their smoothed estimates exceeds a preset threshold, it is determined to be an anomaly and removed; when visual features are missing, interpolation is performed based on the weighted similarity of neighboring frames in the temporal neighborhood, with the weights decaying exponentially with time distance. The visual features after anomaly processing are concatenated with the BeiDou features to form a fused feature vector. A quality scoring function is constructed based on visual confidence, BeiDou confidence, and time alignment error to calculate the quality score corresponding to the fused feature vector at each time step. The quality score is normalized to the (0,1) interval by the Sigmoid function. The output is a fused input sequence consisting of the fused feature vector and its corresponding quality score.

[0009] As a preferred embodiment of the multi-source fusion deep learning visual navigation method described in this invention, the method comprises: generating a dual-branch dynamic mask for both visual and BeiDou navigation based on the confidence level of each modality and cross-modal correlation; and fusing the two types of features using a dynamic reconstruction operator in conjunction with the confidence level modulation vector to obtain a fused feature sequence including: For each frame of the visual image sequence, spatial features are extracted using a convolutional neural network, and the spatial features within a continuous time window are input into a temporal modeling operator to generate visual features. The three-dimensional coordinate sequence in the BeiDou positioning signal is smoothed and filtered, and then concatenated with the quality index that characterizes the signal quality and input into the multilayer perceptron encoder to generate BeiDou features. Based on the visual features and BeiDou features, the visual confidence and BeiDou confidence in the current environment are dynamically calculated using the first confidence scoring network and the second confidence scoring network, respectively. Based on the visual confidence and BeiDou confidence, a weighted combination and normalization by the Sigmoid function are used to generate an overall quality score of the fused features, which is used to characterize the reliability of each modality at the current moment. The fused feature input sequence and its corresponding overall quality score are obtained.

[0010] As a preferred embodiment of the multi-source fusion deep learning visual navigation method described in this invention, the generation of a dual-branch dynamic mask based on the confidence level of each modality and cross-modal correlation includes: Based on the enhanced visual features and BeiDou features after feature encoding, wherein the enhanced visual features are stabilized by introducing position constraint residuals between consecutive frames; Based on the enhanced visual features and BeiDou features, a visual mask vector and a BeiDou mask vector are calculated respectively. The enhanced visual features and BeiDou features are jointly input through a first learnable mapping and activated by Sigmoid to generate a visual mask vector. The BeiDou features and enhanced visual features are jointly input through a second learnable mapping and activated by Sigmoid to generate a BeiDou mask vector.

[0011] As a preferred embodiment of the multi-source fusion deep learning visual navigation method described in this invention, the method involves: combining a confidence modulation vector and fusing two types of features through a dynamic reconstruction operator to obtain a fused feature sequence, including: Obtain the visual confidence and BeiDou confidence at the current moment, and convert the scalar confidence into visual confidence modulation vector and BeiDou confidence modulation vector respectively through the corresponding nonlinear mapping function; The confidence modulation vector is multiplied element-wise with the corresponding mask vector to obtain the modulated visual features and the modulated BeiDou features; the two types of modulated features are input into a dynamic reconstruction operator for nonlinear fusion. The dynamic reconstruction operator includes a linear transformation term for visual features, a linear transformation term for BeiDou features, and an interaction term for the element-wise multiplication of the two, and outputs a fused intermediate representation. The fusion intermediate representation is subjected to dimensionality reduction and regularization mapping to generate the final fusion feature vector; The data at each time point are processed to obtain the fused feature sequence.

[0012] As a preferred embodiment of the multi-source fusion deep learning visual navigation method described in this invention, the fused feature sequence is context-modeled, with global consistency constraints weighted by BeiDou confidence, and combined with trajectory smoothness and modal consistency losses to construct a joint optimization objective function for collaborative training, outputting a robot navigation trajectory including: The fused feature sequence is subjected to context enhancement using a temporal modeling operator based on a multi-head self-attention mechanism to obtain a context-enhanced representation at each time step. This context-enhanced representation is then input into the trajectory decoder to predict the robot's position increment in the local coordinate system, and the navigation trajectory is recovered through recursive integration. The entire temporal modeling and trajectory inference process is jointly optimized using multiple loss functions, as follows: in, The reconstruction loss is the trajectory increment. For trajectory smoothing constraints, For global consistency constraints assisted by BeiDou, For confidence level consistency constraints, This is a hyperparameter used to adjust the relative weights of the various loss components.

[0013] Secondly, the present invention provides a multi-source fusion deep learning visual navigation system, comprising: The multi-source data acquisition and preprocessing module is used to acquire visual image sequences and BeiDou positioning signals obtained by the robot during operation, and to preprocess the visual image sequences and BeiDou positioning signals to generate a fusion input sequence with quality scores. The feature encoding and joint representation module is used to perform dual-channel feature encoding on the visual image sequence and the BeiDou positioning signal, map the visual features and BeiDou features to a unified joint representation space, and dynamically model the reliability of each modality in the current environment through a confidence scoring network. The fusion module is used to generate a dual-branch dynamic mask for vision and BeiDou based on the confidence of each modality and cross-modal correlation. It combines the confidence modulation vector and fuses the two types of features through a dynamic reconstruction operator to obtain a fused feature sequence. The temporal modeling and joint optimization module is used to perform contextual modeling on the fused feature sequence, with global consistency constraints weighted by BeiDou confidence, and combined with trajectory smoothness and modal consistency loss to construct a joint optimization objective function for collaborative training, and output the robot navigation trajectory.

[0014] Thirdly, the present invention provides an electronic device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the multi-source fusion deep learning visual navigation method.

[0015] Fourthly, the present invention provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the multi-source fusion deep learning visual navigation method.

[0016] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention achieves a unified representation of local geometric details and global coordinate constraints through multi-source heterogeneous fusion modeling of visual image sequences and BeiDou positioning signals, combined with cross-frame residual enhancement and attention mechanisms. Compared with traditional methods that rely on a single visual odometry or a single GNSS solution, this invention can effectively alleviate the single-mode degradation problem in complex indoor scenes, significantly improving the accuracy and robustness of pose estimation. This invention proposes a dynamic fusion mechanism based on mask generation and confidence modulation, which takes the lead when visual features are clear and introduces global consistency constraints when BeiDou signals are stable, achieving adaptive complementarity between modes. Compared with traditional fixed-weight or simple weighted fusion strategies, this invention can dynamically adjust the modal contribution according to changes in environmental conditions, reducing navigation risks in dark environments or under signal obstruction.

[0017] The system introduces a multi-head self-attention-based temporal modeling framework in the trajectory estimation stage, combined with trajectory smoothing regularization and global consistency constraints, effectively suppressing the drift problem caused by long-term cumulative visual errors. Compared with existing methods that rely on local matching or short-term window optimization, this invention can maintain the continuity and global consistency of trajectory estimation in long-term tasks, making it more suitable for large-scale indoor inspection tasks. This invention transforms the original multi-source signals into a unified high-dimensional fusion representation through preprocessing steps such as field standardization, time synchronization, noise suppression, and missing data completion, and assigns a quality score to each feature. This mechanism can maintain stable input quality even when some sensors fail or data is missing, thus ensuring that the navigation model remains highly robust under abnormal conditions.

[0018] This invention archives multi-source data, fusion weights, and trajectory estimation results throughout the navigation process, and optimizes fusion parameters and temporal modeling strategies based on historical samples to achieve adaptive evolution. Compared to traditional methods with static parameter configuration, this invention can continuously adapt to changes in indoor lighting, occlusion interference, and BeiDou signal fluctuations, ensuring the efficiency and reliability of the inspection robot in long-term tasks. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a schematic diagram of the process flow of a multi-source fusion deep learning visual navigation method according to an embodiment of the present invention. Detailed Implementation

[0020] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0021] Example 1, referring to Figure 1 As one embodiment of the present invention, this embodiment provides a multi-source fusion deep learning visual navigation method, including: S100: Collects visual image sequences and BeiDou positioning signals acquired by the robot during operation, and preprocesses the visual image sequences and BeiDou positioning signals to generate a fusion input sequence with quality scores; S200: Performs dual-channel feature encoding on visual image sequences and BeiDou positioning signals, maps visual features and BeiDou features to a unified joint representation space, and dynamically models the reliability of each modality in the current environment through a confidence scoring network; S300: Based on the confidence level of each modality and cross-modal correlation, a dual-branch dynamic mask of vision and Beidou is generated, and combined with the confidence modulation vector, the two types of features are fused through a dynamic reconstruction operator to obtain a fused feature sequence. S400: Contextual modeling is performed on the fused feature sequence, global consistency constraints are weighted by BeiDou confidence, and a joint optimization objective function is constructed by combining trajectory smoothness and modal consistency loss for collaborative training, outputting the robot navigation trajectory.

[0022] It should be noted that, as a crucial component of autonomous navigation for mobile robots, visual navigation and localization technologies directly impact the effectiveness of environmental perception, path planning, and task execution. In practical applications, navigation systems are not only essential for the integrity of target recognition and trajectory control but also closely related to obstacle avoidance decisions, task planning, and multi-source perception logic. However, limited by current technological conditions, existing visual navigation methods frequently exhibit problems such as large positioning errors, system instability, and sensitivity to external interference in complex environments, posing challenges to the safety and reliability of autonomous operation. Existing deep learning-based visual navigation solutions mostly focus on end-to-end image processing and path prediction, primarily replacing traditional SLAM or GPS positioning, lacking the ability to deeply fuse multi-source information. Specifically: On the one hand, existing systems struggle to maintain stable performance in complex environments. For example, under conditions of changing lighting, occlusion, rain, fog, or electromagnetic interference, the generalization ability of pure visual navigation models is insufficient, making them prone to positioning drift or target loss. On the other hand, although some solutions introduce external signals such as GNSS / BeiDou as aids, most of them are only used as rough global coordinate references, lacking tight coupling with multi-dimensional information such as vision and inertia, making it difficult to achieve high-precision and continuous navigation in environments with weak signals, multipath interference, or occlusion.

[0023] Meanwhile, existing solutions also have shortcomings in multi-source data utilization and system collaboration. Most methods rely solely on single information sources (such as visual features or inertial estimation), lacking a dual-confirmation mechanism for multi-source states. This makes them prone to misjudgments due to single sensor failure, noise interference, or data conflicts, affecting navigation reliability. At the system level, when visual estimation conflicts with BeiDou positioning, manual intervention is often still required to restore navigation, failing to achieve true local rapid correction and autonomous decision-making, thus limiting the robot's intelligence level in complex tasks. To enhance navigation robustness in complex environments, this invention aims to maintain high accuracy and continuity even under varying lighting conditions, occlusion, severe weather, and electromagnetic interference, avoiding drift accumulation and failure inherent in purely visual methods. It constructs a deep learning-driven BeiDou fusion mechanism, jointly optimizing BeiDou high-precision positioning with multi-dimensional information such as visual features and inertial measurement at both the feature and decision levels, overcoming the limitations of existing technologies that only achieve result-level fusion. A multi-source state dual-confirmation mechanism is established to achieve rapid conflict detection and self-recovery when discrepancies exist between visual estimation and BeiDou positioning, reducing navigation failures caused by single-signal errors. This invention proposes a deep learning-based visual navigation method with vision as the primary driver and BeiDou assistance. By combining multi-source data acquisition and preprocessing, feature encoding and high-dimensional representation, selective fusion mechanisms, and time-series modeling and joint optimization, it constructs an autonomous navigation system with high accuracy, robustness, and environmental adaptability. This achieves deep collaboration and intelligent fusion of visual, BeiDou, and other sensor information, thereby improving the navigation accuracy, reliability, and autonomy of robots in complex environments.

[0024] This invention establishes a unified fusion input data stream by performing multi-source preprocessing on image sequences and BeiDou positioning signals acquired by a robot in an indoor environment, including field standardization, temporal alignment, signal denoising, and missing data completion. This process ensures the continuity and reliability of feature input even under conditions of inconsistency between visual high-frequency and BeiDou low-frequency signals, signal noise interference, and occasional missing data.

[0025] Furthermore, a dual-channel feature encoding method combining visual and BeiDou navigation is proposed. The visual channel extracts spatial texture and dynamic geometric constraints through a convolutional-temporal network, while the BeiDou channel encodes positioning coordinates and quality indicators into high-dimensional feature vectors, achieving mapping and comparison within a unified representation space. Unlike existing single-modal navigation methods, this invention maintains consistency between the two modalities in the semantic space through a joint representation mechanism, thereby providing high-quality input for fusion and optimization.

[0026] Building upon this foundation, this invention designs an improved selective fusion mechanism. Through dynamic reconstruction operators involving cross-frame residual enhancement, dual-branch mask generation, and confidence modulation, the system can prioritize visual features when they are clear, while introducing global constraints when the BeiDou signal is stable, achieving complementarity and dynamic balance between modes. Simultaneously, temporal smoothing constraints are introduced to prevent trajectory discontinuities caused by drastic weight fluctuations, ensuring the stability of navigation results.

[0027] Furthermore, this invention proposes a self-attention-based temporal modeling and joint optimization framework. This framework recursively predicts trajectory increments at a local scale, suppresses accumulated errors at a global scale through BeiDou-assisted constraints, and jointly considers trajectory reconstruction loss, smoothing regularization, and modal consistency constraints to form a complete optimization objective function. Through this closed-loop process, the system can achieve continuous, accurate, and long-term stable navigation trajectory estimation in complex indoor environments, significantly improving the autonomy and task execution efficiency of the inspection robot.

[0028] It should be noted that due to differences in sampling frequency, inconsistencies in units, noise interference, and occasional missing data between visual information and BeiDou positioning signals, directly using unprocessed data will lead to feature temporal misalignment and reduced confidence, thus affecting the accuracy and robustness of fused navigation. Therefore, this invention designs a complete visual-BeiDou data preprocessing workflow, including field standardization and temporal synchronization, denoising and feature extraction, anomaly detection and missing data completion, quality assessment and compression mapping, etc., to provide high-quality, confidence-annotated fusion vector input for upper-layer deep learning fusion.

[0029] In this embodiment of the invention, step S100, which preprocesses the visual image sequence and the BeiDou positioning signal to generate a fused input sequence with a quality score, includes: performing field standardization processing on the visual image sequence and the BeiDou positioning signal respectively; calculating the optimal time delay based on the cross-correlation function between the visual signal and the BeiDou positioning signal; and resampling and interpolating the BeiDou positioning signal according to the optimal time delay to align the BeiDou positioning signal and the visual image sequence on a unified time axis. Feature extraction and denoising processing are performed on the standardized and synchronized visual image sequence and the BeiDou positioning signal to obtain visual features and BeiDou features. The visual features are obtained by extracting spatial geometric and temporal dynamic information from continuous image frames through convolutional neural networks and temporal modeling operators. The BeiDou features are obtained by denoising the BeiDou three-dimensional coordinate sequence and concatenating it with quality indicators that characterize the signal quality.

[0030] Furthermore, suppose the robot is at any time Raw observation data were collected from both the visual and BeiDou subsystems: in, This represents the raw feature quantity extracted from the image frame. This represents the three-dimensional coordinates and their quality indicators obtained from BeiDou positioning calculations.

[0031] Because the units and dimensions of data from different sources are inconsistent, field standardization is required. Define a semantic mapping function. : The standardized numerical fields are then normalized using Z-score: in, fields respectively Mean and standard deviation within a historical window.

[0032] Furthermore, since the visual subsystem typically has high-frequency sampling... BeiDou positioning, however, is limited by the receiver refresh rate. Therefore, the timeline must be standardized. Let the cross-correlation function between visual and BeiDou signals be: The optimal delay is: It should be noted that the present invention is based on The BeiDou data was resampled and interpolated to align it with the visual data to a unified reference timeline. This ensures the consistency of event occurrence times and provides a reliable temporal basis for subsequent joint modeling.

[0033] In short, in complex indoor environments, visual signals may be affected by changes in lighting, reflection, and occlusion; while BeiDou signals may be affected by wall obstruction or multipath effects. Therefore, this invention introduces denoising and feature extraction steps in the preprocessing stage.

[0034] Furthermore, in the visual channel, the original image sequence First, a convolutional neural network feature extractor is used. : Intermediate feature vectors representing spatial geometry and motion are obtained. Subsequently, temporal modeling operators are used. Capture dynamic constraints between consecutive frames in the time dimension: In the BeiDou channel, the positioning result is represented as follows: in, Used to characterize signal quality; This invention utilizes a Kalman smoother. Denoising the BeiDou coordinate sequence: The final splicing coordinates and quality indicators yield the BeiDou features: It should be noted that in actual operation, visual features may be lost due to occlusion or motion blur, and BeiDou signals may also be intermittently lost due to indoor occlusion; in order to avoid erroneous data affecting navigation results, this invention designs an anomaly detection mechanism based on residuals.

[0035] In this embodiment of the invention, step S100 further includes: performing anomaly detection and missing feature completion on visual features and BeiDou features respectively; when the residual between the BeiDou coordinates and its smoothed estimate exceeds a preset threshold, it is determined to be an anomaly and removed; when visual features are missing, interpolation is performed based on the weighted similarity of neighboring frames in the time neighborhood, with the weight decaying exponentially with time distance. The visual features after anomaly processing are concatenated with the BeiDou features to form a fused feature vector. A quality scoring function is constructed based on visual confidence, BeiDou confidence, and time alignment error to calculate the quality score corresponding to the fused feature vector at each time step. The quality score is normalized to the (0,1) interval by the Sigmoid function. The output is a fused input sequence consisting of the fused feature vector and its corresponding quality score.

[0036] It should be noted that the preset threshold in the embodiments of the present invention can be dynamically adjusted according to the actual operating conditions.

[0037] Furthermore, taking BeiDou coordinates as an example, the residual is defined as follows: like If the data at that moment is found to be abnormal, it will be removed.

[0038] For cases where visual features are missing, this invention employs a filling strategy based on the similarity of neighboring frames: in, It represents the time neighborhood, and the weight decays with the time difference.

[0039] Furthermore, after standardization, synchronization, noise reduction, and missing data completion, this invention stitches together visual and BeiDou features to obtain a unified representation as follows: Simultaneously define a quality scoring function to score the input at each time step: in, These represent the confidence levels for visual and BeiDou navigation systems, respectively. Indicates time alignment error. The Sigmoid function makes the final quality score... Normalization to .

[0040] The final fused input sequence is obtained as follows: Deep learning is used to fuse the input sequences, providing a reliable data foundation for high-precision navigation of inspection robots in complex indoor environments.

[0041] It should be noted that after completing the multi-source data acquisition and preprocessing, this invention enters the feature encoding and representation stage. The goal of this stage is twofold: firstly, to extract stable and discriminative deep feature representations from visual and BeiDou data; and secondly, to map different modalities through a unified high-dimensional vector space, enabling subsequent deep learning fusion and navigation estimation to be optimized within the same feature domain.

[0042] In this embodiment of the invention, step S200 generates a dual-branch dynamic mask for vision and BeiDou based on the confidence level of each modality and cross-modal correlation. Combined with the confidence modulation vector, the two types of features are fused using a dynamic reconstruction operator to obtain a fused feature sequence including: For each frame of the visual image sequence, spatial features are extracted using a convolutional neural network, and the spatial features within a continuous time window are input into a temporal modeling operator to generate visual features. Furthermore, regarding the image sequences collected by the inspection robot... First, through a convolutional neural network Extracting spatial features: in, It includes low-level visual features such as edges, textures, and corners, while also preserving high-level abstractions of local structures.

[0043] To capture dynamic information in the time dimension, this invention introduces a time-series modeling operator. : in, Indicates the length of the time window. This characterizes the movement trajectory information of the inspection robot within a local time period.

[0044] It should be noted that by introducing temporal modeling operators, visual features not only reflect the spatial geometry of a single frame image, but also include temporal constraints between consecutive frames, thereby effectively improving stability in environments with weak texture and varying illumination.

[0045] In this embodiment of the invention, step S200 involves smoothing and filtering the three-dimensional coordinate sequence in the BeiDou positioning signal, concatenating it with a quality index characterizing the signal quality, and then inputting it into a multilayer perceptron encoder to generate BeiDou features. Furthermore, the position and quality indicators output by the BeiDou subsystem are denoted as: in, In three-dimensional coordinates, Reliability characteristics related to BeiDou signals.

[0046] To mitigate indoor multipath effects and intermittent signal loss, this invention first employs a smoothing operator. Filter the position sequence: Then, the BeiDou feature vector is constructed: in, It is a multilayer perceptron (MLP) encoder, whose function is to map the original physical quantities into high-dimensional embeddings, so that they can be compared and fused with visual features in the same representation space.

[0047] It should be noted that by performing smoothing filtering on the three-dimensional coordinate sequence in the BeiDou positioning signal, the confidence information of BeiDou positioning is explicitly modeled, ensuring that it can still constrain the system output in signal degradation scenarios.

[0048] Furthermore, in order to unify the two heterogeneous features of vision and BeiDou, this invention adopts a combination of feature splicing and nonlinear mapping to project both onto a unified joint representation space. : in, For fusion mapping networks, this can be implemented using a Transformer-based cross-attention module; symbol This represents a vector concatenation operation. To achieve a fusion representation that is comparable at the semantic level and unified at the spatial scale.

[0049] In this embodiment of the invention, in step S200, the visual confidence and BeiDou confidence in the current environment are dynamically calculated based on visual features and BeiDou features through the first confidence scoring network and the second confidence scoring network, respectively. Based on visual confidence and BeiDou confidence, a weighted combination and normalized by the Sigmoid function are used to generate an overall quality score of the fused features, which is used to characterize the reliability of each modality at the current moment. The fused feature input sequence and its corresponding overall quality score are obtained.

[0050] Furthermore, considering the reliability differences between visual and BeiDou signals in different scenarios, this invention introduces confidence weights in the feature representation layer. For each modality, a confidence function is defined as follows: in, For a first-confidence scoring network, The first confidence scoring network and the second confidence scoring network have their output ranges limited to... .

[0051] The overall quality score of the final fused features is defined as follows: in, For adjustable weight parameters, This is the Sigmoid mapping function.

[0052] It should be noted that, through this mechanism, the system can automatically reduce the impact of the corresponding mode in the event of visual degradation or BeiDou signal attenuation, thereby improving the robustness of the overall feature representation.

[0053] Furthermore, this invention proposes a novel deep learning selective fusion mechanism based on vision-driven and BeiDou-assisted navigation. Its core idea is to use visual features as the primary source of navigation information, while leveraging BeiDou features to provide global position information and confidence level correction, thereby achieving high-precision navigation support for indoor inspection robots in complex environments. This mechanism mainly includes four stages: visual feature enhancement processing, dual-branch mask generation, confidence level modulation, and temporal constraint fusion.

[0054] It should be noted that, considering issues such as uneven lighting, occlusion, and sparse texture in indoor environments, directly using single-frame visual features can easily lead to positioning drift. Therefore, this invention introduces visual feature enhancement processing in the visual processing stage, improving the stability of visual features through cross-frame association and mask weighting mechanisms.

[0055] In this embodiment of the invention, step S300, which generates a dual-branch dynamic mask for vision and BeiDou based on the confidence level of each modality and cross-modal correlation, includes: Based on the enhanced visual features and BeiDou features after feature encoding, the enhanced visual features are stabilized by introducing position constraint residuals between consecutive frames; Based on enhanced visual features and BeiDou features, visual mask vectors and BeiDou mask vectors are calculated separately. The enhanced visual features and BeiDou features are jointly input through a first learnable mapping and activated by Sigmoid to generate a visual mask vector. The BeiDou features and enhanced visual features are jointly input through a second learnable mapping and activated by Sigmoid to generate a BeiDou mask vector.

[0056] Furthermore, let the visual features extracted by the convolutional-temporal network be: in, For convolutional network extractors, It is a timing encoder. For a unified feature dimension.

[0057] To highlight the differences from BeiDou features, this invention introduces position-constrained residuals for visual features: in, This is a learnable weight matrix. This residual term can explicitly strengthen motion constraints between consecutive frames and reduce drift caused by anomalies in a single frame.

[0058] In this invention, the dual-branch mask generation mechanism aims to achieve dynamic adjustment and differentiated utilization of visual and BeiDou features, thereby ensuring the robustness of navigation features in complex indoor environments. Traditional modal fusion methods often combine multi-source information through simple weighting or splicing, but this approach cannot fully capture the complementarity and competition between different modalities, easily leading to mutual interference between modalities. The dual-branch mask generation mechanism proposed in this invention selectively adjusts visual and BeiDou features dimension-by-dimensionally through parallel mask branches, ensuring that visual information remains dominant during the fusion process, while BeiDou information dynamically plays a supporting role based on reliability.

[0059] Specifically, let the input features be the enhanced visual features. Features of BeiDou The calculation process of the dual-branch mask is as follows: in, For learnable parameter matrix, For bias vectors, This is the Sigmoid activation function. The output... These are mask vectors for visual and BeiDou features, respectively, with each element of its dimension corresponding to the selective weight of that modality feature during fusion.

[0060] It should be noted that the dual-branch mask generation mechanism ensures that the visual mask depends not only on the visual features themselves but also on the modulation of BeiDou features, and vice versa. This cross-dependency structure effectively captures the complementary relationships between modalities. For example, when visual features are occluded or interfered with by illumination in certain dimensions, the BeiDou branch can enhance the weight compensation of that dimension through cross-modal adjustment; while when the BeiDou signal is weak or multipath effects are significant, the visual branch can suppress its interference components through its own stable texture or geometric features.

[0061] Furthermore, to ensure the robustness of the mask, this invention introduces sparsity constraints to prevent overactivation. Specifically, during the training phase, a sparsity constraint is applied to the mask. Regularization: in, is the regularization coefficient. This constraint prompts the mask to selectively activate in different dimensions, improving the discriminative power and sparse representation capability of modal features.

[0062] It should be noted that the dual-branch mask generation mechanism not only achieves dimensional adjustment of modal features, but also ensures the complementary use of vision and BeiDou through a cross-modal dependency mechanism. Under this framework, visual features are preferentially preserved due to their stability in indoor environments, while BeiDou features provide correction and compensation in key dimensions through the masking mechanism, enabling the fused representation to have both local perceptual accuracy and global consistency.

[0063] Furthermore, after completing the dual-branch mask generation, this invention proposes a fusion mechanism based on confidence modulation. Unlike traditional methods that rely solely on linear weighting, this mechanism uses confidence-guided nonlinear mapping and dynamic reconstruction operators to ensure visual dominance while fully utilizing BeiDou information to correct the global position estimate.

[0064] In this embodiment of the invention, step S300 combines the confidence modulation vector and fuses the two types of features through a dynamic reconstruction operator to obtain a fused feature sequence, including: obtaining the visual confidence and BeiDou confidence at the current moment, and converting the scalar confidence into the visual confidence modulation vector and the BeiDou confidence modulation vector respectively through the corresponding nonlinear mapping function; The confidence modulation vector is multiplied element-wise with the corresponding mask vector to obtain the modulated visual features and the modulated BeiDou features. The two types of modulated features are then input into a dynamic reconstruction operator for nonlinear fusion. The dynamic reconstruction operator includes a linear transformation term for the visual features, a linear transformation term for the BeiDou features, and an interaction term for the element-wise multiplication of the two features. The output is a fused intermediate representation. The intermediate fusion representation is reduced in dimensionality and regularized to generate the final fusion feature vector. The data at each time point are processed to obtain the fused feature sequence.

[0065] Furthermore, suppose the sensor is at time... The confidence level is This invention defines a confidence modulation vector: in, This is a nonlinear mapping function, which can be implemented using a multilayer perceptron to transform scalar confidence into a dimension-wise modulation factor. The confidence not only acts as an overall scaling factor but also produces differentiated modulation across different feature dimensions.

[0066] After obtaining the modulation vector, this invention introduces a dynamic reconstruction operator. This enables nonlinear fusion of visual and BeiDou features: in, This represents element-wise multiplication. The specific form is defined as follows: in, For learnable parameter matrix, This is a bias term. It can be improved by adding interactive elements. This operator can capture the nonlinear dependence between visual and BeiDou features in a specific dimension, thereby achieving a more complex modulation process than simple weighting.

[0067] To enhance the stability of the fusion results, this invention introduces a confidence consistency constraint during the training process. Specifically, the difference in predicted positions is defined. for: in, and These are the decoding mappings for visual and BeiDou systems, respectively. The consistency constraint loss is then: It should be noted that this constraint forces the system to maintain consistency in its position estimation results when both modes have high confidence levels; while when the confidence level of one mode is low, the constraint is automatically weakened to avoid misleading the fusion results.

[0068] Furthermore, the fused representation after confidence modulation and dynamic reconstruction is as follows: in, To reduce dimensionality and regularize the mapping function, the output is kept in a unified feature space. This mechanism can not only adaptively adjust the contribution ratio of visual and BeiDou features, but also dynamically select a more reliable modal dimension when there is modal conflict, thereby ensuring the positioning accuracy and robustness of the inspection robot in indoor environments.

[0069] Furthermore, to ensure the continuity of robot trajectory estimation, this invention introduces temporal smoothing constraints on mask changes to avoid frequent weight fluctuations between different modes. The temporal regularization term is defined as follows: This constraint ensures a smooth transition in the time dimension for the fusion module, avoiding drastic switching caused by transient signal anomalies.

[0070] This invention at every moment Output fusion features: in, As a mapping network, it is responsible for compressing the spliced ​​modal features into a unified representation dimension.

[0071] The resulting fusion feature sequence: .

[0072] It should be noted that, after obtaining the fused feature sequence, this invention further proposes a trajectory inference method based on temporal modeling and joint optimization. This method explicitly models the motion patterns of the inspection robot at continuous time intervals and introduces a joint optimization mechanism of vision and BeiDou modality to achieve high-precision estimation of local trajectories and consistency correction of global positioning, thereby improving the robustness and reliability of the overall navigation system.

[0073] In this embodiment of the invention, step S400 involves contextual modeling of the fused feature sequence, using a global consistency constraint weighted by BeiDou confidence, and combining trajectory smoothness and modal consistency loss to construct a joint optimization objective function for collaborative training. The output robot navigation trajectory includes: The fused feature sequence is augmented with a temporal modeling operator based on a multi-head self-attention mechanism to obtain the context augmentation representation at each time step. The context augmentation representation is then input into the trajectory decoder to predict the robot's position increment in the local coordinate system, and the navigation trajectory is recovered by recursive integration. Furthermore, considering the continuity of the inspection robot's movement indoors, this invention employs a temporal modeling framework based on a self-attention mechanism to capture the dependency between historical trajectories and current estimates.

[0074] Set time The fusion characteristics are The time series modeling process is then defined as: in, This represents a context modeling operator based on Multi-Head Self-Attention (MHSA), and its output is... For a moment Context-enhanced representation.

[0075] In terms of implementation, The following calculation was performed: Among them, the query matrix The key matrix and value matrix are respectively ,in .

[0076] It should be noted that a time-series modeling framework based on a self-attention mechanism is adopted to capture the dependency between historical trajectories and the current estimate. This framework can comprehensively consider historical trajectory information when estimating the current position and suppress noise interference at a single moment.

[0077] Furthermore, upon obtaining the temporal enhancement representation... Subsequently, the present invention utilizes a trajectory decoder. Output the robot's position increment in the local coordinate system : And the trajectory is recovered through recursion: It should be noted that the process of recovering the trajectory through recursion ensures the continuity and geometric interpretability of the trajectory, and can generate high-precision local trajectory estimates under vision-driven conditions.

[0078] In this embodiment of the invention, step S400 involves jointly optimizing the entire time-series modeling and trajectory inference process using multiple loss functions, as shown below: in, The reconstruction loss is the trajectory increment. For trajectory smoothing constraints, For global consistency constraints assisted by BeiDou, For confidence level consistency constraints, This is a hyperparameter used to adjust the relative weights of the various loss components.

[0079] Specifically, the reconstruction loss of trajectory increment is expressed as: in, For the predicted displacement increment, This represents the actual displacement increment. Specifically, the trajectory smoothing constraint is expressed as: in, The location predicted by the model.

[0080] Furthermore, to avoid long-term drift caused by visual accumulation errors, this invention introduces BeiDou-assisted global constraints during the trajectory optimization stage; specifically, setting the time... The global position measured by BeiDou is Then, the global consistency loss is defined as follows: in, The confidence level of the BeiDou signal is used to dynamically adjust the constraint strength.

[0081] When the BeiDou signal is stable, the system forces the visual trajectory to remain consistent with the BeiDou position; when the BeiDou signal degrades, the constraint is automatically weakened to avoid interfering with the visually-driven estimation results.

[0082] It should be noted that the joint optimization framework can ensure that the system maintains high-precision trajectory estimation in a local range, while avoiding drift through BeiDou correction during long-term operation, thereby realizing the complementary advantages of vision and BeiDou.

[0083] In summary, the method of this invention constructs a full-process intelligent navigation system, from multi-source data acquisition and preprocessing, feature encoding and high-dimensional representation, and an improved selective fusion mechanism, to temporal modeling and joint optimization.

[0084] First, this invention achieves heterogeneous data standardization and confidence modeling of visual image sequences and BeiDou positioning signals through steps such as field standardization, time alignment, signal denoising, anomaly detection, and missing data completion, ensuring the consistency and reliability of input features in the temporal dimension. Second, it constructs dual-channel feature representations of vision and BeiDou using a convolutional-temporal network and a multilayer perceptron encoder, and realizes modal mapping and comparison through a unified high-dimensional joint representation space, providing a stable input foundation for deep learning fusion. On this basis, this invention proposes an improved selective fusion mechanism with vision as the primary driver and BeiDou as the auxiliary driver. Through cross-frame residual enhancement, dual-branch mask generation, confidence modulation, and temporal smoothing constraints, it achieves dimensional dynamic adjustment and modal complementarity, enabling the fused features to possess both local geometric accuracy and global consistency.

[0085] Furthermore, this invention introduces a temporal modeling framework based on multi-head self-attention in the trajectory inference stage, and combines trajectory incremental reconstruction, trajectory smoothing regularization, and global consistency constraints assisted by BeiDou to establish a joint optimization objective function. While ensuring high-precision local estimation, it effectively suppresses the drift problem caused by long-term cumulative visual errors. This forms a closed-loop intelligent navigation system that integrates multi-source heterogeneous data, feature encoding and representation, dynamic fusion modeling, and temporal optimization inference. It not only achieves autonomous navigation capabilities with both high precision and robustness at the single robot level, but also provides a unified and scalable technical framework for intelligent inspection tasks in complex indoor environments, significantly improving the autonomy, stability, and task execution efficiency of inspection robots.

[0086] Example 2: The above example is an illustrative scheme of a multi-source fusion deep learning visual navigation method. It should be noted that the technical solution of this multi-source fusion deep learning visual navigation system belongs to the same concept as the technical solution of the multi-source fusion deep learning visual navigation method described above. Details not described in detail in the technical solution of the multi-source fusion deep learning visual navigation system in this example can be found in the description of the technical solution of the multi-source fusion deep learning visual navigation method described above.

[0087] This embodiment presents a multi-source fusion deep learning visual navigation system, comprising: The multi-source data acquisition and preprocessing module is used to acquire visual image sequences and BeiDou positioning signals obtained by the robot during operation, and to preprocess the visual image sequences and BeiDou positioning signals to generate a fusion input sequence with quality scores. The feature encoding and joint representation module is used to perform dual-channel feature encoding on visual image sequences and BeiDou positioning signals, map visual features and BeiDou features to a unified joint representation space, and dynamically model the reliability of each modality in the current environment through a confidence scoring network. The fusion module is used to generate a dual-branch dynamic mask for vision and BeiDou based on the confidence of each modality and cross-modal correlation. It combines the confidence modulation vector and fuses the two types of features through a dynamic reconstruction operator to obtain a fused feature sequence. The temporal modeling and joint optimization module is used to perform contextual modeling on the fused feature sequence, with global consistency constraints weighted by BeiDou confidence, and combined with trajectory smoothness and modal consistency loss to construct a joint optimization objective function for collaborative training, and output the robot navigation trajectory.

[0088] This embodiment also provides an electronic device applicable to multi-source fusion deep learning visual navigation methods, including: The system includes a memory and a processor. The memory stores computer-executable instructions, and the processor executes these instructions to implement the deep learning visual navigation method for multi-source fusion as proposed in the above embodiments.

[0089] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements the deep learning visual navigation method for multi-source fusion as proposed in the above embodiments.

[0090] The storage medium proposed in this embodiment belongs to the same inventive concept as the deep learning visual navigation method for realizing multi-source fusion proposed in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0091] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0092] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multi-source fusion deep learning visual navigation method, characterized in that, include: The robot acquires visual image sequences and BeiDou positioning signals during operation, and preprocesses the visual image sequences and BeiDou positioning signals to generate a fused input sequence with quality scores. The visual image sequence and the BeiDou positioning signal are subjected to dual-channel feature encoding, and the visual features and BeiDou features are mapped to a unified joint representation space. The reliability of each modality in the current environment is dynamically modeled through a confidence scoring network. Based on the confidence level of each modality and cross-modal correlation, a dual-branch dynamic mask of vision and Beidou is generated. Combined with the confidence modulation vector, the two types of features are fused through a dynamic reconstruction operator to obtain a fused feature sequence. The fused feature sequence is contextually modeled, with global consistency constraints weighted by BeiDou confidence, and combined with trajectory smoothness and modal consistency loss to construct a joint optimization objective function for collaborative training, and output the robot navigation trajectory.

2. The deep learning visual navigation method based on multi-source fusion as described in claim 1, characterized in that, The step of preprocessing the visual image sequence and the BeiDou positioning signal to generate a fused input sequence with a quality score includes: performing field standardization processing on the visual image sequence and the BeiDou positioning signal respectively; calculating the optimal time delay based on the cross-correlation function between the visual signal and the BeiDou positioning signal; and resampling and interpolating the BeiDou positioning signal according to the optimal time delay to align the BeiDou positioning signal and the visual image sequence on a unified time axis. Feature extraction and denoising processing are performed on the standardized and synchronized visual image sequence and the BeiDou positioning signal to obtain visual features and BeiDou features. The visual features are obtained by extracting spatial geometric and temporal dynamic information from continuous image frames through convolutional neural networks and temporal modeling operators. The BeiDou features are obtained by denoising the BeiDou three-dimensional coordinate sequence and concatenating it with quality indicators that characterize signal quality.

3. The deep learning visual navigation method based on multi-source fusion as described in claim 2, characterized in that, Also includes: Anomaly detection and missing feature completion are performed on the visual features and BeiDou features respectively; when the residual between the BeiDou coordinates and their smoothed estimates exceeds a preset threshold, it is judged as an anomaly and removed; when visual features are missing, interpolation is performed based on the weighted similarity of neighboring frames in the time neighborhood, and the weight decays exponentially with time distance. The visual features after anomaly processing are concatenated with the BeiDou features to form a fused feature vector. A quality scoring function is constructed based on visual confidence, BeiDou confidence, and time alignment error to calculate the quality score corresponding to the fused feature vector at each time step. The quality score is normalized to the (0,1) interval by the Sigmoid function. The output is a fused input sequence consisting of the fused feature vector and its corresponding quality score.

4. The deep learning visual navigation method based on multi-source fusion as described in claim 3, characterized in that, Based on the confidence scores of each modality and cross-modal correlations, a dual-branch dynamic mask for both visual and BeiDou systems is generated. Combined with the confidence modulation vector, the two types of features are fused using a dynamic reconstruction operator to obtain a fused feature sequence, including: For each frame of the visual image sequence, spatial features are extracted using a convolutional neural network, and the spatial features within a continuous time window are input into a temporal modeling operator to generate visual features. The three-dimensional coordinate sequence in the BeiDou positioning signal is smoothed and filtered, and then concatenated with the quality index that characterizes the signal quality and input into the multilayer perceptron encoder to generate BeiDou features. Based on the visual features and BeiDou features, the visual confidence and BeiDou confidence in the current environment are dynamically calculated using the first confidence scoring network and the second confidence scoring network, respectively. Based on the visual confidence and BeiDou confidence, a weighted combination and normalization by the Sigmoid function are used to generate an overall quality score of the fused features, which is used to characterize the reliability of each modality at the current moment. The fused feature input sequence and its corresponding overall quality score are obtained.

5. The deep learning visual navigation method based on multi-source fusion as described in claim 4, characterized in that, Based on the confidence levels of each modality and cross-modal correlations, a dual-branch dynamic mask for both vision and BeiDou is generated, including: Based on the enhanced visual features and BeiDou features after feature encoding, wherein the enhanced visual features are stabilized by introducing position constraint residuals between consecutive frames; Based on the enhanced visual features and BeiDou features, a visual mask vector and a BeiDou mask vector are calculated respectively. The enhanced visual features and BeiDou features are jointly input through a first learnable mapping and activated by Sigmoid to generate a visual mask vector. The BeiDou features and enhanced visual features are jointly input through a second learnable mapping and activated by Sigmoid to generate a BeiDou mask vector.

6. The deep learning visual navigation method based on multi-source fusion as described in claim 5, characterized in that, By combining the confidence modulation vector, the two types of features are fused through a dynamic reconstruction operator to obtain the fused feature sequence, which includes: obtaining the visual confidence and BeiDou confidence at the current moment, and converting the scalar confidence into the visual confidence modulation vector and the BeiDou confidence modulation vector respectively through the corresponding nonlinear mapping function; The confidence modulation vector is multiplied element-wise with the corresponding mask vector to obtain the modulated visual features and the modulated BeiDou features; the two types of modulated features are input into a dynamic reconstruction operator for nonlinear fusion. The dynamic reconstruction operator includes a linear transformation term for visual features, a linear transformation term for BeiDou features, and an interaction term for the element-wise multiplication of the two, and outputs a fused intermediate representation. The fusion intermediate representation is subjected to dimensionality reduction and regularization mapping to generate the final fusion feature vector; The data at each time point are processed to obtain the fused feature sequence.

7. The deep learning visual navigation method based on multi-source fusion as described in claim 6, characterized in that, Contextual modeling is performed on the fused feature sequence, with global consistency constraints weighted by BeiDou confidence, and a joint optimization objective function is constructed by combining trajectory smoothness and modal consistency loss for collaborative training. The output robot navigation trajectory includes: The fused feature sequence is subjected to context enhancement using a temporal modeling operator based on a multi-head self-attention mechanism to obtain a context-enhanced representation at each time step. This context-enhanced representation is then input into the trajectory decoder to predict the robot's position increment in the local coordinate system, and the navigation trajectory is recovered through recursive integration. The entire temporal modeling and trajectory inference process is jointly optimized using multiple loss functions, as follows: in, The reconstruction loss is the trajectory increment. For trajectory smoothing constraints, For global consistency constraints assisted by BeiDou, For confidence level consistency constraints, This is a hyperparameter used to adjust the relative weights of the various loss components.

8. A multi-source fusion deep learning visual navigation system, applied to the method described in any one of claims 1-7, characterized in that, include: The multi-source data acquisition and preprocessing module is used to acquire visual image sequences and BeiDou positioning signals obtained by the robot during operation, and to preprocess the visual image sequences and BeiDou positioning signals to generate a fusion input sequence with quality scores. The feature encoding and joint representation module is used to perform dual-channel feature encoding on the visual image sequence and the BeiDou positioning signal, map the visual features and BeiDou features to a unified joint representation space, and dynamically model the reliability of each modality in the current environment through a confidence scoring network. The fusion module is used to generate a dual-branch dynamic mask for vision and BeiDou based on the confidence of each modality and cross-modal correlation. It combines the confidence modulation vector and fuses the two types of features through a dynamic reconstruction operator to obtain a fused feature sequence. The temporal modeling and joint optimization module is used to perform contextual modeling on the fused feature sequence, with global consistency constraints weighted by BeiDou confidence, and combined with trajectory smoothness and modal consistency loss to construct a joint optimization objective function for collaborative training, and output the robot navigation trajectory.

9. An electronic device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the multi-source fusion deep learning visual navigation method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the multi-source fusion deep learning visual navigation method according to any one of claims 1 to 7.

Images