Hierarchical visual language navigation memory enhancement system in cross-floor scene

Abstract

The invention discloses a hierarchical visual language navigation memory enhancement system in a cross-floor scene, relates to the technical field of intelligent navigation, and adopts the technical scheme that the hierarchical visual language navigation memory enhancement system comprises three core parts, namely a hierarchical visual semantic model construction module, a memory enhancement algorithm module and a navigation system integration module. The hierarchical visual semantic model construction module is used for constructing a basic visual feature layer, a floor semantic layer and a cross-floor semantic association layer; the memory enhancement algorithm module comprises a short-term memory module, a long-term memory module and a memory fusion and update strategy module; and the navigation system integration module is used for integrating the hierarchical visual semantic model and a memory enhancement algorithm into the system. Three-dimensional space characterization misalignment and path planning error accumulation can be effectively avoided, the accuracy, environment adaptability and decision-making efficiency of cross-floor navigation of the intelligent agent are remarkably improved, and the method has important technical innovation value and application prospects.

Description

Technical Field

[0001] This invention relates to the field of intelligent navigation technology, and more specifically, to a hierarchical visual language navigation memory enhancement system for cross-floor scenarios. Background Technology

[0002] With the rapid development of intelligent agent technology, visual language navigation is being used more and more widely in indoor scenarios, especially in multi-story buildings, where intelligent agents need to accurately understand and execute instructions such as "Please go to the meeting room on the east side of the third floor." However, current visual language navigation systems face serious challenges in cross-floor scenarios, with the problem of navigation gaps across floors being particularly prominent.

[0003] In multi-story buildings, transitional spaces such as elevator lobbies and emergency exits are essential for changing floors. However, these spaces often share highly similar design styles and decorative elements, lacking distinct visual semantic features. For example, in office buildings, elevator lobbies often employ uniform decoration standards, while emergency exits typically consist of fire doors, similar staircases, and monotonous walls, making it difficult for visual navigation systems to identify the current floor and location.

[0004] When an agent performs cross-floor navigation tasks, the lack of unique visual semantic features in the transitional space prevents the system from accurately updating its 3D spatial representation. 3D spatial representation is the foundation of path planning; inaccuracy leads to path planning deviations, and errors accumulate through repeated passages through the transitional space, manifesting as a spiraling error accumulation that deviates from the target path and fails to reach the target location accurately.

[0005] When an agent performs a cross-floor navigation task, the lack of unique visual semantic features in transitional spaces prevents the navigation system from accurately updating its 3D spatial representation when changing floors. 3D spatial representation is the foundation for path planning in navigation systems; inaccurate 3D spatial representation leads to deviations in subsequent path planning. Moreover, this deviation is not independent but rather accumulates in a spiral pattern as the agent moves between different floors. For example, when the agent travels from the first floor to the third floor, the errors caused by inaccurate spatial representation accumulate each time it passes through the elevator or emergency exit, ultimately causing the agent to deviate from the target path and fail to accurately reach the conference room on the east side of the third floor.

[0006] Existing methods primarily rely on single visual recognition algorithms or simple floor markers, both of which have shortcomings: visual algorithms struggle to extract distinguishing features, while floor markers require additional hardware or fixed identifiers, resulting in high deployment costs and susceptibility to failure. For example, while the "Charlie Rides the Elevator" project supports elevator operation, it heavily relies on environment-specific robotic arm mechanisms and lacks generalization capabilities; methods like OCR-RCNN can recognize elevator buttons but lack contextual understanding for floor recognition. Therefore, a hierarchical visual-language navigation memory enhancement mechanism is urgently needed to strengthen cross-floor navigation capabilities and improve the accuracy and reliability of navigation in multi-story buildings. Summary of the Invention

[0007] The purpose of this invention is to provide a hierarchical visual language navigation memory enhancement system for cross-floor scenarios, overcoming the shortcomings of existing technologies in cross-floor visual language navigation. This enables intelligent agents to accurately understand and execute cross-floor navigation commands in multi-story buildings, avoiding the problems of inaccurate 3D spatial representation and accumulated path planning errors caused by the lack of transitional spatial visual semantic features.

[0008] The above-mentioned technical objective of the present invention is achieved through the following technical solution: a hierarchical visual language navigation memory enhancement system for cross-floor scenarios, comprising a hierarchical visual semantic model construction module, a memory enhancement algorithm module, and a navigation system integration module; the hierarchical visual semantic model construction module is used to construct a basic visual feature layer, a floor semantic layer, and a cross-floor semantic association layer; the memory enhancement algorithm module includes a short-term memory module, a long-term memory module, and a memory fusion and update strategy module; and the navigation system integration module is used to integrate the hierarchical visual semantic model and the memory enhancement algorithm into an intelligent agent navigation system.

[0009] The present invention is further configured such that: the basic visual feature layer construction method is: using a deep convolutional neural network to process the scene image inside a multi-story building, introducing a channel attention mechanism to highlight key visual features, and using a multi-scale feature fusion strategy to capture information at different scales to obtain basic visual features.

[0010] The present invention is further configured such that: the method for constructing the floor semantic layer is as follows: based on the building layout, semantic labels are assigned to floors and sub-regions to construct a floor semantic graph describing regional relationships; the graph is trained using a graph convolutional network, and combined with a dynamic semantic coding mechanism, the semantic features are updated according to time and environmental factors.

[0011] The present invention is further configured such that: the method for constructing the cross-floor semantic association layer is as follows: feature modeling is performed on the transitional space to construct a cross-floor semantic graph representing the relationship between nodes between floors; a spatiotemporal attention mechanism is introduced to adjust the association weights between nodes according to time and space information to achieve cross-floor semantic reasoning.

[0012] The present invention is further configured such that: the short-term memory module uses a queue structure to store multimodal information collected in real time by the agent, controls the information storage duration through a dynamic time window mechanism, updates the information based on information similarity, and provides a retrieval function based on query conditions.

[0013] The present invention is further configured such that: the long-term memory module uses a knowledge graph to store building structure and semantic information, and uses a graph neural network to realize knowledge reasoning; and updates the knowledge graph based on navigation results and environmental changes through a reinforcement learning algorithm.

[0014] The present invention is further configured such that: at the key decision point of the agent, the memory fusion and update strategy module calculates the correlation between short-term memory and long-term memory and determines the adaptive fusion weight to achieve information fusion; updates the short-term memory and long-term memory respectively according to the set rules, and optimizes the relevant parameters based on the performance feedback after the update.

[0015] The present invention is further configured such that: the navigation system integration module connects the hierarchical visual semantic model and memory enhancement algorithm with the motion control module and sensor module of the intelligent agent, and uses a unified interface protocol to realize data interaction; the integrated navigation system is tested by designing diverse test scenarios, and the model and algorithm are optimized based on the performance evaluation results.

[0016] In summary, the present invention has the following beneficial effects:

[0017] 1. Significantly Improved Navigation Accuracy: The memory fusion mechanism accurately calculates the visual and semantic correlation between short-term memory units and long-term memory nodes, and uses this to determine adaptive fusion weights, achieving deep integration of real-time information from short-term memory and stable knowledge from long-term memory. When the agent faces critical decision points such as elevators and floor changes, the fused memory information... It can provide a more comprehensive and accurate basis for navigation decisions, effectively avoid the inaccuracy of three-dimensional spatial representation caused by the lack of transitional spatial visual semantic features, reduce the accumulation of spiral errors in path planning, and greatly improve the accuracy of the agent reaching the target position.

[0018] 2. Enhanced Environmental Adaptability: The dynamic update strategy endows the system with strong environmental adaptability. The short-term memory update mechanism dynamically adjusts the time window length according to environmental changes and achieves efficient management of real-time information through similarity calculation, ensuring that short-term memory always stores information most relevant to the current navigation task. Long-term memory updates utilize reinforcement learning algorithms, based on navigation results and environmental change detection, using a reward function to evaluate the effectiveness of update actions, and continuously optimizing the update strategy through a deep Q-network. This enables the long-term memory knowledge graph to adapt to dynamic environments such as building structure adjustments and functional area changes in a timely manner, ensuring the continuous and stable operation of the navigation system in complex and ever-changing scenarios.

[0019] 3. Improved Navigation Decision Efficiency: The adaptive weight fusion strategy quickly and accurately integrates short-term and long-term memory information, reducing the processing time of the agent on a large amount of redundant information during the decision-making process. When faced with complex cross-floor navigation tasks, the agent can make decisions quickly based on the fused memory information, avoiding delays caused by information filtering and analysis, effectively improving overall navigation efficiency, and enabling the agent to complete navigation tasks more efficiently, especially suitable for emergency navigation scenarios with high time requirements.

[0020] 4. Reduce Navigation System Uncertainty: By quantifying the correlation between visual features and semantic labels, and combining it with adaptive weight calculation, this strategy reduces subjective judgment and uncertainty in the navigation decision-making process. Whether in transitional spaces with similar visual features or regions with ambiguous semantic information, the system can derive reliable navigation basis through precise calculation and reasoning, enhancing the stability and reliability of the navigation system and providing a solid guarantee for the autonomous navigation of intelligent agents in multi-story buildings.

[0021] 5. Achieving continuous knowledge optimization and accumulation: The reinforcement learning and updating mechanism of long-term memory enables the system to continuously learn and optimize from navigation experience. The results of each navigation task and environmental changes become the basis for system improvement. Through continuous iterative training, the accuracy and completeness of the long-term memory knowledge graph are constantly improved. This not only helps to successfully complete the current navigation task, but also accumulates richer and more accurate knowledge for future navigation decisions, forming a virtuous cycle and driving the continuous improvement of navigation system performance. Attached Figure Description

[0022] Figure 1 This describes the workflow of the multimodal navigation system in this embodiment of the invention.

[0023] Figure 2 This is a cross-floor hierarchical visual semantic navigation model in this embodiment of the invention. Detailed Implementation

[0024] The following is in conjunction with the appendix Figure 1-2The present invention will be described in further detail below.

[0025] Example 1: Data Acquisition and Preprocessing

[0026] (a) Data collection

[0027] Equipment Deployment: High-definition cameras should be evenly deployed throughout the multi-story building, including corridors, rooms, elevator lobbies, and safety passages. Camera installation locations must ensure coverage of main passageways and key nodes in each area, with a certain degree of overlap between adjacent cameras to eliminate blind spots. Camera resolution should be set to at least 1920×1080 pixels, and the frame rate to 25 frames per second to ensure clear and continuous scene images.

[0028] Data Acquisition Strategy: Comprehensive image acquisition of the building will be conducted at different times (e.g., weekday daytime, weekday nighttime, weekends) and under varying lighting conditions (natural light on sunny days, cloudy days, artificial lighting, etc.). For each floor and area, at least 1000 images will be acquired from different angles and in different scenes to encompass rich visual information. Simultaneously, positioning devices (such as high-precision GPS or indoor positioning systems) will be used to record the precise location coordinates at the time of each image acquisition. ,in Indicates the floor number.

[0029] (ii) Data preprocessing

[0030] Image cropping and scaling: Based on the camera's field of view and navigation task requirements, the acquired original images are cropped to remove irrelevant edge areas and retain the effective scene portion. The cropped images are then uniformly scaled to a fixed size, such as 224×224 pixels, to meet the input requirements for subsequent model training.

[0031] Normalization: This process normalizes the pixel values ​​of an image, adjusting their range to [0, 1]. The specific formula is as follows:

[0032]

[0033] in, These are the original image pixel values. and These are the minimum and maximum pixel values ​​of the original image, respectively. These are the normalized image pixel values.

[0034] Data annotation: Based on the actual layout and functional area division of the building, professional annotation tools are used to annotate the images. Each image is assigned a corresponding floor number, area name (e.g., "first floor lobby", "third floor east corridor"), and semantic label (e.g., "elevator entrance", "conference room door"). Simultaneously, the location and category information of key visual elements in the images (e.g., stairs, fire doors, signs) are annotated to form a detailed annotated dataset.

[0035] II. Training of Hierarchical Visual Semantic Model

[0036] (I) Training of basic visual feature layers

[0037] Model selection: An improved ResNet-50 was chosen as the basic visual feature extraction network. Based on the original ResNet-50 architecture, a channel attention mechanism and a multi-scale feature fusion module were embedded.

[0038] Training parameter settings: The batch size was set to 32, the training epochs to 50, the optimizer to Adam, the initial learning rate to 0.001, and the learning rate decayed by 0.1 times every 10 epochs. The cross-entropy loss function was used to measure the difference between the model's predicted features and the ground truth labeled features.

[0039] Training Process: The preprocessed image dataset is divided into a training set (80%), a validation set (10%), and a test set (10%). During training, the training set images are input into the network, output features are calculated through forward propagation, the loss value is calculated according to the loss function, and the network parameters are updated using the backpropagation algorithm. After each epoch of training, the model performance is evaluated on the validation set, and hyperparameters such as the learning rate are adjusted based on the validation set loss and accuracy. After training, a final performance test is performed on the test set, and the optimal model weights are saved for subsequent applications.

[0040] (II) Training of Floor Semantic Layer

[0041] Graph Construction: Based on the building floor plan and functional area division, an initial floor semantic graph is manually constructed. Node set It includes floor nodes and sub-region nodes, and assigns an initial semantic label to each node. Edge set. The weights of the edges are initialized based on the actual connectivity between regions. It is 0.

[0042] Model Training: A Graph Convolutional Network (GCN) was used to train the floor semantic graph. The features output from the basic visual feature layer were used as the initial features for each node. The GCN had 3 layers and 64 hidden units per layer. Training parameters were set similarly to the basic visual feature layer, using the Adam optimizer with an initial learning rate of 0.001 and a training epoch of 30. During training, feature propagation and update operations within the GCN enabled nodes to learn more discriminative semantic features. Simultaneously, edge weights were dynamically adjusted based on the relationships between regions in the labeled data. Optimize the semantic graph structure of the floors.

[0043] Dynamic semantic encoding training: Collect relevant data on buildings under different time periods and usage states (such as pedestrian density and records of changes in area functions), and combine this data with sensor data (such as access control system records and lighting status data) to construct a dynamic semantic feature dataset. Use recurrent neural networks (RNNs) or long short-term memory networks (LSTMs) to model the dynamic semantic features, and fuse static and dynamic semantic features during training, adjusting the weight coefficients accordingly. This enables the model to accurately capture the dynamic changes in floor semantics.

[0044] (III) Training of the cross-floor semantic association layer

[0045] Data preparation: Extract images and related information of transitional spaces (elevator shafts, safety passages) from the labeled dataset, including connected floors and their relationship with the target area. Construct a cross-floor semantic association dataset, where each data sample contains an image of the transitional space, connecting floor nodes, target area nodes, and semantic relationship annotations between them.

[0046] Model Training: A graph neural network (GNN) is used to construct a semantic graph across floors. The GNN is set to have 4 layers, with 128 hidden units per layer. During training, edge weights between nodes are calculated based on the cross-layer semantic association dataset. The cross-floor semantic graph structure is optimized. Simultaneously, a spatiotemporal attention mechanism is introduced, encoding temporal information (such as different times of day and different dates of week) and spatial information (the agent's current position and target position) into feature vectors. These vectors are then fused with node features during training to learn the spatiotemporal attention weights. The training parameters were set to batch size 16, training epochs 40, the optimizer Adam, and the learning rate 0.0005.

[0047] III. Implementation of Memory Enhancement Algorithm

[0048] (a) Implementation of the short-term memory module

[0049] Storage Structure Construction: In the navigation system of an intelligent agent, a queue data structure is used to implement the short-term memory module. Set the maximum capacity of the queue. 100. Each memory unit according to The information is stored in the format of, where For image data, For visual feature vectors, For semantic tags, For timestamps, These are the position coordinates.

[0050] Implementation of Dynamic Time Window and Update Mechanism: Write program code to implement a dynamic time window mechanism, which calculates the difference between the current time and the earliest stored information time in real time. The time window length is dynamically adjusted based on the complexity of the environment. During the agent's movement, whenever new information is collected... Calculate its similarity to existing memory units in the queue. Perform update or add operations based on similarity and queue status. Set a similarity threshold. The threshold is 0.8. If there is a memory unit with a similarity exceeding this threshold, the unit is updated; otherwise, it is determined whether the queue is full, and new information is stored according to the rules.

[0051] Implementation of search function: Design of search function Based on the input query conditions Retrieve relevant memory units from the short-term memory queue (e.g., based on specific visual features, semantic labels, or location ranges). Use efficient search algorithms (e.g., hash search or K-nearest neighbor search) to quickly find units with similarity exceeding the retrieval threshold. A memory cell (set to 0.7) is used to provide a reference for navigation decisions.

[0052] (II) Implementation of Long-Term Memory Module

[0053] Knowledge graph construction: Using knowledge graph construction tools (such as Neo4j) to build a long-term memory knowledge graph storage structure. Add floor nodes, sub-region nodes, transitional space nodes, and semantic concept nodes to the node set. Establish an edge set based on the building structure and semantic relationships. and initialize the edge weights. .

[0054] Knowledge Reasoning Implementation: This section describes the implementation of knowledge reasoning functionality based on a Graph Neural Network (GNN) algorithm. Code is written to implement the node feature update formula for the GNN, setting the number of GNN layers to 3-5. Through multiple iterations, node features are updated, allowing each node to learn rich semantic information. During navigation decision-making, semantic analysis and path planning are performed using the reasoned node features based on the current scene information.

[0055] Reinforcement Learning Update Implementation: A reinforcement learning update strategy for long-term memory is implemented using a Deep Q-Network (DQN) framework. State space is defined. Action space and reward function Set the learning rate The discount factor is 0.001. The accuracy is 0.9. After the agent completes the navigation task, it triggers a long-term memory update based on the task results and environmental change detection. By interacting with the environment, it collects state-action-reward data, trains the DQN network, learns the optimal update strategy, and continuously optimizes the long-term memory knowledge graph.

[0056] (III) Implementation of Memory Fusion and Update Strategies

[0057] Memory fusion implementation: When the agent encounters a critical decision point, a program is written to implement the memory fusion mechanism. This involves calculating the visual feature correlation between short-term memory units and long-term memory nodes. and semantic relevance Thus, the total correlation degree is obtained. Adaptive fusion weights for short-term and long-term memory are calculated based on the correlation. and And perform aggregation operations on short-term memory. Reasoning operations on long-term memory Ultimately, the merged memory information is obtained. .

[0058] Memory update implementation: Update strategies for short-term and long-term memory are implemented separately. For short-term memory, updates are performed based on similarity calculation and dynamic time window rules; for long-term memory, updates are performed using reinforcement learning algorithms based on navigation results and environmental change detection. After each update, the performance of the memory module is evaluated, and relevant parameters (such as similarity threshold, reward function weight coefficients, etc.) are adjusted based on performance feedback to optimize the memory fusion and update strategies.

[0059] IV. Navigation System Integration and Testing

[0060] (I) System Integration

[0061] Module Connection: The trained hierarchical visual semantic model (basic visual feature layer, floor semantic layer, cross-floor semantic association layer), the implemented memory enhancement algorithm (short-term memory module, long-term memory module, memory fusion and update strategy), and the agent's motion control module and sensor modules (camera, positioning sensor, etc.) are connected. A unified interface protocol is used to realize data transmission and interaction between the modules, ensuring smooth information transmission.

[0062] System debugging: Conduct comprehensive debugging of the integrated navigation system to check whether each module is working properly and whether data transmission is accurate. During the debugging process, promptly fix any problems found (such as data format mismatches or interface call errors) to ensure the overall stability of the system.

[0063] (II) System Testing

[0064] Test Scenario Design: Diverse test scenarios were designed within a real-world multi-story building environment, including navigation tasks from different starting points to different destinations (e.g., from the first-floor lobby to the second-floor west conference room, or from the third-floor east corridor to the fifth-floor central office), covering the usage of various transitional spaces (elevator lobbies, different stairwells). Simultaneously, tests were conducted simulating different environmental changes (e.g., renovations in some areas altering the path, or temporary closure of certain passageways).

[0065] Performance Evaluation: In each test scenario, the navigation system was run multiple times, recording metrics such as the agent's navigation path, time to reach the target location, and navigation accuracy. Statistical data, including average navigation time, average path length, and the percentage of correct target arrivals out of total tests, were calculated to evaluate the navigation system's performance. Based on the test results, problems with the system were analyzed (e.g., low navigation accuracy in complex transitional spaces, inefficient path planning), and the model and algorithm were optimized and adjusted accordingly until the design requirements were met. The working principle of the system described in this invention is as follows:

[0066] 1. Construction of a hierarchical visual semantic model

[0067] Constructing a basic visual feature layer: Deep convolutional neural networks (CNNs) are used to process various scene images within multi-story buildings, extracting low-level visual features such as edges, textures, and colors. These basic visual features form the basis for subsequent analysis and can capture the fundamental visual elements of the scene.

[0068] In the complex and similar transitional spatial environment of multi-story buildings, an improved deep convolutional neural network architecture is adopted to accurately extract effective visual features. (Input image) ,in Indicates the image height. Indicates the image width. This indicates the number of image channels.

[0069] First, preliminary feature extraction is performed using convolutional layers. The convolution operation can be represented as:

[0070]

[0071] in, It is the first Feature map of the layer It is the first The first layer Each feature map It is the first The first layer One convolutional kernel, It is the first Number of feature maps in a layer This represents the convolution operation. It is the bias vector of the l-th layer. For activation functions, such as the ReLU function .

[0072] Considering that similar visual elements in transitional spaces can easily cause feature confusion, a channel attention mechanism is introduced. For feature maps... First, perform global average pooling and global max pooling to obtain the channel descriptor. and :

[0073]

[0074]

[0075] in, Representation of feature map The One channel.

[0076] Will and After concatenation, the data is processed by a multilayer perceptron (MLP) to obtain the channel attention weights. :

[0077]

[0078] in, , It is the weight matrix of the MLP. It's the dimensionality reduction ratio. It is an activation function such as ReLU. This represents the vector concatenation operation. The final weighted feature map is obtained: :

[0079] Furthermore, to capture visual information at different scales, a multi-scale feature fusion strategy is employed. This is achieved by setting convolutional kernels of different sizes (e.g., ...). , , Parallel feature extraction yields feature maps at different scales. , , After adjusting these feature maps to the same size using bilinear interpolation, a weighted summation method is used for fusion.

[0080]

[0081] in, , , The weight coefficients are learned from the training data, and .

[0082] After the above processing, the final basic visual features are obtained. It can more effectively capture detailed information in multi-story building scenes, distinguish subtle differences between similar transition spaces, and provide more discriminative underlying features for the construction of subsequent floor semantic layers and cross-floor semantic association layers.

[0083] Establish a semantic layer for each floor: Based on the building's floor layout and functional area division, assign a specific semantic label to each floor. For example, label the first floor as the "lobby floor" and the third floor as the "office area floor," etc. At the same time, combine the spatial structure information of the floor, such as the location of the elevator lobby and the direction of the staircase, to construct a semantic map of the floor, describing the relationships between the various areas within the floor.

[0084] To accurately assign specific semantic information to each floor of a multi-story building and construct the regional relationships within each floor, thereby overcoming the problem of spatial cognitive ambiguity in cross-floor navigation, a method for constructing a floor semantic layer that integrates spatial topology and visual features is designed.

[0085] First, based on the building floor plan and functional area division, an initial set of semantic tags is generated for each floor. ,in This represents the total number of floors. For example, in an office building scenario, It can mean "Lobby Level". This indicates areas such as "office floor". Additionally, for each floor's sub-areas, such as elevator lobbies, meeting rooms, and corridors, a set of semantic tags for each sub-area is defined. , This represents the number of sub-regions.

[0086] To establish semantic relationships between different areas within a floor, a floor semantic graph is constructed. Among them, the node set Includes floor nodes and sub-region nodes, and an edge set. This indicates the connection relationships between nodes. For adjacent region nodes... and Edge weight Defined as:

[0087]

[0088] in, For nodes and The shortest path distance between them (which can be calculated from the building floor plan). This is the semantic association coefficient for the region, and its value range is... This is learned through expert annotation or historical navigation data. For example, the elevator lobby and corridor... The value is higher because the two functions are closely related; while the values ​​for restrooms and meeting rooms are... The value is relatively low.

[0089] Features extracted using the basic visual feature layer Based on this, a Graph Convolutional Network (GCN) is introduced to propagate and update features of the floor semantic graph. The layer graph convolution operation is represented as:

[0090]

[0091] in, It is the feature matrix of the nodes in the k-th layer. Initialize to use basic visual features Feature vector encoded with node semantic labels; It is an adjacency matrix with self-loops added. This is the original adjacency matrix; yes The degree matrix of the diagonal nodes; It is the first Layer weight matrix; For example, ReLU is an activation function.

[0092] Considering the dynamic changes in semantic features across different floors in a multi-story building, such as functional changes in certain floors during specific time periods, a dynamic semantic encoding mechanism is adopted. For each node... Its dynamic semantic features The calculation is as follows:

[0093]

[0094] in, It is a node The static semantic features (determined by the initial semantic label and basic visual features). These are temporary semantic features that are dynamically updated based on factors such as time and environment (e.g., obtained through sensor data or external information). These are weighting coefficients, adjusted through online learning to balance static and dynamic semantic information.

[0095] After the above processing, the floor semantic layer can effectively integrate the building spatial structure, regional semantic relationships and dynamic change information, providing the intelligent agent with accurate floor and regional semantic cognition when navigating across floors, and laying the foundation for the construction of the subsequent cross-floor semantic association layer and navigation decision-making.

[0096] Constructing a cross-floor semantic association layer: By analyzing the visual features and semantic information of transitional spaces between different floors, a cross-floor semantic association model is established. This model can identify the changing patterns of transitional spaces such as elevator lobbies and safety passages on different floors, as well as their relationships with target floor areas. For example, through learning, it was discovered that although the elevator lobbies from the first to the third floor in a certain building appear visually similar, the exit direction of the third-floor elevator lobbies has a specific angular relationship with the east-facing conference room.

[0097] To address the navigation gaps across floors caused by transitional spaces such as elevator lobbies and safety passages in multi-story buildings, a method for constructing a cross-floor semantic association layer that integrates spatiotemporal information and semantic topology is designed to achieve effective association and transmission of semantic information between different floors.

[0098] First, feature modeling is performed on the transitional spaces. These transitional spaces are considered as key nodes connecting different floors, with each transitional space node... Having visual feature vectors and semantic tags .in, It is obtained by feature extraction and fusion from the basic visual feature layer, and can be specifically represented as the basic visual feature layer. Based on this, feature clipping and weighted aggregation are performed on transitional spatial regions:

[0099]

[0100] in, Represents transitional space nodes The set of pixel indices in an image. For pixels The weights are obtained by calculating the reciprocal of the distance between the pixel and the center of the transition space and then normalizing it. For pixels To the transitional space center The Euclidean distance.

[0101] Next, construct a cross-floor semantic graph. Node set Includes all floor nodes, sub-region nodes, and transitional space nodes; edge set This indicates the connection relationships between nodes on different floors. For nodes connecting across floors... (Located on the floor) ) and nodes (Located on the floor) ), edge weight The calculation is as follows:

[0102]

[0103] in, For nodes and Cosine similarity of visual features is used to measure the degree of similarity between visual features. For floors and floors The distance between them can be calculated based on the vertical structure of the building; For nodes and The relevance of semantic tags is calculated through the similarity of predefined relation matrices or semantic embedding vectors of semantic tags;

[0104] , , Let be the weighting coefficient, satisfying Optimization is achieved through reinforcement learning to balance the roles of visual, spatial, and semantic information in cross-floor associations.

[0105] To enhance the dynamic adaptability of cross-floor semantic association, a spatiotemporal attention mechanism is introduced. During agent navigation, attention is adjusted based on the current time. and location Calculate the spatiotemporal attention weights :

[0106]

[0107] in, For nodes and The correlation score under spatiotemporal conditions is calculated as follows:

[0108] , , These are the weighting coefficients. The value is determined by the influence of time factors and is quantified based on data such as pedestrian flow and functional usage of the building at different time periods. The value representing the influence of location factors is calculated based on the relationship between the agent's current location and the target location, as well as the traffic frequency in different areas.

[0109] Finally, based on the cross-floor semantic graph and spatiotemporal attention weights, cross-floor semantic association reasoning is performed. For the target node... Its cross-floor semantic association features The calculation is as follows:

[0110] Through the above steps, the cross-floor semantic association layer can effectively capture the semantic connections between different floors, dynamically adjust the association weights by combining spatiotemporal information, provide accurate semantic guidance for the agent when navigating across floors, and avoid the accumulation of navigation errors caused by similar transitional spatial features.

[0111] 2. Memory Augmentation Algorithm Design

[0112] Short-term memory module: During navigation tasks, the agent records the visual features and related semantic information of the current scene in real time, forming short-term memory. Short-term memory primarily stores scene data experienced by the agent within a recent period, such as the current floor, the area traversed, and obvious visual landmarks observed. When the agent enters a transitional space, short-term memory retains the floor and location information from before entry for subsequent comparison and calibration.

[0113] In multi-story building cross-floor navigation scenarios, intelligent agents face challenges such as dynamic environmental changes and high similarity of transition spaces. To achieve efficient storage and rapid retrieval of real-time information, a short-term memory module based on dynamic time windows and multimodal fusion is designed.

[0114] First, define the short-term memory storage structure. Short-term memory modules store information in a queue format, denoted as [queue name missing]. ,in Indicates the first One memory unit, This represents the maximum capacity of the queue. Each memory unit... It contains multimodal information, specifically represented as:

[0115]

[0116] in, Image data collected by the agent at the current moment; It is an image visual feature vector extracted from the basic visual feature layer. , For feature dimensions; The semantic labels for the corresponding scenarios are obtained through reasoning from the floor semantic layer and the cross-floor semantic association layer; For information collection timestamps; These are the position coordinates of the agent when it collects information. , , For planar coordinates, For floor identification.

[0117] To adapt to the dynamic information changes of the intelligent agent during navigation, a dynamic time window mechanism is introduced. The time window length is set. At each time step Determine the current time With the earliest stored information time The difference:

[0118] like If the earliest timestamp is deleted, the queue will only store information from the most recent period. Simultaneously, the time window length (L) is dynamically adjusted based on the complexity of the building environment, using the following formula:

[0119]

[0120] in, This represents the standard deviation of the visual features of the current scene, used to measure the degree of visual change in the scene. The mean standard deviation; and These are the preset maximum standard deviation and minimum mean, respectively; The coefficients were determined through experimental optimization. When the scene visuals change drastically, the time window length is increased to retain more information; conversely, it is decreased.

[0121] Regarding information updates, when the intelligent agent collects new information... When new information is added, the similarity between it and existing memory units in the queue is calculated. For memory unit \(e_i\), the similarity \(sim(e_i, e_{new})\) between it and \(e_{new}\) is calculated as follows:

[0122]

[0123] in, Cosine similarity of visual feature vectors; For semantic tag matching degree, if but Otherwise, it is 0; The Euclidean distance between the position coordinates; , , These are the weighting coefficients, and The algorithm is optimized through training to balance the roles of different modalities in similarity calculation. If any similarity exceeds a threshold... If the memory cell is updated, then the information of that cell is updated to... Otherwise Add to the queue.

[0124] During the information retrieval phase, when the agent makes navigation decisions, it retrieves relevant information from short-term memory based on current needs. For example, when navigating across floors to enter a transitional space, it retrieves memory units with similar visual features and semantic labels to the current transitional space, providing a reference for navigation. Retrieval Function Defined as:

[0125]

[0126] in, To retrieve search criteria, visual features, semantic tags, or location information can be included; The retrieval threshold was determined experimentally.

[0127] Through the aforementioned dynamic time window and multimodal fusion mechanism, the short-term memory module can efficiently store and manage real-time information of the agent during cross-floor navigation, quickly respond to navigation decision requirements, and assist the agent in accurately navigating in complex and ever-changing multi-story building environments.

[0128] Long-term memory module: This module stores the overall structural information of the multi-story building, the semantic features of each floor, and cross-floor semantic relationships. This information is obtained through pre-training and learning on the building. During navigation, the long-term memory module continuously interacts and updates with the short-term memory, providing the agent with global navigation references. For example, when the agent becomes disoriented on a floor, the long-term memory module can provide possible directions and path suggestions to the target floor based on the stored floor semantic graph and cross-floor relationship information.

[0129] In the scenario of navigation across floors in multi-story buildings, in order to provide the intelligent agent with a stable and comprehensive building knowledge system and overcome the problem of navigation gaps in transitional spaces, a long-term memory module based on knowledge graphs and reinforcement learning update strategies is designed to achieve efficient storage and dynamic maintenance of building structure and semantic information.

[0130] First, construct a knowledge graph storage structure for long-term memory. Node set Includes floor nodes Sub-region nodes Transitional space nodes and semantic concept nodes ,in , , , These represent the number of corresponding nodes. Edge set This represents the semantic relationships between nodes, such as the "containment relationship" between floors and sub-regions, the "connection relationship" between transitional spaces and floors, and the "attribute relationship" between semantic concepts and nodes. Each edge... Attachment weight This is used to quantify the tightness of the relationship between nodes, and its calculation method is as follows:

[0131]

[0132] in, Represents a node and The frequency of the relationship between them can be obtained by analyzing and statistically analyzing historical navigation data or design drawings of buildings. The confidence level of the relationship, with a value range of [value range missing]. The values ​​are determined by expert annotation or by evaluation based on machine learning algorithms.

[0133] To achieve efficient knowledge reasoning based on knowledge graphs, a graph neural network (GNN) algorithm is employed. For nodes... In its first Layer feature representation Update using the following formula:

[0134]

[0135] in, For nodes The set of neighboring nodes, This represents the number of neighboring nodes; and It is the first The trainable weight matrix of the layer; The activation function is ReLU. Through iterative updates of multiple GNNs, each node can aggregate semantic information from its neighbors, thereby learning more representative features and supporting semantic reasoning by the agent during cross-floor navigation.

[0136] Considering potential structural changes in multi-story building environments (such as adjustments to zoning functions and renovations), a dynamic update strategy based on reinforcement learning is designed. The state space is defined. The action space is the set of all possible states of a long-term memory knowledge graph. This includes operations such as adding, deleting, and modifying nodes, as well as adjusting edge relationship weights. Reward function. Designed as follows:

[0137]

[0138] in, To perform the action The state after; The improvement in accuracy of the updated knowledge graph in navigation tasks can be measured by the ratio of the actual number of successful navigation attempts to the total number of navigation attempts. The consistency score of the knowledge graph is used to evaluate the logical rationality of semantic relationships in the updated graph, and its value ranges from [value range missing]. ; To perform the action The complexity metric is such that adding multiple nodes is more complex than modifying the weight of a single edge; , , The weighting coefficients are determined through experimental optimization to balance accuracy, consistency, and operational complexity.

[0139] The intelligent agent interacts with the environment (i.e., the actual situation of a multi-story building) and adjusts its actions based on the current state. Select Action Receive a reward after performing the action. The system updates the long-term memory knowledge graph. The optimal strategy is learned using a Deep Q-Network (DQN) algorithm. Its core formula is:

[0140]

[0141] in, State-Action Pairs The value function; The learning rate; This serves as a discount factor, used to balance the importance of current and future rewards. Through continuous iterative training, the agent learns optimal strategies for updating long-term memory under different building changes, ensuring that the long-term memory module always provides accurate and effective knowledge support for cross-floor navigation.

[0142] Memory Fusion and Update Strategy: A memory fusion algorithm is designed to effectively integrate short-term and long-term memory. When the agent encounters key decision points, such as reaching an elevator or a floor transition, the short-term and long-term memories are weighted and fused based on the visual features and semantic information of the current scene to generate more accurate navigation decisions. Simultaneously, as the agent moves through the building, the short-term and long-term memories are continuously updated to ensure the timeliness and accuracy of the information. For example, when the agent detects a change in a previously marked visual landmark, the information in the relevant area of ​​the long-term memory is updated promptly for more accurate subsequent navigation.

[0143] In multi-story building navigation scenarios, intelligent agents face the challenge of balancing dynamically changing environmental information and the timeliness of short-term memory with the stability of long-term memory. To achieve accurate navigation, a memory fusion and update strategy based on adaptive weight fusion and dynamic feedback optimization is designed to effectively integrate information from short-term and long-term memory modules, thereby improving the accuracy and reliability of navigation decisions.

[0144] (1) Memory fusion mechanism

[0145] The information representation of the short-term memory module is defined as follows: ,in For the first Each memory unit; the knowledge graph representation of the long-term memory module is as follows: When the agent encounters a critical decision point (such as arriving at the elevator or a floor changeover), it performs a memory fusion operation.

[0146] First, calculate the short-term memory unit. Nodes in long-term memory knowledge graph correlation For visual feature correlation, cosine similarity is used for calculation:

[0147]

[0148] in, It is a short-term memory unit Visual feature vectors, It is a node in the long-term memory knowledge graph. The visual feature vector.

[0149] For semantic relevance, it is calculated based on the degree of matching of semantic tags:

[0150]

[0151] in, It is a short-term memory unit semantic tags, It is a node in the long-term memory knowledge graph. Semantic tags.

[0152] Combining visual and semantic relevance, the overall relevance between short-term memory units and long-term memory nodes is obtained:

[0153] in, and These are the weighting coefficients, and The algorithm is optimized through training to balance the roles of visual and semantic information in relational calculation.

[0154] Based on the correlation, adaptive fusion weights for short-term and long-term memory are calculated. For the weights of short-term memory... The calculation method is as follows:

[0155]

[0156] in, The number of nodes in a knowledge graph for long-term memory. Let be the number of edges. For the edge The weights of long-term memory. .

[0157] Final fusion of memory information for:

[0158]

[0159] in, This is for performing aggregation operations on short-term memory (such as average pooling and weighted summation). This information is obtained through reasoning based on a long-term memory knowledge graph.

[0160] (2) Memory update strategy

[0161] Memory renewal is divided into two parts: short-term memory renewal and long-term memory renewal.

[0162] Short-term memory update: continuously collecting new information as the agent moves. .calculate Similarity to existing memory units in short-term memory The calculation method is the same as the similarity calculation in the short-term memory module. If a similarity exceeds a threshold... If the memory cell is not found, then update the cell to... If all similarities are below the threshold, then determine if the short-term memory queue is full. If not, proceed... Add to the queue; if the queue is full, add the oldest memory unit after deleting it, according to the dynamic time window mechanism. At the same time, the time window length is dynamically adjusted according to environmental changes, and the adjustment formula is consistent with that in the short-term memory module.

[0163] Long-term memory update: After an agent completes a navigation task, a long-term memory update is triggered based on the navigation result (success or failure) and environmental change detection (such as sensing changes in building structure through sensors). This utilizes the reward function in reinforcement learning algorithms. Assess the current state of the long-term memory knowledge graph Execute update action The effect.

[0164] First, environmental changes are detected using sensor data (such as changes in camera images and building management system information) to determine if long-term memory needs updating. If a change is detected, the agent adjusts its current state accordingly. From the action space Select update action After performing the action, a new state is obtained. And calculate the reward .

[0165] The long-term memory knowledge graph is updated using a Deep Q-Network (DQN) algorithm. Based on the reward... and the maximum value function of subsequent states Value function for updating state-action pairs :

[0166]

[0167] in, For learning rate, This serves as a discount factor. Through continuous iterative training, the agent learns the optimal long-term memory update strategy to adapt to the dynamic changes in the built environment, ensuring the accuracy and effectiveness of the long-term memory knowledge graph.

[0168] By employing the aforementioned memory fusion and update strategies, the real-time nature of short-term memory and the stability of long-term memory can be effectively integrated. This allows the intelligent agent to dynamically adjust memory information based on environmental changes during navigation across multiple floors in a multi-story building, providing a more reliable basis for navigation decisions and improving navigation performance.

[0169] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of the claims of the present invention.

Claims

1. A hierarchical visual language navigation memory enhancement system for cross-floor scenes, characterized by: It includes a hierarchical visual semantic model construction module, a memory enhancement algorithm module, and a navigation system integration module; The hierarchical visual semantic model construction module is used to construct a basic visual feature layer, a floor semantic layer, and a cross-floor semantic association layer. The memory enhancement algorithm module includes a short-term memory module, a long-term memory module, and a memory fusion and update strategy module; The navigation system integration module is used to integrate a hierarchical visual semantic model and a memory enhancement algorithm into the system.

2. The hierarchical visual language navigation memory enhancement system for cross-floor scenes according to claim 1, characterized in that: The method for constructing the basic visual feature layer is as follows: a deep convolutional neural network is used to process the scene image inside a multi-story building, a channel attention mechanism is introduced to highlight key visual features, and a multi-scale feature fusion strategy is used to capture information at different scales to obtain basic visual features. Specifically, in the complex and similar transitional spatial environment of multi-story buildings, an improved deep convolutional neural network architecture is adopted to accurately extract effective visual features, with the input image... ,in Indicates the image height. Indicates the image width. Indicates the number of image channels; First, preliminary feature extraction is performed through convolutional layers. The convolution operation can be represented as: in, It is the first Feature map of the layer It is the first The first layer Each feature map It is the first The first layer One convolutional kernel, It is the first Number of feature maps in a layer This represents the convolution operation. It is the bias vector of the l-th layer. For activation functions, such as the ReLU function ; Considering that similar visual elements in transitional spaces can easily cause feature confusion, a channel attention mechanism is introduced for feature maps. First, perform global average pooling and global max pooling to obtain the channel descriptor. and : in, Representation of feature map The One channel; will and After concatenation, the data is processed by a multilayer perceptron (MLP) to obtain the channel attention weights. : in, , It is the weight matrix of the MLP. It's the dimensionality reduction ratio. It is an activation function such as ReLU. This represents the vector concatenation operation; the final result is the weighted feature map: : ; Furthermore, to capture visual information at different scales, a multi-scale feature fusion strategy is adopted. By setting convolutional kernels of different sizes and extracting features in parallel, feature maps at different scales are obtained. , , After adjusting these feature maps to the same size using bilinear interpolation, a weighted summation method is used for fusion: in, , , The weight coefficients are learned from the training data, and .

3. The hierarchical visual language navigation memory enhancement system for cross-floor scenes according to claim 1, characterized in that: The method for constructing the floor semantic layer is as follows: based on the building layout, semantic labels are assigned to floors and sub-regions to construct a floor semantic graph describing regional relationships; the graph is trained using a graph convolutional network, and combined with a dynamic semantic coding mechanism, the semantic features are updated according to time and environmental factors. Specifically, to accurately assign specific semantic information to each floor of a multi-story building and construct the regional relationships within each floor to overcome the problem of spatial cognitive ambiguity in cross-floor navigation, a method for constructing a floor semantic layer that integrates spatial topology and visual features is designed. First, based on the building floor plan and functional area division, an initial set of semantic tags is generated for each floor. ,in The total number of floors; simultaneously, for each sub-region within a floor, a set of semantic tags for that sub-region is defined. , This represents the number of sub-regions. To establish semantic relationships between different areas within a floor, a floor semantic graph is constructed. Among them, the set of nodes Includes floor nodes and sub-region nodes, and an edge set. This represents the connection relationship between nodes; for adjacent region nodes... and Edge weight Defined as: in, For nodes and The shortest path distance between them This is the semantic association coefficient for the region, and its value range is... ; Features extracted using the basic visual feature layer Based on this, a Graph Convolutional Network (GCN) is introduced to perform feature propagation and update of the floor semantic graph; The layer graph convolution operation is represented as: in, It is the feature matrix of the nodes in the k-th layer. Initialize to use basic visual features Feature vector encoded with node semantic labels; It is an adjacency matrix with self-loops added. This is the original adjacency matrix; yes The degree matrix of the diagonal nodes; It is the first Layer weight matrix; For activation functions; Considering the dynamic changes in semantic features across different floors in a multi-story building, a dynamic semantic coding mechanism is adopted; for each node... Its dynamic semantic features The calculation is as follows: in, It is a node Static semantic features These are temporary semantic features that are dynamically updated based on factors such as time and environment. These are weighting coefficients, adjusted through online learning to balance static and dynamic semantic information.

4. The hierarchical visual language navigation memory enhancement system for cross-floor scenes according to claim 1, characterized in that: The method for constructing the cross-floor semantic association layer is as follows: feature modeling is performed on the transitional space to construct a cross-floor semantic graph representing the relationship between nodes between floors; a spatiotemporal attention mechanism is introduced to adjust the association weights between nodes according to time and space information to achieve cross-floor semantic reasoning. First, feature modeling is performed on the transitional spaces, treating them as key nodes connecting different floors. Each transitional space node... Having visual feature vectors and semantic tags ;in, It is obtained by feature extraction and fusion from the basic visual feature layer, and can be specifically represented as the basic visual feature layer. Based on this, feature clipping and weighted aggregation are performed on transitional spatial regions: in, Represents transitional space nodes The set of pixel indices in an image. For pixels The weights are obtained by calculating the reciprocal of the distance between the pixel and the center of the transition space and normalizing it. For pixels To the transitional space center The Euclidean distance; Next, construct a cross-floor semantic graph. Node set Includes all floor nodes, sub-region nodes, and transitional space nodes; edge set This indicates the connection relationship between nodes on different floors; for nodes located on a floor... Cross-floor connection nodes and located on the floor nodes Edge weight The calculation is as follows: in, For nodes and Cosine similarity of visual features is used to measure the degree of similarity between visual features. For floors and floors The distance between them can be calculated based on the vertical structure of the building; For nodes and The relevance of semantic tags is calculated through the similarity of predefined relation matrices or semantic embedding vectors of semantic tags; , , Let be the weighting coefficient, satisfying Optimization is achieved through reinforcement learning to balance the roles of visual, spatial, and semantic information in cross-floor associations; To enhance the dynamic adaptability of cross-floor semantic association, a spatiotemporal attention mechanism is introduced; during agent navigation, attention is adjusted based on the current time. and location Calculate the spatiotemporal attention weights : in, For nodes and The correlation score under spatiotemporal conditions is calculated as follows: ; , , These are the weighting coefficients. The value is determined by the influence of time factors and is quantified based on data such as pedestrian flow and functional usage of the building at different time periods. The value representing the influence of location factors is calculated based on the relationship between the agent's current location and the target location, as well as the traffic frequency in different areas. Finally, based on the cross-floor semantic graph and spatiotemporal attention weights, cross-floor semantic association reasoning is performed; for the target node... Its cross-floor semantic association features The calculation is as follows: .

5. The hierarchical visual language navigation memory enhancement system for cross-floor scenes according to claim 1, characterized in that: The short-term memory module uses a queue structure to store multimodal information collected in real time by the agent, controls the information storage duration through a dynamic time window mechanism, updates the information based on information similarity, and provides a retrieval function based on query conditions. Specifically, the following steps are taken: First, define the short-term memory storage structure. Short-term memory modules store information in a queue format, denoted as... ,in Indicates the first One memory unit, Maximum queue capacity; each memory unit It contains multimodal information, specifically represented as: in, Image data collected by the agent at the current moment; It is an image visual feature vector extracted from the basic visual feature layer. , For feature dimensions; The semantic labels for the corresponding scenarios are obtained through reasoning from the floor semantic layer and the cross-floor semantic association layer; For information collection timestamps; These are the position coordinates of the agent when it collects information. , , For planar coordinates, For floor identification; To adapt to the dynamic information changes of the intelligent agent during navigation, a dynamic time window mechanism is introduced, and the length of the time window is set. At each time step Determine the current time With the earliest stored information time The difference: ; like If the earliest timestamp is deleted, the queue will only store information from the most recent period. Simultaneously, the time window length (L) is dynamically adjusted based on the complexity of the building environment, using the following formula: in, This represents the standard deviation of the visual features of the current scene, used to measure the degree of visual change in the scene. The mean standard deviation; and These are the preset maximum standard deviation and minimum mean, respectively; To adjust the coefficients, the time window length was determined through experiments: when the scene visuals change drastically, the time window length is increased to retain more information; conversely, it is decreased. Regarding information updates, when the intelligent agent collects new information... When new information is added, the similarity between it and existing memory units in the queue is calculated. For memory unit \(e_i\), the similarity \(sim(e_i, e_{new})\) between it and \(e_{new}\) is calculated as follows: in, Cosine similarity of visual feature vectors; For semantic tag matching degree, if but Otherwise, it is 0; The Euclidean distance between the position coordinates; , , These are the weighting coefficients, and The similarity calculation is optimized through training to balance the roles of different modalities; if the similarity exceeds a threshold... If the memory cell is updated, then the information of that cell is updated to... Otherwise Add to queue; During the information retrieval phase, when the agent makes navigation decisions, it retrieves relevant information from its short-term memory based on current needs; retrieval function Defined as: in, To retrieve search criteria, visual features, semantic tags, or location information can be included; The retrieval threshold was determined experimentally.

6. The hierarchical visual language navigation memory enhancement system for cross-floor scenes according to claim 1, characterized in that: The long-term memory module uses a knowledge graph to store building structure and semantic information, and uses a graph neural network to realize knowledge reasoning; it updates the knowledge graph based on navigation results and environmental changes through reinforcement learning algorithms. Specifically, the process is as follows: First, construct a knowledge graph storage structure for long-term memory. Node set Includes floor nodes Sub-region nodes Transitional space nodes and semantic concept nodes ,in , , , These represent the number of corresponding nodes and the set of edges, respectively. Representing the semantic relationships between nodes, each edge Attachment weight This is used to quantify the tightness of the relationship between nodes, and its calculation method is as follows: in, Represents a node and The frequency of the relationship between them can be obtained by analyzing and statistically analyzing historical navigation data or design drawings of buildings. The confidence level of the relationship, with a value range of [value range missing]. Determined by expert annotation or based on machine learning algorithm evaluation; To achieve efficient knowledge reasoning based on knowledge graphs, a graph neural network (GNN) algorithm is adopted; for nodes... In its first Layer feature representation Update using the following formula: in, For nodes The set of neighboring nodes, This represents the number of neighboring nodes. and It is the first The trainable weight matrix of the layer; The activation function is used; through multi-layer GNN iterative updates, each node can aggregate semantic information from neighboring nodes, thereby learning more representative features and supporting semantic reasoning of the agent when navigating across floors. Considering the potential structural changes in multi-story building environments, a dynamic update strategy based on reinforcement learning is designed; the state space is defined. The action space is the set of all possible states of a long-term memory knowledge graph. This includes adding, deleting, and modifying nodes, adjusting edge relationship weights, and other related operations; reward function. Designed as follows: in, To perform the action The state after; The improvement in accuracy of the updated knowledge graph in navigation tasks can be measured by the ratio of the actual number of successful navigation attempts to the total number of navigation attempts. The consistency score of the knowledge graph is used to evaluate the logical rationality of semantic relationships in the updated graph, and its value ranges from [value range missing]. ; To perform the action The complexity metric; , , The weighting coefficients were determined through experimental optimization to balance accuracy, consistency, and operational complexity. The intelligent agent interacts with the environment and adjusts its actions based on the current state. Select Action Receive a reward after performing the action. It also updates the long-term memory knowledge graph; and employs a deep Q-network (DQN) algorithm to learn the optimal strategy. Its core formula is: in, State-Action Pairs The value function; The learning rate; This is a discount factor used to balance the importance of current rewards and future rewards.

7. The hierarchical visual language navigation memory enhancement system for cross-floor scenes according to claim 1, characterized in that: The memory fusion and update strategy module calculates the correlation between short-term memory and long-term memory and determines the adaptive fusion weight at the agent's key decision points to achieve information fusion; it updates short-term memory and long-term memory according to the set rules, and optimizes relevant parameters based on the performance feedback after the update. Specifically as follows: (1) Memory fusion mechanism The information representation of the short-term memory module is defined as follows: ,in For the first One memory unit; The knowledge graph representation of the long-term memory module is as follows When the agent encounters a critical decision point, it performs a memory fusion operation. First, calculate the short-term memory unit. Nodes in long-term memory knowledge graph correlation For visual feature correlation, cosine similarity is used for calculation: in, It is a short-term memory unit Visual feature vectors, It is a node in the long-term memory knowledge graph. Visual feature vectors; For semantic relevance, it is calculated based on the degree of matching of semantic tags: in, It is a short-term memory unit semantic tags, It is a node in the long-term memory knowledge graph. semantic tags; Combining visual and semantic relevance, the overall relevance between short-term memory units and long-term memory nodes is obtained: ; in, and These are the weighting coefficients, and The training optimization aims to balance the roles of visual and semantic information in the correlation calculation. Based on the correlation, adaptive fusion weights for short-term and long-term memory are calculated; for the weights of short-term memory... The calculation method is as follows: in, The number of nodes in a knowledge graph for long-term memory. Let be the number of edges. For the edge Weights; weights of long-term memory ; Final fusion of memory information for: in, To perform aggregation operations on short-term memory, Information obtained through reasoning based on long-term memory knowledge graphs; (2) Memory update strategy Memory updating is divided into two parts: short-term memory updating and long-term memory updating. Short-term memory update: continuously collecting new information as the agent moves. ;calculate Similarity to existing memory units in short-term memory The calculation method is the same as the similarity calculation in the short-term memory module; if the similarity exceeds the threshold... If the memory cell is not found, then update the cell to... If all similarities are below the threshold, then determine if the short-term memory queue is full; if not, then... Add to the queue; if the queue is full, add the oldest memory unit after deleting it, according to the dynamic time window mechanism. At the same time, the time window length is dynamically adjusted according to environmental changes, and the adjustment formula is consistent with that in the short-term memory module. Long-term memory update: After the agent completes a navigation task, the long-term memory is updated based on the navigation results and environmental change detection; this is achieved using the reward function in reinforcement learning algorithms. Assess the current state of the long-term memory knowledge graph Execute update action The effect; First, the agent detects environmental changes using sensor data to determine if long-term memory needs updating; if a change is detected, the agent adjusts its current state accordingly. From the action space Select update action After performing the action, a new state is obtained. And calculate the reward ; The long-term memory knowledge graph is updated using the Deep Q-Network (DQN) algorithm; based on rewards... and the maximum value function of subsequent states Value function for updating state-action pairs: ： in, For learning rate, The discount factor is used; through continuous iterative training, the agent learns the optimal long-term memory update strategy to adapt to the dynamic changes in the built environment, ensuring the accuracy and effectiveness of the long-term memory knowledge graph.

8. The hierarchical visual language navigation memory enhancement system for cross-floor scenes according to claim 1, characterized in that: The navigation system integration module connects the hierarchical visual semantic model and memory enhancement algorithm with the motion control module and sensor module of the intelligent agent, and uses a unified interface protocol to realize data interaction. The integrated navigation system is tested by designing diverse test scenarios, and the model and algorithm are optimized based on the performance evaluation results.

Images