Robot map asynchronous establishment method, electronic device and storage medium

Abstract

This application relates to an asynchronous robot map creation method, electronic device, and storage medium, belonging to the field of robot map construction technology. The method includes: asynchronously receiving multi-source data streams; constructing an implicit network model, which takes sampled spatial coordinates, viewpoint direction, and linguistic features as input, and outputs the occupancy probability and semantic feature vector of the corresponding spatial coordinates to form a three-dimensional implicit occupancy map; fusing the semantic features corresponding to the query point in the three-dimensional implicit occupancy map with the geometric features in the LiDAR point cloud map across modalities to obtain fused features, and constructing and maintaining an environment map for the robot. This invention integrates online construction of a three-dimensional implicit occupancy map, efficient cross-modal fusion based on a dual sparsity strategy, and specialized adaptation for the dynamic characteristics of bipedal robots through a complete asynchronous processing framework, generating a unified environment model with both accurate geometry and rich semantics.

Description

Technical Field

[0001] This application relates to the field of robot environmental perception and mapping technology, specifically to a method for asynchronous robot mapping, an electronic device, and a storage medium. Background Technology

[0002] Bipedal humanoid robots, due to their human-like locomotion and maneuvering potential, have broad application prospects in performing complex tasks such as domestic services, disaster relief, and industrial inspection. Achieving high-level autonomous navigation, such as understanding and executing natural language commands that integrate spatial location and semantic function descriptions, like "Go to the bedroom and get me the water glass from the bedside table," poses a significant challenge to the robot's environmental perception and mapping capabilities.

[0003] In existing technologies, environmental map construction methods for robot navigation are mainly divided into two categories: One type is environmental semantic representations generated based on visual or visual language navigation (VLN) frameworks, such as 3D implicit occupancy maps. These maps map spatial coordinates to occupancy probabilities and semantic features via neural networks, encoding rich information about object categories and appearances. However, their geometric accuracy is limited by the distribution of training data and the network's generalization ability, easily leading to scale distortion and blurred boundaries in real-world scenarios, making it difficult to provide reliable spatial constraints for robot physical obstacle avoidance and fine manipulation.

[0004] Another type is point cloud maps generated using LiDAR (LiDAR) simultaneous localization and mapping (SLAM) technology. These maps have millimeter-level geometric accuracy and can accurately depict the three-dimensional structure of the environment. However, they are essentially "dumb" data lacking semantic labels. Robots cannot directly associate the geometric clusters in the point cloud with semantic concepts in navigation commands (such as "door" or "cup"), limiting their ability to execute semantic commands.

[0005] Existing methods that attempt to link the two, such as projecting visual semantic segmentation results onto point clouds for coloring, are merely superficial and static data annotations, failing to establish deep, reasonable semantic-geometric correspondences. Furthermore, the inherent problems of bipedal humanoid robots during walking, such as torso swaying, asynchronous sensor data, and limited onboard computing resources, all pose significant challenges to the robustness, real-time performance, and integration of existing map-building methods.

[0006] Therefore, there is an urgent need for a method that can efficiently and deeply integrate asynchronous multimodal perception data, and is particularly adapted to the dynamic instability characteristics of bipedal humanoid robots, so as to generate a unified environment model with both accurate geometry and rich semantics. Summary of the Invention

[0007] In order to overcome the above-mentioned defects, this application is proposed to solve or at least partially solve the technical problem that the semantic information and geometric information representation in environmental maps are separated and weakly correlated, making it difficult for robots to simultaneously understand the semantic functions and precise spatial boundaries of the environment.

[0008] According to a first aspect of the present invention, an asynchronous robot map creation method is provided, comprising: The system receives multi-source data streams asynchronously. The multi-source data streams include at least lidar point cloud data and natural language command data. Each data stream is preprocessed to obtain the corresponding lidar point cloud map and language features. An implicit network model is constructed. The implicit network model takes the sampled spatial coordinates, the view direction inferred based on the robot's current pose, and the language features as inputs, and outputs the occupancy probability and semantic feature vector corresponding to the spatial coordinates to form a three-dimensional implicit occupancy map. The spatial coordinates are the positions of three-dimensional sampling points obtained by sampling along the observation ray during ray propagation rendering. The semantic features corresponding to the query point in the three-dimensional implicit placeholder map are fused with the geometric features retrieved through the sparse attention mechanism in the lidar point cloud map across modes to obtain the fused features. Using the fusion features, an environmental map for the robot is constructed and maintained.

[0009] By constructing semantic and geometric maps in parallel and deeply fusing them through an innovative dual-sparse attention mechanism, the shortcomings of traditional methods in separating semantic understanding and geometric accuracy are overcome. This enables robots to simultaneously acquire high-level semantic understanding of the environment (such as recognizing "cup" and "door") and precise geometric spatial perception (such as determining the boundaries of cups and the position of door frames) within a unified framework, providing a complete cognitive foundation for realizing complex semantic navigation commands such as "go get the cup on the coffee table".

[0010] In the above technical solution, the query point is a discrete location sampled in the three-dimensional implicit occupancy map and whose occupancy probability is higher than a threshold. The step of fusing the semantic features corresponding to the query point in the 3D implicit placeholder map with the geometric features retrieved from the lidar point cloud map through a sparse attention mechanism across modalities includes: The query points are sampled from the three-dimensional implicit placeholder map, and the corresponding semantic feature vectors are obtained through the implicit network model to form a query set; Geometric feature vectors are extracted from the laser radar point cloud map. These geometric feature vectors are obtained by performing super-point clustering and lightweight PointNet encoding on the laser point cloud to form a key-value set. A sparse mask is generated based on a dual sparsification strategy of geometric proximity and semantic saliency. Based on the sparse mask, spatially neighboring and semantically related geometric feature vectors are retrieved from the key value set for each query point, and sparse attention is calculated to obtain the fused feature.

[0011] In any of the above technical solutions, the dual sparsity strategy includes: For each query point, the nearest neighbor points in the LiDAR point cloud map are found based on its three-dimensional coordinates to obtain the geometric proximity mask. Calculate the correlation score between the semantic features of the query point and all key-value features, retain the key-value pairs with the highest scores, and obtain the semantic saliency mask; The union of the geometric proximity mask and the semantic saliency mask is taken as the sparse mask.

[0012] In any of the above technical solutions, sparse attention computation is performed to obtain fused features, including: For each query point, attention is computed only within the range of the geometric feature vectors determined by the sparse mask; Calculate the degree of matching between the query point and each geometric feature vector within the sparse mask; The matching degree is normalized and converted into probability weights; Based on the probability weights, the geometric feature vectors within the sparse mask are weighted and summed to obtain the fusion feature of the query point. The fusion feature includes the semantic information of the query point and the spatially adjacent and semantically related geometric information.

[0013] In any of the above technical solutions, the multi-source data stream further includes image data. Updating the parameters of the implicit network model using online incremental learning includes: When a new image arrives in the image data, the current 3D implicit placeholder map generates a rendered image from the robot's current perspective. The photometric loss and semantic loss between the rendered image and the monitored image are calculated. The weight parameters and bias parameters inside the implicit network model are updated asynchronously using the gradient descent method to update the 3D implicit placeholder map expressed by the implicit network model.

[0014] In any of the above technical solutions, the multi-source data stream further includes inertial measurement data; Before acquiring the lidar point cloud map, the following steps are included: The laser point cloud is processed using the inertial measurement data to remove motion distortion, the robot pose is estimated using laser inertial odometry, and the laser radar point cloud map is constructed and output.

[0015] In any of the above technical solutions, in the sparse attention calculation, the nearest neighbor search radius is dynamically adjusted based on the shaking amplitude estimated by the inertial measurement data; For image data frames that degrade due to jitter, reduce the weight of the image data frames in the online training of the implicit network model.

[0016] In any of the above technical solutions, estimating the robot's pose using a laser inertial odometry system includes: registering the current frame point cloud with the historical frame point cloud, and combining the inertial measurement data to estimate the robot's relative displacement and rotation. Constructing the lidar point cloud map includes: transforming the current frame point cloud to a global coordinate system based on the estimated pose, stitching it together with the historical frame point cloud, and outputting the lidar point cloud map at the current moment and the local geometric features of each point.

[0017] In any of the above technical solutions, the periodic gait phase of the robot's bipedal motion is extracted based on the inertial measurement data, and the gait phase is encoded into a motion embedding vector; The motion embedding vector and the view direction are input into the implicit network model, so that the implicit network model learns the view-appearance mapping relationship under different time phases; During the inference phase, the current gait phase is used to predict the viewpoint direction at future moments, and images of the future viewpoint are rendered in advance for path planning.

[0018] In any of the above technical solutions, when new information arrives in the three-dimensional implicit occupancy map or the lidar point cloud map, a fusion update is initiated, and the affected local area is updated using the new information without waiting for the synchronization data of other branch maps. Using the fused features, an environmental map for the robot is constructed and maintained, including: inputting the fused features into a decoder network to predict enhanced occupancy probabilities, refine semantic labels, and access costs. The environment map uses a sparse voxel hash table as a hybrid representation. Each voxel stores: the fused occupancy probability, the refined semantic features, and the local geometric descriptor extracted from the lidar point cloud map. The local geometric descriptor includes a normal vector and curvature.

[0019] According to another aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the asynchronous robot map creation method described in any of the above technical solutions.

[0020] According to another aspect of the present invention, an electronic device is also provided, including a processor and a memory, wherein a computer program is stored in the memory, and when the processor executes the computer program, it implements the asynchronous robot map creation method as described in any of the above technical solutions.

[0021] This invention selectively and adaptively fuses the rich semantics of the 3D implicit occupancy map with the precise geometry of the LiDAR point cloud map through a sparse cross-modal attention mechanism. This overcomes the defect of information separation between the two in traditional methods, enabling robots to understand the semantic functions of the environment and the precise spatial boundaries at the same time, laying a cognitive foundation for executing complex semantic navigation commands.

[0022] An innovative dual sparsity strategy based on geometric proximity and semantic saliency is introduced, significantly reducing the computational complexity of traditional global attention from O(|Q|·|K|) to O(|Q|·c), where c is a constant and much smaller than |K|. This enables complex cross-modal fusion to run efficiently and in real-time on the limited onboard computing resources of bipedal robots.

[0023] A dynamic characteristic adaptation module for bipedal robots was specifically designed. IMU-guided motion distortion removal corrected point cloud distortion caused by walking sway; adaptive adjustment of feature search radius and image frame weights ensured the stability of feature matching and the quality of map construction in non-stationary states.

[0024] It allows each sensor module to operate independently and asynchronously at its own inherent frequency, and to be fused and updated through an event-triggered mechanism, avoiding the waiting delay and data loss introduced by traditional synchronization methods, and greatly improving the system's information throughput and response speed. Attached Figure Description

[0025] The disclosure of this application will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Wherein: Figure 1 This is a flowchart of an asynchronous robot map creation method according to an embodiment of this application; Figure 2 This is a schematic diagram of the overall architecture for asynchronous robot map creation according to an embodiment of this application; Figure 3 This is a schematic diagram of the overall architecture for asynchronous robot map creation according to an embodiment of the present invention. Detailed Implementation

[0026] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.

[0027] In the description of this application, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, memory, and may also include software components, such as program code, or a combination of software and hardware. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" or "this" can also include plural forms.

[0028] like Figure 1 As shown, an asynchronous robot map creation method according to an embodiment of the present invention may include the following main steps: Step 102: Receive multi-source data streams asynchronously. The multi-source data streams include at least LiDAR point cloud data and natural language command data. Preprocess each data stream to obtain the corresponding LiDAR point cloud map and language features.

[0029] Step 104: Construct an implicit network model. The implicit network model takes the sampled spatial coordinates, the viewpoint direction derived from the robot's current pose, and language features as inputs, and outputs the occupancy probability and semantic feature vector of the corresponding spatial coordinates to form a three-dimensional implicit occupancy map. The spatial coordinates are the positions of three-dimensional sampling points obtained by sampling along the observation ray during the rendering of light travel.

[0030] Step 106: The semantic features corresponding to the query point in the 3D implicit placeholder map are fused with the geometric features retrieved from the lidar point cloud map through the sparse attention mechanism to obtain the fused features.

[0031] Step 108: Utilize fusion features to construct and maintain an environmental map for the robot.

[0032] By constructing semantic and geometric maps in parallel and deeply fusing them through an innovative dual-sparse attention mechanism, the shortcomings of traditional methods in separating semantic understanding and geometric accuracy are overcome. The constructed environment map integrates semantic and geometric maps, forming a globally unified semantic-geometric map. This enables the robot to simultaneously acquire high-level semantic understanding of the environment (such as recognizing "cup" and "door") and precise geometric spatial perception (such as determining the boundaries of the cup and the position of the door frame) within a unified framework, providing a complete cognitive foundation for realizing complex semantic navigation commands such as "go get the cup on the coffee table."

[0033] Existing technologies typically treat vision-based semantic map construction and LiDAR-based geometric map construction as two independent or loosely coupled processes. While semantic maps can encode object category information, their geometric boundaries are vague; geometric maps, while highly accurate, lack semantic labels. This information gap makes it difficult for robots to associate semantic concepts in navigation commands (such as "cup") with precise locations and boundaries in physical space. This solution addresses this problem through an organic combination of the following techniques: First, an implicit network model is constructed, taking spatial coordinates, viewpoint direction, and linguistic features as inputs, and outputting occupancy probability and semantic feature vectors to form a semantically rich 3D implicit occupancy map. Second, a high-precision LiDAR point cloud map is obtained through preprocessing of LiDAR point cloud data. Then, the semantic features of query points in the 3D implicit occupancy map are fused with the geometric features in the LiDAR point cloud map across modalities to obtain fused features. Finally, a globally unified semantic-geometric map is constructed using the fused features. The generated unified map possesses both the rich semantic understanding capabilities from the implicit network model (such as identifying object categories) and the precise geometric perception capabilities from the lidar point cloud map (such as determining object boundaries), overcoming the defects of information fragmentation and providing a complete and unified cognitive foundation for robots to execute complex instructions such as "go get the cup on the coffee table" that integrate spatial location and semantic concepts.

[0034] In real-world robotic systems, sensors such as vision cameras, LiDAR, and IMU typically operate asynchronously at different frequencies (e.g., camera 30Hz, LiDAR 10Hz, IMU 400Hz). Existing map-building methods are mostly based on synchronous or strong temporal alignment assumptions, employing a strategy of "waiting for a complete frame of data to align before processing." This inevitably introduces processing latency or results in data being discarded due to timeouts, reducing information utilization. For bipedal humanoid robots that require high-frequency state estimation to maintain dynamic equilibrium, such latency is unacceptable. This solution addresses this problem through the following technical means: During the data reception phase, a strategy of "receiving multi-source data streams asynchronously" and "preprocessing each data stream separately" is explicitly adopted. This allows each sensor module to operate independently and in parallel at its inherent frequency without waiting for each other, thereby eliminating the processing latency introduced by forced synchronization, maximizing the real-time utilization of all available perceptual information, significantly improving the system's data throughput and response speed, and meeting the stringent real-time requirements of bipedal robots.

[0035] While traditional LiDAR point cloud maps possess millimeter-level geometric accuracy, they are essentially "dumb" data lacking semantic labels. Robots cannot directly associate geometric clusters in the point cloud with semantic concepts in natural language commands (such as "door" or "chair"), limiting their semantic navigation capabilities. This solution addresses this issue through the following techniques: When constructing the implicit network model, its input explicitly includes "linguistic features" obtained from preprocessing natural language command data. These linguistic features are used as conditional variables, inputting them along with spatial coordinates and viewpoint direction to predict the semantic feature vectors for the corresponding spatial coordinates. The generated 3D implicit placeholder map establishes a direct alignment with the natural language commands in the feature space, giving spatial locations on the map queryable semantic attributes. The robot can then directly map received natural language commands (such as "go to the bedroom") to the corresponding semantic regions on the map, achieving end-to-end conversion from high-level language understanding to low-level spatial cognition, thus endowing the robot with the ability to understand and execute semantic navigation commands.

[0036] It should be noted that the online addition and updating of the 3D implicit placeholder map and the construction of the LiDAR point cloud map are independent of each other.

[0037] In step 104, the query point is a discrete location sampled in the three-dimensional implicit occupancy map and whose occupancy probability is higher than a threshold. In step 106, the semantic features corresponding to the query point in the 3D implicit placeholder map are fused across modally with the geometric features retrieved from the LiDAR point cloud map through a sparse attention mechanism, including: Query points are sampled from a 3D implicit placeholder map, and corresponding semantic feature vectors are obtained through an implicit network model to form a query set; Independent of the construction of the query set, geometric feature vectors are extracted from the LiDAR point cloud map. The geometric feature vectors are obtained by performing super-point clustering and lightweight PointNet encoding on the LiDAR point cloud, forming a key-value set. A sparse mask is generated based on a dual sparsification strategy of geometric proximity and semantic saliency. Based on the sparse mask, spatially neighboring and semantically related geometric feature vectors are retrieved from the key value set for each query point, and sparse attention is calculated to obtain fused features.

[0038] Traditional cross-modal attention mechanisms require calculating the correlation between each element in the query set and all elements in the key-value set, with a computational complexity of O(|Q|·|K|), where |Q| is the number of query points and |K| is the number of geometric feature vectors. In real-world scenarios, LiDAR point cloud maps typically contain hundreds of thousands or even millions of points. Directly calculating global attention would incur enormous computational overhead, making it difficult to run in real-time on the limited onboard computing resources of bipedal robots. This solution addresses the problem through the following technical means: First, when constructing the query set, only "discrete locations with an occupancy probability higher than a threshold" are sampled from the 3D implicit occupancy map as query points, rather than densely sampling the entire space, effectively reducing the query set size |Q|. Second, when constructing the key-value set, "super-point clustering" is performed on the laser point cloud to compress the original massive point cloud into a significantly reduced number of superpoints, and the geometric feature vector of each superpoint is extracted through lightweight PointNet encoding, effectively reducing the key-value set size |K|. Most importantly, a "dual sparsity strategy based on geometric proximity and semantic saliency is used to generate a sparse mask," pre-selecting a candidate key subset much smaller than |K| for each query point, and then performing precise attention calculation only within this subset. Through triple optimization of query point filtering, key-value set clustering compression, and dual sparsity strategies, the computational complexity of attention is significantly reduced from a global O(|Q|·|K|) to O(|Q|·c), where c is the size of the sparse mask and c << |K|. This enables the originally computationally intensive cross-modal fusion process to be executed in real time on the onboard processor of the bipedal robot, ensuring the system's high responsiveness and deployability.

[0039] When fusing semantic and geometric features, not all geometric features contribute to the current semantic query. For example, when the query semantics are "cup," the geometric features of "walls" or "floors" in space are irrelevant information. Indiscriminately including them in the fusion not only wastes computational resources but may also introduce noise interference, reducing the fusion quality. This solution addresses this problem through the following techniques: When constructing the key-value set, a "super-point clustering" operation is used to abstract the original laser point cloud. Super-point clustering aggregates spatially adjacent points with similar geometric features into a unit (superpoint) with macroscopic geometric meaning. This process effectively filters out outlier noise points and redundant details in the original point cloud while preserving the main geometric structure information of the environment. Then, each superpoint is encoded using a lightweight PointNet to extract robust local geometric feature vectors. Each feature vector in the key-value set represents a stable geometric attribute of a local region, rather than the accidental features of a single noise point, thus enhancing the robustness of the feature representation. Subsequent cross-modal attention computation on this high-quality key-value set can more accurately capture the correspondence between semantic query points and related geometric structures, avoid the contamination of fusion results by noise points, and improve the accuracy and reliability of fusion features.

[0040] Simple "fully connected" feature fusion cannot reflect the differentiated needs of different semantic queries for different geometric regions. For example, the semantic query "door handle" should focus more on the local point cloud near the door handle, while the semantic query "corridor" may need to focus on the geometric information of the walls over a longer distance. This solution solves this problem through the following technical means: It prepares the query set and key-value set separately in a "query set-independent" manner, maintaining the independence of the two modal features; based on this, it introduces a sparse attention mechanism. The core of this mechanism is to "retrieve" the most relevant geometric feature vector in the key-value set according to the specific semantic features of each query point and assign different attention weights. Each semantic query point can adaptively and selectively aggregate spatially adjacent and semantically relevant geometric context information from the LiDAR point cloud map, rather than passively receiving fixed, undifferentiated geometric features. This gives the fusion process "content awareness": different query points (such as "door handle" and "corridor") will retrieve completely different subsets of geometric features and form customized fusion features, thus more accurately serving subsequent semantic understanding and navigation decisions.

[0041] Among them, the dual sparsification strategy includes: For each query point, find the nearest neighbor points in the LiDAR point cloud map based on its three-dimensional coordinates to obtain the geometric proximity mask; Calculate the relevance score between the semantic features of the query point and all key-value features, retain the key-value pairs with the highest scores, and obtain the semantic saliency mask; The union of the geometric proximity mask and the semantic saliency mask is taken as the sparse mask.

[0042] In cross-modal fusion, semantically related objects typically exhibit physical proximity. For example, if a query point is located near a "table," its associated geometric features are highly likely to originate from the point cloud of the tabletop rather than the point cloud of a distant wall. Based on this prior knowledge, this scheme introduces a technique of "finding the nearest neighbor points in the LiDAR point cloud map based on its 3D coordinates to obtain a geometric proximity mask." This drastically reduces the scope of attention computation from the entire key-value set (of size |K|) to a very small local spatial neighborhood (of size k, typically k is 32 or 64, much smaller than |K|). This operation reduces the size of the search space by several orders of magnitude, fundamentally guaranteeing the improvement in computational efficiency.

[0043] Geometric proximity filtering, which relies purely on spatial distance, has limitations. In some cases, semantically relevant geometric features may not be absolute spatial nearest neighbors. For example, when the query semantics are "door handle," its nearest point cloud might be from the "door panel," while the point cloud of the "door frame," which is slightly further away, is equally crucial for understanding the structure and location of the "door handle." Furthermore, due to occlusion or sparse point clouds, truly relevant geometric features may rank lower in spatial distance. This solution effectively overcomes this deficiency by introducing a technique that "calculates the relevance scores between the semantic features of the query point and all key-value features, retains the top-scoring key-value pairs, and obtains a semantic saliency mask." Building upon geometric proximity filtering, a "soft filter" based on semantic relevance is added. Even if a geometric feature is not absolutely nearest in space, it can be included in the candidate set as long as it is highly semantically relevant to the query point. This ensures that crucial, semantically relevant geometric context is not overlooked, thereby improving the accuracy and recall of cross-modal fusion.

[0044] Geometric proximity masks ensure that key local geometric contexts (such as supporting planes and object boundaries) are captured; semantic saliency masks ensure that potential matches with cross-modal semantic relevance (such as spatially distant related structures due to occlusion) are not missed. This scheme organically combines the advantages of both by using the union of geometric proximity masks and semantic saliency masks as sparse masks. The final sparse mask includes spatially nearest geometric features (ensuring local geometric accuracy) and also supplements semantically highly relevant but spatially non-nearest neighbor features (ensuring semantic association integrity). This "double insurance" strategy ensures that all information valuable for semantic understanding and geometric localization around each query point is included in subsequent accurate calculations, achieving maximum information aggregation with minimal computational overhead and reaching the optimal balance between computational efficiency and fusion quality.

[0045] Furthermore, in step 106, sparse attention calculation is performed to obtain fused features, including: For each query point, attention is computed only within the range of the geometric feature vectors determined by the sparse mask; Calculate the degree of matching between the query point and each geometric feature vector within the sparse mask; The matching degree is normalized and converted into probability weights; Based on probability weights, the geometric feature vectors within the sparse mask are weighted and summed to obtain the fusion feature of the query point. The fusion feature contains the semantic information of the query point and the spatially adjacent and semantically related geometric information.

[0046] After determining the sparse mask (i.e., a subset of candidate geometric features), the key is to efficiently aggregate information within this subset. This scheme clarifies the specific process of sparse attention computation: First, attention computation is performed "only within the range of geometric feature vectors determined by the sparse mask," rather than global computation. This precisely allocates valuable computational resources to the most valuable feature interactions, avoiding invalid computations on irrelevant features and achieving efficient allocation of computational resources.

[0047] Different geometric features contribute differently to the same semantic query. For example, for the query "bottom of the cup," the point cloud features of the table directly below it should have higher reference value than the point cloud features of the table legs on the side. This solution achieves adaptive weighting of candidate geometric features through a series of techniques, including "calculating the matching degree," "normalizing to probability weights," and "weighted summation." The attention mechanism automatically learns and assigns higher probability weights to geometric features that are more semantically similar to the query and have a closer spatial relationship, while assigning lower weights to geometric features with lower relevance. The final fused feature obtained through weighted summation is a "soft" aggregation of all relevant geometric information within the sparse mask, rather than a simple averaging or splicing. This allows the fused feature to highlight the most important geometric context, suppress secondary or slightly irrelevant information, and generate a content-adaptive feature representation that is optimal for the current query.

[0048] The final fused feature "contains semantic information of the query point and spatially adjacent and semantically related geometric information." This fused feature is not an abstract intermediate variable, but a highly condensed feature vector that deeply fuses information from two modalities. On one hand, it inherits the semantic feature vector from the implicit network model, preserving high-level semantic attributes such as object category and function; on the other hand, it aggregates precise geometric information (such as local shape, normal vectors, and boundaries) from the LiDAR point cloud map through an attention mechanism. Therefore, this fused feature becomes a complete representation that simultaneously encodes "what it is" (semantics) and "where / what shape it is" (geometry). When this feature is input into the subsequent decoder or planning module, downstream tasks no longer need to query the semantic map and geometric map separately; they can directly obtain the fused comprehensive information, greatly simplifying the complexity of subsequent processing and improving the overall system efficiency and performance.

[0049] In step 102, the multi-source data stream may further include image data. Updating the parameters of the implicit network model using online incremental learning includes: When a new image arrives in the image data, the current 3D implicit placeholder map generates a rendered image from the robot's current perspective. The photometric loss and semantic loss between the rendered image and the monitored image are calculated. The weight parameters and bias parameters inside the implicit network model are updated asynchronously using the gradient descent method to update the 3D implicit placeholder map expressed by the implicit network model.

[0050] Traditional implicit map construction methods often employ offline training, where the network is trained using all collected data at once, and the map remains fixed after training. This approach cannot adapt to new objects or scene changes in the environment (such as furniture moving or doors opening). This solution addresses this problem by employing an "online incremental learning method to update the parameters of the implicit network model." Specifically, whenever a new image arrives in the image data, the system triggers a learning process: rendering an image from a new perspective based on the current map, calculating the photometric and semantic losses between the rendered image and the actual monitored image, and then updating the network parameters using gradient descent. The 3D implicit placeholder map is no longer static but continuously absorbs new observation information during robot exploration, dynamically optimizing and refining its representation of the environment. When the robot enters a previously unseen area or the environment changes, the online learning mechanism can instantly integrate this new information into the map, ensuring the map remains consistent with the real environment and giving the system continuous adaptability to dynamic environments.

[0051] If all historical image data is used for each update (i.e., full learning), the computational load and memory consumption will increase indefinitely as the exploration range expands, making it difficult to run on airborne equipment for extended periods. If only the latest frame data is used each time (i.e., fully online learning), it may lead to "catastrophic forgetting" of earlier observations, compromising the global consistency of the map. This scheme implicitly employs incremental learning techniques, such as maintaining a keyframe queue (as described in the specific implementation). Each parameter update only needs to be based on the loss between the current rendered image and the real image. The computation graph only involves the network activation region near the current viewpoint, resulting in a relatively constant computational load that does not increase linearly with the amount of historical data. Simultaneously, the network parameters themselves are preserved and continuously refined as a compressed representation of all historical observations, avoiding the overhead of explicitly storing all historical data. This design effectively utilizes historical information to maintain global map consistency while keeping the computational overhead of a single update within an acceptable range, achieving a good balance between map quality and computational efficiency.

[0052] Online training of implicit network models involves gradient backpropagation and parameter updates, which are computationally intensive operations. If this process is executed synchronously with laser inertial odometry or path planning modules that have extremely high real-time requirements, it will compete for valuable computing resources, leading to overall system response delays. This solution explicitly adopts the technique of "asynchronously updating the weight and bias parameters inside the implicit network model." The parameter update of the implicit network model runs as an independent background task, without blocking or waiting for other modules with higher real-time requirements (such as LiDAR point cloud processing and state estimation). This ensures that the robot's core localization and mapping functions are not affected by the semantic learning process, maintains the smoothness and high responsiveness of the overall system operation, and achieves harmonious coexistence of computationally intensive tasks and real-time critical tasks on the same airborne platform.

[0053] Training an implicit map using only photometric loss (comparing pixel color differences) may result in a network learning representations that are only visually realistic but semantically incorrect (e.g., incorrectly reconstructing a shadow on a white wall as a geometric indentation). This approach employs both photometric and semantic losses when calculating the loss. The semantic loss is calculated by comparing the rendered semantic feature map with a semantic feature map extracted from a real image (e.g., using a pre-trained semantic segmentation model). In addition to supervision of color appearance, higher-level semantic information is added as supervision. This guides the implicit network model to learn not only the appearance of the scene but also its semantic structure and object category information. The resulting 3D implicit placeholder map has internal feature representations that are more aligned with real-world semantic concepts, more accurate semantic label predictions, and clearer boundaries, providing a higher-quality map foundation for subsequent semantic-based navigation and interaction tasks.

[0054] In step 102, the multi-source data stream also includes inertial measurement data; Before obtaining the LiDAR point cloud map, the following steps are required: Motion distortion removal is performed on the laser point cloud using inertial measurement data, robot pose is estimated using laser inertial odometry, and a laser radar point cloud map is constructed and output.

[0055] Acquiring a complete point cloud frame using LiDAR through rotation or scanning takes a certain amount of time (e.g., 100 milliseconds). During this time, if the robot is walking, its position and posture continuously change, resulting in different LiDAR points within the same frame being acquired under different robot poses. Directly using these points to construct a map will produce severe motion distortion, manifested as point cloud warping and ghosting, leading to a decrease in the accuracy of subsequent geometric feature extraction and localization. The periodic swaying of bipedal robots during walking exacerbates this problem. This solution addresses this issue through the following technical means: before acquiring the LiDAR point cloud map, "motion distortion removal is performed on the LiDAR point cloud using inertial measurement data." The inertial measurement unit (IMU) can measure the robot's angular velocity and acceleration at high frequencies (e.g., 400Hz), and through integration, the minute motion at each moment within the LiDAR scanning cycle can be accurately estimated. Using these high-frequency pose estimates, each LiDAR point within a frame can be compensated to a unified coordinate system based on a reference time (e.g., the frame start time). This effectively eliminates point cloud distortion caused by the robot's own movement, making the processed point cloud appear as if it were collected while the robot was stationary, thus accurately reflecting the geometric structure of the environment. This lays a solid foundation for subsequent construction of high-precision LiDAR point cloud maps and accurate geometric feature extraction.

[0056] Relying solely on laser point clouds for pose estimation (such as the ICP algorithm) is prone to failure in geometrically degraded scenarios (such as long corridors or open areas) or during rapid movement; relying solely on IMU for pose estimation suffers from rapid drift over time. This solution employs a technique of "estimating robot pose using laser inertial odometry to construct and output a laser radar point cloud map," tightly coupling and fusing laser radar point cloud data with inertial measurement data. The laser inertial odometry utilizes laser point cloud registration to provide absolute pose constraints, eliminating the cumulative drift of the IMU, while simultaneously using the high-frequency motion information of the IMU to provide good initial values ​​for laser point cloud registration and compensate for the limitations of laser in rapid movement or geometrically degraded conditions. The complementary advantages of the two sensors achieve more robust and accurate real-time robot pose estimation than a single sensor. The laser radar point cloud map constructed based on this accurate pose exhibits higher global consistency and local accuracy, maintaining stable and reliable map construction quality even when the bipedal robot experiences swaying during walking or traverses areas with sparse environmental features.

[0057] In the sparse attention calculation, the nearest neighbor search radius is dynamically adjusted based on the shaking amplitude estimated from the inertial measurement data; for image data frames that degrade due to jitter, the weight of the image data frames in the online training of the implicit network model is reduced.

[0058] During the walking cycle of a bipedal robot, the sway amplitude changes dynamically. During the single-leg support phase, the robot is relatively stable with minimal sway; however, during the double-leg support phase and at the moment the heel touches the ground, the impact is severe, and the sway amplitude increases significantly. If the spatial nearest neighbor search radius remains fixed, sensor observations become highly uncertain when swaying is severe. The query point and geometric feature point corresponding to the original location may exceed the fixed search radius due to instantaneous deviations in pose estimation, leading to the omission of correct matching pairs and failure of cross-modal fusion. This solution addresses this problem by dynamically adjusting the nearest neighbor search radius based on the sway amplitude estimated from inertial measurement data. The system can perceive the robot's motion state in real time based on IMU data. When the sway amplitude is small, a smaller search radius is used to ensure computational efficiency; when the sway amplitude exceeds a threshold, the nearest neighbor search radius is automatically and proportionally expanded. This adaptive adjustment strategy increases the "fault tolerance" of feature matching, ensuring that the correct geometric context is included in the candidate set even in non-stationary states, thereby guaranteeing the stability and robustness of cross-modal attention fusion throughout the robot's entire motion cycle.

[0059] When bipedal robots walk, the impact of their feet hitting the ground causes severe camera shake, resulting in serious motion blur or rolling shutter effects in the captured images. The visual features contained in these degraded image frames are unreliable. If they are used with normal weights in the online incremental training of the implicit network model, noise and errors will be introduced into the 3D implicit placeholder map, leading to degraded map quality, blurred semantic boundaries, and even global divergence. This solution addresses this problem by reducing the weight of image data frames degraded due to shaking during online training of the implicit network model. The system can automatically identify and "penalize" low-quality data frames. By reducing their training weights (even to zero in extreme degradation cases), the system effectively suppresses the impact of noisy data on network parameter updates, ensuring that only high-quality, reliable visual observations make a major contribution to map construction. This mechanism acts like an intelligent "data filter," protecting the implicit map learning process from instantaneous motion noise and guaranteeing the quality and stability of the final generated semantic map.

[0060] The process of estimating the robot's pose using laser inertial odometry includes: registering the current frame point cloud with the historical frame point cloud, and estimating the robot's relative displacement and rotation by combining inertial measurement data; and constructing a lidar point cloud map includes: transforming the current frame point cloud to the global coordinate system based on the estimated pose, stitching it together with the historical frame point cloud, and outputting the lidar point cloud map at the current moment and the local geometric features of each point.

[0061] For pose estimation, this solution clarifies that laser observation constraints are obtained by "registering the current frame point cloud with historical frame point clouds," while kinematic constraints are obtained by "combining inertial measurement data." These two methods are then integrated to "estimate the robot's relative displacement and rotation." For map construction, the solution clarifies the steps of "transforming the current frame point cloud to the global coordinate system based on the estimated pose and stitching it with historical frame point clouds." This provides a clear implementation path for those skilled in the field, ensuring the reproducibility of the technical solution. By combining laser point cloud registration (providing a low-frequency absolute pose reference) with IMU integration (providing high-frequency relative motion estimation), high-precision, high-frequency pose output is achieved. Furthermore, by continuously stitching together the distortion-reduced and registered point clouds, incremental, globally consistent map construction is achieved.

[0062] This further specifies that while constructing the LiDAR point cloud map, the output should include "local geometric features of each point." These local geometric features (such as normal vectors, curvature, and local point density) describe the geometric shape attributes of the local neighborhood of each point in the point cloud. The output of the LiDAR point cloud map is no longer just a raw set of 3D coordinates, but also includes feature vectors describing the local geometric structure. This provides a direct input or a good initial feature base for the step of "extracting geometric feature vectors from the LiDAR point cloud map." By preprocessing, extracting, and storing these local geometric features, time-consuming local geometric calculations (such as neighborhood search and eigenvalue decomposition) can be avoided during cross-modal fusion module operation, thereby further accelerating the entire system's processing flow and improving real-time performance.

[0063] Furthermore, the periodic gait phase of the robot's bipedal motion is extracted based on inertial measurement data, and the gait phase is encoded into a motion embedding vector; The motion embedding vector and the view direction are input into the implicit network model, enabling the implicit network model to learn the view-appearance mapping relationship under different time phases. During the inference phase, the current gait phase is used to predict the viewpoint direction at future moments, and images of the future viewpoint are rendered in advance for path planning.

[0064] This solution passively suppresses motion noise by reducing the weight of degraded frames, a "stopgap" strategy, while proposing a proactive, "root cause" approach. Bipedal walking exhibits significant periodicity; different gait phases (e.g., left foot support, right foot swing, both feet support) correspond to different torso postures and movement patterns, thus affecting the camera's perspective. The solution involves "extracting the periodic gait phases of the robot's bipedal movement based on the inertial measurement data, encoding these gait phases into motion embedding vectors," and inputting these vectors "along with the perspective direction" into the implicit network model. During training, the implicit network model not only learns "how the scene looks from a certain perspective" but also "how the scene appearance should be adjusted for a certain perspective at a certain gait phase." This allows the network to proactively understand and compensate for periodic perspective disturbances caused by gait. For example, the neural network can learn to automatically perform some form of correction or feature adjustment on the rendered image during the "heel strike" (maximum impact) phase. This proactive modeling mechanism fundamentally improves the implicit map's adaptability to bipedal movement characteristics, making it more advanced and effective than simple passive noise reduction strategies.

[0065] Traditional path planning relies solely on the currently constructed map, lacking the ability to predict future perception outcomes. For dynamically walking bipedal robots, the drivability of the space seen from the current perspective may change after taking a step due to a change in perspective. This solution endows the system with a kind of "mental imagination" capability by using the technique of "predicting the future perspective direction using the current gait phase during the inference phase and pre-rendering the image of the future perspective for path planning." Based on the current gait phase and movement trend, the robot can predict the camera perspective one or several gait cycles later and use a trained implicit network model to pre-render the environmental image that will be seen from that future perspective. This allows the path planning module not only to assess the drivability of the current location but also to pre-assess "what I will see if I take another step forward, whether I will encounter blind spots or collision risks." This forward-looking information greatly improves the safety and efficiency of path planning, enabling the robot to make more intelligent and predictive movement decisions.

[0066] Furthermore, when new information arrives in the 3D implicit occupancy map or the LiDAR point cloud map, a fusion update is initiated, using the new information to update the affected local area without waiting for synchronization data from other branch maps. The fusion features are used to construct and maintain an environmental map for the robot, including: inputting the fusion features into a decoder network to predict enhanced occupancy probabilities, refined semantic labels, and passage costs. The environmental map uses a sparse voxel hash table as a hybrid representation, with each voxel storing: the fused occupancy probability, refined semantic features, and a local geometric descriptor extracted from the LiDAR point cloud map, wherein the local geometric descriptor includes a normal vector and curvature.

[0067] In systems with multiple parallel processing branches, requiring all branch data to be aligned before fusion introduces latency. This solution constructs an event-driven asynchronous update mechanism by immediately initiating fusion updates when new information arrives in the 3D implicit placeholder map or the LiDAR point cloud map, using the new information to update the affected local areas without waiting for synchronization data from other branch maps. Whenever any branch generates new information (such as a new area added to the LiDAR point cloud map, or the implicit map updating the semantics of an object through online learning), the fusion module responds immediately, updating only the local voxels in the unified map affected by the new information. This mechanism minimizes the latency between information generation and utilization, ensuring the unified map reflects the latest environmental state as quickly as possible, improving the overall system real-time performance and response speed. Simultaneously, the local update strategy avoids full map reconstruction, saving significant computational resources.

[0068] The aforementioned fused features are high-dimensional feature vectors, which cannot be directly used by traditional navigation modules such as localization and path planning. This solution builds a bridge from the "implicit feature space" to the "explicit task space" by inputting the fused features into a decoder network to predict and enhance occupancy probability, refine semantic labels, and assess passage costs. It is not a simple binary judgment of geometric "empty / occupied," but a more intelligent occupancy estimate that incorporates semantic information (e.g., "This is a table; the space beneath it is geometrically empty but impassable"), providing safer spatial constraints for path planning. After incorporating geometric boundary information, the accuracy of semantic labels is improved, providing more reliable target guidance for semantic-level navigation tasks. It directly provides the planner with passage cost values ​​for each area (e.g., low cost on flat ground, high cost on stairs), enabling the path planning algorithm to use it directly and simplifying the design of the navigation module. Furthermore, if a dense 3D mesh is used to store a unified map, memory consumption will increase cubically with the scene volume, making it unsuitable for large-scale mapping. This solution addresses this problem by using a sparse voxel hash table as a hybrid representation for the environment map. The sparsity property means that storage space is allocated only to spatial regions actually occupied by objects or with semantic information, while empty areas do not occupy memory, thus decoupling map size from environment scale and supporting map construction for large-scale or even borderless scenes. The hash table structure provides fast random access capability with an average time complexity of O(1), allowing the robot to instantly query all attributes (occupancy probability, semantic features, geometric descriptors) of a location based on its 3D coordinates. Hybrid representation means that each voxel simultaneously encapsulates "semantic occupancy" (occupancy probability, semantic features) and "geometric structure" (normal vector, curvature) information, forming a highly integrated environmental information unit, providing a "one-stop" data service for the robot's real-time decision-making.

[0069] See Figure 2 and Figure 3 The overall architecture for asynchronous robot map creation provided in this embodiment of the invention mainly includes: an input layer, an implicit map branch, a geometric map branch, a sparse cross-modal attention fusion module, a unified semantic-geometric map generation module, and an output layer.

[0070] Input layer: Receives multi-source data streams asynchronously. These streams include at least LiDAR point cloud data and natural language command data, and may also include image data and inertial measurement data. For example, LiDAR point cloud data is acquired by a LiDAR sensor, natural language command data is input by the user (e.g., "Go to the sofa in the living room and get the remote control"), image data is acquired by a monocular or RGB-D camera, and inertial measurement data is acquired by an IMU.

[0071] Parallel processing branches: Implicit map branch (3D implicit placeholder map branch) and geometric map branch (LiDAR point cloud map branch).

[0072] Implicit Map Branch: This branch is responsible for constructing a semantically rich 3D implicit placeholder map. It includes preprocessing of natural language instruction data to obtain language features, an implicit network model, and an online incremental learning update module.

[0073] Geometric Map Branch: This branch is responsible for building high-precision laser point cloud maps. It includes preprocessing of laser point cloud data, a laser inertial odometry module, and a local laser point cloud map construction module.

[0074] The core fusion module, namely the sparse cross-modal attention fusion module, receives semantic features from the implicit map branch and geometric features from the geometric map branch, and performs cross-modal fusion based on the sparse attention mechanism to obtain fused features.

[0075] Output layer: Constructs and maintains an environmental map for the robot using fused features, which is then used by downstream tasks such as localization, path planning, and motion control.

[0076] The input layer involves asynchronous multi-source data preprocessing: the system receives data from different sensors asynchronously. The multi-source data streams include at least laser point cloud data and natural language command data. For each arriving frame, each data stream is preprocessed to obtain the corresponding laser point cloud map and language features.

[0077] Laser point cloud data preprocessing: The original laser point cloud is downsampled and denoised to reduce the data volume and filter out outliers. Downsampling can be achieved using a voxel grid filter, and denoising can be achieved using statistical filtering or radius filtering. Optionally, local geometric features such as the normal vector of each point are calculated.

[0078] Natural language instruction data preprocessing: The natural language instruction data is encoded into fixed-dimensional semantic vectors, i.e., language features, using a text encoder (e.g., the BERT model). These language features capture key semantic information in the instructions, such as target objects, actions, and spatial relationships.

[0079] Image data preprocessing: When the multi-source data stream also includes image data, the images are subjected to distortion correction and illumination normalization. Optionally, a pre-trained visual encoder (e.g., a ResNet network) is used to extract multi-scale visual features.

[0080] Inertial measurement data (IMU) preprocessing: When the multi-source data stream also includes inertial measurement data, the high-frequency acceleration and angular velocity measurements are zero-biased to eliminate sensor system errors and are used for subsequent motion compensation and pose estimation.

[0081] Construction of a 3D implicit placeholder map: The implicit network model can be implemented using a multilayer perceptron. Its input is a vector composed of three concatenated parts: The spatial coordinates of the sampled points are x = (x, y, z): These spatial coordinates are the positions of the three-dimensional sampling points obtained by sampling along the observation ray during ray propagation rendering. Specifically, based on the robot's current preliminary pose (provided by the laser inertial odometry) and the known intrinsic parameters of the camera, an observation ray is emitted from the camera's optical center, passing through a certain pixel on the image, and sampling is performed along this observation ray with a step size δ to obtain a series of three-dimensional sampling point positions.

[0082] The viewpoint direction d=(dx, dy, dz) derived from the robot's current pose is the direction vector of the observed ray, which is derived from the robot's current pose.

[0083] Language feature f_lang: A fixed-dimensional semantic vector obtained after preprocessing natural language instruction data.

[0084] The implicit network model outputs a triple corresponding to the spatial coordinates: color information c, volume density σ, and semantic feature vector s.

[0085] Wherein, the volume density σ is used to derive the occupancy probability O(x) at spatial coordinate x using the formula O(x)=1-exp(-σ·δ), and δ is the sampling step size.

[0086] Through the above process, the implicit network model establishes a mapping from arbitrary spatial coordinates to occupancy probabilities and semantic feature vectors, forming a continuous, differentiable three-dimensional implicit occupancy map.

[0087] When the multi-source data stream also includes image data, the parameters of the implicit network model are updated using online incremental learning.

[0088] The system maintains a global implicit network model parameter set (including weight parameters and bias parameters) and a keyframe queue.

[0089] When a new image arrives in the image data, the following online incremental learning operation is performed: Rendering: Using the current implicit network model parameters, a rendered image is generated from the robot's current perspective. Specifically, based on the robot's current pose and camera intrinsic parameters, an observation ray is emitted that passes through the image pixels. Spatial coordinates are sampled on the observation ray, color information is queried through the implicit network model, and a rendered image is synthesized.

[0090] Loss Calculation: Calculate the photometric loss and semantic loss between the rendered image and the monitored image (i.e., the actually acquired image). The photometric loss can be the mean squared error loss or L1 loss; the semantic loss can be the cross-entropy loss between the rendered semantic feature map and the prediction result of the pre-trained semantic segmentation model.

[0091] Parameter Update: Based on photometric and semantic losses, gradient descent is used to asynchronously update the weight and bias parameters within the implicit network model to update the 3D implicit occupancy map represented by the implicit network model. This update process is decoupled from the laser point cloud data processing cycle and executed independently.

[0092] Through this online incremental learning mechanism, the 3D implicit occupancy map is continuously optimized during the robot's exploration process, gradually absorbing and integrating the visual appearance of the scene and semantic information related to task instructions.

[0093] Cross-modal fusion of semantic features from 3D implicit placeholder maps and geometric features from laser point cloud maps. Figure 3 This process is illustrated schematically.

[0094] First, feature preparation is performed.

[0095] Query set construction: Query points are discrete locations sampled from a 3D implicit occupancy map that have an occupancy probability higher than a threshold. Query points are sampled from the 3D implicit occupancy map, and their corresponding semantic feature vectors are obtained through an implicit network model to form the query set Q. Sampling can be performed along the observation ray or uniformly sampled in areas with high occupancy probability. The occupancy probability threshold can be set, for example, to 0.5.

[0096] Key-value set construction: Independent of query set construction, geometric feature vectors are extracted from the laser point cloud map. Specifically, super-point clustering is performed on the laser point cloud (e.g., based on voxel mesh downsampling or region growing algorithms), and then a local geometric feature vector is extracted for each super-point using lightweight PointNet encoding. The geometric feature vectors of all super-points constitute the key-value set K.

[0097] Then, sparse mask generation is performed.

[0098] A sparse mask is generated based on a dual sparsification strategy of geometric proximity and semantic saliency. For each query point in the query set Q, the following operations are performed: Obtaining the geometric proximity mask: For each query point, find the nearest neighbor points in the laser point cloud map based on its 3D coordinates to obtain the geometric proximity mask M. geo Specifically, the k nearest neighbors can be quickly found using spatial indexing data structures such as KD-Tree. The value of k can be set according to the point cloud density and empirical values, such as k=32 or k=64.

[0099] Obtaining the semantic saliency mask: Calculate the relevance score between the semantic features of the query point and all key-value features, retain the key-value pairs with the highest scores, and obtain the semantic saliency mask M. sem Relevance scores can be calculated using methods such as dot product, bilinear, or additive attention. For example, W q and W k W is a learnable projection matrix. q ·s i The semantic feature vector s represents the query point. i Perform a linear transformation, W k ·g j The geometric eigenvector g represents the key point. j Perform a linear transformation.

[0100] The m highest-scoring key values ​​are retained. The value of m is usually much smaller than |K|, for example, m=16 or m=32.

[0101] Generation of sparse masks: The sparse mask M is the union of the geometric proximity mask and the semantic saliency mask, i.e., M = M geo ∪M sem This ensures that only geometric feature vectors that are spatially adjacent, or semantically highly related but not the closest, are included in subsequent precise calculations.

[0102] Next, sparse attention calculations are performed.

[0103] Based on a sparse mask, spatially neighboring and semantically related geometric feature vectors are retrieved from the key set for each query point, and sparse attention is calculated to obtain fused features. Specifically, this includes: For each query point, attention is computed only within the range of the geometric feature vectors determined by the sparse mask.

[0104] Calculate the degree of match between the query point and each geometric feature vector within the sparse mask. The degree of match can be calculated using scaled dot product attention: Where d is the dimension of the feature vector, W' q and W' k s is a learnable projection matrix. i It is query point q i The corresponding semantic feature vector (dimension d).

[0105] The matching degree is normalized and converted into probability weights. Normalization can be achieved using the softmax function.

[0106] Based on probability weights, the geometric feature vectors within the sparse mask are weighted and summed to obtain the fused features of the query point. The fused features contain the semantic information of the query point and the spatially adjacent and semantically related geometric information.

[0107] It should be noted that the generation of key sets is not limited to extraction from superpoints in local laser point cloud maps. Another approach is to directly sample a certain number of points from the keyframe laser point cloud as keys, or to establish a geometric correspondence between laser scan lines and visual image pixels, and then use the visual features of the corresponding pixels as keys.

[0108] In addition to PointNet, other point cloud feature extraction networks such as PointNet++, DGCNN, and KPConv can also be used to extract richer local geometric features.

[0109] Relevance score calculation can also use an additive attention mechanism: r ij =v T ·tanh(W q· s i +W k· g j ), where v is a learnable weight vector.

[0110] Finally, feature decoding is integrated with the construction of a unified semantic-geometric map.

[0111] The fused features are input into the decoder network to predict the enhanced occupancy probability, refined semantic labels, and passage cost. The decoder network can be a lightweight multilayer perceptron or convolutional network.

[0112] Occupancy probability integrates geometric structure information and semantic accessibility information. For example, the semantic information of a "table" will assign a higher occupancy probability to the geometrically empty area beneath it, preventing robots from crawling into it. The injection of geometric boundary information makes semantic labels sharper and more accurate at object edges. This directly provides a cost map for path planning, guiding the robot to choose smoother and safer paths.

[0113] Construction and maintenance of the unified semantic-geometry map (i.e., the environment map): The unified semantic-geometry map uses a sparse voxel hash table as a hybrid representation. Each voxel (e.g., a cube with a side length of 5 cm) stores: The occupancy probability after fusion; Refined semantic features; Local geometric descriptors extracted from laser point cloud maps, including normal vectors and curvature.

[0114] When the implicit network model parameters are updated (i.e., the 3D implicit placeholder map is updated) or a new local laser point cloud map is generated, the fusion module is immediately triggered, updating only the affected local regions in the unified semantic-geometric map (e.g., voxels within a certain radius around the robot's current position). This event-driven local update mechanism ensures low latency and high responsiveness of the system.

[0115] In addition to using sparse voxel hash tables, the unified semantic-geometric map can also employ octree maps or implicit representations based on neural radiation fields. Sparse voxel hash tables achieve a good balance between access speed and memory efficiency, making them particularly suitable for real-time mapping in large-scale scenes.

[0116] Before acquiring the laser point cloud map, motion distortion removal can be performed on the laser point cloud using inertial measurement data. Since it takes a certain amount of time (e.g., 100 milliseconds) for the laser sensor to acquire a frame of point cloud through rotation or scanning, the robot's movement during this period can cause point cloud distortion. Using high-frequency inertial measurement data (e.g., 400Hz), the precise sensor pose corresponding to the emission time of each laser point can be interpolated and calculated. This point can then be compensated to a unified frame coordinate system, thus achieving motion distortion removal.

[0117] Estimating robot pose using laser inertial odometry includes: registering the current frame point cloud with historical frame point clouds (e.g., using the ICP algorithm or its variants), and combining inertial measurement data to estimate the robot's relative displacement and rotation. Laser point cloud registration provides absolute pose constraints to eliminate cumulative drift from inertial measurements, while the high-frequency motion information from the inertial measurement data provides good initial values ​​for point cloud registration and compensates for the limitations of laser in rapid motion or geometric degradation.

[0118] This embodiment also describes adaptation measures for the dynamic instability of the robot during walking.

[0119] Dynamic adjustment of the nearest neighbor search radius: In sparse attention computation, the nearest neighbor search radius is dynamically adjusted based on the sway amplitude estimated from inertial measurement data. The sway amplitude can be estimated by calculating the variance of the angular velocity over a period of time. When the sway amplitude exceeds a preset threshold, the radius of the spatial nearest neighbor search is dynamically and proportionally increased (i.e., the k-value or search radius of the k-nearest neighbor search is increased). When the robot is stable, a smaller search radius is used to ensure computational efficiency; when the sway is severe, the search radius is increased to increase the fault tolerance of feature matching and ensure the robustness of fusion.

[0120] Elastic update strategy: For image data frames degraded due to jitter, the weight of these frames in the online training of the implicit network model is reduced. The system evaluates the blurriness of the current image frame (e.g., by analyzing the Laplacian variance of the image). For degraded image data frames with quality below a preset standard, a lower weight (e.g., 0.1) is assigned when calculating the loss, or even, in extreme cases, the parameter update is skipped. This effectively suppresses the contamination of the 3D implicit placeholder map by poor-quality data.

[0121] Periodic gait phases of the robot's bipedal motion are extracted from inertial measurement data and encoded into motion embedding vectors. When constructing the implicit network model, these motion embedding vectors and the view direction are input together, enabling the model to learn the view-appearance mapping relationship under different gait phases. During the inference phase, the view direction at future moments is predicted using the current gait phase, and images of the future view are rendered in advance for path planning. This allows the path planning module to pre-assess the drivability of future locations, improving the foresight and safety of motion decisions.

[0122] According to an embodiment of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the asynchronous robot map creation method described in any of the above technical solutions.

[0123] The asynchronous robot map creation method described in this invention is essentially a series of executable computational steps and algorithmic flows. These steps are solidified into computer program code and stored in various media. Users can fully reproduce the asynchronous robot map creation method of this invention on any compatible computing device by reading the medium and executing the program, without needing to understand complex algorithmic details or retrain the model. This solidifies the technical solution of this invention into a replicable, distributable, and deployable product form, facilitating industrial replication and large-scale deployment in industrial scenarios.

[0124] An electronic device according to an embodiment of the present invention includes a processor and a memory, wherein a computer program is stored in the memory, and when the processor executes the computer program, it implements the asynchronous robot map creation method as described in any of the above technical solutions.

[0125] In summary, this invention organically integrates online construction of a 3D implicit placeholder map, efficient cross-modal fusion based on a dual sparsity strategy, and specialized adaptation for robot dynamic characteristics through a complete asynchronous processing framework, ultimately generating a unified environment model with both accurate geometry and rich semantics. This solution effectively overcomes the shortcomings of existing technologies and significantly improves the feasibility and reliability of robots performing high-level semantic navigation tasks in complex real-world environments.

[0126] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of this application, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application and therefore will also fall within the protection scope of this application.

[0127] Those skilled in the art will understand that all or part of the processes in the method of the above-described embodiment can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0128] The technical solution of this application has been described above with reference to one embodiment shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.

Claims

1. A method for asynchronously building a robot map, characterized in that, include: The system receives multi-source data streams asynchronously. The multi-source data streams include at least lidar point cloud data and natural language command data. Each data stream is preprocessed to obtain the corresponding lidar point cloud map and language features. An implicit network model is constructed. The implicit network model takes the sampled spatial coordinates, the view direction inferred based on the robot's current pose, and the language features as inputs, and outputs the occupancy probability and semantic feature vector corresponding to the spatial coordinates to form a three-dimensional implicit occupancy map. The spatial coordinates are the positions of three-dimensional sampling points obtained by sampling along the observation ray during ray propagation rendering. The semantic features corresponding to the query point in the three-dimensional implicit placeholder map are fused with the geometric features retrieved through the sparse attention mechanism in the lidar point cloud map across modes to obtain the fused features. Using the fusion features, an environmental map for the robot is constructed and maintained.

2. The asynchronous robot map creation method according to claim 1, characterized in that, The query point is a discrete location sampled in the three-dimensional implicit occupancy map and whose occupancy probability is higher than a threshold. The step of fusing the semantic features corresponding to the query point in the 3D implicit placeholder map with the geometric features retrieved from the lidar point cloud map through a sparse attention mechanism across modalities includes: The query points are sampled from the three-dimensional implicit placeholder map, and the corresponding semantic feature vectors are obtained through the implicit network model to form a query set; Geometric feature vectors are extracted from the laser radar point cloud map. These geometric feature vectors are obtained by performing super-point clustering and lightweight PointNet encoding on the laser point cloud to form a key-value set. A sparse mask is generated based on a dual sparsification strategy of geometric proximity and semantic saliency. Based on the sparse mask, spatially neighboring and semantically related geometric feature vectors are retrieved from the key value set for each query point, and sparse attention is calculated to obtain the fused feature.

3. The asynchronous robot map creation method according to claim 2, characterized in that, The dual sparsity strategy includes: For each query point, the nearest neighbor points in the LiDAR point cloud map are found based on its three-dimensional coordinates to obtain the geometric proximity mask. Calculate the correlation score between the semantic features of the query point and all key-value features, retain the key-value pairs with the highest scores, and obtain the semantic saliency mask; The union of the geometric proximity mask and the semantic saliency mask is taken as the sparse mask.

4. The method according to claim 3, characterized in that, Sparse attention computation is performed to obtain fused features, including: For each query point, attention is computed only within the range of the geometric feature vectors determined by the sparse mask; Calculate the degree of matching between the query point and each geometric feature vector within the sparse mask; The matching degree is normalized and converted into probability weights; Based on the probability weights, the geometric feature vectors within the sparse mask are weighted and summed to obtain the fusion feature of the query point. The fusion feature includes the semantic information of the query point and the spatially adjacent and semantically related geometric information.

5. The asynchronous robot map creation method according to claim 1, characterized in that, The multi-source data stream also includes image data. Updating the parameters of the implicit network model using online incremental learning includes: When a new image arrives in the image data, the current 3D implicit placeholder map generates a rendered image from the robot's current perspective. The photometric loss and semantic loss between the rendered image and the monitored image are calculated. The weight parameters and bias parameters inside the implicit network model are updated asynchronously using the gradient descent method to update the 3D implicit placeholder map expressed by the implicit network model.

6. The asynchronous robot map creation method according to claim 2, characterized in that, The multi-source data stream also includes inertial measurement data; Before acquiring the lidar point cloud map, the following steps are included: The laser point cloud is processed using the inertial measurement data to remove motion distortion, the robot pose is estimated using laser inertial odometry, and the laser radar point cloud map is constructed and output.

7. The asynchronous robot map creation method according to claim 6, characterized in that, In the sparse attention calculation, the nearest neighbor search radius is dynamically adjusted based on the sway amplitude estimated from the inertial measurement data; For image data frames that degrade due to jitter, reduce the weight of the image data frames in the online training of the implicit network model.

8. The asynchronous robot map creation method according to claim 6, characterized in that, Estimating the robot's pose using laser inertial odometry includes: registering the current frame point cloud with the historical frame point cloud, and combining the inertial measurement data to estimate the robot's relative displacement and rotation. Constructing the lidar point cloud map includes: transforming the current frame point cloud to a global coordinate system based on the estimated robot pose, stitching it together with the historical frame point cloud, and outputting the lidar point cloud map at the current moment and the local geometric features of each point.

9. The asynchronous robot map creation method according to claim 6, characterized in that, Based on the inertial measurement data, the periodic gait phase of the robot's bipedal motion is extracted, and the gait phase is encoded into a motion embedding vector; The motion embedding vector and the view direction are input into the implicit network model, so that the implicit network model learns the view-appearance mapping relationship under different time phases; During the inference phase, the current gait phase is used to predict the viewpoint direction at future moments, and images of the future viewpoint are rendered in advance for path planning.

10. The asynchronous robot map creation method according to claim 1, characterized in that, When new information arrives in the 3D implicit occupancy map or the lidar point cloud map, a fusion update is initiated to update the affected local area using the new information, without waiting for synchronization data from other branch maps. Using the fused features, an environmental map for the robot is constructed and maintained, including: inputting the fused features into a decoder network to predict enhanced occupancy probabilities, refine semantic labels, and access costs. The environment map uses a sparse voxel hash table as a hybrid representation. Each voxel stores: the fused occupancy probability, the refined semantic features, and the local geometric descriptor extracted from the lidar point cloud map. The local geometric descriptor includes a normal vector and curvature.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the asynchronous robot map creation method according to any one of claims 1-10.

12. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the asynchronous robot map creation method according to any one of claims 1-10.

Images