A robot environment perception method and system, a robot, a medium, and a product

Abstract

The application discloses a robot environment perception method and system, a robot, a medium and a product. The method comprises the following steps: acquiring key frame pose data containing image data, scene structure data representing the association between an image frame and a target environment space, and semantic processing frames determined from the image frame; performing semantic perception processing on the semantic processing frames to obtain first semantic instance information, and / or matching and obtaining second semantic instance information from a semantic model; and performing fusion processing based on the obtained semantic instance information to generate or update semantic map data containing semantic road signs. The scheme decouples the geometric data acquisition and the semantic reasoning task in time sequence, so that the real-time positioning and tracking are not blocked by the semantic reasoning delay, the semantic reasoning can be triggered as needed, and the real-time positioning performance and the semantic map construction quality are considered under limited computing resources.

Description

Technical Field

[0001] This application relates to the field of mobile robot technology, and more particularly to robot environmental perception methods and systems, robots, media, and products. Background Technology

[0002] On resource-constrained embedded mobile platforms such as indoor companion robots, it is necessary to simultaneously achieve high-frequency real-time environmental geometric localization and high-quality semantic map construction to support the robot's autonomous navigation and intelligent interaction. Since embedded platforms have extremely limited GPU and CPU computing power and video memory resources, how to efficiently complete semantic understanding and map backfilling of indoor scenes while ensuring hard real-time geometric localization remains a persistent technical challenge in this field.

[0003] In related technologies, to achieve the above goals, a tightly coupled parallel processing architecture is typically adopted, where geometric odometry front-end tracking and deep learning semantic inference nodes run synchronously within the same task cycle. Such solutions can achieve relatively comprehensive environmental perception results under conditions of sufficient hardware resources. However, their inherent mechanism dictates that on embedded platforms, the two types of computational tasks directly compete for limited GPU / CPU and video memory bandwidth resources. Furthermore, the indiscriminate frame-by-frame semantic inference strategy generates a large amount of redundant computation, continuously consuming computing power from a fully observed conventional perspective, further exacerbating resource constraints. The problems of tightly coupled computation and dynamic processing lag ultimately mean that on typical edge devices such as NVIDIA, Jetson, Orin, and Nano, existing solutions cannot simultaneously achieve hard real-time geometric tracking accuracy and high-quality, detailed semantic mapping of the scene. Therefore, how to eliminate the temporal mutual exclusion conflict between real-time geometric tracking and high-load semantic inference under limited computing power conditions, and overcome the inherent defects of delayed dynamic object detection and semantic computation redundancy, has become a key technical problem urgently needing to be solved in this field. Summary of the Invention

[0004] The purpose of this application is to provide a technical solution for addressing the inherent limitations of delayed dynamic object detection and redundant semantic computation by eliminating the temporal mutual exclusion between real-time geometric tracking and high-load semantic reasoning under limited computing power. To solve the aforementioned technical problems, a robot environment perception method, system, robot, computer-readable storage medium, and computer program product are provided.

[0005] A first aspect of this application provides a robot environment perception method, comprising: acquiring keyframe pose data, scene structure data, and semantic processing frames; wherein the keyframe pose data includes image data collected by the robot for a target environment; the scene structure data includes image frame association data used to characterize the spatial association relationship between multiple image frames in the image data and the target environment; the semantic processing frame is an object to be processed determined from multiple image frames in the image data; performing semantic perception processing on the semantic processing frame to obtain corresponding first semantic instance information, and / or matching the semantic processing frame from a semantic model to obtain corresponding second semantic instance information; and performing fusion processing on semantic instances in the target environment based on the first semantic instance information and / or the second semantic instance information to generate or update semantic map data corresponding to the target environment, the semantic map data including semantic landmarks used to characterize objects in the target environment.

[0006] A second aspect of this application provides a robot environment perception system, comprising: a data acquisition module for acquiring keyframe pose data, scene structure data, and semantic processing frames; wherein the keyframe pose data includes image data collected by the robot for a target environment; the scene structure data includes image frame association data for characterizing the spatial relationship between multiple image frames in the image data and the target environment; the semantic processing frame is an object to be processed determined from multiple image frames in the image data; a semantic recognition module for performing semantic perception processing on the semantic processing frame to obtain corresponding first semantic instance information, and / or matching the semantic processing frame from a semantic model to obtain corresponding second semantic instance information; and a map fusion module for performing fusion processing on semantic instances in the target environment based on the first semantic instance information and / or the second semantic instance information to generate or update semantic map data corresponding to the target environment, wherein the semantic map data includes semantic landmarks for characterizing objects in the target environment.

[0007] A third aspect of this application provides a robot that includes the environmental perception system provided in the second aspect above.

[0008] A fourth aspect of the embodiments of this application provides a computer-readable storage medium for storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the method provided in any implementation of the first aspect described above.

[0009] A fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method provided in any implementation of the first aspect described above.

[0010] In real-time environmental perception of embedded mobile robots, geometric tracking and semantic reasoning are tightly coupled and run in parallel, competing for limited computing power, making it impossible to simultaneously guarantee positioning accuracy and high-quality semantic mapping. This application provides a robot environmental perception method, system, robot, computer-readable storage medium, and computer program product. By acquiring keyframe pose data, scene structure data, and semantic processing frames, and then performing semantic perception processing on the semantic processing frames and fusing them to generate semantic map data, geometric data acquisition and semantic reasoning are decoupled in time. Thus, the geometric tracking process is no longer delayed by semantic reasoning, ensuring the stability and accuracy of real-time positioning; while semantic reasoning can be performed independently using the acquired geometric prior data, without real-time constraints, enabling the construction of a high-quality semantic map. This effectively solves the problem of positioning frame loss and map drift caused by computing power contention on embedded platforms, achieving a good balance between real-time geometric positioning and high-precision semantic mapping. Attached Figure Description

[0011] The embodiments of this application are further described below with reference to the accompanying drawings and specific implementation details.

[0012] Figure 1 These are the implementation steps of a robot environment perception method provided in the embodiments of this application.

[0013] Figure 2 This is a specific implementation of the method for obtaining keyframe pose data provided in the embodiments of this application.

[0014] Figure 3 This is a specific implementation method for obtaining scene structure data in the embodiments of this application.

[0015] Figure 4 This is a specific implementation of the method for determining the dynamic unit provided in the embodiments of this application.

[0016] Figure 5 This is a specific structure of a robot environmental perception system provided in the embodiments of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the embodiments of this application.

[0018] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0019] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0020] In the field of robot environmental perception, a common technical solution to enable robots to locate themselves in real time and construct semantic maps for interaction and navigation is to tightly couple visual-inertial odometry with a deep learning-based semantic reasoning module. Specifically, this system simultaneously performs geometric feature tracking and frame-by-frame semantic segmentation within the same clock cycle, acquiring pose and object labels through parallel computation, while sharing image data input from the camera. Its design aims to achieve synchronous acquisition of geometric and semantic information to ensure temporal consistency of output.

[0021] However, when this technical solution is deployed on embedded platforms where computing power, power consumption, and storage space are strictly constrained, its performance is unsatisfactory. The fundamental contradiction of this solution lies in the fact that, in pursuit of hard real-time geometric tracking, the front-end visual odometry needs to operate stably at a frequency of no less than 30Hz. The parallel execution of this odometry and semantic inference leads to competition for GPU and CPU resources. Given limited computing power, ensuring stable pose output without frame drops necessitates compressing the computational overhead of semantic inference; conversely, running large-scale neural networks to obtain high-quality semantic maps would clog the geometric front-end processing pipeline, resulting in the loss of keyframe pose data. Specifically, when a service robot first enters an unfamiliar home environment for rapid exploration, the embedded processor is prone to computational overload due to the simultaneous processing of camera image features and semantic segmentation, causing visual odometry frame drops, which in turn leads to map drift and long-term accumulated errors.

[0022] In-depth analysis revealed that the root cause of the aforementioned contradiction lies in the forced temporal coupling between two tasks with drastically different real-time requirements: geometric data acquisition and semantic data parsing. Geometric tracking relies on feature extraction and pose calculation within tens of milliseconds to ensure accuracy, while semantic inference naturally allows for processing delays on the order of seconds; the two follow different time scales. The parallel architecture forces semantic inference to be completed within the hard real-time window of geometric tracking, causing the former to become the performance bottleneck of the latter. Furthermore, performing semantic inference frame by frame introduces a large amount of redundant computation. The viewpoint changes between adjacent keyframes are small, and there is a high degree of repetition in semantic information. These redundant calculations further exacerbate the computational strain.

[0023] To overcome the aforementioned contradictions, this application proposes a different technical approach. Instead of tightly coupling geometric mapping and semantic reasoning in parallel processing, it employs a two-scan delayed mapping strategy, decoupling the high-frame-rate geometric data acquisition stage from the semantic data parsing stage, which allows for non-real-time processing. During the first scan, all semantic reasoning nodes are shut down, concentrating all computing power to complete high-frequency pose estimation and scene geometry extraction, generating keyframe pose maps and room topology maps. Then, in a second mode—either online replay or offline batch processing—semantic reasoning is performed based on optimized precise poses and scene priors, either triggered on demand or with delayed backfilling, and the semantic results are injected into the 3D map. This architecture ensures that geometric tracking is no longer blocked by semantic reasoning delays, and semantic reasoning does not need to sacrifice model capacity or accuracy under hard real-time constraints, thus achieving stable 30Hz geometric tracking and high-quality semantic maps simultaneously on embedded platforms.

[0024] See Figure 1 One embodiment of this application provides a robot environment perception method. This method can be applied to a mobile robot platform equipped with a binocular global shutter camera, a six-axis microelectromechanical inertial measurement unit (MEMS), and an optional time-of-flight depth sensor. Its computational core is an embedded processing unit containing a central processing unit and a graphics processing unit, and it is equipped with a non-volatile external storage device based on the NVMe protocol. The method includes the following steps: First, keyframe pose data, scene structure data, and semantic processing frames are acquired. The keyframe pose data includes image data collected by the robot in relation to the target environment. The scene structure data includes image frame association data that characterizes the spatial relationship between multiple image frames in the image data and the target environment. The semantic processing frames are the objects to be processed determined from the multiple image frames in the image data. Secondly, semantic processing is performed on the semantic processing frame to obtain the corresponding first semantic instance information, and / or, the semantic processing frame is matched from the semantic model to obtain the corresponding second semantic instance information; Finally, based on the first semantic instance information and / or the second semantic instance information, the semantic instances in the target environment are fused to generate or update semantic map data corresponding to the target environment. The semantic map data includes semantic landmarks used to represent objects in the target environment. Specifically, keyframe pose data includes image data collected by the robot for the target environment; scene structure data includes image frame association data used to represent the spatial relationships between multiple image frames in the image data and the target environment; and semantic processing frames are the objects to be processed determined from multiple image frames in the image data.

[0025] In this embodiment, keyframe pose data generally refers to the pose information composed of image frames stored in time-stamp order after sliding window optimization from image data acquired by the robot in the target environment. This includes the position and orientation of each keyframe in the world coordinate system, as well as the relative constraints between keyframes. For example, it may include keyframe pose maps and common edges obtained through visual-inertial odometry and sliding window optimization (such as MSCKF); or any data that can describe the 6-DOF pose of the camera at the acquisition time and its uncertainties. As an example, keyframe pose data can be represented as: ,Include Precise pose of each keyframe And the co-visibility side parameter representing the i-th keyframe and the j-th keyframe. For example, this could include disabling the semantic reasoning node during the robot's initial exploration, running only visual-inertial odometry, plane extraction, and room segmentation, and storing keyframe images by timestamp; or any operational mode that delays or disables semantic reasoning to ensure hard real-time performance of geometric tracking. Scene structure data broadly refers to structured information extracted from image data that describes the geometric layout of the target environment, as well as image frame association data used to characterize the spatial relationships between multiple image frames and the target environment. For example, this could include wall intersection data and ground boundary data obtained through plane extraction, and association data obtained by subspace segmentation based on these data and associated with image frames; or any topological and geometric information that can provide geometric priors for subsequent semantic reasoning, where the room topology can be represented by the following formula: ,in For room nodes, This is the adjacency matrix. A semantic processing frame generally refers to a frame or intra-frame region determined from multiple image frames in image data that is to undergo semantic-aware processing. For example, it may include keyframe images marked as requiring semantic reasoning, image regions cropped via ROI, or image frames selected through cost scoring trigger conditions.

[0026] In this embodiment, refer to Figure 2The acquisition of keyframe pose data includes: extracting multiple feature points from the image data (e.g., extracting FAST corner points from the image stream of a stereo camera, the number of which can be configured to be 150 to 200); performing sliding window optimization on the sequence formed by multiple feature points (e.g., using a sliding window based on multi-state constrained Kalman filtering, jointly processing visual reprojection error and inertial measurement unit pre-integration error, adjusting the pose of all keyframes and inertial measurement unit bias within the window, and outputting the optimized keyframe pose map); and storing the optimized image data in timestamp order.

[0027] In this embodiment, refer to Figure 3 The acquisition of scene structure data includes: extracting wall intersection line data and ground boundary data from multiple image frames in the image data; wherein, the wall intersection line data is the line segment data formed by the intersection of different walls, and the ground boundary data is the contour line data at the junction of the ground and the surrounding structure (for example, by performing planar segmentation based on the random sampling consensus algorithm on the 3D point cloud, extracting ground planes and wall planes with high inlier rates, and calculating the spatial intersection lines between adjacent walls and the boundary lines between walls and the ground); performing subspace segmentation based on the wall intersection line data and the ground boundary data (for example, dividing the walls and ground that constitute a closed area into a room and assigning a unique identifier); associating the segmented subspace with multiple image frames to obtain the image frame association data. Through the above steps, the system obtains accurate geometric prior information—keyframe pose data and scene structure data.

[0028] The aforementioned method of processing keyframe pose data and scene structure data ensures the integrity and orderliness of the original observation data. The system writes all original images selected as keyframes into an external non-volatile storage device in precise system timestamp order. This ordered image sequence constitutes the original material library for high-precision offline semantic backfilling. Because the pose map has been optimized, the timestamps and pose information are precisely correlated, allowing the semantic inference module to index the corresponding precise shooting pose via the timestamp when backfilling any historical image. Using this method, even after several hours, semantic information can be fused into a unified map coordinate system without error. This provides direct and powerful support for subsequent offline semantic backfilling, utilizing precise poses for information fusion and leveraging structured information such as room layout to improve inference efficiency and accuracy, jointly ensuring high-quality acquisition of the geometric foundation required for semantic mapping.

[0029] In this embodiment, the first semantic instance information corresponds to the first semantic reasoning (offline backfilling), and the second semantic instance information corresponds to the second semantic reasoning (online on-demand triggering). The first semantic instance information generally refers to the semantic instance attributes obtained after semantic perception processing (such as full-frame high-precision semantic segmentation, cross-frame instance association, and voxel fusion) of semantic processing frames when the robot is idle. For example, it may include the category, 3D centroid, bounding box, appearance feature mask, and probability distribution of the identified static object (such as furniture) in the voxel map. The second semantic instance information generally refers to the semantic instance attributes obtained by matching semantic processing frames from the semantic model when triggering conditions are met, or the attributes of new instances created based on semantic processing frames. For example, it may include matching existing instances in the semantic model by calculating the 3D attributes of the semantic processing frame, returning the instance identifier and update information when the match is successful; or the category, location, geometry, and appearance features of new instances created after a failed match. The semantic model generally refers to a database or model that stores and manages identified semantic instances. For example, it could refer to a global instance database, where each semantic instance records fields such as category, probability distribution, 3D centroid, bounding box, appearance features, room affiliation, and status; or it could refer to a semantic segmentation network used for inference and its weights. The model supports cross-frame matching, instance creation and updating, and can distinguish between dynamic and static instances.

[0030] Specifically, when the robot is detected to be in an idle state (the idle state is defined as the robot's available computing power for semantic perception exceeding a third preset threshold per unit time, such as the robot charging at night or during off-peak periods), the robot performs semantic perception processing on the semantic processing frames to obtain the corresponding first semantic instance information. At this time, the robot reads all keyframe image sequences stored in an external storage device and uses a more complex and accurate deep learning model than real-time tracking to perform complete semantic instance segmentation and recognition on each keyframe image (e.g., object areas such as tables, sofas, and appliances), inferring the first semantic instance information. Subsequently, based on the first semantic instance information, semantic instances in the target environment are fused to generate or update semantic map data corresponding to the target environment. The semantic map data includes semantic landmarks used to represent objects in the target environment. Semantic map data generally refers to map data generated or updated after fusing semantic instances in the target environment, including semantic landmarks used to represent objects in the target environment. For example, it may include backprojecting a mask of a static semantic instance onto a voxel map, updating the semantic voxel map obtained by fusing semantic probabilities from multiple frames and spatially smoothing using Log-Odds Bayes; or a topological map containing object-level semantic landmarks. Semantic landmarks refer to map elements in semantic map data used to represent objects in the target environment, with each semantic landmark corresponding to a confirmed static semantic instance. For example, it may include a set of voxels labeled "sofa" in the voxel map and its probability field; or 3D bounding boxes, category labels, and feature vectors recorded in the global instance database. Before fusion processing, the repetition probability value between the first semantic instance information and other semantic instance information with the same spatial association can also be calculated; if the repetition probability value is greater than a fourth preset threshold, the first semantic instance information is not fused to avoid redundant labeling. Idle state refers to the state in which the robot is in when the computing power available for semantic perception per unit time is greater than a third preset threshold, i.e., a period when the real-time tracking deadline is not required. For example, it may include the idle period when the robot returns to the charging station after exploration (such as at night); or a period where any processing delay can exceed 100ms without affecting the robot's navigation and localization.

[0031] In this embodiment, when the step of matching semantic processing frames from the semantic model to obtain the corresponding second semantic instance information is performed, the triggering conditions include: calculating the overhead score of the zero neural network corresponding to the semantic processing frame, where the overhead score is greater than a fourth preset threshold; wherein, the overhead score includes an evaluation of one or more of the following: a pose difference factor for characterizing the change in camera viewpoint, a raster coverage factor for characterizing the coverage of unobserved areas, an image quality factor for evaluating image sharpness, and a semantic confidence factor for characterizing the uncertainty of the constructed semantic model within the field of view; or, identifying that the semantic processing frame is not associated with subspace information. The pose difference factor generally refers to the factor in the overhead score used to evaluate the change in the current camera pose compared to the pose during the most recent semantic inference. For example, it can be calculated based on a weighted sum of translation distance and rotation angle. The raster coverage factor generally refers to the factor in the overhead score used to evaluate the proportion of spatial rasteres within the current field of view that are not yet covered by the semantic model. For example, the coverage factor can be obtained by dividing the number of unobserved or unlabeled rasteres in the voxel map by the total number of rasteres. Image quality factors refer to the factors in the overhead score used to evaluate the quality of the current semantic processing frame, such as sharpness, blurriness, and lighting conditions. For example, Laplacian variance or Brenner gradient can be used to calculate the image sharpness score. Semantic confidence factors refer to the factors in the overhead score used to evaluate the degree of uncertainty of the semantic model already constructed within the current field of view. For example, the probability distribution entropy of instances in the semantic model can be calculated, or the weighted average of the inverse confidence of each instance can be used. The system calculates this score in real time, and only initiates the relatively time-consuming second semantic inference when the expected information gain is higher than a preset threshold, thus avoiding unnecessary consumption of computing power due to frequent initiation. At the same time, second semantic inference is also triggered when the robot determines that it has entered a room that has not been fully labeled before. This strategy ensures that when the robot enters a new environmental area, it can immediately begin to build a semantic map of that area, guaranteeing the map's coverage integrity and immediate availability. By employing this purely numerically-based triggering strategy, the frequency of semantic reasoning is significantly reduced from the highest possible frame rate of the hardware (e.g., 30Hz) to only occur during critical, novel, and high-quality moments (e.g., 1-2Hz), saving considerable GPU computing power. This ensures that the localization module always has sufficient computing resources to maintain hard real-time operation and achieves an optimal balance between map coverage integrity and construction efficiency under resource constraints.

[0032] In this embodiment, after the triggering condition is met, obtaining the corresponding second semantic instance information by matching the semantic processing frame from the semantic model specifically includes: calculating the three-dimensional attributes of the semantic processing frame (e.g., obtaining the position, size, and orientation of the object in three-dimensional space based on keyframe pose data and depth information), matching the corresponding semantic instance in the semantic model based on the three-dimensional attributes; when a corresponding semantic instance cannot be matched from the semantic model, creating a new semantic instance based on the semantic processing frame and updating it in the semantic model. This may include performing semantic segmentation on the region after ROI cropping of the current frame, calculating the three-dimensional centroid, appearance features, and bounding box of the object, matching and updating it with existing instances in the semantic model, or creating new instances; and backfilling static instances into the voxel map. This inference allows the time taken per frame to reach several times the real-time cycle without affecting geometric tracking. For example, the semantic model may be a global instance database, where each semantic instance records fields such as category, probability distribution, three-dimensional centroid, bounding box, appearance features, room affiliation, and status; or it may refer to the semantic segmentation network used for inference and its weights. This model supports cross-frame matching, instance creation and updating, and can distinguish between dynamic and static instances. Its execution relies on specific triggering conditions designed to maximize information gain and minimize computational waste. For example, when a robot autonomously revisits a mapped environment, it remains silent if the scene within its current field of view has already been sufficiently observed and labeled. The triggering conditions are only met when the robot enters a completely new room, or when its perspective changes significantly, allowing it to observe a large number of unlabeled areas, thus initiating the second semantic inference. The semantic model contains information about identified object instances, accelerating the online inference process. In this way, the frequency of online semantic inference can be significantly reduced from frame-by-frame inference (e.g., 30Hz) to 1-2Hz, saving substantial GPU computing power and ensuring that hard real-time geometric tracking remains undisturbed.

[0033] Specifically, by using the keyframe pose data and depth information associated with the semantic processing frame, the two-dimensional mask region of the semantic processing frame is back-projected into a three-dimensional space in the world coordinate system, forming a three-dimensional point set. Based on this point set, a set of physically meaningful three-dimensional attributes of the semantic processing frame can be calculated. These three-dimensional attributes may include its three-dimensional centroid in the world coordinate system, a compact three-dimensional axis-aligned bounding box, and an appearance feature vector calculated based on the visual features of the region (such as feature vectors obtained through global average pooling).

[0034] Subsequently, the semantic model is accessed to query all known instances of the same category stored therein. The matching process is implemented by designing a comprehensive distance metric or similarity score. This scoring function integrates the Euclidean distance between the 3D centroids and the cosine similarity between the appearance feature vectors. The score between the 3D attributes of the current semantic processing frame and the attributes of each candidate instance in the semantic model is calculated. If there is a candidate instance whose matching score is higher than a preset confirmation threshold, then the two are determined to be different observations of the same physical object. In this case, a new instance is not created. Instead, the newly observed 3D attributes and appearance features are fused and updated into the matched semantic instance using methods such as moving average, so that its attributes become increasingly accurate and complete over time.

[0035] Conversely, if no matching instance with a score higher than the threshold is found in the semantic model, it is determined that a completely new object has been observed. In this case, a new globally unique instance identifier is generated for the semantic processing frame, and a new semantic instance entry is created in the semantic model based on its three-dimensional attributes. This new instance will include its category, three-dimensional attributes, appearance features, the room identifier of the first observation, and an initial state (such as "candidate").

[0036] As an example, for a semantic instance of a new observation Its three-dimensional properties in the world coordinate system can be represented as: centroid : The arithmetic mean of the effective depth pixels within the mask projected onto the world coordinate system; appearance features Global average pooling vector; 3D bounding box : Axis-aligned bounding box in the world coordinate system after mask back projection.

[0037] The semantic model instance database can be represented as New observations Find the best match among similar instances: ; in, This is an indicator function; it takes the value 1 if the condition inside the parentheses is true, and 0 otherwise. , and These represent the category, centroid, and appearance feature values ​​of the j-th instance in the semantic model instance database, respectively. Geometric weights; Weights for appearance features; The threshold is set to spatial distance. If the maximum score of a newly observed semantic instance and an instance in the database exceeds a preset threshold, the newly observed semantic instance is considered to be the same instance as an instance in the database. Through this mechanism, multiple frames of two-dimensional fragments from different perspectives and times are organically linked and aggregated into unique, attribute-rich object-level semantic landmarks. This representation method significantly reduces the storage size of the entire room map from a massive number of voxels or points to only a limited number of semantic instances, and directly supports task-oriented object-level queries such as "finding the location of the sofa," greatly improving the map's practical value and storage efficiency.

[0038] In this embodiment, before performing the second type of semantic reasoning, height band filtering and / or subspace semantic prior can be applied to the semantic processing frame based on scene structure data. Height band filtering narrows the range of semantic instances in the semantic model based on height information associated with the semantic processing frame (e.g., restricting objects near the ground to furniture and objects near the ceiling to lighting fixtures). Subspace semantic prior narrows the range of semantic instances in the semantic model based on subspace information associated with the semantic processing frame (e.g., prioritizing matching kitchenware semantic models within the kitchen subspace). Height band filtering generally refers to the operation of narrowing the range of semantic instances in the semantic model based on height information associated with the semantic processing frame before performing semantic perception processing. For example, the height band from 0.2m to 1.0m above the ground can be determined using camera extrinsic parameters and plane equations as the possible height range for specific categories such as "sofa" and "bed," and semantic segmentation or matching can be performed only within this height band, while areas outside the band are not processed or have their probability set to zero. Subspace semantic prior generally refers to the operation of constraining the search results of semantic reasoning using identified subspace (room) type information. For example, if the current semantic processing frame is determined to be located in a bedroom, the matching probability of bedroom-type instances such as beds and wardrobes is increased during semantic model matching, while non-bedroom-type instances such as refrigerators and stoves are suppressed; or the semantic instance search range is directly limited to a subset of categories associated with this subspace type. Before fusion processing, the repetition probability value between the second semantic instance information and other semantic instance information with the same spatial association can also be calculated; if the repetition probability value is greater than the fifth preset threshold, the second semantic instance information is not fused. This avoids the unnecessary computational overhead that would still occur if semantic segmentation is performed directly on the full-resolution image after the second type of semantic reasoning is triggered, and the lack of context-guided reasoning is prone to false detections among similar objects in the room. Through the synergistic effect of the height band filtering and room semantic prior, a two-pronged approach is taken in both physical and semantic spaces, greatly reducing the effective search range of the semantic model. For example, areas in the image that are not furniture heights are physically cropped, and irrelevant categories are logically suppressed. This preprocessing process significantly reduces the equivalent latency of the second type of semantic reasoning, greatly reduces the GPU computing load, and effectively reduces the false detection rate of categories caused by similar appearance in indoor environments, ultimately improving the efficiency of online backfilling and map quality.

[0039] In this embodiment, semantic inference and map backfilling can be performed on demand or periodically using an optimized pose graph in offline batch processing mode or online lazy loading mode; or any operation mode that no longer forces synchronization between semantic inference and geometric tracking. Semantic inference in this mode allows single-frame processing time exceeding 33ms without causing tracking loss. It can be an online inference program that is activated during autonomous navigation replay when entering a new room or when the viewpoint change meets the information gain score threshold, using a lightweight semantic segmentation network (such as FastViT) to infer and update the semantic model for a portion of the current frame; or any soft real-time semantic processing unit activated on demand. Triggering conditions refer to a set of criteria for determining whether to initiate the second type of semantic inference. For example, it can include calculating a zero-neural network overhead score for the semantic processing frame, triggering when the score is greater than a third preset threshold; or forcibly triggering when the current room is identified as a new room. For another example, the score can be a combination of at least one of a pose difference factor, a raster coverage factor, an image quality factor, and a semantic confidence factor, and the third preset threshold can be set to a value between 0.5 and 0.8. In this embodiment, refer to... Figure 4 The method further includes identifying whether a semantic processing frame is a dynamic unit. Specifically, the steps are: predicting the position of the static feature points of the semantic processing frame corresponding to the next frame to obtain the predicted position; calculating the optical flow residual between the actual position and the predicted position of the static feature points; if the proportion of static feature points whose optical flow residual exceeds a first preset threshold exceeds a second preset threshold, then the semantic processing frame is marked as a dynamic unit; different semantic perception processing is performed on semantic processing frames marked as dynamic units and those not marked as dynamic units (e.g., dynamic units do not participate in the long-term fusion of semantic maps, but are only temporarily labeled). Dynamic units generally refer to regions or objects in the environment that exhibit independent movement and should not be considered part of a static background. For example, they may include image grid regions detected by using IMU-predicted optical flow and actual optical flow residuals, where the proportion of feature points whose residual exceeds a first preset threshold (e.g., 5 pixels) exceeds a second preset threshold (e.g., 30%); or instances already labeled as dynamic in the semantic model, such as pedestrians or pets. Preferably, the features of dynamic units do not participate in geometric tracking and static map construction, or different semantic perception processing is performed on semantic processing frames marked as dynamic units and semantic processing frames not marked as dynamic units.

[0040] As an example, the time interval can be predicted using IMU data. Semantic processing of inter-frame rotation components With translation components ; Static feature points tracked by the front end The prediction of its position in the next frame using IMU can be expressed as: ; in For camera projection model, The depth of the feature points; If the actual optical flow of a certain semantic processing frame Compared with IMU predictions optical flow residual between (Default 5px) Feature point ratio exceeds (Default 30%), then the semantic processing frame is marked as a dynamic unit.

[0041] This design enables the pre-labeling of dynamic regions at millisecond levels (e.g., less than 5 milliseconds) before or simultaneously with feature point extraction at the visual front end, without any neural network computational overhead. Its advantage lies in the fact that feature points within the grid regions labeled as dynamic units are immediately excluded by the back-end optimizer, effectively preventing the contamination of feature tracking, triangulation, and bundle adjustment optimization by fast-moving objects such as children and pets, and avoiding tracking loss due to landmark drift. This solves the millisecond-level dynamic response lag problem commonly found in pure vision or pure neural network solutions. This "coarse screening" result can also complement the "fine screening" result of the subsequent cognitive module. By adopting this preferred scheme, localization robustness is significantly enhanced. By employing this dynamic / static separation mechanism based on model instance attribute annotation, the final constructed static semantic map is protected from contamination by dynamic objects, while the information from dynamic objects is also effectively utilized, achieving intelligent separation and processing of static / dynamic information.

[0042] In the online backfilling process of the second type of semantic reasoning, to effectively distinguish between persistent static structures and transient dynamic objects in the environment and avoid including pedestrians, pets, etc., in the static map, this application also provides the following preferred solutions. In a preferred embodiment, semantic instances in the semantic model are labeled as dynamic units or static units. Here, the semantic model refers to a database that accumulates information on environmental object instances. When constructing this model, each semantic instance stored therein (e.g., a "person," a "chair"), in addition to its category, geometry, appearance, and other attributes, also contains an inherent attribute label to identify whether the instance itself is dynamic or static. This design enables the second semantic reasoning module to perform differentiated processing based on this label when performing online reasoning. For example, when a semantic processing frame is identified by the second semantic reasoning module and matched with an instance in the semantic model, the module will query the dynamic / static attributes of the instance. If the matched semantic instance is labeled as a dynamic unit, then the region corresponding to the semantic processing frame in the current frame will also be identified as a dynamic region. While this dynamic information is not backfilled into the static 3D voxel map to represent environmental structures, it can be sent separately to the dynamic object cache for motion trajectory prediction, providing crucial information for the robot's navigation module's local obstacle avoidance. Conversely, only semantic processing frames identified as matching static units in the semantic model have their semantic information reliably used to update the static voxel map. By employing this dynamic-static separation mechanism based on model instance attribute annotation, the final constructed static semantic map is protected from dynamic object contamination, while the information from dynamic objects is effectively utilized, achieving intelligent separation and processing of static / dynamic information.

[0043] In this embodiment, the repetition probability values ​​between the first semantic instance information and other semantic instance information with the same spatial association, and the repetition probability values ​​between the second semantic instance information and other semantic instance information with the same spatial association, can also be calculated separately. If the repetition probability value is greater than the corresponding threshold, the first semantic instance information or the second semantic instance information is not fused. The repetition probability value refers to the metric used to determine whether newly identified semantic instance information (first semantic instance information or second semantic instance information) is repeated with other labeled semantic instances in the same spatial association (such as the same room). For example, the attribute similarity between the semantic processing frame and other labeled semantic instances in the same room can be calculated, including the three-dimensional intersection-union ratio, appearance feature cosine similarity, and category consistency, and the repetition probability value is obtained by combining them. When the repetition probability value of the first semantic instance information is greater than the fourth preset threshold (such as 0.8), the first semantic instance information is not fused; when the repetition probability value of the second semantic instance information is greater than the fifth preset threshold (such as 0.8), the second semantic instance information is not fused.

[0044] Specifically, the process first determines the room to which the semantic processing frame belongs, and then strictly limits the potential matching comparison range to the existing set of instances within that room. For each labeled semantic instance of the same category within the room, a repetition probability representing that the two are the same object is calculated. This probability calculation can integrate multiple similarity indicators, such as the overlap of the 3D bounding boxes of the two instances, the spatial proximity of their centroids, and the cosine similarity of their appearance features. If the repetition probability with an existing instance exceeds a preset threshold, such as 95%, the module makes a "repetition labeling" judgment. The key advantage of this mechanism lies in its room context awareness capability. It sets an important principle: semantic units belonging to different room attributes, even if the calculated appearance and geometric similarity are extremely high, should not be judged as repetition labels. This principle effectively prevents furniture located in different rooms but with the same appearance (e.g., two identical dining chairs in two different rooms) from being mistakenly merged into the same instance. By introducing this repetition labeling verification, it ensures that each semantic landmark in the map uniquely and atomically corresponds to a physical object in the real world, thereby eliminating map redundancy and maintaining the consistency and accuracy of the object-level map.

[0045] As an example, the fusion process preferably includes fusion of 2D mask backprojection, voxel mesh mapping, Log-Odds Bayesian update, and conditional random field spatial smoothing to generate or update semantic map data. Preferably, for a semantic instance identified as static, all associated historical observation frames are retrieved. For each frame observation, the instance is backprojected from the 2D semantic mask of that frame into the 3D world coordinate system using the frame's precise pose. For each voxel traversed by the backprojection ray or covered by the mask, a Bayesian update based on a log-odds model is performed. This update process continuously accumulates the probability that the voxel belongs to a specific semantic category and smooths out occasional erroneous observations or noise. For example, a voxel may be observed from different angles in multiple frames, with most observations classifying it as "sofa," but one frame may misclassify it as "background" due to occlusion or shadows. Through Bayesian update, the voxel eventually converges stably to the correct category of "sofa." By strictly limiting semantic backfilling to static semantic instances, the erroneous backprojection and fusion of masks of dynamic objects such as pedestrians and pets into the static voxel map representing the persistent environment structure is fundamentally prevented, ensuring the accuracy and reliability of occupancy queries and semantic queries. Information about dynamic objects is maintained independently and used for other non-mapping tasks, achieving a clean map content and decoupling of functionality.

[0046] As an application scenario example, in a commercial application of a home service robot, the robot first enters a newly renovated residence to explore its environment. The user guides the robot to move through the rooms via remote control, at which point the environmental perception method is implemented. First, the robot acquires image data at a 30Hz frame rate using a binocular global shutter camera on its head, while simultaneously outputting 200Hz inertial data from a nearby six-axis microelectromechanical inertial measurement unit. The attitude acquisition module extracts multiple feature points from the image data (such as FAST corner points), performs inter-frame tracking using KLT optical flow, and performs sliding window optimization on the sequence formed by multiple feature points. Multi-state constrained Kalman filtering optimization is then performed, combining visual reprojection error and inertial pre-integration error, to obtain high-precision keyframe pose data. The original keyframe images are stored in timestamp order on an external NVMe solid-state storage device. Meanwhile, the pose acquisition module extracts wall intersection data and ground boundary data from the image data—specifically, iteratively extracting the plane equations of the ground and walls from the 3D point cloud using a random sampling consensus algorithm, performing subspace segmentation based on the wall intersections and ground boundaries (each room is considered a subspace), assigning an identifier to each subspace, and constructing a room topology map as image frame association data, thereby obtaining scene structure data. Among all selected keyframes, the objects to be processed (such as each image frame) determined from multiple image frames are used as semantic processing frames. At night, the robot automatically returns to its charging dock, recognizing that it is in an idle state—the computing power available for semantic perception per unit time is greater than a third preset threshold—thus triggering semantic perception processing on the semantic processing frames. The first semantic reasoning module sequentially reads the keyframe image sequence precisely associated with the keyframe pose data from solid-state storage, and performs first-type semantic reasoning on each semantic processing frame using the known high-precision pose map and room topology information. This reasoning uses the FastViT semantic segmentation model specifically for indoor use to identify instance masks and categories of various furniture and appliances such as sofas, beds, dining tables, refrigerators, and cabinets, obtaining the corresponding first semantic instance information. Since there are no hard constraints on real-time performance, the inference time for a single frame is allowed to be within one hundred milliseconds. The inferred 2D semantic labels are back-projected onto the 3D voxel map based on the corresponding keyframe pose data. Semantic instances in the target environment are fused based on the first semantic instance information—probabilistic fusion is performed through log-probability-based Bayesian updates to generate dense and consistent static semantic map data. This includes semantic landmarks representing objects in the target environment (such as a six-seater sofa in the living room, which is associated as a unique semantic instance and persisted as a semantic landmark after matching its centroid, appearance feature vector, and 3D axis-aligned bounding box). At this point, a high-precision environmental geometry (i.e., the first keyframe pose map) has been established and is available. Therefore, the main task of the pose acquisition module is no longer to reconstruct the geometric map, but to use existing geometric priors for high-precision, robust real-time localization. By dawn, the robot has completed the backfilling of the semantic map for the entire house.The next day, the user gave the voice command, "Bring me the remote control from the coffee table." The robot autonomously navigated to the living room. At this point, the pose acquisition module used the established keyframe pose map for relocalization and continuous tracking, continuing to acquire keyframe pose data. During its journey, the family dog ​​suddenly darted past the robot. The predicted position was obtained by predicting the position of the static feature points in the next frame corresponding to the predicted position of the static feature points, and the optical flow residual between the actual and predicted positions of the static feature points was calculated. When the number of static feature points with an optical flow residual exceeding five pixels in a certain image grid cell reached more than 30% of the total, the grid was immediately identified as a dynamic cell. For semantic processing frames marked as dynamic cells, their feature points were immediately excluded and not included in subsequent optimization; furthermore, the regions corresponding to these dynamic cells were not used for fusion processing in subsequent semantic perception processing. Meanwhile, in the online-triggered cognitive reasoning, second semantic instance information is obtained by matching semantic processing frames from a pre-built semantic model: the three-dimensional attributes of the semantic processing frame (e.g., via voxel coordinates) are calculated, and the corresponding semantic instance is matched in the semantic model based on the three-dimensional attributes (e.g., the pet dog instance has been pre-labeled with motion and static attributes). When a match is successful, its mask is sent to the dynamic object cache for trajectory prediction, instead of being backfilled into the static voxel map. The robot thus achieves robust perception and navigation of fast-moving objects. Throughout the process, the following operations are also performed: before performing the second type of semantic reasoning, the semantic processing frame is highly band filtered and / or subspace semantic prior is performed based on scene structure data—for example, when the current frame is identified as being located in the living room subspace, the range of instances in the semantic model is narrowed down to common items in the living room; when the cost score of the zero neural network corresponding to the semantic processing frame (including one or more of the pose difference factor, raster coverage factor, image quality factor, and semantic confidence factor) is greater than a fourth preset threshold, matching the semantic processing frame from the semantic model is triggered; if the semantic processing frame is identified as not being associated with subspace information, the matching is also triggered. Furthermore, during the fusion process, the repetition probability value between the first semantic instance information and other semantic instance information with the same spatial association is calculated. If the repetition probability value is greater than a fourth preset threshold, the first semantic instance information is not fused. The same repetition probability value is calculated for the second semantic instance information; if it is greater than a fifth preset threshold, it is not fused. These mechanisms ensure the accuracy and consistency of the semantic map.

[0047] Figure 5This is a schematic diagram of the architecture of a robot environment perception system according to an embodiment of this application. The environment perception system can be mounted on a mobile robot, such as an indoor companion robot. A sensor assembly is fixed to the robot body, including a binocular global shutter camera, a six-axis microelectromechanical inertial measurement unit (MEMS), and an optional time-of-flight depth sensor. The binocular camera is positioned directly in front of the robot's head with a preset tilt angle, and the inertial measurement unit is mounted adjacent to the camera; the two are time-stamp aligned via a hardware synchronization signal line. The sensor assembly is connected to an embedded processing unit via a board-level data bus. The embedded processing unit includes a central processing unit (CPU) and a graphics processing unit (GPU), and is equipped with a non-volatile external storage device, such as an NVMe-based solid-state memory, used to cache keyframe image sequences from the camera and intermediate data generated during processing. The CPU and GPU work together to execute program instructions related to environment perception. The program loaded in the embedded processing unit constitutes at least a data acquisition module, a semantic recognition module, and a map fusion module. The data acquisition module is configured to acquire keyframe pose data, scene structure data, and semantic processing frames. The keyframe pose data includes image data collected by the robot from the target environment. The scene structure data includes image frame association data representing the spatial relationships between multiple image frames in the image data and the target environment. The semantic processing frame is the object to be processed determined from multiple image frames in the image data. The semantic recognition module is configured to perform semantic perception processing on the semantic processing frames to obtain corresponding first semantic instance information, and / or, match the semantic processing frames from the semantic model to obtain corresponding second semantic instance information. The map fusion module is configured to perform fusion processing on semantic instances in the target environment based on the first semantic instance information and / or the second semantic instance information, generating or updating semantic map data corresponding to the target environment. The semantic map data includes semantic landmarks representing objects in the target environment. Furthermore, the data acquisition module may specifically include: a feature point extraction unit, a sliding window optimization unit, and a temporal storage unit for acquiring keyframe pose data; and a wall intersection extraction unit, a ground boundary extraction unit, a subspace segmentation unit, and an image frame association unit for acquiring scene structure data. The semantic recognition module may further include a dynamic unit recognition submodule, used to predict the position of static feature points and calculate optical flow residuals to determine whether a semantic processing frame is a dynamic unit, and accordingly apply different semantic perception processing. The recognition module may also include a height band filtering submodule and / or a subspace semantic prior submodule, used to narrow down the range of semantic instances in the semantic model before performing the second type of semantic inference. The map fusion module may further include a repetition probability calculation submodule, used to calculate the repetition probability value between the first semantic instance information or the second semantic instance information and other semantic instance information, and skip the fusion process if the repetition probability exceeds a corresponding threshold.Each module operates in coordination according to the set timing and triggering logic: when the robot is detected to be in an idle state, semantic perception processing of the semantic processing frame is triggered to obtain the first semantic instance information; when the cost score of the semantic processing frame exceeds a preset threshold or the semantic processing frame is not associated with subspace information, matching from the semantic model is triggered to obtain the second semantic instance information. Finally, semantic map data that can be directly used by the mobile robot for navigation and interaction tasks is formed.

[0048] In several embodiments provided in this application, the robot environmental perception method and related devices can be applied to electronic devices with certain computing and network access capabilities. These electronic devices can be desktop computers, laptops, tablets, smartphones, or servers. Specifically, the electronic device includes a processor, a memory, and a network access module for network communication. The server can be an electronic device with strong data processing capabilities; alternatively, it can refer to a server cluster formed by multiple electronic devices, or a quantum server built using a quantum computer.

[0049] In addition to the methods, apparatus, and devices described above, embodiments of this disclosure may also be computer program products, comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the robot environment perception methods provided in the various embodiments of this disclosure. The computer program product may be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code may be executed entirely on a user computing device, partially on a user device, as a standalone software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server. Furthermore, embodiments of this disclosure may also be storage media (e.g., computer-readable storage media) storing a computer program (or instructions) that, when executed by a processor, cause the processor to perform the steps of the methods provided in the various embodiments of this disclosure. The storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. Readable storage media may include, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatus and methods can be implemented in other ways.For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Additionally, the functional units in the various embodiments of this disclosure may be integrated into a similar region segmentation unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. If the function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks. The above descriptions are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A robot environmental perception method, characterized in that, Includes the following steps: Acquire keyframe pose data, scene structure data, and semantic processing frames; wherein, the keyframe pose data includes image data collected by the robot for the target environment; the scene structure data includes image frame association data used to characterize the spatial relationship between multiple image frames in the image data and the target environment; the semantic processing frames are the objects to be processed determined from the multiple image frames in the image data; The semantic processing frame is subjected to semantic perception processing to obtain the corresponding first semantic instance information, and / or the semantic processing frame is matched from the semantic model to obtain the corresponding second semantic instance information; Based on the first semantic instance information and / or the second semantic instance information, the semantic instances in the target environment are fused to generate or update the semantic map data corresponding to the target environment. The semantic map data includes semantic landmarks used to represent objects in the target environment.

2. The robot environment perception method according to claim 1, characterized in that, Obtaining the keyframe pose data specifically includes: Extract multiple feature points from the image data; Perform sliding window optimization on the sequence formed by multiple feature points; Optimized image data is stored in timestamp order.

3. The robot environment perception method according to claim 1, characterized in that, Obtaining the scene structure data specifically includes: Extract wall intersection line data and ground boundary data from multiple image frames in the image data; wherein, the wall intersection line data is the line segment data formed by the intersection of different walls, and the ground boundary data is the contour line data at the junction of the ground and the surrounding structure; Subspace segmentation is performed based on the wall intersection data and the ground boundary data; The image frame association data is obtained by associating the segmented subspace with the multiple image frames.

4. The robot environment perception method according to claim 1, characterized in that, The method further includes identifying whether the semantic processing frame is a dynamic unit, specifically including: The predicted position is obtained by predicting the position of the static feature points of the semantic processing frame corresponding to the position of the next frame; Calculate the optical flow residual between the actual and predicted positions of static feature points; If the proportion of static feature points with optical flow residuals exceeding a first preset threshold to the total exceeds a second preset threshold, then the semantic processing frame is marked as a dynamic unit. Different semantic awareness processing is performed on semantic processing frames that are marked as dynamic units and those that are not marked as dynamic units.

5. The robot environment perception method according to claim 1, characterized in that, The method further includes performing high-band filtering and / or subspace semantic prior on the semantic processing frame based on scene structure data before performing the second semantic reasoning; The height band filtering is used to narrow the range of semantic instances in the semantic model based on the height information associated with the semantic processing frame; the subspace semantic prior is used to narrow the range of semantic instances in the semantic model based on the subspace information associated with the semantic processing frame.

6. The robot environment perception method according to claim 1, characterized in that, The method further includes: When the robot is detected to be in an idle state, the step of performing semantic perception processing on the semantic processing frame to obtain the corresponding first semantic instance information is executed; the idle state is defined as the computing power value of the robot available for semantic perception per unit time being greater than a third preset threshold.

7. The robot environment perception method according to claim 1, characterized in that, The triggering conditions for performing the matching of the semantic processing frame from the semantic model to obtain the corresponding second semantic instance information include: Calculate the overhead score of the zero neural network used for the semantic processing frame, wherein the overhead score is greater than a fourth preset threshold; wherein the overhead score includes an evaluation of one or more of the following: a pose difference factor for characterizing the change in camera viewpoint, a grid coverage factor for characterizing the coverage of unobserved areas, an image quality factor for evaluating image sharpness, and a semantic confidence factor for characterizing the uncertainty of the constructed semantic model within the field of view. Alternatively, it may be identified that the semantic processing frame is not associated with subspace information.

8. The robot environment perception method according to claim 1, characterized in that, The step of matching the semantic processing frame from the semantic model to obtain the corresponding second semantic instance information specifically includes: Calculate the three-dimensional attributes of the semantic processing frame. Based on the three-dimensional attributes, the corresponding semantic instance is matched in the semantic model; And / or, when a corresponding semantic instance cannot be matched from the semantic model, a new semantic instance is created based on the semantic processing frame and updated to the semantic model.

9. A robot environment perception method according to claim 1, characterized in that, The method further includes: Calculate the repetition probability value between the first semantic instance information and other semantic instance information that have the same spatial association relationship; When the repetition probability value is greater than the fourth preset threshold, the first semantic instance information is not fused.

10. A robot environment perception method according to claim 1, characterized in that, The method further includes: Calculate the repetition probability value between the second semantic instance information and other semantic instance information that have the same spatial association relationship; When the repetition probability value is greater than the fifth preset threshold, the second semantic instance information will not be fused.

11. A robot environmental perception system, characterized in that, The system includes: The data acquisition module is used to acquire keyframe pose data, scene structure data, and semantic processing frames; wherein, the keyframe pose data includes image data collected by the robot in relation to the target environment; the scene structure data includes image frame association data used to characterize the spatial relationship between multiple image frames in the image data and the target environment; and the semantic processing frames are objects to be processed determined from multiple image frames in the image data. The semantic recognition module is used to perform semantic perception processing on the semantic processing frame to obtain the corresponding first semantic instance information, and / or to match the semantic processing frame from the semantic model to obtain the corresponding second semantic instance information; The map fusion module is used to perform fusion processing on semantic instances in the target environment based on the first semantic instance information and / or the second semantic instance information, to generate or update semantic map data corresponding to the target environment, wherein the semantic map data includes semantic landmarks used to represent objects in the target environment.

12. A robot, characterized in that, The robot includes the environmental perception system as described in claim 11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images