Multi-level semantic map construction method based on scene recognition and target detection

Abstract

The invention provides a multi-level semantic map construction method based on multi-sensor fusion, and the method carries out the construction of an environment grid layer, and comprises the steps: constructing an environment grid map in real time through fusing perception data; scene semantic layer construction: extracting image scene semantic probability distribution by using a deep convolutional network, fusing time sequence observation through Bayesian filtering, and mapping a scene category to a grid unit by using an occupation probability model; constructing an object semantic layer, namely identifying an object by adopting a target detection network in which an information aggregation-distribution mechanism is introduced, extracting an object point cloud, and dynamically updating object semantic attributes of grid units through multi-source observation fusion; and scene atlas generation: constructing a hierarchical scene atlas which takes the marker object as a reference core and comprises a spatial topological relation. According to the method, the dynamic environment adaptability and the multi-modal data fusion precision of semantic mapping are improved, a more visual environment understanding mode is provided for the robot, and the practicability of the semantic map in robot positioning and navigation is improved.

Description

Technical Field

[0001] The present invention belongs to the field of robot intelligent environment perception and scene understanding, and specifically relates to a technical solution for constructing a multi-level semantic map based on scene recognition and target detection and generating a scene atlas. Background Art

[0002] As mobile robot applications become increasingly complex, traditional geometrically-based environmental maps (such as 2D grid maps and 3D point cloud maps) are no longer able to meet the robots' need for semantic understanding. For example, in home services or industrial inspections, robots must not only perceive the location of obstacles but also identify scene information, object information, and their spatial relationships within the environment. However, existing semantic mapping methods suffer from insufficient adaptability to dynamic environments and low accuracy in multimodal data fusion, limiting the practical application of semantic maps for robot positioning and navigation. Summary of the Invention

[0003] In order to solve the problems existing in the prior art, the present invention provides a technical solution for constructing a multi-level semantic map based on scene recognition and target detection and generating a scene atlas.

[0004] The present invention provides a multi-level semantic map construction method based on multi-sensor fusion, which performs the following process: Environmental grid layer construction, including real-time construction of environmental grid maps by fusing perception data; The construction of the scene semantic layer includes extracting the semantic probability distribution of the image scene using a deep convolutional network, fusing temporal observations through Bayesian filtering, and mapping the scene categories to grid cells using an occupancy probability model; The object semantic layer is constructed, including identifying objects using a target detection network that introduces an information aggregation and distribution mechanism, extracting object point clouds by combining depth information and image segmentation, and dynamically updating the object semantic attributes of grid cells through multi-source observation fusion; Scene graph generation includes constructing a hierarchical scene graph with landmark objects as the reference core and containing spatial topological relationships based on the association between the scene semantic layer and the object semantic layer.

[0005] Moreover, when constructing the environment grid map in real time, the branch-and-bound algorithm is used to solve the optimal pose transformation between the laser point cloud and the sub-map; and the closed-loop error is minimized in the pose graph model.

[0006] Furthermore, the scene semantic layer construction includes modeling scene classification as a Bayesian estimation process; and defining a scene occupancy probability model of grid cells.

[0007] Moreover, the target detection network that introduces the information aggregation-distribution mechanism is implemented by improving the YOLOv8 target detection algorithm, including realizing multi-scale feature aggregation and distribution through the information aggregation-distribution mechanism, introducing an attention scale sequence fusion strategy, and performing structured pruning based on layer adaptive amplitude.

[0008] Moreover, the multi-scale feature aggregation and distribution is achieved through the information aggregation-distribution mechanism, and the implementation method includes setting Low-GD branch fusion to extract and fuse large-scale feature maps, setting High-GD branches to extract and fuse small-scale feature maps, and setting a feature alignment module, a feature information fusion module and a feature information distribution module in each branch.

[0009] Moreover, the multi-source observation fusion is implemented by representing the semantic map as a joint probability distribution of each semantic grid unit based on the robot posture sequence and sensor data set, and calculating the probability of obstacle existence and non-existence for each semantic grid unit based on the Bayesian probability framework to achieve dynamic semantic grid state update.

[0010] Moreover, when constructing a scene graph, landmark objects are screened based on the frequency of appearance and position stability of objects in the scene, and a local semantic relationship graph is established with the landmark objects as the center; a three-level hierarchical index structure is constructed, with the first-level index being the scene category label, the second-level index being the landmark object nodes, and the third-level index being the object nodes associated with each landmark object. The nodes are connected by relative distance edges to form a semantic relationship graph structure; the semantic relationship graph structure supports the robot to quickly retrieve positioning reference objects in the scene according to task requirements.

[0011] On the other hand, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and runnable on the processor. When the processor executes the program, the multi-level semantic map construction method based on scene recognition and target detection as described above is implemented.

[0012] On the other hand, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-level semantic map construction method based on scene recognition and target detection as described above.

[0013] On the other hand, the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the multi-level semantic map construction method based on scene recognition and target detection as described above.

[0014] This paper proposes a multi-level semantic map construction framework based on scene recognition and object detection. This framework constructs a multi-level map of "scene → object → geometry." This framework then uses a Bayesian incremental mapping algorithm to optimize the dynamic update mechanism of semantic information, achieving a hierarchical map representation that combines both scene and object semantics. Finally, a scene atlas is generated based on this multi-level semantic map, providing robots with a more intuitive way to understand their environment.

[0015] The present invention has the following advantages: The proposed method for constructing and generating a scene graph using multi-level semantic maps based on scene recognition and object detection provides a more intuitive solution for environmental understanding, addressing the difficulties faced by traditional geometric feature-based environmental mapping methods in meeting the robot's semantic understanding needs, as well as the shortcomings of existing semantic mapping methods in adapting to dynamic environments and inaccurate multimodal data fusion. This method effectively combines scene information and object information in the robot's environment, enhancing the map's expressive power by introducing semantic features. This improves the performance of traditional maps in environments with simple structures or sparse features. Generating a scene graph based on the constructed multi-level semantic map fully utilizes both global and local features of the environment, combining scene information (such as environmental features like offices and corridors) with object information (such as semantic identifiers like doors and cabinets), improving the map's adaptability and practicality and providing the robot with a more intuitive and accurate representation of the environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of the process of constructing a multi-layer semantic map and scene graph according to an embodiment of the present invention.

[0017] Figure 2 Schematic diagram of the principle of constructing a multi-layer semantic map according to an embodiment of the present invention.

[0018] Figure 3 Figure 1 is a schematic diagram of the graph construction of an embodiment of the present invention, which includes (a) the overall office environment; (b) the raster map construction result; (c) scene recognition; (d) scene semantic mapping process 1; (e) scene semantic mapping process 2; (f) the final scene layer semantic map; (g) target detection; (h) object semantic mapping process 1; (i) object semantic mapping process 2; (j) the final object layer semantic map; and (k) the office scene graph.

[0019] Figure 4 This is a feature fusion framework diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to facilitate ordinary technicians in this field to understand and implement the present invention, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0021] The present invention first uses a robot to obtain data from various sensors. The front end achieves real-time pose estimation through incremental matching, and the back end uses global loop detection and pose graph optimization to eliminate cumulative errors and construct an environmental grid map. Then, the current image scene probability is obtained through a network model, and the scene semantic information is mapped to the grid map to construct a scene-level semantic map. Subsequently, the object category and detection frame are obtained in real time through the target detection algorithm. The object outline is extracted according to the corresponding detection frame position in the camera depth map, and the point cloud data within the outline is mapped to the grid map in real time. The Bayesian method is used to gradually improve and update the object semantic information to construct an object-level semantic map. After obtaining the multi-level semantic map, the scene information and object information are combined to generate a scene map of the indoor environment. The present invention improves the dynamic environment adaptability and multimodal data fusion accuracy of the semantic mapping method, provides a more intuitive way for robots to understand the environment, and improves the practicality of semantic maps in robot positioning and navigation.

[0022] An embodiment of the present invention provides a method for constructing a multi-level semantic map based on scene recognition and target detection. The specific implementation can rely on a hardware platform, including: a robot platform, a main control machine, a depth camera, and a lidar; the main control machine is connected to the robot platform, depth camera, and lidar respectively.

[0023] See also Figure 1-3 The embodiment provides a method for constructing a multi-level semantic map based on scene recognition and target detection, comprising the following steps: Step 1: Constructing the environment grid layer: The robot obtains data from sensors such as lidar, odometry, and IMU. The front-end achieves real-time pose estimation through incremental matching of local sub-maps. The back-end then uses global loop closure detection and pose graph optimization to eliminate accumulated errors and construct an environment grid map.

[0024] In the embodiment, the environment grid map is preferably constructed by using the Cartographer algorithm (a real-time SLAM algorithm based on pose graph optimization). The specific implementation of step 1 includes the following processing: (1) The Cartographer algorithm inserts continuous LiDAR frame data into a local submap through scan matching. Specifically, it uses a global matching algorithm based on branch and bound to solve the optimal pose transformation between the current scan point cloud and the submap. Its objective function can be expressed as:

[0025] Where, is the optimal pose transformation matrix, T is the pose transformation matrix to be determined, Indicates the accumulation of index i from 1 to N, p i is the coordinate of the i-th laser point, T○p i Represents the pose transformation T applied to point p i The new coordinates after , M(•) represents the grid probability value of the sub-map.

[0026] (2) Loop detection mechanism, which uses a fast constraint search method based on branch and bound. When a closed loop is detected, the system constructs a pose graph model that includes constraints such as odometer, IMU, and sub-map nodes:

[0027] Where E(X) is the total error function of the pose graph, X is the set of all node poses, represents the error term between the observation constraint and the predicted pose, Z ij is the observed relative pose transformation, i and j are the indices of the nodes in the pose graph, and each node represents the pose of the system at a certain moment, that is, and Represent the positions of node i and node j in the global coordinate system, is the relative pose predicted based on the current pose graph, is the information matrix.

[0028] Step 2: Scene semantic layer construction: First, the scene probability of the current image is obtained through a deep convolutional network, and then its scene semantic information is mapped to a two-dimensional grid map to construct a scene layer semantic map.

[0029] In the embodiment, the Places365-VGG16 network model is preferably used to obtain the scene probability of the current image, and the scene layer semantic map is constructed in the following manner: (1) Convert the semantic scene classification problem into a probability estimation task. Assume that the tth image of the current scene captured by the camera is I t , then the discrete probability distribution p(x i |I t ), where x i is the scene category label with index i. At the same time, the label set of all scene categories is defined as:

[0030] Where, is an ordered set of 365 scene category labels of the Places365 dataset, x iis the scene category label with index i, i=0,1,…n, where n is the maximum index 364.

[0031] (2) For all scene category labels, the scene recognition model is used to calculate the probability distribution of the acquired image data stream, which is recorded as , where I {0～t} Represents the image sequence from time 0 to time t, from which the likelihood function can be obtained:

[0032] Where, Represents the image sequence I from time 0 to time t using the scene recognition model for all scene category labels {0～t} Perform probability distribution calculations. Represents the discrete probability estimate of the t-th frame image on the n-th scene category, scene category Independent of each other.

[0033] (3) Traditional scene recognition methods only process each frame independently, while this paper proposes to treat the scene classification problem as a Bayesian estimation process by utilizing the temporal continuity between adjacent frames. Assuming that this process satisfies the first-order Markov property, a more coherent scene classification result can be obtained. Therefore, the update process of the Bayesian filter can be expressed as:

[0034] in, Indicates that the current frame image I has been observed at the current time t t The scene category state under the condition The probability distribution of Indicates that the current scene state is Under the condition of t The probability of Indicates that the previous frame image I has been observed at the previous time t-1 t-1 Under the condition of The probability distribution of .

[0035] By treating scene classification as a Bayesian estimation problem, additional prior information can be incorporated into the calculation process. Given that the Places365-VGG16 dataset covers many indoor scenes, it is possible to use the existing prior probability In addition, this prior knowledge is incorporated into the Bayesian filtering formula to further improve the accuracy of scene recognition:

[0036] (4) After obtaining the scene probability of the current image, its scene semantic information is mapped onto a two-dimensional grid map. Through a gridding calculation method, the semantic information of the scene category is integrated into the two-dimensional grid, and the semantic information of each scene category is represented as a specific spatial unit.

[0037] Assume that the grid unit is m, then the scene category semantic information x in the unit is n The occupation probability is The probability is updated based on the current camera image information:

[0038] Where x n is the scene category semantic information, Represents the semantic information of scene category x n The estimated value of is the scene category semantic information in grid cell m under image information I from time 0 to t The occupation probability, I t is the camera image information at the current time t, is the image information from time 0 to t-1, is the image information I at the current time t t Semantic information of scene category in the lower grid cell m The conditional probability of is the scene category semantic information under the image information from time 0 to t-1 The historical posterior probability of is the scene category semantic information in grid cell m under the image information from time 0 to t-1 The historical posterior probability of is the scene category semantic information The prior probability of is the scene category semantic information in grid cell m The prior probability of .

[0039] The probability of grid cell m occupying the scene category semantic grid depends on the current image information I t By fusing the prior probability of scene classification and current observation information, the semantic distribution of the scene can be effectively expressed in the spatial grid.

[0040] Step 3: Construct the object semantic layer: First, use the target detection algorithm to obtain the category and detection frame of objects in the environment in real time. Then, use the GrabCut algorithm to extract the object outline based on the position of the detection frame corresponding to the camera depth map. Then, map the point cloud data within the outline to the raster map in real time, and use the Bayesian method to gradually improve and update the object semantic information of the environment to construct an object layer semantic map.

[0041] The method for constructing an object-level semantic map based on target detection preferably adopted in the embodiment is as follows: (1) Optimization of Yolov8 algorithm: As an advanced target detection algorithm, YOLOv8 has the characteristics of fast speed and high accuracy, and is very suitable for real-time perception tasks of robots.

[0042] Since YOLOv8’s feature fusion method uses the traditional PANet, it can only completely fuse the features of adjacent layers, and the information of other layers can only be indirectly “recursively”, which will cause problems such as slow speed and information loss. To this end, this patent proposes a new feature fusion framework, see Figure 4 First, based on the different-scale features B1-B5 extracted by the Backbone network (for example, 640×640×3, 320×320×64, 80×80×256, 40×40×512, and 20×20×1024, respectively), a Neck network is established. This involves using a Gather and Distribute (GD) mechanism to uniformly collect features from different levels for comprehensive information fusion. This information-rich feature is then distributed back to each level, enhancing the model's expressive power and reducing information loss during the fusion process. Furthermore, the attention-based scale sequence fusion strategy of Advanced Feature Selection (ASF) is introduced to further refine feature selection and strengthen the model's ability to recognize objects of different scales. Specifically, based on the original YOLOv8 model, the GD mechanism, the ASF attention module, and the pruning operation are sequentially added.

[0043] By comparing the improved YOLOv8 with the original YOLOv8, and comparing it with the current mainstream YOLO series algorithms v5 and v6 in terms of accuracy and speed, it can be found that the accuracy and speed of the improved YOLOv8 target detection algorithm after this patent are both optimized, providing important value for mobile robots to extract semantic information of objects in indoor environments.

[0044] (2) Implementation of GD mechanism: The GD mechanism implements aggregation and distribution through three modules: the Feature Alignment Module (FAM), the Information Fusion Module (IFM), and the Inject module. To improve the model's detection capabilities for objects of varying sizes, the GD mechanism consists of two branches: Low-GD and High-GD. Low-GD extracts and fuses large-scale feature maps, while High-GD extracts and fuses small-scale feature maps.

[0045] Low-GD is used to extract and fuse large-scale feature maps. By fusing feature maps B2, B3, B4, and B5, high-resolution features that retain small target information are obtained.

[0046] [1] Low-FAM alignment module, selects B4 of moderate scale as the benchmark. For the larger scale feature maps B2 and B3, down-sample by average pooling (avgpool), and for the smaller scale feature map B5, up-sample by bilinear interpolation (bilinear) to achieve the unification of feature map size, and then concat to obtain the merged feature Falign; [2] Low-IFM fusion module, including convolution (Conv), re-parameterized convolution (RepBlock) module, split operation (Split), the feature Falign obtained by Low-FAM is input into the RepBlock module to obtain Ffuse fusion feature. The role of Conv is to adjust the channel. Ffuse is split into F on the channel through Split. inj_P3 and F inj_P4 , and then perform the next step of feature fusion with different levels; [3] Inject information distribution module, whose input is the current layer feature x_local (B3 and B4) to be distributed, and the global feature x_global (F generated by IFM inj_P3 and F inj_P4 ), in the fusion process, average pooling or bilinear interpolation is used to ensure the size consistency, and finally RepBlock is used to further extract and fuse information. inj_P3 Fusion to obtain P3, B4 and F inj_P4 P4 is obtained by fusion, and finally P3, P4 and P5 (B5) are input into the High-GD module.

[0047] The High-GD module is used to extract and fuse small-scale feature maps. The features P3, P4, and P5 obtained by Low-GD fusion are further fused by High-GD: [1] The High-FAM alignment module consists of average pooling, which unifies the size of the input features. First, P3 and P4 are reduced to the size of P5, and then the feature maps are merged through Concat to obtain Falign. [2] The High-IFM fusion module consists of a Transformer module and a split operation. The Transformer module contains L blocks, each of which consists of a multi-head attention module, a feedforward network, and a residual connection. First, the feature Falign obtained by High-FAM is processed by the Transformer to obtain Ffuse. After the Conv1×1 convolution adjusts the channel of Ffuse, the feature map is sliced ​​into F through the split operation. inj_N4 and F inj_N5 ; [3] The high-level information distribution module is exactly the same as the low-level one, through P4 and F inj_N4 Fusion to obtain N4, P5 and F inj_N5 The fusion gave N5.

[0048] (3) ASF’s attention scale sequence fusion strategy In the ASF module, the feature maps B3, B4, and B5 extracted by the backbone network are normalized to the same size using Attentional Scale Sequence Fusion (SSFF), upsampled, stacked, and fed into a 3D convolution to combine multi-scale features. The multi-scale information obtained by SSFF is then combined with the P3 branch, and a Channel and Position Attention Mechanism (CPAM) is introduced to leverage high-order multi-scale features and detailed features. This allows the model to dynamically focus on more important feature channels and spatial regions, improving its ability to capture fine-grained features and adaptability to complex scenes. The resulting N3 is fed into the detection head along with N4 and N5 for final detection.

[0049] Since computing resources on mobile robot platforms are often limited, it is crucial to keep the model lightweight while ensuring the accuracy of the target detection algorithm. To this end, this patent adopts the LAMP pruning method, which uses layer-adaptive amplitude pruning to score, as shown in the following formula.

[0050]

[0051] like but score(u;W) is the pruning score of node u under the weight set W, W[u] is the weight value of node u in the weight set W, and W[v] is the weight value of node v in the weight set W.

[0052] This pruning method can more precisely customize the sparsity of each layer, rather than relying on fixed sparsity settings or manual adjustments. This adaptability can better preserve important connections in the network while removing redundant weights, significantly improving the sparsity of the model without sacrificing performance, thereby reducing computing resource consumption and improving model efficiency.

[0053] (4) The GrabCut algorithm is used to extract the outline of the object, remove the background information, and map the point cloud data corresponding to the object outline in the camera depth map to the grid map in real time: Set the target detection box parameter to a rectangular area When the algorithm is initialized, the area outside the detection frame is defined as the determined background, and the area inside the frame is the possible foreground, that is:

[0054] n is the pixel, α n Used to identify whether pixel n is within the detection box R, α n =0 means pixel n is outside the detection frame, α n =1 means pixel n is inside the detection box.

[0055] Perform K-means clustering on the pixels in the box and calculate the mean of each cluster and covariance :

[0056] Where, is the i-th cluster, is the number of pixels in the i-th cluster, is the RGB color vector, μ k,i is the mean of the i-th cluster, Σ k,i is the covariance of the ith cluster. For each pixel , calculate the probability of it belonging to the foreground (k=1) or background (k=0):

[0057] Where I is the total number of clusters, i is used to identify each specific cluster, k is a conditional parameter that can only be 0 or 1, k=0 means the background area is considered, k=1 means the foreground area is considered, is the label of the n-th pixel.

[0058] The Gaussian distribution Defined as:

[0059] Σ is the covariance matrix of the Gaussian distribution, |Σ| is the determinant of the covariance matrix Σ, Σ -1is the inverse matrix of the covariance matrix Σ, exp represents the exponential function, and μ is a general symbol representing the mean of the Gaussian distribution.

[0060] Minimize the energy function by graph cuts , its data term and smoothing term can be expanded as follows:

[0061] In the formula, the data item is , for each pixel Calculate the likelihood that its color value belongs to the current label The smoothing term is , if two adjacent pixels m and n have similar colors ( →0) and the distance is close ( →1), the smoothing term value approaches γ, tending to assign the same label. α is the category to which pixel n is assigned, γ is the weight of the smoothing term, and σ is the standard deviation parameter in the smoothing term. is the sum of all adjacent pixel pairs (m,n), C represents the set of adjacent pixel pairs, is the denominator of the smoothness term, representing the spatial distance between pixels m and n.

[0062] The segmentation result mask M is a binary image that satisfies:

[0063] M(x,y) represents the pixel value at position (x,y). It is a binary image segmentation result mask, where x and y are the pixel positions in the image.

[0064] Point cloud data in the depth map Filter by mask:

[0065] Where i is the index of a single point in the point cloud data, for point The depth value of the point The distance from the camera in 3D space, and for point The x and y coordinates in two-dimensional space.

[0066] (5) Coordinate transformation process: The coordinate transformation relationship between the camera coordinate system and the robot coordinate system can be expressed as:

[0067] Where R cr With T crThey are the rotation matrix and translation vector of the camera coordinate system relative to the robot coordinate system, respectively. Their forms depend on the specific installation position of the camera on the robot. r 、Y r 、Z r Indicates the three-dimensional coordinate value in the robot coordinate system, X c 、Y c 、Z c Represents the three-dimensional coordinate value in the camera coordinate system.

[0068] The robot system used in this patent includes:

[0069] t x , t y , t z They represent the translation in the x, y, and z directions respectively. They determine the position of the origin of the camera coordinate system in the robot coordinate system.

[0070] The coordinate transformation relationship between the robot coordinate system and the world coordinate system can be expressed as:

[0071] Where Xw, Yw, and Zw represent the three-dimensional coordinate values ​​in the world coordinate system, Rwr and Twr are the rotation matrix and translation vector of the world coordinate system relative to the robot coordinate system, and the position and posture of the robot at time t are obtained by the SLAM algorithm. Decision: For this patent:

[0072] Where t is the time moment, which is used to identify the posture of the robot at different time points. In robot navigation and positioning, the posture of the robot changes with time, so the time moment t is used to distinguish the posture states at different time points.

[0073] Finally, the object semantic information obtained by the robot is converted into the grid coordinate system:

[0074] Where, Represents the coordinates of the semantic object in the world coordinate system, Represents the coordinates of the semantic object in the grid coordinate system, int represents the rounding function, which is used to convert floating-point numbers into integers, and R is the grid map resolution, which is set to 0.05m in this patent.

[0075] (6) Multi-source observation fusion method based on Bayesian probability framework: Assume the robot pose sequence is x 1:t={x1, x2, …, xt}, represents the robot’s pose set at t time steps. The sensor dataset is z 1:t ={z1, z2, …, zt}, represents the data set collected by the sensor at t time steps. Then the semantic map m can be represented as each semantic grid unit m i The joint probability distribution of :

[0076] This formula expresses the probability of a semantic map m being the probability of all semantic grid cells mi in a given robot pose sequence x 1:t and sensor dataset z 1:t The product of the probabilities under the conditions.

[0077] Based on Bayesian estimation theory, this patent establishes a dynamic semantic grid state update model. For each semantic grid unit, the probability of obstacle presence and absence is calculated: The obstacle probability calculation formula is:

[0078] This formula is used to calculate the probability p(x1:t,z1:t) of an obstacle in semantic grid cell mi. p(mi|zt,xt) is the probability that an obstacle exists in mi given the robot pose xt and sensor data zt at the current time t; p(zt|xt) is the probability that the sensor observes zt at the current time t given the robot pose xt; p(mi|z1:t-1,x1:t-1) is the probability that an obstacle exists in mi given the robot pose and sensor data from the previous t-1 time steps. p(mi) is the prior probability that an obstacle exists in mi; p(zt|z1:t-1,x1:t) is the probability that zt is observed at the current time t given the sensor data from the previous t-1 time steps and the robot pose from t time steps.

[0079] The calculation formula for non-obstacle probability is:

[0080] Where, is the probability that there is no obstacle in mi given the robot pose xt and sensor data zt at the current time t; is the probability that there is no obstacle in mi given the robot pose and sensor data in the previous t-1 time steps; is the prior probability that there is no obstacle in mi; is the probability that there is no obstacle in mi given the robot pose and sensor data in the first t time steps.

[0081] Comparing the two and taking their logarithms gives:

[0082] Simply written as:

[0083] Where, is the semantic grid state at the current moment; The semantic grid state at the previous moment; is the inverse observation model of the sensor; Represents the prior probability. At the initial moment, the occupied probability and idle probability of the semantic grid are both 0.5, so .

[0084] Step 4: Scene map generation: After obtaining the multi-level semantic map, the scene information and object information are combined to generate a scene map of the indoor environment.

[0085] The present invention proposes that the scene map is based on the semantic map, integrates the objects detected in each scene and their spatial distribution information, and selects objects with stable positions and high significance as marker objects to construct the core reference points of the scene. The role of the marker object is to provide an accurate positioning reference for the robot, which can help the robot to quickly achieve positioning. In addition, the scene map also records the relative spatial relationship between the marker object and the surrounding associated objects. In the process of scene map construction, the object category, position, quantity and distribution characteristics of each scene are marked in detail and associated with the scene information to form a multi-level, multi-dimensional indoor scene semantic model. Taking the marker object as the benchmark, other objects around the marker object are regarded as its associated objects. In specific implementation, they can be clearly distinguished and displayed. For example, the marker object is represented by a red circle in the scene map, and other associated objects are represented by gray circles.

[0086] The implementation method of constructing a scene graph based on a multi-level semantic map proposed in the embodiment is as follows: On the basis of constructing a multi-level semantic map, for each type of scene (such as offices, corridors, etc.), the frequency of occurrence of various objects in this type of scene is counted, and the objects with the highest appearance frequency and the strongest spatial position stability are selected as the landmark objects of this type of scene. These objects are used to construct the core reference points of the scene and provide a stable and recognizable positioning basis for the mobile robot.

[0087] The algorithm then traverses all associated objects within a specified range (e.g., 5 meters) around each landmark, extracting their category information and distance relationships relative to the landmark, and constructing a local semantic relationship graph centered on the landmark. Each landmark is then assigned unique identification information that combines its category, spatial location, and the categories and distance relationships of its associated objects to enhance distinguishability and recognition accuracy.

[0088] The generated scene graph, based on this semantic information, is organized and managed using a hierarchical index structure: the first-level index represents the scene category, used to distinguish different functional areas; the second-level index represents the landmark object nodes within the scene, which serve as the core reference for that area; and the third-level index represents the object nodes associated with each landmark object, representing the semantic composition of its local environment. Landmark object nodes and their associated object nodes are connected by edges representing relative distances, forming a semantic relationship graph. This structure facilitates the robot's rapid retrieval of available positioning reference objects in the scene based on task requirements, achieving efficient and accurate autonomous positioning.

[0089] For simplicity of description, the method embodiments are described as a series of actions. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because certain steps can be performed in other orders or simultaneously according to the embodiments of the present invention. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0090] In specific implementation, the method proposed in the technical solution of the present invention can be automatically run by those skilled in the art using computer software technology. System devices that implement the method, such as computer-readable storage media that store the corresponding computer program of the technical solution of the present invention and computer equipment that runs the corresponding computer program, should also be within the scope of protection of the present invention.

[0091] The following embodiments describe the electronic device provided by the present invention. The electronic device described below and the multi-level semantic map construction method based on scene recognition and target detection described above can be referenced to each other.

[0092] The electronic device may include: a processor, a communications interface, a memory, and a communications bus, wherein the processor, the communications interface, and the memory communicate with each other via the communications bus. The processor may call logic instructions in the memory to execute a multi-level semantic map construction method based on scene recognition and object detection, which mainly includes the software processing portion of the above steps.

[0093] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0094] On the other hand, an embodiment of the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the software processing part of the multi-level semantic map construction method based on scene recognition and target detection provided by the above methods.

[0095] On the other hand, an embodiment of the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the software processing part of the multi-level semantic map construction method based on scene recognition and target detection provided by the above-mentioned methods.

[0096] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0097] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A multi-level semantic map construction method based on multi-sensor fusion, characterized in that: Follow the process below: Environmental grid layer construction, including real-time construction of environmental grid maps by fusing perception data; The construction of the scene semantic layer includes extracting the semantic probability distribution of the image scene using a deep convolutional network, fusing temporal observations through Bayesian filtering, and mapping the scene categories to grid cells using an occupancy probability model; The object semantic layer is constructed, including identifying objects using a target detection network that introduces an information aggregation and distribution mechanism, extracting object point clouds by combining depth information and image segmentation, and dynamically updating the object semantic attributes of grid cells through multi-source observation fusion; Scene graph generation includes constructing a hierarchical scene graph with landmark objects as the reference core and containing spatial topological relationships based on the association between the scene semantic layer and the object semantic layer.

2. The method according to claim 1, wherein: When constructing the environment grid map in real time, the branch-and-bound algorithm is used to solve the optimal pose transformation between the laser point cloud and the sub-map; and the closed-loop error is minimized in the pose graph model.

3. The method according to claim 1, wherein: The scene semantic layer construction includes modeling scene classification as a Bayesian estimation process; and defining a scene occupancy probability model of grid cells.

4. The method according to claim 1, wherein: The target detection network that introduces the information aggregation-distribution mechanism is implemented by improving the YOLOv8 target detection algorithm, including realizing multi-scale feature aggregation and distribution through the information aggregation-distribution mechanism, introducing an attention scale sequence fusion strategy, and performing structured pruning based on layer adaptive amplitude.

5. The method according to claim 4, characterized in that: The multi-scale feature aggregation and distribution is achieved through the information aggregation-distribution mechanism, and the implementation method includes setting Low-GD branch fusion to extract and fuse large-scale feature maps, setting High-GD branches to extract and fuse small-scale feature maps, and setting a feature alignment module, a feature information fusion module and a feature information distribution module in each branch.

6. The method according to claim 1, wherein: The multi-source observation fusion is implemented by representing the semantic map as a joint probability distribution of each semantic grid unit based on the robot pose sequence and sensor data set. The probability of obstacle existence and non-existence is calculated for each semantic grid unit based on the Bayesian probability framework to achieve dynamic semantic grid state update.

7. The method according to claim 1, wherein: When constructing a scene graph, landmark objects are screened based on the frequency of appearance and position stability of objects in the scene, and a local semantic relationship graph is established with the landmark objects as the center; a three-level hierarchical index structure is constructed, with the first-level index being the scene category label, the second-level index being the landmark object nodes, and the third-level index being the object nodes associated with each landmark object. The nodes are connected by relative distance edges to form a semantic relationship graph structure; the semantic relationship graph structure supports the robot to quickly retrieve positioning reference objects in the scene according to task requirements.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the multi-level semantic map construction method based on scene recognition and target detection as described in any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-level semantic map construction method based on scene recognition and target detection as described in any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program, characterized in that: When the computer program is executed by a processor, the multi-level semantic map construction method based on scene recognition and target detection as described in any one of claims 1 to 7 is implemented.

Images