An indoor multi-scene semantic map construction method based on a semantic relation graph

Abstract

The application discloses a kind of indoor multi-scene semantic map construction method based on semantic relation graph, comprising: based on laser radar using grid map algorithm constructs 2D grid map;Using binocular camera obtains scene image, obtains depth map by stereo matching and identifies object in environment using trained target detection model, obtains the class and coordinate of landmark, image data adopts bayesian filtering technology to combine prior information and carries out scene classification, obtains the semantic label of current scene, using the semantic label obtained using the semantic map of occupancy grid method;Based on target identification dataset constructs prior relation graph, fuses space relation knowledge graph and prior relation knowledge graph to construct graph layer, obtains multi-scene semantic map.The semantic relation map of the present application based on deep learning and knowledge graph construction stores a large amount of prior knowledge, gives map comprehensive and accurate environmental information, can realize quick search in multi-scene rescue and daily service.

Description

Technical Field

[0001] This invention relates to the field of robot target search technology, specifically to a method for constructing an indoor multi-scene semantic map based on semantic relationship graphs. Background Technology

[0002] Currently, robots have gradually developed into multifunctional systems integrating environmental perception, decision-making and planning, and behavior control, showing strong development momentum in fields such as disaster relief, smart homes, and medical care.

[0003] In the fields of robotic rescue and home services, target search is a highly significant task. For example, rescue robots need to locate survivors in the post-disaster environment, and service robots need to identify and locate targets when providing services to users. Robots face numerous challenges in performing these search tasks, among which constructing information-rich semantic maps is a key challenge for achieving rapid target search. In existing technologies, some semantic map construction methods have achieved certain results in various fields. For instance, Chinese patent CN117146841A discloses a semantic map construction method based on a combination of LiDAR and binocular vision. Combining the advantages of LiDAR and binocular vision, it can obtain both rich semantic information and accurate geometric information. Another Chinese patent CN116740710A discloses a semantic map construction method for a subway maintenance workshop based on an improved YOLOv5 model. This method inserts an improved YOLOv5 model into the ORB-SLAM2 algorithm to extract two-dimensional semantic information from color images, uses depth images acquired by an RGB-D camera to obtain a three-dimensional point cloud map, and fuses the two-dimensional semantic information with the three-dimensional point cloud map to construct the semantic map, thus improving the ability to construct multi-angle semantic maps. However, in rescue and home service search tasks, existing semantic maps cannot effectively solve the time cost problem caused by the search, and the efficiency of target search needs to be improved. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a method for constructing indoor multi-scene semantic maps based on semantic relation graphs. The semantic map constructed by this method stores a large amount of prior information and uses semantic relation graphs to associate prior knowledge and semantic information, giving the map more comprehensive and accurate environmental information and improving target search efficiency.

[0005] The technical solution adopted in this invention is: a method for constructing indoor multi-scene semantic maps based on semantic relationship graphs, comprising the following steps:

[0006] S1. Construct a 2D grid map based on LiDAR using a grid map construction algorithm;

[0007] S2. Use a binocular camera to acquire image data, apply Bayesian filtering to the image data and combine it with prior information to classify the scene and obtain scene semantic labels, detect and locate landmark objects in the image data and obtain landmark semantic labels; then, based on the constructed 2D grid map, construct a semantic map using the occupancy grid method with the scene semantic labels and landmark semantic labels.

[0008] S3. Based on scene semantic label information and landmark semantic label information, construct a spatial knowledge graph in the form of triples, and construct a prior relation graph using object detection dataset and common sense information; integrate the spatial knowledge graph and the prior relation graph to form a multi-scene semantic map based on the semantic relation graph.

[0009] As a further improvement of the present invention, in step S2, the construction of the semantic map using the occupancy grid method specifically involves:

[0010] S21. Scene semantic label information is predicted by the classification model trained on the scene classification dataset. Using the correlation of image sequences, Bayesian filtering technology is used in combination with prior information to classify the scene, and the maximum semantic label value obtained by classification is assigned to the grid of the LiDAR area in the field of view of the visual sensor.

[0011] S22. Landmark semantic label information is acquired through a target detection network model trained on a target detection dataset and a stereo camera. This includes the landmark category and the three-dimensional coordinates of the landmark. The planar coordinates parallel to the 2D grid in the three-dimensional coordinates are matched with the 2D grid map, and the grid is assigned a value using the maximum semantic label value.

[0012] As a further improvement of the present invention, in step S3, the construction of the spatial relationship knowledge graph specifically involves:

[0013] S310. Construct a spatial topology graph using the space after scene classification. The topology graph is represented in the form of triples (entity-relationship-entity), and the triple form is: <Scene 1, Connectivity, Scene 2>.

[0014] S311. Construct a relationship between landmarks and scenes according to the form of triples (entity-relationship-entity), where the triple is: <landmark, location, scene>;

[0015] S312. Represent the landmark information in the form of a triple (entity-attribute-attribute value), where the triple is in the form of: <landmark, coordinates, coordinate value>.

[0016] S313. The triples from steps S310, S311, and 312 are used to construct a spatial relationship knowledge graph.

[0017] As a further improvement of the present invention, in step S3, the construction of the prior relation knowledge graph specifically involves:

[0018] S320. Divide the object detection dataset into two categories: landmark objects and target objects.

[0019] Landmark objects: W = {w0, w1, w2, w3, w4, w5...}, totaling S.

[0020] Target objects: T = {t0, t0, t2, t3, t4, t5...}, a total of M objects.

[0021] Among them, landmark objects are fixed objects or easily detectable objects in the scene, and target objects are moving objects or small objects in the scene;

[0022] S321. Traverse all images in the dataset and calculate the relationship values ​​between the target object and the landmark object:

[0023]

[0024] In the formula, P sm Landmark object W s With target object T m The relationship values ​​are: Co(,) represents the number of times the landmark object and the target object co-occur, and Ap() represents the number of times the target object appears in the dataset.

[0025] S322. Based on the constructed knowledge graph ontology, construct a relational knowledge graph from the calculated relationship values ​​between targets and landmarks in the form of triples. The triples are:

[0026] <Target, Relationship Value, Landmark>;

[0027] S323. Construct a spatial knowledge graph of objects based on the patterns in the space they occupy. The triples are:

[0028] <Object, belonging to the scene class>.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] The map constructed in this invention stores a large amount of prior information and uses a semantic relation graph to associate prior knowledge and semantic information, giving the map more comprehensive and accurate environmental information. By connecting scene information, landmark information, and prior information through a knowledge graph, indoor robots can use the map constructed in this invention to understand the environment when facing complex and diverse problems in rescue and service scenarios, thereby completing tasks quickly and efficiently. The target prior relation graph constructed in this invention can provide the location probability of target objects in rescue search and service target search tasks, thereby improving target search efficiency. Attached Figure Description

[0031] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] Figure 1 This is a flowchart of the indoor multi-scene semantic map construction method based on semantic relationship graph of the present invention;

[0033] Figure 2 This is a semantic topology diagram of an indoor environment according to an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram of a priori relation graph according to an embodiment of the present invention;

[0035] Figure 4 This is a semantic map diagram of one embodiment of the present invention. Detailed Implementation

[0036] like Figure 1 The diagram shows a flowchart of the indoor multi-scene semantic map construction method based on semantic relationship graph of the present invention. The method includes the following steps:

[0037] Step S1: Construct a 2D grid map based on the LiDAR using a grid map construction algorithm.

[0038] Step S2: Acquire image data using a binocular camera. Perform scene classification on the image data using Bayesian filtering combined with prior information. Detect and locate landmarks in the image data. Then, based on the 2D grid map constructed in Step S1, construct a semantic map using the occupancy grid method by combining scene semantic labels and landmark semantic labels. The specific steps for constructing the semantic map using the occupancy grid method are as follows:

[0039] Scene categories are predicted by scene classification models trained on scene classification datasets. Since real-time model classification may result in misclassification, Bayesian filtering is used in conjunction with prior information to classify scenes by utilizing the correlation of image sequences. The maximum semantic label value obtained from the classification is then assigned to the grid of the LiDAR region within the field of view of the visual sensor.

[0040] Landmark information is obtained through a target detection network model trained on a target recognition dataset and a stereo camera. This includes the landmark category and its 3D coordinates. The location information of the landmark's semantic labels is matched with a 2D raster map, and the raster is assigned a value based on the maximum semantic label value. Specifically, landmark objects are fixed or easily detectable objects in the scene, such as beds, sofas, refrigerators, wardrobes, and displays. The target category information obtained by processing the image data acquired by the stereo camera through the target detection model is matched with the depth map obtained through stereo matching to obtain the landmark category and its 3D coordinates. The planar coordinates parallel to the 2D raster in the landmark's 3D coordinates are then matched with the 2D raster map, and the raster is assigned a value based on the maximum semantic label value.

[0041] Step 3: Using the scene label information and landmark label information obtained in Step 2, construct a spatial knowledge graph in the form of triples. Use the object detection dataset and common sense information to construct a prior relation graph. Finally, merge the spatial knowledge graph and the prior relation graph to form the multi-scene semantic map based on semantic relation graph proposed in this invention.

[0042] The construction of a spatial relationship knowledge graph includes the following steps:

[0043] After classifying the scenes, a spatial topology map is constructed. The constructed spatial topology map is as follows: Figure 2 As shown.

[0044] The relationship between landmarks and scenes is constructed according to the form of triples (entity-relationship-entity), and the triple form is as follows:

[0045] <Landmarks, locations, scenes>;

[0046] The information of the landmark is represented in the form of a triple (entity-attribute-attribute value), and the triple is in the following form:

[0047] <Landmark, coordinates, coordinate values>;

[0048] The aforementioned triples are used to construct a spatial relationship knowledge graph.

[0049] The construction of a priori relation knowledge graph includes the following steps:

[0050] The target recognition dataset is divided into two categories: landmark objects and target objects.

[0051] Landmark objects: W = {w0, w1, w2, w3, w4, w5...}, totaling S.

[0052] Target objects: T = {t0, t0, t2, t3, t4, t5...}, a total of M objects.

[0053] Landmark objects are fixed or easily detectable objects in the scene, such as sofas, refrigerators, wardrobes, and monitors; target objects are moving or small objects in the scene, such as mice, ashtrays, and handbags.

[0054] Iterate through all images in the dataset and calculate the relationship values ​​between the target object and the landmark objects:

[0055]

[0056] In the formula, P sm Landmark object W s With target object T m The relationship values ​​are Co(,), which represents the number of times the landmark object and the target object co-occur, and Ap(), which represents the number of times the target object appears in the dataset.

[0057] Based on the constructed knowledge graph ontology, the calculated relationship values ​​between targets and landmarks are used to construct a relational knowledge graph in the form of triples. The triples are as follows:

[0058] <Target, Relationship Value, Landmark>;

[0059] A well-constructed knowledge graph of the relationship between goals and landmarks, such as Figure 3 As shown, the central circle represents the target, the other circles represent landmarks, and the values ​​between the lines represent relationship values.

[0060] Construct an object space knowledge graph based on the patterns of the space in which the object is located. The triples are:

[0061] <Object, belonging to the scene class>.

[0062] The obtained spatial relationship knowledge graph and the constructed prior relationship knowledge graph are merged to construct the graph layer of the map.

[0063] like Figure 4 The image shown is an example of a multi-scene semantic map constructed in this invention, which mainly includes a grid layer, a semantic layer, and a graph layer.

[0064] In summary, this invention constructs a 2D grid map based on LiDAR using a grid map construction algorithm. It acquires scene images using a binocular camera, obtains depth maps using stereo matching, and uses a trained object detection model to identify objects in the environment, obtaining the categories and coordinates of landmarks. Furthermore, it uses Bayesian filtering combined with prior information to classify the scene and obtain semantic labels for the current scene. These semantic labels are then used to generate a semantic map using the occupancy grid method. Based on the object recognition dataset, a prior relation graph is constructed using the idea of ​​object co-occurrence. A spatial knowledge graph is constructed based on the obtained landmark scene relationships. The spatial relation knowledge graph and the prior relation knowledge graph are fused to construct the map's graph layer, ultimately yielding the semantic graph map of this invention.

[0065] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes that can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention are all within the protection scope of the claims of the present invention.

Claims

1. A method for constructing indoor multi-scene semantic maps based on semantic relationship graphs, characterized in that, Includes the following steps: S1. Construct a 2D grid map based on LiDAR using a grid map construction algorithm; S2. Use a binocular camera to acquire image data, apply Bayesian filtering to the image data and combine it with prior information to classify the scene and obtain scene semantic labels, detect and locate landmark objects in the image data and obtain landmark semantic labels; then, based on the constructed 2D grid map, construct a semantic map using the occupancy grid method with the scene semantic labels and landmark semantic labels. S3. Based on scene semantic label information and landmark semantic label information, construct a spatial knowledge graph in the form of triples, and construct a prior relation graph using object detection dataset and common sense information; integrate the spatial knowledge graph and the prior relation graph to form a multi-scene semantic map based on the semantic relation graph. The construction of the spatial knowledge graph specifically involves: S310. Construct a spatial topology graph using the space after scene classification. The topology graph is represented in the form of triples (entity-relationship-entity), and the triple form is: <Scene 1, Connectivity, Scene 2>. S311. Construct relationships between landmarks and scenes according to the form of triples (entity-relationship-entity), where the triple is: <landmark, location, scene>; S312. Represent the information of the landmark in the form of a triple (entity-attribute-attribute value), where the triple is in the form of: <landmark, coordinates, coordinate value>. S313. Construct a spatial relationship knowledge graph by combining the triples from steps S311, S312 and S313. The construction of the prior relation graph specifically involves: S320. Divide the object detection dataset into two categories: landmark objects and target objects. Landmark objects: There are S in total. Target object: There are M in total. Among them, landmark objects are fixed objects or easily detectable objects in the scene, and target objects are moving objects or small objects in the scene; S321. Traverse all images in the dataset and calculate the relationship values ​​between the target object and the landmark object: ， In the formula, Landmark objects With the target object Relationship values Indicates the number of times a landmark object and a target object co-occur. This indicates the number of times the target object appears in the dataset. S322. Based on the constructed knowledge graph ontology, construct a relational knowledge graph from the calculated relationship values ​​between targets and landmarks in the form of triples. The triples are: <Target, Relationship Value, Landmark>; S323. Construct a spatial knowledge graph of objects based on the patterns in the space they occupy. The triples are: <Object, belongs to, Scene class>.

2. The method for constructing indoor multi-scene semantic maps based on semantic relationship graphs according to claim 1, characterized in that, In step S2, the construction of the semantic map using the occupancy grid method specifically involves: S21. Scene semantic label information is predicted by the classification model trained on the scene classification dataset. Using the correlation of image sequences, Bayesian filtering technology is used in combination with prior information to classify the scene, and the maximum semantic label value obtained by classification is assigned to the grid of the LiDAR area in the field of view of the visual sensor. S22. Landmark semantic label information is acquired through a target detection network model trained on a target detection dataset and a stereo camera. This includes the landmark category and the three-dimensional coordinates of the landmark. The planar coordinates parallel to the 2D grid in the three-dimensional coordinates are matched with the 2D grid map, and the grid is assigned a value using the maximum semantic label value.

Images