Robot semantic map construction method based on multi-modal large model and graph RAG

Abstract

The invention discloses a robot semantic map construction method based on a multi-modal large model and a graph RAG. The method comprises two main links of constructing a priori semantic database and constructing a semantic map. The method comprises the following steps: firstly, screening an internet picture related to a robot task through a multi-modal large model, extracting semantic information of an object, and constructing a prior semantic database; then, when the current environment data is processed, an enhanced cue word is generated by utilizing a graph RAG technology, the multi-modal large model is guided to analyze semantic information of the current environment, and a semantic map containing detailed semantic information is constructed; according to the method, by introducing the prior semantic database and the multi-modal graph RAG technology, the accuracy and reliability of the semantic map are effectively improved, meanwhile, richer semantic information is provided, and the method can adapt to various complex environments and task requirements.

Description

Technical Field

[0001] The present invention relates to the field of robotics, and in particular to SLAM technology, multimodal large model technology and multimodal graph RAG technology of mobile robots, and in particular to a robot semantic map construction method based on multimodal large model and graph RAG. Background Art

[0002] In the field of mobile robotics, semantic map construction is one of the key technologies to achieve autonomous navigation and intelligent interaction of robots. Traditional semantic map construction methods mainly rely on geometric information, visual features or auxiliary guidance information. Specifically:

[0003] Methods based on geometric information: Use 3D laser radar, ultrasonic ranging sensors and other distance sensors to scan the surrounding environment and generate a point cloud map. After using filtering algorithms to remove noise and non-interest areas, use algorithms such as point cloud segmentation and clustering to divide the point cloud into different areas and assign a semantic label to each area. However, such methods can only capture the geometric structure of semantic objects and cannot obtain other important information such as texture and color. In addition, for unstructured complex environments, such as open spaces indoors, point clouds may be difficult to segment, resulting in poor results for this method.

[0004] Methods based on visual features: Use images taken by visual sensors such as monocular cameras, binocular cameras, RGBD cameras, etc. as input. After preprocessing such as denoising, dedistortion, and image enhancement, use semantic segmentation and classification algorithms to obtain image semantic information and add it to the corresponding map area to form a semantic map. During the map construction process, computer vision algorithms or neural network models are used to extract image features from the acquired images and feature matching algorithms are used to match them to ensure that the same area under different perspectives has consistent semantic information. This type of method is greatly affected by image quality. For scenes with drastic changes in lighting or camera movement and vibration, it is easy to have blurred images and other situations where semantic information cannot be extracted; in addition, for low-resolution images, some key semantic information may be lost.

[0005] Methods based on auxiliary guidance information: This type of method uses additional data provided by human users to build semantic maps. Common data includes QR codes containing semantic information pasted at specific locations in the scene. The mobile robot will recognize these QR codes and record their semantic information in the corresponding areas of the map during the map construction process. Or use human wearable devices to record the user's activity trajectory and behavior, and associate them with predefined semantic tags to form a semantic map. However, it requires a lot of manpower and time to collect data; at the same time, the behavior of different participants may have errors and uncertainties, which will affect the accuracy of the semantic map.

[0006] Method of introducing multimodal large models: This method introduces a multimodal large model to replace the original algorithm when processing image, trajectory and other data. For example, when identifying objects in an image or classifying objects, a large model is used to replace the object recognition and classification algorithm, and a detailed description of the target object is directly obtained with appropriate prompt words. These descriptions can be added to the map as object-level semantic information. This type of method is greatly affected by the hallucination of the multimodal large model. When the large model gives wrong information, it cannot be effectively corrected; in addition, the effect of the large model strongly depends on the quality of the prompt words, and the prompt words need to be continuously optimized and adjusted so that the large model can give comprehensive and accurate semantic information in most scenarios.

[0007] Comprehensive methods: This type of method combines multiple of the above solutions and uses semantic fusion, data alignment and other algorithms to fuse semantic maps from different sources to achieve better results. For example, 3D lidar and camera images can be used to build semantic maps separately, and then corrected by combining human prior information. In the process of fusing semantic maps from different sources, this type of method may cause semantic conflicts or inconsistencies, requiring a lot of experiments to adjust the fusion algorithm and repeatedly check to achieve the best results.

[0008] Therefore, the existing technology has many deficiencies in the accuracy and reliability of semantic maps, and it is difficult to meet the needs of mobile robots in complex tasks. Obviously, the method has problems such as incomplete semantic information, low reliability, reliance on manual intervention and insufficient fusion efficiency. Summary of the invention

[0009] Based on this, an embodiment of the present application provides a robot semantic map construction method based on a multimodal large model and graph RAG, which can improve the intelligence level of mobile robots in complex environments.

[0010] In a first aspect, a method for constructing a robot semantic map based on a multimodal large model and a graph RAG is provided, the method comprising constructing a priori semantic database and constructing a semantic map, wherein:

[0011] The construction of the prior semantic database specifically includes:

[0012] Generate search keywords based on the task description, collect Internet pictures, remove duplicates through feature similarity, and use a multimodal large model to verify whether the pictures meet the environmental requirements, pre-process the pictures that meet the environmental requirements, and process the pictures through multiple models to obtain semantic information; wherein the multiple models include at least a semantic segmentation model, a depth estimation model, and an object detection model;

[0013] Use a multi-modal large model to extract the semantic information to generate a detailed semantic description, verify the consistency between the segmented region and the detailed semantic description, after the consistency verification passes, vectorize the semantic information, construct an object-centered semantic subgraph, and finally merge similar subgraphs and store them in the database to form a prior semantic database with environmental associations;

[0014] The construction of the semantic map specifically includes:

[0015] Generate a dense point cloud map based on SLAM, segment it into blocks according to a preset size, and associate the corresponding image data. Perform semantic segmentation, depth point cloud registration, and object detection on the pictures of each map block, generate a preliminary semantic description in combination with a multi-modal large model, retrieve the prior semantic database through vector similarity, combine the object co-occurrence subgraph and the current detection results, generate an enhanced prompt word containing a list of alternative objects, use the enhanced prompt word to extract and verify the semantic information, vectorize the results and then construct an object subgraph with environmental coordinates, and finally integrate all map blocks to form a robot semantic map.

[0016] Optionally, generate search keywords based on the task description, collect Internet pictures, remove duplicates through feature similarity, and use a multi-modal large model to verify whether the pictures meet the environmental requirements, specifically including:

[0017] For the obtained pictures, first use a depth model based on a CNN convolutional network to extract picture features, and perform a cosine similarity comparison with the picture feature library stored in the database. If the similarity is greater than the threshold, it is judged as a duplicate picture and discarded, and a new picture is collected again;

[0018] If it is a non-duplicate picture, use a multi-modal large model to identify and verify whether it meets the task and environmental descriptions. If it meets, proceed to the next step and store the picture features already extracted from the current picture. If it does not meet, discard the current picture and collect it again.

[0019] Optionally, preprocess the pictures that meet the environmental requirements, and obtain semantic information by processing the pictures through multiple models, including:

[0020] Process the pictures into a unified preset size: for pictures larger than the preset size, crop and retain the central region of the picture; for pictures smaller than the preset size, use linear interpolation to enlarge them to the preset size, where pictures smaller than 80% of the preset size will be discarded;

[0021] For each picture, use the following depth models to process the pictures and temporarily store the processing results respectively:

[0022] Use an image semantic segmentation model to obtain all masks of the picture;

[0023] Use a monocular image depth estimation model to obtain the depth information of the picture and convert it into a point cloud; use a point cloud segmentation model to obtain the bounding boxes of each part after segmentation;

[0024] Use a multi-modal large model to describe the picture and obtain the descriptions of the objects in the picture;

[0025] Use an object detection model to obtain the categories and bounding boxes of the objects in the picture.

[0026] Optionally, construct an object-centered semantic subgraph, and finally merge the similar subgraphs and store them in the database to form a prior semantic database with environmental associations, including:

[0027] Traverse all the objects in the picture, and respectively establish subgraphs with the semantic vectors of each object as the central nodes. The sub-nodes of the subgraph are the semantic vectors of other objects that appear in the same picture, and their edges are the number of times the sub-node objects appear; when adding a sub-node, if the object category of the newly added sub-node is the same as that of an existing sub-node, only retain the existing sub-node and add one to the number recorded in the edge to obtain a subgraph that describes the frequency of other objects appearing around each object. This subgraph characterizes the environment around each object;

[0028] Traverse all the sub-nodes in the subgraph, check the object categories of the sub-nodes. If they have the same object category as the sub-nodes of an existing stored subgraph, add the values of the corresponding edges; otherwise, add the sub-node and the edge to the merged subgraph and store it; if there is no same central node object category, store it directly.

[0029] Optionally, after constructing an object-centered semantic subgraph, finally merging the similar subgraphs and storing them in the database to form a prior semantic database with environmental associations, it also includes:

[0030] According to a preset threshold, judge whether to continue adding data;

[0031] Loop through the above steps until the amount of data added to the database reaches the preset threshold. After the loop is completed, obtain a prior semantic database containing object environment subgraphs and object semantic vectors, which can be used for the graph RAG technology in semantic map construction.

[0032] Optionally, use enhanced prompt words to extract and verify semantic information, vectorize the results, and construct an object subgraph with environmental coordinates. Finally, integrate all the map tiles to form a robot semantic map, including:

[0033] Use a multi-modal large model to analyze each regional image, and describe the semantic information of all the objects contained therein, including position, category, shape, and texture information, and save the semantic information of each object in the form of a Chinese character string;

[0034] Vectorize the object categories, query the top k objects with the highest similarity in the prior semantic database according to the cosine similarity, generate enhanced prompt words based on the object shape and texture description, and use the multi-modal large model to verify whether the description of each object is accurate according to this prompt word;

[0035] Vectorize the complete image, the image region described by the position part in the semantic information, and the semantic information string respectively as the semantic vectors of each object, and store the above data; among them, the semantic vector is used as the key value to index the information of the object;

[0036] Traverse all objects in the image, and establish a subgraph with the semantic vector of each object as the central node. The sub-nodes of the subgraph are the semantic vectors of other objects that appear in the same point cloud map block, and their edges are the central coordinates of the point cloud map block;

[0037] Store all subgraphs with the central coordinates of the current point cloud map block as the index;

[0038] Check whether all map blocks have been traversed. If the traversal is not completed, continue to process the next map block;

[0039] Loop through the above steps until all map blocks are traversed. After the loop is completed, obtain the database of the current environment as the semantic map, and finally integrate all map blocks to form the robot semantic map.

[0040] In a second aspect, a robot semantic map construction system based on a multi-modal large model and Graph RAG is provided. The system includes a prior semantic database construction module and a semantic map construction module, where:

[0041] The prior semantic database construction module is specifically used for:

[0042] Generate search keywords based on the task description, collect Internet images, remove duplicates through feature similarity, and use the multi-modal large model to verify whether the images meet the environmental requirements. Preprocess the images that meet the environmental requirements, and process the images through multiple models to obtain semantic information; among them, the multiple models at least include a semantic segmentation model, a depth estimation model, and an object detection model;

[0043] Use the multi-modal large model to extract the semantic information to generate a detailed semantic description, verify the consistency between the segmentation region and the detailed semantic description, and after the consistency verification passes, vectorize the semantic information, construct a semantic subgraph centered on the object, and finally merge the similar subgraphs and store them in the database to form a prior semantic database with environmental associations;

[0044] The semantic map construction module is specifically used for:

[0045] Generate a dense point cloud map based on SLAM, segment it into blocks according to a preset size, and associate the corresponding image data. Perform semantic segmentation, depth point cloud registration, and object detection on the pictures of each map block. Combine with a multimodal large model to generate a preliminary semantic description. Retrieve the prior semantic database through vector similarity, combine the object co-occurrence subgraph and the current detection results to generate an enhanced prompt word containing a list of alternative objects. Use the enhanced prompt word to extract and verify semantic information, vectorize the results, and construct an object subgraph with environmental coordinates. Finally, integrate all map blocks to form a robot semantic map.

[0046] In a third aspect, an electronic device is provided, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the robot semantic map construction method described in any one of the first aspects above is implemented.

[0047] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the robot semantic map construction method described in any one of the first aspects above is implemented.

[0048] In a fifth aspect, a computer program product is provided, including a computer program / instructions. When the computer program / instructions are executed by a processor, the robot semantic map construction method described in any one of the first aspects above is implemented.

[0049] The beneficial effects brought by the technical solutions provided in the embodiments of the present application at least include:

[0050] (1) Due to the introduction of the multimodal large model and the prior semantic database, more scenarios can be covered. For example, in an environment with poor imaging quality, the generated enhanced prompt word ensures that the multimodal large model can infer the possible semantic information in the current environment based on low-quality pictures and similar scenarios in the prior semantic database.

[0051] (2) Compared with other simple large model introduction schemes, the multiple verifications of the enhanced prompt word in this scheme can effectively alleviate the large model hallucination and improve the accuracy of the semantic information provided by the large model.

[0052] (3) The constructed semantic map based on the database contains detailed object-level semantic descriptions and object surrounding environment information. Compared with most map-based semantic maps, it can provide richer information, and this information is stored in the database in a structured format and can be directly applied to various large model-related tasks, including large model-based language-vision navigation, knowledge graphs, etc., without the need for more processing steps. Description of the Drawings

[0053] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only exemplary, and for those of ordinary skill in the art, without creative efforts, other implementation drawings can also be obtained based on the provided drawings.

[0054] Figure 1 Schematic diagram of the process for constructing a prior semantic database provided by an embodiment of the present application;

[0055] Figure 2 Schematic diagram of the process for constructing a semantic map provided by an embodiment of the present application;

[0056] Figure 3 Block diagram of a robot semantic map construction system provided by an embodiment of the present application;

[0057] Figure 4 Schematic diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0058] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the following further details the present application in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] In the description of the present invention, the terms "include", "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily limit to the clearly listed steps or units, but may also include other steps or units inherent to these processes, methods, products or devices that are not clearly listed, or steps or units added based on the further optimized solutions conceived by the present invention.

[0060] In the field of mobile robot technology, semantic map construction is one of the key technologies for realizing autonomous navigation and intelligent interaction of robots. Traditional semantic map construction methods mainly rely on geometric information, visual features or auxiliary guidance information. Specifically:

[0061] 1.1 Methods based on geometric information

[0062] This type of method uses range sensors such as 3D lidar and ultrasonic ranging sensors to scan the surrounding environment and generate a point cloud map. After using filtering algorithms to remove noise and non-concerned areas, point cloud segmentation, clustering and other algorithms are used to divide the point cloud into different regions, and a semantic label is assigned to each region.

[0063] 1.2 Methods based on visual features

[0064] This type of method uses images taken by visual sensors such as monocular cameras, binocular cameras, and RGBD cameras as input. After preprocessing such as denoising, dedistortion, and image enhancement, semantic segmentation and classification algorithms are used to obtain image semantic information and add it to the corresponding map area to form a semantic map. During the map construction process, computer vision algorithms or neural network models are used to extract image features from the acquired images and feature matching algorithms are used to match them to ensure that the same area under different perspectives has consistent semantic information.

[0065] 1.3 Methods based on auxiliary guidance information

[0066] This type of method uses additional data provided by human users to build semantic maps. Common data includes QR codes containing semantic information pasted at specific locations in the scene. The mobile robot will recognize these QR codes and record the corresponding areas of their semantic information in the map during the map construction process. Or use human wearable devices to record the user's activity trajectory and behavior, and associate them with predefined semantic tags to form a semantic map.

[0067] 1.4 Methods for Introducing Multimodal Large Models

[0068] This type of method introduces a multimodal large model to replace the original algorithm when processing image, trajectory and other data. For example, when identifying objects in an image or classifying objects, a large model is used to replace the object recognition and classification algorithm, and a detailed description of the target object is directly obtained with appropriate prompt words. These descriptions can be added to the map as object-level semantic information.

[0069] 1.5 Comprehensive approach

[0070] This type of method combines multiple of the above solutions and uses semantic fusion, data alignment and other algorithms to fuse semantic maps from different sources to achieve better results. For example, 3D lidar and camera images can be used to build semantic maps separately, and then combined with human prior information for correction.

[0071] The above existing methods have the following disadvantages:

[0072] 2.1 Objective shortcomings of methods based on geometric information

[0073] This type of method can only capture the geometric structure of semantic objects and cannot obtain other important information such as texture, color, etc.; and for unstructured complex environments, such as open spaces indoors, point clouds may be difficult to segment, resulting in poor results for this method.

[0074] 2.2 Objective shortcomings of methods based on visual features

[0075] Such methods are greatly affected by image quality. In scenarios with drastic changes in lighting, camera movement, vibration, etc., it is easy to have situations where the image is blurred and semantic information cannot be extracted; in addition, for low-resolution images, some key semantic information may be lost.

[0076] 2.3 Objective Disadvantages of the Method Based on Auxiliary Guidance Information

[0077] Such methods require a large amount of human and time costs to collect data; at the same time, the behaviors of different participants may have errors and uncertainties, which will affect the accuracy of the semantic map.

[0078] 2.4 Method of Introducing Multimodal Large Models

[0079] Such methods are greatly affected by the hallucinations of multimodal large models. When the large model gives incorrect information, it cannot be effectively corrected; in addition, the effect of the large model strongly depends on the quality of the prompt words, and the prompt words need to be continuously optimized and adjusted to enable the large model to give comprehensive and accurate semantic information in most scenarios.

[0080] 2.5 Objective Disadvantages of the Method Based on Comprehensive Method

[0081] In the process of fusing semantic maps from different sources, such methods may have semantic conflicts or inconsistencies, and a large number of experiments are required to adjust the fusion algorithm and check repeatedly to obtain the best effect.

[0082] The present invention is a method for constructing a semantic map of a mobile robot based on a multimodal large model and multimodal graph RAG. The purpose of the present invention is to improve the reliability, accuracy, and usability of the semantic map, provide more comprehensive semantic information for the mobile robot in embodied intelligence, improve the intelligence level of the mobile robot in complex tasks, and can provide data support for other robot-related tasks based on multimodal large models, such as visual language navigation, etc. This method consists of two links. First, a prior semantic database constructed from prior data is constructed, and then when processing the current map data, graph RAG technology is used to extract relevant data from the prior semantic database to form an enhanced prompt word. The multimodal large model uses this prompt word to analyze and reason the semantic information of the current environmental data, and constructs the current environmental database as the semantic map. The semantic information of the map includes the category, location, shape, and texture information of all objects in the scene. In addition to saving these semantic information, the database will also save other multimodal data, including graphs describing the environment around the objects. Since there is a large amount of prior data as a reference and the multimodal large model is introduced to process and verify the data in multiple links, the quality of the semantic map can be improved.

[0083] To implement the method of this patent, the system hardware required for collecting on-site environmental data includes a mobile robot equipped with sensors such as cameras and radars, or a data collection device equipped with sensors such as cameras and radars and necessary components such as batteries. The mobile robot includes necessary components such as a computer host and a mobile robot chassis, and may include auxiliary modules such as an IMU and a collision detection module. The system hardware required for running the algorithm includes a local or cloud server. In addition, if a commercial large model is called, the system also requires the API KEY of the commercial large model and a necessary network environment.

[0084] The following separately describes the two steps involved in the invention: a) constructing a prior semantic database; b) constructing a current environment database as a semantic map:

[0085] Constructing the prior semantic database specifically includes generating search keywords based on task descriptions, collecting Internet images, removing duplicates through feature similarity, and using a multi-modal large model to verify whether the images meet the environmental requirements. Preprocess the images that meet the environmental requirements, and process the images through multiple models to obtain semantic information; among them, the multiple models at least include a semantic segmentation model, a depth estimation model, and an object detection model;

[0086] Use a multi-modal large model to extract the semantic information to generate a detailed semantic description, verify the consistency between the segmentation region and the detailed semantic description. After the consistency verification passes, vectorize the semantic information, construct an object-centered semantic subgraph, and finally merge similar subgraphs and store them in the database to form a prior semantic database with environmental associations;

[0087] Please refer to Figure 1 , which shows a schematic diagram of the process of constructing a prior semantic database provided by an embodiment of the present application, and specifically may include the following steps:

[0088] (1) Collect images

[0089] Describe in detail the tasks that the mobile robot may perform and the relevant environment, use a large model to generate search keywords and search. For the obtained images, first use a depth model based on a CNN convolutional network to extract image features, and compare them with the image feature library stored in the database through cosine similarity. If the similarity is greater than the threshold, it is determined as a duplicate image and discarded, and new images are collected again. If it is a non-duplicate image, use a multi-modal large model to identify and verify whether it meets the task and environment descriptions. If it meets, proceed to the next step and store the image features already extracted from the current image. If it does not meet, discard the current image and collect again.

[0090] Examples of the core part of the prompt words used during the search:

[0091] ……

[0092] - Target -

[0093] The current robot is: {Robot description}, and it needs to perform the following tasks in the {Related environment description} environment: {Task description}. Please search for a picture that meets this requirement.

[0094] - Steps -

[0095] 1. Analyze the robot description and the environment description

[0096] 2. Analyze the task description

[0097] 3. Generate search keywords

[0098] 4. Search and return the picture

[0099] ……

[0100] Examples of the core part of the prompt used for verification:

[0101] ……

[0102] - Objectives -

[0103] Please describe the current picture and verify whether the environment in this picture meets the requirements for the robot {Robot description} to perform tasks in the {Related environment description} environment: {Task description}.

[0104] - Steps -

[0105] 1. Analyze the robot description and the environment description

[0106] 2. Analyze the task description

[0107] 3. Describe the current picture

[0108] 4. Return whether it meets the requirements and describe the reasons

[0109] ……

[0110] (2) Image preprocessing

[0111] Process the picture into a unified preset size: For pictures larger than the preset size, crop and retain the central area of the picture; for pictures smaller than the preset size, use linear interpolation to enlarge them to the preset size, and pictures smaller than 80% of the preset size will be discarded.

[0112] (3) Process the picture using multiple models

[0113] For each picture, use the following deep models to process the picture and temporarily store the processing results:

[0114] a. Use an image semantic segmentation model to obtain all masks of the picture;

[0115] b. Obtain the depth information of the picture using a monocular image depth estimation model and convert it into a point cloud. Use a point cloud segmentation model to obtain the bounding boxes of each part after segmentation;

[0116] c. Describe the picture using a multimodal large model to obtain the descriptions of the objects in the picture. Examples of the core parts of the prompts used:

[0117] ……

[0118] - Target -

[0119] This is a picture of a {description of the relevant environment} environment. Please describe the content.

[0120] - Steps -

[0121] 1. Describe whether the picture belongs to an indoor, outdoor or other environment

[0122] 2. Describe the lighting conditions of the picture

[0123] 3. Describe whether the picture contains glass

[0124] 4. Describe the objects contained in the picture and their detailed shape and texture features

[0125] 5. Describe the positions of the objects in the picture

[0126] ……

[0127] d. Use an object detection model to obtain the categories and bounding boxes of the objects in the picture.

[0128] (4) Extract semantic information using a multimodal large model

[0129] Use a multimodal large model to synthesize the results in steps (3).c and (3).d, and for each object contained, describe its semantic information, including position, category, shape and texture information. Save the semantic information of each object as a Chinese character string. Examples of the core parts of the prompts used:

[0130] ……

[0131] - Target -

[0132] This is a picture of a {description of the relevant environment} environment. It is known that the following objects may exist in the picture: {results of step (3).c}{results of step (3).d} Please describe the content of the picture.

[0133] - Steps -

[0134] 1. Describe whether the picture belongs to an indoor, outdoor or other environment

[0135] 2. Describe the lighting conditions of the picture

[0136] 3. Describe whether the picture contains glass

[0137] 4. Describe the objects contained in the picture and their detailed shapes and texture features

[0138] 5. Describe the positions of the respective objects in the picture

[0139] ……

[0140] (5) Semantic information verification

[0141] For each region corresponding to a mask in step (3).a and each region in the picture mapped by the depth information for each bounding box in (3).b, verify whether it contains an object and that the object is included in the result of step (4). Example of the core part of the prompt used:

[0142] ……

[0143] - Target -

[0144] This is a complete picture of an environment of {description of the relevant environment} and a picture containing a partial region thereof. The following objects may exist in the known picture: {result of step (4)}. Please describe the content of the region.

[0145] - Steps -

[0146] 1. Describe whether the picture belongs to an indoor, outdoor or other environment

[0147] 2. Describe the lighting conditions of the picture

[0148] 3. Describe whether the picture contains glass

[0149] 4. Describe whether the region contains an object? If it contains an object, please describe its detailed shape and texture features

[0150] 5. Describe the position of the object in the picture

[0151] 6. Please describe whether the object is included in the list of known objects

[0152] ……

[0153] Manually check the results returned by the large model again to ensure accuracy.

[0154] (6) Semantic information vectorization

[0155] For each object in the result of step (5), vectorize the complete picture, the picture region described by the position part in the semantic information, and the semantic information string respectively as the semantic vector of each object. Store the above data, where the semantic vector is used as the key value to index the information of the object.

[0156] (7)Construct object subgraphs

[0157] Traverse all objects in the image. For each object, build a subgraph with its semantic vector as the central node. The sub-nodes of the subgraph are the semantic vectors of other objects that appear in the same image, and their edges are the number of times the sub-node objects appear. When adding a sub-node, if the object category of the newly added sub-node is the same as that of an existing sub-node, only retain the existing sub-node and increment the count recorded in the edge by one. After this step, subgraphs that describe the frequency of other objects appearing around each object will be obtained, and these subgraphs characterize the possible environments around each object.

[0158] (8)Store object semantic vectors and subgraphs and merge subgraphs of the same object

[0159] Store the object semantic vectors in the prior semantic database. Check the object category of the central node of each subgraph. If it is the same as the object category of the central node of an existing stored subgraph in the database, then merge the subgraphs. The method is as follows: Traverse all sub-nodes in the subgraph, check the object category of the sub-nodes. If it has the same object category as a sub-node of an existing stored subgraph, then add the values of the corresponding edges; otherwise, add the sub-node and the edge to the merged subgraph and store it. If there is no same object category of the central node, then store it directly.

[0160] (9)Judge whether to continue adding data

[0161] Judge whether to continue adding data according to a preset threshold.

[0162] Loop through the above steps until the amount of data added to the database reaches the preset threshold. After the loop is completed, a prior semantic database containing object environment subgraphs and object semantic vectors is obtained, which can be used for the graph RAG technology during semantic map construction.

[0163] The construction of the semantic map specifically includes: generating a dense point cloud map based on SLAM, dividing it into blocks according to a preset size, and associating the corresponding image data. Perform semantic segmentation, depth point cloud registration, and object detection on the images of each map block, generate a preliminary semantic description by combining a multi-modal large model, retrieve the prior semantic database through vector similarity, combine the object co-occurrence subgraph and the current detection results to generate an enhanced prompt word containing a list of alternative objects, use the enhanced prompt word to extract and verify semantic information, vectorize the results and construct an object subgraph with environmental coordinates, and finally integrate all map blocks to form a robot semantic map.

[0164] Please refer to Figure 2 , which shows a schematic diagram of the process for constructing a semantic map provided by an embodiment of the present application, and specifically may include the following steps:

[0165] (1)SLAM mapping

[0166] Based on the recorded images and point cloud data in the current environment, use SLAM technology to construct a dense point cloud map.

[0167] (2) Filter data

[0168] Set the map segmentation unit size according to the image resolution and the camera field of view. Divide the dense point cloud map into several blocks according to this size. For each block of the point cloud map, manually select several pictures from the image data that can cover all the objects in this block of the map. The subsequent steps in each loop will be based on each block of the point cloud map and its related pictures.

[0169] (3) Process pictures using multiple models

[0170] For each block of the point cloud map and its related pictures, use the following depth models to process and temporarily store the processing results respectively:

[0171] a. Use an image semantic segmentation model to obtain all masks of the picture;

[0172] b. Use a monocular image depth estimation model to obtain the depth information of the picture and convert it into a point cloud. Use a point cloud segmentation model to obtain the bounding boxes of each part after segmentation;

[0173] c. For each block of the point cloud map, filter outlier points, and then use a point cloud segmentation model to obtain the bounding boxes of each part after segmentation;

[0174] d. Use a point cloud registration model to register the point cloud obtained from the monocular image depth estimation in step b above with the single-block map point cloud, calculate the IoU of the bounding boxes in step b and step c above after registration, and consider the parts with IoU greater than the threshold as the same object, and merge their bounding boxes to complete deduplication;

[0175] e. Use a multimodal large model to describe the picture and obtain the descriptions of the objects in the picture. Examples of the core parts of the prompts used:

[0176] ……

[0177] - Target -

[0178] This is a picture of the {current environment description} environment. Please describe the content in it.

[0179] - Steps -

[0180] 1. Describe whether the picture belongs to an indoor, outdoor or other environment

[0181] 2. Describe the lighting conditions of the picture

[0182] 3. Describe whether the picture contains glass

[0183] 4. Describe the objects contained in the picture, their detailed shapes, and texture features

[0184] 5. Describe the positions of the objects in the picture

[0185] ……

[0186] f. Use an object detection model to obtain the categories and bounding boxes of the objects in the picture.

[0187] (4) Vectorize the regions of the image to be inferred and the object description information

[0188] For the mask in step (3).a, obtain the image of the corresponding region of the mask; for the bounding box obtained in step (3).d, find the image of the corresponding region according to the point cloud and depth information in step (3).b; for the bounding box in step (3).f, obtain the image of the corresponding region; call the Embedding API of the multimodal large model to vectorize the above images, and vectorize the description of the object in step (3).e and the object category in step (3).f.

[0189] (5) Use the Graph RAG technology to compose enhanced prompt words

[0190] Based on the results of step (4), query the following data in the prior semantic database to obtain a list of alternative objects:

[0191] For each image vector in step (4), find the top k region image vectors with the highest similarity in the prior semantic database according to the cosine similarity, and obtain their object categories as the alternative object list 1;

[0192] Take each object in the alternative object list 1 as the central node, query the top k child nodes with the largest edge values in its subgraph, and obtain the object categories of these child nodes as the alternative object list 2;

[0193] Take the object categories in step (3).e and the object categories in step (3).f as the alternative object list 3;

[0194] For each object category vector in step (4), find the top k object semantic information vectors with the highest similarity in the prior semantic database according to the cosine similarity, and obtain their object categories as the alternative object list 4.

[0195] Merge the above four lists of alternative objects and remove duplicate object categories, and generate enhanced prompt words according to this list. Example of the core part of the prompt words:

[0196] ……

[0197] - Target -

[0198] This is a complete picture of the {related environment description} environment and a picture containing some of its areas. The following objects may be present in the picture: {list of alternative objects}. Please describe the content of the picture.

[0199] - Steps -

[0200] 1. Describe whether the picture belongs to an indoor, outdoor, or other environment

[0201] 2. Describe the lighting conditions of the picture

[0202] 3. Describe whether the picture contains glass

[0203] 4. Describe the objects contained in the picture and their detailed shape and texture features

[0204] 5. Describe the positions of the objects in the picture

[0205] ……

[0206] (6) Use a multimodal large model to extract semantic information based on enhanced prompts

[0207] Based on the enhanced prompts obtained in step (5), use a multimodal large model to analyze each regional image. For all objects contained therein, describe their semantic information, including position, category, shape, and texture information. Save the semantic information for each object in the form of a Chinese character string.

[0208] (7) Generate enhanced prompts and verify the semantic information

[0209] For each object in the result of step (6), vectorize its object category and query the top k objects with the highest similarity in the prior semantic database according to the cosine similarity. Generate enhanced prompts based on the object shape and texture description. Use the multimodal large model to verify whether the description of each object in step (6) is accurate according to this prompt. Example of the core part of the prompt used:

[0210] ……

[0211] - Objectives -

[0212] This is a picture of the {related environment description} environment. For the objects in the picture: {object categories in step (6)}, please determine whether the following descriptions are accurate and give reasons:

[0213] Description 1: {shape and texture description in step (6)};

[0214] Description 2: {shape and texture description of the 1st object retrieved};

[0215] ……

[0216] Description k+1: {The k-th shape and texture description obtained by query}

[0217] - Steps -

[0218] 1. Describe whether the picture belongs to an indoor, outdoor or other environment

[0219] 2. Describe the lighting condition of the picture

[0220] 3. Describe whether the picture contains glass

[0221] 4. Locate the target object

[0222] 5. Compare each description in detail

[0223] 6. Judge whether the description matches the target and give reasons

[0224] ……

[0225] (8) Semantic information vectorization

[0226] For each object in the result of step (7), vectorize the complete picture, the picture area described by the position part in the semantic information, and the semantic information string respectively as the semantic vector of each object. Store the above data, where the semantic vector is used as the key value to index the information of the object.

[0227] (9) Construct object subgraphs

[0228] Traverse all objects in the picture, and establish subgraphs with the semantic vectors of each object as the central nodes respectively. The sub-nodes of the subgraphs are the semantic vectors of other objects that appear in the same point cloud map block, and their edges are the central coordinates of the point cloud map block. This will construct a subgraph describing the object and its surrounding environment.

[0229] (10) Store subgraphs

[0230] Store all subgraphs with the central coordinates of the current point cloud map block as the index.

[0231] (11) Judge whether to continue adding data

[0232] Check whether all map blocks have been traversed. If the traversal is not completed, continue to process the next map block.

[0233] Loop through the above steps until all map blocks are traversed. After the loop is completed, obtain the database of the current environment as the semantic map.

[0234] In summary, the key points of this application are as follows:

[0235] The present invention is a semantic map construction method based on a multimodal large model and multimodal graph RAG technology:

[0236] a) Prior semantic database constructed based on Internet image data: Use a multimodal large model to screen Internet images for specific environments and tasks, and then construct a prior semantic database that contains object-level semantic vectors and graphs describing the environment around the object. This database can be used for multimodal graph RAG tasks related to semantic maps.

[0237] b) Generation of semantic map task prompts based on multimodal graph RAG technology: Use multimodal graph RAG technology to generate enhanced and optimized prompts, which include reliable object-level prior semantic data that has been verified by the large model multiple times.

[0238] c) Construction of semantic maps based on multimodal graph RAG technology: Based on multimodal graph RAG technology, use a multimodal large model to verify semantic information multiple times to ensure its reliability and accuracy. According to the verified semantic information, construct a database containing semantic vectors, graphs describing the environment, and other multimodal data as a semantic map. This database can be used for other multimodal graph RAG tasks.

[0239] Compared with the prior art, due to the introduction of the multimodal large model and the prior semantic database, this solution can cover more scenarios. For example, in an environment with poor imaging quality, the enhanced prompts generated in steps 2.2(5) and 2.2(7) ensure that the multimodal large model can infer the possible semantic information in the current environment based on low-quality images and similar scenarios in the prior semantic database.

[0240] Compared with other solutions that simply introduce large models, the enhanced prompts in this solution and the multiple verifications in steps 2.2(6) and 2.2(7) can effectively alleviate the large model hallucination and improve the accuracy of the semantic information provided by the large model.

[0241] The semantic map based on the database constructed in 2.2 of this solution contains detailed object-level semantic descriptions and information about the environment around the object. Compared with most map-based semantic maps, it can provide richer information, and this information is stored in the database in a structured format and can be directly applied to various large model-related tasks, including large model-based language-vision navigation, knowledge graphs, etc., without the need for more processing steps.

[0242] Please refer to Figure 3 which shows a block diagram of a robot semantic map construction system provided by an embodiment of the present application. As Figure 3 shown, the system may include:

[0243] The module for constructing the prior semantic database is specifically used for:

[0244] Generate search keywords based on task descriptions, collect Internet images, remove duplicates through feature similarity, and use a multimodal large model to verify whether the images meet the environmental requirements. Preprocess the images that meet the environmental requirements, and process the images through multiple models to obtain semantic information. Among them, the multiple models at least include a semantic segmentation model, a depth estimation model, and an object detection model.

[0245] Use a multimodal large model to extract the semantic information to generate a detailed semantic description, verify the consistency between the segmented region and the detailed semantic description. After the consistency verification passes, vectorize the semantic information, construct an object-centered semantic subgraph, and finally merge the similar subgraphs and store them in the database to form a prior semantic database with environmental associations.

[0246] The module for constructing the semantic map is specifically used for:

[0247] Generate a dense point cloud map based on SLAM, segment it into blocks according to a preset size, and associate the corresponding image data. Perform semantic segmentation, depth point cloud registration, and object detection on the images of each map block. Combine with a multimodal large model to generate a preliminary semantic description. Retrieve the prior semantic database through vector similarity. Combine the object co-occurrence subgraph and the current detection results to generate an enhanced prompt word containing a list of alternative objects. Use the enhanced prompt word to extract and verify the semantic information. After vectorizing the results, construct an object subgraph with environmental coordinates, and finally integrate all map blocks to form a robot semantic map.

[0248] For the specific limitations of the robot semantic map construction system based on the multimodal large model and Graph RAG, reference can be made to the limitations of the robot semantic map construction method based on the multimodal large model and Graph RAG in the above text, which will not be elaborated here. Each module in the above robot semantic map construction system based on the multimodal large model and Graph RAG can be implemented in whole or in part by software, hardware, and their combinations. The above modules can be embedded in the processor of the computer device in hardware form or be independent of it, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0249] In one embodiment, an electronic device is provided. The electronic device can be a computer, and its internal structure diagram can be as Figure 4As shown in the figure. The electronic device includes a processor, a memory, and a network interface connected through a system bus. Among them, the processor of the device is used to provide computing and control capabilities. The memory of the device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to construct data for the robot semantic map based on the multimodal large model and the graph RAG. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it realizes a method for constructing a robot semantic map based on the multimodal large model and the graph RAG.

[0250] Those skilled in the art can understand that the structure shown in Figure 4 is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0251] In one embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, covering all or part of the processes in the method of the above embodiment.

[0252] In one embodiment, a computer program product is also provided, including a computer program / instructions, covering all or part of the processes in the method of the above embodiment.

[0253] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in M forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Symchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0254] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0255] The above embodiments only represent several implementation manners of the present application. The description is relatively specific and detailed, but it cannot be understood as a limitation to the scope of the patent application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A robot semantic map construction method based on a multimodal large model and graph RAG, characterized in that: The method comprises constructing a priori semantic database and constructing a semantic map, wherein: The construction of the prior semantic database specifically includes: Generate search keywords based on the task description, collect Internet pictures, remove duplicates through feature similarity, and use a multimodal large model to verify whether the pictures meet the environmental requirements, pre-process the pictures that meet the environmental requirements, and process the pictures through multiple models to obtain semantic information; wherein the multiple models include at least a semantic segmentation model, a depth estimation model, and an object detection model; The semantic information is extracted using a multimodal large model to generate a detailed semantic description, and the consistency between the segmented area and the detailed semantic description is verified. After the consistency verification is passed, the semantic information is vectorized to construct a semantic subgraph centered on the object, and finally similar subgraphs are merged and stored in a database to form a priori semantic database with environmental associations; The construction of the semantic map specifically includes: Based on SLAM, a dense point cloud map is generated, divided into blocks according to a preset size, and the corresponding image data is associated. Semantic segmentation, deep point cloud registration and object detection are performed on the image of each map block. A preliminary semantic description is generated by combining a multimodal large model. The prior semantic database is retrieved through vector similarity. The object co-occurrence sub-graph and the current detection results are combined to generate enhanced prompt words containing a list of candidate objects. The enhanced prompt words are used to extract and verify semantic information. The results are vectorized to construct an object sub-graph with environmental coordinates. Finally, all map blocks are integrated to form a robot semantic map.

2. The robot semantic map construction method according to claim 1, characterized in that: Generate search keywords based on the task description, collect Internet images, remove duplicates through feature similarity, and use a multimodal large model to verify whether the images meet the environmental requirements, including: For the obtained pictures, we first use a deep model based on CNN convolutional network to extract picture features, and compare the cosine similarity with the picture feature library stored in the database. If the similarity is greater than the threshold, it is judged as a duplicate picture and discarded, and a new picture is collected; If it is a non-repeated image, a multimodal large model is used for recognition to verify whether it meets the task and environment description. If so, proceed to the next step and store the extracted image features of the current image. If not, discard the current image and re-collect it.

3. The robot semantic map construction method according to claim 1, characterized in that: Preprocess the images that meet the requirements of the environment and obtain semantic information through multiple models, including: Process the images to a uniform preset size: For images larger than the preset size, crop and retain the central area of ​​the image; for images smaller than the preset size, use linear interpolation to enlarge them to the preset size, and images smaller than 80% of the preset size will be discarded; For each image, use the following deep models to process the image and temporarily store the processing results: Use the image semantic segmentation model to obtain all the masks of the image; Use the monocular image depth estimation model to obtain the image depth information and convert it into a point cloud; use the point cloud segmentation model to obtain the bounding box of each part after segmentation; Use a large multimodal model to describe the image and obtain descriptions of each object in the image; Use the object detection model to obtain the category and bounding box of the object in the image.

4. The robot semantic map construction method according to claim 1, characterized in that: Construct object-centered semantic subgraphs, merge similar subgraphs and store them in the database to form a priori semantic database with environmental associations, including: Traverse all objects in the image, and establish subgraphs with the semantic vector of each object as the central node. The subnodes of the subgraph are the semantic vectors of other objects that appear in the same image, and their edges are the number of times the subnode objects appear. When adding a subnode, if the object category of the newly added subnode is the same as that of an existing subnode, only the existing subnode is retained and the number of times recorded in the edge is increased by one, so as to obtain a subgraph that describes the frequency of other objects appearing around each object. This subgraph represents the environment around each object. Traverse all child nodes in the subgraph and check the object category of the child node. If it has the same object category as a child node of a stored subgraph, add the values ​​of the corresponding edges; otherwise, add the child node and edge to the merged subgraph and store it; if there is no identical central node object category, store it directly.

5. The robot semantic map construction method according to claim 1, characterized in that: After constructing object-centered semantic subgraphs, merging similar subgraphs and storing them in the database to form a priori semantic database with environmental associations, the following are also included: According to the preset threshold, determine whether to continue adding data; The loop is executed until the amount of data added to the database reaches a preset threshold. After the loop is completed, a priori semantic database containing object environment subgraphs and object semantic vectors is obtained, which can be used for graph RAG technology when constructing semantic maps.

6. The robot semantic map construction method according to claim 1, characterized in that: The enhanced prompt words are used to extract and verify semantic information, and the results are vectorized to construct object sub-maps with environmental coordinates. Finally, all map blocks are integrated to form a robot semantic map, including: Use a multimodal large model to analyze each regional image, describe the semantic information of all objects contained therein, including location, category, shape and texture information, and save the semantic information of each object in the form of Chinese character strings; Vectorize the object categories and search for the top k objects with the highest similarity in the prior semantic database based on cosine similarity. Generate enhanced prompt words based on the object shape and texture description. Use the multimodal large model to verify whether the description of each object is accurate based on the prompt words. The complete image, the image area described by the position part in the semantic information, and the semantic information string are respectively vectorized as the semantic vector of each object, and the above data are stored; wherein the semantic vector is used as a key value to index the information of the object; Traverse all objects in the image, and build subgraphs with the semantic vector of each object as the central node. The subnodes of the subgraph are the semantic vectors of other objects that appear in the same point cloud map block, and their edges are the central coordinates of the point cloud map block. Use the center coordinates of the current point cloud map block as the index to store all sub-maps; Check whether all map blocks have been traversed. If the traversal is not completed, continue to process the next map block; The loop is executed until all map blocks are traversed. After the loop is completed, the database of the current environment is obtained as the semantic map, and finally all map blocks are integrated to form the robot semantic map.

7. A robot semantic map construction system based on a multimodal large model and graph RAG, characterized in that: The system includes a module for constructing a priori semantic database and a module for constructing a semantic map, wherein: The module for constructing a priori semantic database is specifically used for: Generate search keywords based on the task description, collect Internet pictures, remove duplicates through feature similarity, and use a multimodal large model to verify whether the pictures meet the environmental requirements, pre-process the pictures that meet the environmental requirements, and process the pictures through multiple models to obtain semantic information; wherein the multiple models include at least a semantic segmentation model, a depth estimation model, and an object detection model; The semantic information is extracted using a multimodal large model to generate a detailed semantic description, and the consistency between the segmented area and the detailed semantic description is verified. After the consistency verification is passed, the semantic information is vectorized to construct a semantic subgraph centered on the object, and finally similar subgraphs are merged and stored in a database to form a priori semantic database with environmental associations; The module for constructing a semantic map is specifically used for: Based on SLAM, a dense point cloud map is generated, divided into blocks according to a preset size, and the corresponding image data is associated. Semantic segmentation, deep point cloud registration and object detection are performed on the image of each map block. A preliminary semantic description is generated by combining a multimodal large model. The prior semantic database is retrieved through vector similarity. The object co-occurrence sub-graph and the current detection results are combined to generate enhanced prompt words containing a list of candidate objects. The enhanced prompt words are used to extract and verify semantic information. The results are vectorized to construct an object sub-graph with environmental coordinates. Finally, all map blocks are integrated to form a robot semantic map.

8. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images