Robot semantic map construction method based on multi-modal large model and graph rag

Abstract

The application discloses a robot semantic map construction method based on a multi-modal large model and a graph RAG. The method includes two main links: constructing a prior semantic database and constructing a semantic map. First, the multi-modal large model is used to filter Internet pictures related to the robot task, extract semantic information of objects, and construct a prior semantic database. Then, when processing current environment data, the graph RAG technology is used to generate enhanced prompt words to guide the multi-modal large model to analyze the semantic information of the current environment, and a semantic map containing detailed semantic information is constructed. The application effectively improves the accuracy and reliability of the semantic map by introducing the prior semantic database and the multi-modal graph RAG technology, and provides more rich semantic information, which can adapt to various complex environments and task requirements.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, in particular to the SLAM technology of mobile robots, the multi-modal large model technology and the multi-modal graph RAG technology, and specifically relates to a robot semantic map construction method based on multi-modal large model and graph RAG. BACKGROUND

[0002] In the field of mobile robot technology, semantic map construction is one of the key technologies to realize autonomous navigation and intelligent interaction of robots. Traditional semantic map construction methods mainly rely on geometric information, visual features or auxiliary guidance information. Specifically:

[0003] Geometric information-based method: using 3D laser radar, ultrasonic ranging sensor and other distance sensors to scan the surrounding environment and generate point cloud maps, after using filtering algorithms to remove noise and non-interest areas, using point cloud segmentation, clustering and other algorithms to divide the point cloud into different areas, and assigning a semantic label to each area. However, such methods can only capture the geometric structure of semantic objects and cannot obtain other important information such as texture, color, etc.; and for unstructured complex environments, such as open spaces indoors, point clouds may be difficult to segment, resulting in poor performance of this method.

[0004] Visual feature-based method: using monocular cameras, binocular cameras, RGBD cameras and other visual sensors to capture images as input, after pre-processing such as denoising, distortion removal, image enhancement, using semantic segmentation and classification algorithms to obtain image semantic information and add it to the corresponding map area, forming a semantic map. In the process of map construction, computer vision algorithms or neural network models are used to extract image features and match using feature matching algorithms to ensure that the same area under different perspectives has consistent semantic information. Such methods are greatly affected by image quality. For scenes with dramatic changes in lighting or camera movement, shaking, etc., image blurring may occur, making it impossible to extract semantic information; in addition, for low-resolution images, some key semantic information may be lost.

[0005] Auxiliary guidance information-based method: this method uses additional data provided by human users to construct a semantic map. Common data includes pasting QR codes containing semantic information at specific locations in the scene, and the mobile robot identifies these QR codes and records their semantic information in the corresponding area of the map during the map construction process. Or using human wearable devices to record user activity trajectories and behaviors, and associating them with predefined semantic labels to form a semantic map. However, it requires a lot of manpower and time cost to collect data; at the same time, the behaviors from different participants may have errors and uncertainties, which will affect the accuracy of the semantic map.

[0006] The method of introducing a multi-modal large model: when processing image, trajectory and other data, a multi-modal large model is introduced to replace the original algorithm. For example, when identifying objects in an image or classifying objects, a large model is used to replace the object recognition and classification algorithm, and a suitable prompt word is used to directly obtain detailed descriptions of the target object. These descriptions can be added to the map as object-level semantic information. This method is greatly affected by the multi-modal large model illusion, and when the large model gives incorrect information, it cannot be effectively corrected; in addition, the effect of the large model is strongly dependent on the quality of the prompt word, and the prompt word needs to be constantly optimized and adjusted to enable the large model to give comprehensive and accurate semantic information in most scenarios.

[0007] Comprehensive method: this kind of method combines multiple previous schemes, and uses semantic fusion, data alignment and other algorithms to fuse semantic maps from different sources to obtain better results. For example, 3D laser radar and camera images can be used to construct semantic maps respectively, and then combined with human priori information for correction. In the process of fusing semantic maps from different sources, this method may have semantic conflicts or inconsistencies, and a large number of experiments are needed to adjust the fusion algorithm and repeatedly check to obtain the best results.

[0008] Therefore, the prior art has many deficiencies in the accuracy and reliability of the semantic map, and it is difficult to meet the needs of mobile robots in complex tasks. Obviously, the method has the problems of incomplete semantic information, low reliability, dependence on manual intervention and insufficient fusion efficiency. SUMMARY

[0009] Therefore, the prior art has many deficiencies in the accuracy and reliability of the semantic map, and it is difficult to meet the needs of mobile robots in complex tasks. Obviously, the method has the problems of incomplete semantic information, low reliability, dependence on manual intervention and insufficient fusion efficiency.

[0010] In a first aspect, a robot semantic map construction method based on a multi-modal large model and a graph RAG is provided, which includes constructing a priori semantic database and constructing a semantic map, wherein:

[0011] The constructing a priori semantic database specifically includes:

[0012] Based on the task description, search keywords are generated, internet pictures are collected, feature similarity is used for deduplication, and a multi-modal large model is used to verify whether the pictures meet the environmental requirements. The pictures that meet the environmental requirements are preprocessed, and the pictures are processed by multiple models to obtain semantic information; wherein the multiple models at least include a semantic segmentation model, a depth estimation model, and an object detection model;

[0013] The semantic information is extracted using a multi-modal large model to generate a detailed semantic description, consistency of the segmented region and the detailed semantic description is verified, the semantic information is vectorized after the consistency verification is passed, a semantic subgraph centered on an object is constructed, and finally the same type of subgraphs are merged and stored in a database to form a prior semantic database associated with an environment;

[0014] The semantic map is constructed in particular as follows:

[0015] A dense point cloud map is generated based on SLAM, segmented into blocks according to a preset size, and associated with corresponding image data, the pictures of each block of the map are subjected to semantic segmentation, depth point cloud registration and object detection, a preliminary semantic description is generated in combination with a multi-modal large model, an enhanced prompt word containing an alternative object list is generated by searching a prior semantic database based on vector similarity, the semantic information is extracted and verified using the enhanced prompt word, and an object subgraph with environmental coordinates is constructed after the result is vectorized, and finally all map blocks are integrated to form a robot semantic map.

[0016] Optionally, a search keyword is generated based on a task description, internet pictures are collected, and it is verified whether the pictures meet the environmental requirements using a multi-modal large model, in particular as follows:

[0017] For the obtained pictures, first, a deep model based on a CNN convolutional network is used to extract picture features, and a cosine similarity comparison is performed with a picture feature library stored in the database, if the similarity is greater than a threshold value, the picture is judged as a repeated picture and discarded, and a new picture is collected again;

[0018] If the picture is not a repeated picture, a multi-modal large model is used for identification to verify whether it meets the task and environmental description, if it meets the requirements, the next step is performed and the picture features extracted from the current picture are stored, and if it does not meet the requirements, the current picture is discarded and a new picture is collected.

[0019] Optionally, the pictures meeting the environmental requirements are preprocessed, and the semantic information is obtained by processing the pictures using multiple models, including:

[0020] The pictures are processed to a uniform preset size: for pictures larger than the preset size, the central region of the picture is cropped and retained; for pictures smaller than the preset size, the pictures are enlarged to the preset size using linear interpolation, wherein pictures smaller than 80% of the preset size are discarded;

[0021] For each picture, the following deep models are used to process the picture and temporarily store the processing results:

[0022] An image semantic segmentation model is used to obtain all masks of the picture;

[0023] Obtain picture depth information using monocular image depth estimation model and convert to point cloud; obtain bounding box of each part after segmentation using point cloud segmentation model;

[0024] Use multi-modal large model to describe the picture and obtain the description of each object in the picture;

[0025] Use object detection model to obtain the category and bounding box of the object in the picture.

[0026] Optionally, construct an object-centered semantic subgraph, and finally merge the same type of subgraph and store it in the database to form a prior semantic database with environment association, including:

[0027] Traverse all objects in the picture, and establish a subgraph with the semantic vector of each object as the center node. The subnodes of the subgraph are the semantic vectors of other objects appearing in the same picture, and the edges are the number of times the subnode objects appear. When adding a subnode, if the object category of the newly added subnode is the same as that of an existing subnode, only the existing subnode is retained and the number recorded in the edge is incremented by one. A subgraph describing the frequency of other objects appearing around each object is obtained, which represents the environment around each object.

[0028] Traverse all subnodes in the subgraph, check the object category of the subnode, and if it has the same object category as a stored subnode, add the corresponding edge value. Otherwise, add the subnode and edge to the merged subgraph and store it. If there is no same center node object category, it is directly stored.

[0029] Optionally, after constructing an object-centered semantic subgraph and finally merging the same type of subgraph and storing it in the database to form a prior semantic database with environment association, it further includes:

[0030] According to the preset threshold, determine whether to continue adding data;

[0031] Repeat the above steps until the amount of data added to the database reaches the preset threshold. After the loop is completed, a prior semantic database containing object environment subgraphs and object semantic vectors is obtained, which can be used for graph RAG technology in semantic map construction.

[0032] Optionally, use enhanced prompt words to extract and verify semantic information, and construct an object subgraph with environment coordinates after vectorizing the results. Finally, integrate all map tiles to form a robot semantic map, including:

[0033] Use a multi-modal large model to analyze each regional image and describe the semantic information of all objects contained therein, including position, category, shape, and texture information. Save the semantic information of each object in the form of a Chinese string;

[0034] vectorize its object category, and query the top k objects with the highest similarity in the prior semantic database according to the cosine similarity, generate enhanced prompt words according to the object shape and texture description, and use a multi-modal large model to verify whether the description of each object is accurate according to the prompt words;

[0035] vectorize the complete picture, the picture area described by the position part of the semantic information, and the semantic information string as the semantic vector of each object, and store the above data; wherein the semantic vector is used as a key value to index the information of the object;

[0036] traverse all objects in the picture to establish a subgraph with the semantic vector of each object as the center node, and the subnodes of the subgraph are the semantic vectors of other objects appearing in the same point cloud map block, and the edges are the center coordinates of the point cloud map block;

[0037] store all subgraphs with the center coordinates of the current point cloud map block as an index;

[0038] check whether all map blocks have been traversed, and if not, continue processing the next map block;

[0039] loop the above steps until all map blocks are traversed, and obtain the database of the current environment as a semantic map after the loop is completed, and finally integrate all map blocks to form a robot semantic map.

[0040] In a second aspect, a robot semantic map construction system based on a multi-modal large model and a graph RAG is provided, and the system comprises a prior semantic database construction module and a semantic map construction module, wherein:

[0041] The prior semantic database construction module is specifically configured to:

[0042] generate a search keyword based on the task description, collect Internet pictures, remove duplicates through feature similarity, and use a multi-modal large model to verify whether the pictures meet the environmental requirements, preprocess the pictures that meet the environmental requirements, process the pictures through multiple models to obtain semantic information; wherein the multiple models at least include a semantic segmentation model, a depth estimation model, and an object detection model;

[0043] extract the semantic information using a multi-modal large model to generate detailed semantic descriptions, verify the consistency of the segmented areas and the detailed semantic descriptions, vectorize the semantic information after the consistency verification is passed, construct a semantic subgraph centered on the object, and finally merge the same type of subgraphs and store them in the database to form a prior semantic database associated with the environment;

[0044] The semantic map construction module is specifically configured to:

[0045] The dense point cloud map is generated based on SLAM, segmented into blocks according to a preset size, and associated with corresponding image data, semantic segmentation, depth point cloud registration and object detection are performed on the pictures of each block of the map, a preliminary semantic description is generated in combination with a multi-modal large model, a prior semantic database is searched through vector similarity, an enhanced prompt word containing an alternative object list is generated in combination with an object co-occurrence subgraph and a current detection result, semantic information is extracted and verified by using the enhanced prompt word, an object subgraph with environmental coordinates is constructed after the result is vectorized, and finally all map blocks are integrated to form a robot semantic map.

[0046] In a third aspect, an electronic device is provided, including a memory and a processor, the memory stores a computer program, and the processor implements the robot semantic map construction method of any one of the first aspect when executing the computer program.

[0047] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the robot semantic map construction method of any one of the first aspect.

[0048] In a fifth aspect, a computer program product is provided, including computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the robot semantic map construction method of any one of the first aspect.

[0049] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0050] (1) Due to the introduction of the multi-modal large model and the prior semantic database, more scenes can be covered, for example, in an environment with poor imaging quality, the generated enhanced prompt word ensures that the multi-modal large model can infer the semantic information that the current environment may contain according to the low-quality picture and the similar scene in the prior semantic database;

[0051] (2) Compared with other simple introduction of large model schemes, the multiple verifications of the enhanced prompt word of the present scheme can effectively alleviate the large model illusion and improve the accuracy of the semantic information provided by the large model;

[0052] (3) The database-based semantic map constructed contains detailed object-level semantic descriptions and object surrounding environment information, compared with most map-based semantic maps, it can provide more abundant information, and these information are stored in a structured format in the database, without the need for more processing steps, it can be directly applied to various large model related tasks, including large model-based language visual navigation, knowledge graph, etc. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings required in the embodiments or prior art description. Obviously, the drawings in the following description are only exemplary, and for those skilled in the art, other drawings can also be obtained from the provided drawings without creative labor.

[0054] Figure 1 The prior semantic database construction process schematic diagram provided for the embodiments of the present application;

[0055] Figure 2 The semantic map construction process schematic diagram provided for the embodiments of the present application;

[0056] Figure 3 The block diagram of the robot semantic map construction system provided for the embodiments of the present application;

[0057] Figure 4 The schematic diagram of an electronic device provided for the embodiments of the present application. DETAILED DESCRIPTION

[0058] In order to make the objects, technical solutions and advantages of the present application clearer, the following will further describe the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0059] In the description of the present application, the terms "comprising", "having" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not have to be limited to only those steps or units explicitly listed, but can also include other steps or units inherent to the process, method, product or device, or steps or units added based on further optimization of the inventive concept.

[0060] In the field of mobile robots, semantic map construction is one of the key technologies to realize autonomous navigation and intelligent interaction of robots. Traditional semantic map construction methods mainly rely on geometric information, visual features or auxiliary guidance information. Specifically:

[0061] 1.1 Method based on geometric information

[0062] This kind of method uses 3D laser radar, ultrasonic ranging sensor and other distance sensors to scan the surrounding environment and generate point cloud map, and after using filtering algorithm to remove noise and non-concerned areas, uses point cloud segmentation, clustering and other algorithms to divide the point cloud into different areas, and assigns a semantic label to each area.

[0063] 1.2 Method based on visual features

[0064] This kind of method uses images taken by visual sensors such as monocular cameras, binocular cameras, RGBD cameras, etc. as input. After preprocessing such as denoising, distortion correction, image enhancement, etc., semantic segmentation and classification algorithms are used to obtain image semantic information and add it to the corresponding map area to form a semantic map. In the process of map construction, computer vision algorithms or neural network models are used to extract image features from the obtained pictures, and feature matching algorithms are used for matching to ensure that the same area under different perspectives has consistent semantic information.

[0065] 1.3 Methods based on auxiliary guidance information

[0066] This kind of method uses additional data provided by human users to construct semantic maps. Common data includes pasting QR codes containing semantic information at specific locations in the scene. Mobile robots will identify these QR codes during the process of constructing maps and record their semantic information in the corresponding area of the map. Or use human wearable devices to record user activity trajectories and behaviors, and associate them with predefined semantic labels to form a semantic map.

[0067] 1.4 Methods introducing multi-modal large models

[0068] This kind of method introduces multi-modal large models to replace the original algorithms when processing image, trajectory and other data. For example, when identifying objects in images or classifying objects, large models are used to replace object recognition and classification algorithms to directly obtain detailed descriptions of target objects with appropriate prompts. These descriptions can be added to the map as object-level semantic information.

[0069] 1.5 Comprehensive methods

[0070] This kind of method combines multiple previous schemes and uses semantic fusion, data alignment and other algorithms to fuse semantic maps from different sources to achieve better results. For example, 3D laser radar and camera images can be used to construct semantic maps respectively, and then combined with human prior information for correction.

[0071] The following shortcomings exist for the above existing methods respectively:

[0072] 2.1 Objective shortcomings of methods based on geometric information

[0073] This kind of method can only capture the geometric structure of semantic objects and cannot obtain other important information such as texture, color, etc.; and for unstructured complex environments such as indoor open spaces, point clouds may be difficult to segment, which can lead to poor results of this method.

[0074] 2.2 Objective shortcomings of methods based on visual features

[0075] Such methods are greatly affected by image quality. For scenes with severe changes in lighting or camera movement, shaking, etc., it is easy to have image blur and other conditions that cannot extract semantic information; in addition, for low-resolution images, some key semantic information may be lost.

[0076] 2.3 Objective shortcomings of the method based on auxiliary guidance information

[0077] Such methods require a large amount of manpower and time cost to collect data; at the same time, the behaviors from different participants may have errors and uncertainties, which will affect the accuracy of the semantic map.

[0078] 2.4 Method of introducing a multi-modal large model

[0079] Such methods are greatly affected by the illusion of multi-modal large models. When the large model gives incorrect information, it cannot be effectively corrected; in addition, the effect of the large model is strongly dependent on the quality of the prompt words, which need to be constantly optimized and adjusted to make the large model give comprehensive and accurate semantic information in most scenarios.

[0080] 2.5 Objective shortcomings of the method based on comprehensive methods

[0081] In such methods, during the fusion of semantic maps from different sources, semantic conflicts or inconsistencies may occur, which requires a large number of experimental adjustments of the fusion algorithm and repeated checks to obtain the best results.

[0082] The present application is a method for constructing a semantic map of a mobile robot based on a multi-modal large model and a multi-modal graph RAG. The purpose of the present application is to improve the reliability, accuracy and usability of the semantic map, to provide more comprehensive semantic information for the mobile robot in embodied intelligence, to improve the intelligence level of the mobile robot in complex tasks, and to provide data support for other robot-related tasks based on multi-modal large models, such as visual language navigation. The present method comprises two steps, first constructing a prior semantic database constructed from prior data, and then using the graph RAG technology to extract relevant data from the prior semantic database to form enhanced prompt words when processing current map data, and using the prompt words to analyze and infer the semantic information of the current environment data using the multi-modal large model to construct a current environment database as a semantic map. The semantic information of the map includes the category, position, shape and texture information of all objects in the scene. In addition to saving these semantic information, the database also saves other multi-modal data including graphs describing the environment around the object. Since there is a large amount of prior data as a reference, and multi-modal large models are introduced in multiple steps to process and verify the data, the quality of the semantic map can be improved.

[0083] To implement the method of the present patent, the system hardware required for collecting on-site environmental data includes a mobile robot equipped with sensors such as cameras and radars, or a data collection device equipped with necessary components such as cameras, radars, and batteries. The mobile robot includes a computer host, a mobile robot chassis, and other necessary components, and can include auxiliary modules such as IMUs and collision detection modules. The system hardware required for running the algorithm includes servers on the local or cloud. In addition, if a commercial large model is called, the system also needs the API KEY of the commercial large model and the necessary network environment.

[0084] The following describes the two aspects involved in the invention: a) constructing a prior semantic database; and b) constructing a current environment database as a semantic map:

[0085] Constructing a prior semantic database specifically includes generating search keywords based on task descriptions, collecting Internet pictures, removing duplicates by feature similarity, and verifying whether the pictures meet the environmental requirements using a multi-modal large model. The pictures that meet the environmental requirements are pre-processed, and the semantic information is obtained by processing the pictures through multiple models. The multiple models at least include a semantic segmentation model, a depth estimation model, and an object detection model.

[0086] The semantic information is extracted using a multi-modal large model to generate detailed semantic descriptions, and the consistency of the segmented regions and the detailed semantic descriptions is verified. After the consistency verification is passed, the semantic information is vectorized, a semantic subgraph centered on an object is constructed, and finally the same type of subgraphs are merged and stored in the database to form a prior semantic database associated with the environment.

[0087] Please refer to Figure 1 which shows a process diagram for constructing a prior semantic database according to an embodiment of the present application, which can specifically include the following steps:

[0088] (1) Collecting pictures

[0089] Detailed description of the tasks that the mobile robot can perform and the related environment, using a large model to generate search keywords and search. For the obtained pictures, first extract the picture features using a depth model based on a CNN convolutional network, and compare the cosine similarity with the picture feature library already stored in the database. If the similarity is greater than a threshold, it is judged as a duplicate picture and discarded, and a new picture is collected. If it is a non-duplicate picture, a multi-modal large model is used for identification to verify whether it meets the task and environmental description. If it meets the requirements, the next step is performed and the picture features extracted from the current picture are stored. If it does not meet the requirements, the current picture is discarded and a new picture is collected.

[0090] Examples of the core part of the prompt words used in the search:

[0091] …

[0092] -Target-

[0093] The current robot is: {robot description}, which needs to perform the following tasks in the {related environment description} environment: {task description}. Please search for a picture that meets this requirement.

[0094] - Steps -

[0095] 1. Analyze the robot description and environment description

[0096] 2. Analyze the task description

[0097] 3. Generate search keywords

[0098] 4. Search and return the picture

[0099] …

[0100] Example of the core part of the prompt word used for verification:

[0101] …

[0102] - Target -

[0103] Please describe the current picture and verify whether the environment in this picture meets the requirements for the robot {robot description} to perform the task: {task description} in the {related environment description} environment.

[0104] - Steps -

[0105] 1. Analyze the robot description and environment description

[0106] 2. Analyze the task description

[0107] 3. Describe the current picture

[0108] 4. Return whether the requirements are met and describe the reasons

[0109] …

[0110] (2) Picture preprocessing

[0111] Process the picture to a uniform preset size: for pictures larger than the preset size, crop and retain the central area of the picture; for pictures smaller than the preset size, use linear interpolation to enlarge to the preset size, where pictures smaller than 80% of the preset size will be discarded.

[0112] (3) Use multiple models to process the picture

[0113] For each picture, use the following deep models to process the picture and temporarily store the processing results:

[0114] a. Use the image semantic segmentation model to obtain the full mask of the picture;

[0115] b. Obtain picture depth information using monocular image depth estimation model and convert to point cloud. Obtain bounding box of each part after segmentation using point cloud segmentation model;

[0116] c. Describe the picture using a multi-modal large model, obtain the description of each object in the picture, and the core part of the prompt word example:

[0117] …

[0118] -Target-

[0119] This is a picture of the {related environment description} environment. Please describe the content in it.

[0120] -Steps-

[0121] 1. Describe whether the picture belongs to an indoor, outdoor or other environment

[0122] 2. Describe the lighting conditions of the picture

[0123] 3. Describe whether the picture contains glass

[0124] 4. Describe the objects contained in the picture and their detailed shape and texture features

[0125] 5. Describe the position of each object in the picture

[0126] …

[0127] d. Use object detection model to obtain the category and bounding box of objects in the picture.

[0128] (4) Use multi-modal large model to extract semantic information

[0129] Use multi-modal large model to integrate the results of steps (3).c, (3).d, and describe the semantic information of each object contained therein, including position, category, shape and texture information. Save the semantic information of each object in the form of Chinese string. The core part of the prompt word used is as follows:

[0130] …

[0131] -Target-

[0132] This is a picture of the {related environment description} environment. It is known that the following objects may exist in the picture: {the results of step (3).c} {the results of step (3).d} Please describe the content of the picture.

[0133] -Steps-

[0134] 1. Describe whether the picture belongs to an indoor, outdoor or other environment

[0135] 2. Describe the lighting conditions of the picture

[0136] 3. Describe whether the picture contains glass

[0137] 4. Describe the objects contained in the picture and their detailed shape and texture features

[0138] 5. Describe the location of each object in the picture

[0139] …

[0140] (5) Semantic information verification

[0141] For each mask corresponding region in step (3).a, and each bounding box corresponding region in the picture mapped by depth information in (3).b, verify whether the region contains an object and the object is contained in the result of step (4). The prompt word core part example is used:

[0142] …

[0143] -Target-

[0144] This is a complete picture of the {related environment description} environment and a picture containing part of the region. It is known that the following objects may exist in the picture: {the result of step (4)}. Please describe the content in the region.

[0145] -Step-

[0146] 1. Describe whether the picture belongs to indoor, outdoor or other environment

[0147] 2. Describe the lighting condition of the picture

[0148] 3. Describe whether the picture contains glass

[0149] 4. Describe whether the region contains objects? If it contains objects, please describe their detailed shape and texture features

[0150] 5. Describe the location of the object in the picture

[0151] 6. Please describe whether the object is contained in the known object list

[0152] …

[0153] The results returned by the large model are manually checked again to ensure accuracy.

[0154] (6) Semantic information vectorization

[0155] For each object in the result of step (5), the complete picture, the picture region described in the location part of the semantic information, and the semantic information string are respectively vectorized as the semantic vector of each object. Store the above data, where the semantic vector is used as the key value to index the information of the object.

[0156] (7) Constructing object subgraph

[0157] All objects in the picture are traversed, and a subgraph is established for each object with the semantic vector of the object as the center node. The subnodes of the subgraph are the semantic vectors of other objects appearing in the same picture, and the edges are the number of times the subnode objects appear. When adding a subnode, if the object category of the newly added subnode is the same as that of an existing subnode, only the existing subnode is retained and the number recorded in the edge is incremented by one. After this step, a subgraph describing the frequency of other objects appearing around each object is obtained, which represents the possible environment around each object.

[0158] (8) Store object semantic vectors and subgraphs and merge the same object subgraphs

[0159] The object semantic vector is stored in the prior semantic database. The object category of the center node of each subgraph is checked, and if it is the same as the object category of the center node of a stored subgraph in the database, the subgraph is merged. The method is as follows: traverse all subnodes in the subgraph, check the object category of the subnode, and if it has the same object category as a subnode of an existing stored subgraph, add the corresponding edge value. Otherwise, add the subnode and edge to the merged subgraph and store it. If there is no same center node object category, it is directly stored.

[0160] (9) Determine whether to continue adding data

[0161] According to the preset threshold, it is determined whether to continue adding data.

[0162] The above steps are executed in a loop until the amount of data added to the database reaches the preset threshold. After the loop is completed, a prior semantic database containing object environment subgraphs and object semantic vectors is obtained, which can be used for RAG technology in semantic map construction.

[0163] The semantic map construction specifically includes: generating a dense point cloud map based on SLAM, dividing it into blocks according to a preset size, and associating with corresponding image data, performing semantic segmentation, depth point cloud registration and object detection on the pictures of each block of the map, generating a preliminary semantic description combined with a multi-modal large model, retrieving the prior semantic database through vector similarity, generating an enhanced prompt word containing a list of candidate objects combined with the object co-occurrence subgraph and the current detection result, extracting and verifying semantic information using the enhanced prompt word, and constructing an object subgraph with environmental coordinates after vectorizing the result. Finally, all map blocks are integrated to form a robot semantic map.

[0164] Please refer to Figure 2 which shows a semantic map construction process provided by an embodiment of the present application, which can specifically include the following steps:

[0165] (1) SLAM mapping

[0166] Based on the images and point cloud data recorded in the current environment, a dense point cloud map is constructed using SLAM technology.

[0167] (2) Screening data

[0168] According to the image resolution and the size of the camera field of view, the map segmentation unit size is set. The dense point cloud map is divided into several blocks according to this size. For each block of point cloud map, several pictures that can cover all objects in the block map are manually selected from the image data. The subsequent steps in each loop will be based on each block of point cloud map and its related pictures.

[0169] (3) Processing pictures using multiple models

[0170] For each block of point cloud map and related pictures, the following depth models are used to process and store the results respectively:

[0171] a. Use the image semantic segmentation model to obtain the full mask of the picture;

[0172] b. Use monocular image depth estimation model to obtain picture depth information and convert it to point cloud. Use point cloud segmentation model to obtain bounding box of each part after segmentation;

[0173] c. For each block of point cloud map, filter outliers, then use point cloud segmentation model to obtain bounding box of each part after segmentation;

[0174] d. Use point cloud registration model to register the point cloud obtained by monocular image depth estimation in the above step b with the single block map point cloud, calculate the IoU of each bounding box in the above step b and step c after registration, and consider the parts with IoU greater than the threshold as the same object, merge their bounding boxes to complete the deduplication;

[0175] e. Use multi-modal large model to describe the picture and obtain the description of each object in the picture. The core part of the prompt word is as follows:

[0176] …

[0177] -Target-

[0178] This is a picture of the {current environment description} environment. Please describe the content.

[0179] -Steps-

[0180] 1. Describe whether the picture belongs to indoor, outdoor or other environment

[0181] 2. Describe the lighting conditions of the picture

[0182] 3. Describe whether the picture contains glass

[0183] 4. Describe the objects contained in the picture and their detailed shape and texture features

[0184] 5. Describe the position of each object in the picture

[0185] …

[0186] f. Use the object detection model to obtain the category and bounding box of the object in the picture.

[0187] (4) Vectorize the image area to be inferred and object description information

[0188] For the mask in step (3).a, obtain the image of the area corresponding to the mask; for the bounding box obtained in step (3).d, find the image of the corresponding area according to the point cloud and depth information in step (3).b; for the bounding box in step (3).f, obtain the image of the corresponding area; call the multi-modal large model EmbeddingAPI, and vectorize the above image, and vectorize the description of the object in step (3).e and the object category in step (3).f.

[0189] (5) Use the graph RAG technique to form enhanced prompt words

[0190] Based on the results of step (4), query the following data in the prior semantic database to obtain a list of alternative objects:

[0191] For each image vector in step (4), find the top k region image vectors with the highest similarity in the prior semantic database according to the cosine similarity, and obtain their object categories as the first alternative object list;

[0192] Take each object in the first alternative object list as the center node, query the top k subnodes with the largest edge value in its subgraph, and obtain the object categories of these subnodes as the second alternative object list;

[0193] Take the object category in step (3).e and the object category in step (3).f as the third alternative object list;

[0194] For each object category vector in step (4), find the top k object semantic information vectors with the highest similarity in the prior semantic database according to the cosine similarity, and obtain their object categories as the fourth alternative object list.

[0195] Merge the above four alternative object lists and remove duplicate object categories, generate enhanced prompt words according to the list, and the core part of the prompt word example is:

[0196] …

[0197] -Target-

[0198] This is a complete picture of the {related environment description} environment and a picture containing part of the area. The following objects may exist in the picture: {alternative object list} Please describe the content of the picture.

[0199] - Steps -

[0200] 1. Describe whether the picture belongs to an indoor, outdoor or other environment

[0201] 2. Describe the lighting conditions of the picture

[0202] 3. Describe whether the picture contains glass

[0203] 4. Describe the objects contained in the picture and their detailed shape and texture characteristics

[0204] 5. Describe the location of each object in the picture

[0205] …

[0206] (6) Extract semantic information using multi-modal large model based on enhanced prompt

[0207] Based on the enhanced prompt obtained in step (5), use a multi-modal large model to analyze each region image. For all objects contained therein, describe their semantic information, including location, category, shape and texture information. Save the semantic information of each object in the form of a Chinese string.

[0208] (7) Generate enhanced prompts and verify semantic information

[0209] For each object in the results of step (6), vectorize its object category and query the top k objects with the highest similarity in the prior semantic database according to cosine similarity. According to the object shape and texture description, generate an enhanced prompt. Use a multi-modal large model to verify whether the description of each object in step (6) is accurate according to this prompt. Examples of the core part of the prompt used:

[0210] …

[0211] - Goal -

[0212] This is a picture of the {related environment description} environment. For the objects in the picture: {object categories in step (6)}, please determine whether the following descriptions are accurate and give reasons:

[0213] Description 1: {shape and texture description in step (6)};

[0214] Description 2: {the first shape and texture description obtained by querying};

[0215] …

[0216] Description k+1: {kth query profile and texture description};

[0217] -Step-

[0218] 1. Describe the picture belongs to indoor, outdoor or other environment

[0219] 2. Describe the picture lighting conditions

[0220] 3. Describe whether the picture contains glass

[0221] 4. Find the target object

[0222] 5. Detailed comparison of each description

[0223] 6. Determine whether the description is consistent with the target and give reasons

[0224] …

[0225] (8) Semantic information vectorization

[0226] For each object in the result of step (7), vectorize the complete picture, the picture area described by the position part of the semantic information, and the semantic information string as the semantic vector of each object. Store the above data, with the semantic vector as the key value, for indexing the information of the object.

[0227] (9) Construct object subgraph

[0228] Traverse all objects in the picture, and establish a subgraph with the semantic vector of each object as the center node. The child nodes of the subgraph are the semantic vectors of other objects appearing in the same point cloud map block, and their edges are the center coordinates of the point cloud map block. This will construct a subgraph describing the object and its surrounding environment.

[0229] (10) Store subgraph

[0230] Store all subgraphs with the center coordinates of the current point cloud map block as the index.

[0231] (11) Determine whether to continue adding data

[0232] Check if all map blocks have been traversed, and if not, continue processing the next map block.

[0233] Loop the above steps until all map blocks are traversed. After the loop is completed, the database of the current environment is obtained as a semantic map.

[0234] As can be seen from the above, the key point of the present application is:

[0235] The present application is a semantic map construction method based on multi-modal large model and multi-modal graph RAG technology:

[0236] a) Prior semantic database constructed based on internet picture data: using multimodal large model to filter internet pictures of specific environment and task, then constructing a prior semantic database containing object-level semantic vector and graph describing the environment around the object, which can be used for multimodal graph RAG task related to semantic map.

[0237] b) Semantic map task prompt word generation based on multimodal graph RAG technology: using multimodal graph RAG technology to generate enhanced and optimized prompt words, which include reliable object-level prior semantic data verified by large model multiple times.

[0238] c) Semantic map construction based on multimodal graph RAG technology: based on multimodal graph RAG technology, using multimodal large model to verify semantic information multiple times to ensure its reliability and accuracy. According to the verified semantic information, a database containing semantic vector, graph describing the environment and other multimodal data is constructed as a semantic map, which can be used for other multimodal graph RAG tasks.

[0239] Compared with the prior art, the present scheme can cover more scenarios due to the introduction of multimodal large model and prior semantic database. For example, in the environment with poor imaging quality, the enhanced prompt words generated in steps 2.2(5) and 2.2(7) ensure that the multimodal large model can infer the semantic information that the current environment may contain according to the low-quality picture and the similar scene in the prior semantic database.

[0240] Compared with other schemes that simply introduce large models, the enhanced prompt words of the present scheme and the multiple verifications in steps 2.2(6) and 2.2(7) can effectively alleviate the large model hallucination and improve the accuracy of the semantic information provided by the large model.

[0241] The database-based semantic map constructed in 2.2 of the present scheme contains detailed object-level semantic description and environment information around the object. Compared with most map-based semantic maps, it can provide more abundant information, and these information are stored in a structured format in the database, which can be directly applied to various large model related tasks without the need for more processing steps, including large model-based language visual navigation, knowledge graph, etc.

[0242] Please refer to Figure 3 , which shows a block diagram of a robot semantic map construction system based on multimodal large model and graph RAG provided by an embodiment of the present application. As Figure 3 shown, the system can include:

[0243] The construction prior semantic database module is specifically configured to:

[0244] The task description is used to generate a search keyword, collect Internet pictures, remove pictures through feature similarity, and verify whether the pictures meet the environmental requirements by using a multi-modal large model. The pictures meeting the environmental requirements are preprocessed, and semantic information is obtained by processing the pictures through multiple models. The multiple models at least include a semantic segmentation model, a depth estimation model, and an object detection model.

[0245] A multi-modal large model is used to extract the semantic information to generate a detailed semantic description, and the consistency of the segmentation area and the detailed semantic description is verified. After the consistency verification is passed, the semantic information is vectorized, a semantic subgraph centered on an object is constructed, and finally the same type of subgraph is merged and stored in a database to form a prior semantic database associated with an environment.

[0246] The semantic map construction module is specifically used for:

[0247] A dense point cloud map is generated based on SLAM, segmented into blocks according to a preset size, and associated with corresponding image data. The pictures of each block of the map are subjected to semantic segmentation, depth point cloud registration, and object detection. A multi-modal large model is used to generate a preliminary semantic description. The prior semantic database is searched through vector similarity. An enhanced prompt word containing a list of alternative objects is generated by combining the object co-occurrence subgraph and the current detection result. The semantic information is extracted and verified by using the enhanced prompt word. The result is vectorized to construct an object subgraph with environmental coordinates. Finally, all map blocks are integrated to form a robot semantic map.

[0248] The specific limitations of the robot semantic map construction system based on the multi-modal large model and the graph RAG can be referred to the limitations of the robot semantic map construction method based on the multi-modal large model and the graph RAG in the foregoing, which will not be repeated here. Each module in the robot semantic map construction system based on the multi-modal large model and the graph RAG can be realized by software, hardware, or a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0249] In one embodiment, an electronic device, which can be a computer, is provided. The internal structure diagram of the electronic device can be as shown in Figure 4As shown. The electronic device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the device is used to provide computing and control capabilities. The memory of the device includes a non-volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to construct the robot semantic map data based on the multi-modal large model and the graph RAG. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement a robot semantic map construction method based on a multi-modal large model and a graph RAG.

[0250] Those skilled in the art can understand that the structure shown in Figure 4 the above embodiments is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0251] In one embodiment, a computer readable storage medium having a computer program stored thereon is also provided, which relates to all or part of the processes in the above-mentioned embodiment methods.

[0252] In one embodiment, a computer program product is also provided, including computer programs / instructions, which relates to all or part of the processes in the above-mentioned embodiment methods.

[0253] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (SyMchliMk) DRAM (SLDRAM), memory bus (RaMbus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0254] The technical features of the above-mentioned embodiments can be combined in any way. In order to make the description simple, all possible combinations of the technical features in the above-mentioned embodiments are not described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0255] The above-mentioned embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent application. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of the patent protection of the present application should be subject to the appended claims.

Claims

1. A robot semantic map construction method based on a multi-modal large model and a graph RAG, characterized in that, The method comprises constructing a prior semantic database and constructing a semantic map, wherein: The constructing a prior semantic database specifically comprises: Based on the task description, search keywords are generated, internet pictures are collected, feature similarity is used for deduplication, and a multi-modal large model is used to verify whether the pictures meet the environmental requirements. The pictures that meet the environmental requirements are preprocessed, and the semantic information of the pictures is obtained through multiple models; wherein the multiple models at least include a semantic segmentation model, a depth estimation model, and an object detection model; A multi-modal large model is used to extract the semantic information to generate detailed semantic descriptions, and the consistency of the segmented areas and the detailed semantic descriptions is verified. After the consistency verification is passed, the semantic information is vectorized, a semantic subgraph centered on the object is constructed, and finally the same type of subgraphs are merged and stored in the database to form a prior semantic database associated with the environment; The constructing a semantic map specifically comprises: A dense point cloud map is generated based on SLAM, segmented into blocks according to a preset size, and associated with corresponding image data. The pictures of each block are subjected to semantic segmentation, depth point cloud registration, and object detection. A multi-modal large model is used to generate preliminary semantic descriptions. The prior semantic database is searched through vector similarity. The object co-occurrence subgraph and the current detection result are combined to generate an enhanced prompt word containing a list of alternative objects. The semantic information is extracted and verified using the enhanced prompt word. The result is vectorized to construct an object subgraph with environmental coordinates. Finally, all map blocks are integrated to form a robot semantic map. 2.The robot semantic map construction method of claim 1, wherein, Based on the task description, search keywords are generated, internet pictures are collected, feature similarity is used for deduplication, and a multi-modal large model is used to verify whether the pictures meet the environmental requirements, specifically including: For the obtained pictures, first, the picture features are extracted using a depth model based on a CNN convolutional network, and the picture feature library stored in the database is compared for cosine similarity. If the similarity is greater than a threshold value, it is judged as a duplicate picture and discarded, and a new picture is collected; If it is a non-duplicate picture, a multi-modal large model is used for identification to verify whether it meets the task and environmental description. If it meets the requirements, the next step is performed and the picture features of the current picture are stored. If it does not meet the requirements, the current picture is discarded and a new picture is collected. 3.The robot semantic map construction method of claim 1, wherein, The pictures that meet the environmental requirements are preprocessed, and the semantic information of the pictures is obtained through multiple models, including: The pictures are processed to a uniform preset size: for pictures larger than the preset size, the central area of the picture is cropped and retained; for pictures smaller than the preset size, they are enlarged to the preset size using linear interpolation, wherein pictures smaller than 80% of the preset size are discarded; For each picture, the following depth models are used to process the picture and temporarily store the processing results: An image semantic segmentation model is used to obtain all masks of the picture; A monocular image depth estimation model is used to obtain the depth information of the picture and convert it into a point cloud; a point cloud segmentation model is used to obtain the bounding box of each part after segmentation; A multi-modal large model is used to describe the picture and obtain the description of each object in the picture; An object detection model is used to obtain the category and bounding box of the object in the picture. 4.The robot semantic map construction method of claim 1, wherein, Building object-centered semantic sub-graphs, finally merging the same sub-graphs and storing them in the database to form a prior semantic database with environment association, including: Traverse all objects in the picture, and establish a sub-graph with the semantic vector of each object as the center node. The sub-nodes of the sub-graph are the semantic vectors of other objects appearing in the same picture, and the edges are the number of times the sub-node objects appear. When adding a sub-node, if the object category of the newly added sub-node is the same as that of an existing sub-node, only the existing sub-node is retained and the number recorded in the edge is incremented by one. The sub-graph describing the frequency of other objects appearing around each object is obtained, which represents the environment around each object; Traverse all sub-nodes in the sub-graph, check the object category of the sub-node, and if it has the same object category as a sub-node of a stored sub-graph, add the corresponding edge value. Otherwise, add the sub-node and edge to the merged sub-graph and store it. If there is no same center node object category, it is directly stored. 5.The robot semantic map construction method of claim 1, wherein, After building object-centered semantic sub-graphs, finally merging the same sub-graphs and storing them in the database to form a prior semantic database with environment association, it further includes: According to the preset threshold, it is judged whether it is necessary to continue adding data; Loop until the amount of data added in the database reaches the preset threshold, and the prior semantic database containing object environment sub-graph and object semantic vector is obtained after the loop is completed, which can be used for graph RAG technology in semantic map construction. 6.The robot semantic map construction method of claim 1, wherein, Extract and verify semantic information using enhanced prompt words, vectorize the results, and build object sub-graphs with environment coordinates. Finally, integrate all map tiles to form a robot semantic map, including: Use a multi-modal large model to analyze each regional image and describe the semantic information of all objects in it, including position, category, shape, and texture information. Save the semantic information of each object in Chinese string form; Vectorize the object category and query the top k objects with the highest similarity in the prior semantic database based on cosine similarity. Generate enhanced prompt words based on the object shape and texture description. Use a multi-modal large model to verify whether the description of each object is accurate based on the prompt words; Vectorize the complete picture, the picture area described by the position part in the semantic information, and the semantic information string as the semantic vector of each object. Store the above data. The semantic vector is used as a key value to index the information of the object. Traverse all objects in the picture, and establish a sub-graph with the semantic vector of each object as the center node. The sub-nodes of the sub-graph are the semantic vectors of other objects appearing in the same picture, and the edges are the number of times the sub-node objects appear. When adding a sub-node, if the object category of the newly added sub-node is the same as that of an existing sub-node, only the existing sub-node is retained and the number recorded in the edge is incremented by one. The sub-graph describing the frequency of other objects appearing around each object is obtained, which represents the environment around each object; Store all sub-graphs with the center coordinates of the current point cloud map block as the index. Check if all map blocks have been traversed. If not, continue processing the next map block. Loop until all map blocks are traversed. After the loop is completed, the database of the current environment is obtained as a semantic map. Finally, integrate all map blocks to form a robot semantic map.

7. A robot semantic map construction system based on a multi-modal large model and a graph RAG, characterized in that, The system includes a prior semantic database construction module and a semantic map construction module, wherein: The prior semantic database construction module is specifically used for: The task description is used to generate search keywords, collect Internet pictures, remove pictures through feature similarity, and verify whether the pictures meet the environmental requirements by using a multi-modal large model. The pictures that meet the environmental requirements are preprocessed, and the semantic information is obtained by processing the pictures through multiple models. The multiple models at least include a semantic segmentation model, a depth estimation model, and an object detection model. A multi-modal large model is used to extract the semantic information to generate a detailed semantic description, and the consistency of the segmentation area and the detailed semantic description is verified. After the consistency verification is passed, the semantic information is vectorized, an object-centered semantic subgraph is constructed, and finally the same subgraph is merged and stored in a database to form a prior semantic database with environmental association. The semantic map construction module is specifically used for: Based on SLAM, a dense point cloud map is generated, which is segmented into blocks according to a preset size and associated with corresponding image data. The pictures of each block are subjected to semantic segmentation, depth point cloud registration and object detection, and a multi-modal large model is used to generate a preliminary semantic description. The prior semantic database is searched through vector similarity, and an enhanced prompt word containing an alternative object list is generated by combining the object co-occurrence subgraph and the current detection result. The semantic information is extracted and verified by using the enhanced prompt word, and an object subgraph with environmental coordinates is constructed after the result is vectorized. Finally, all map blocks are integrated to form a robot semantic map.

8. An electronic device, comprising: The computer program is stored in the memory and executed by the processor to realize the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer program is stored in the memory and executed by the processor to realize the steps of the method according to any one of claims 1 to 6.

10. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to realize the steps of the method according to any one of claims 1 to 6.

Images