Semantic map creation and use methods, systems, robots, and media

Abstract

The application provides a semantic map creation and use method, system, robot and medium, belongs to the field of artificial intelligence and robot technology, and aims to solve the problem that the existing semantic map construction method depends on pre-defined categories and cannot discover unknown semantic concepts. The method comprises the following steps: collecting multi-modal data in an environment and attaching spatial position information; extracting multi-modal features; clustering the multi-modal features by using an unsupervised clustering algorithm to form data clusters; generating natural language semantic labels for the data clusters by using a pre-trained multi-modal large model; constructing a multi-dimensional semantic map; and responding to a natural language query. Through the core process of unsupervised clustering first and large model labeling later, the application realizes automatic discovery of unknown semantics, and improves the environmental understanding ability of the robot and the naturalness of human-computer interaction.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and robotics, and in particular to a method, system, robot, and medium for creating semantic maps for robot environmental perception and human-computer interaction. Background Technology

[0002] In robotics, environmental maps are typically built to enable robots to navigate autonomously and perform tasks within their environments. Traditional maps, such as grid maps or point cloud maps, primarily record the geometric and spatial information of the environment, lacking a deep understanding of its content. To enhance a robot's environmental understanding, semantic maps have emerged, adding semantic labels, such as "door" or "table," to objects or regions within the environment based on geometric maps.

[0003] In recent years, with the development of multimodal technologies and large-scale language models, some existing technical solutions have attempted to integrate data from multiple sensors such as vision and LiDAR, and use large models to directly identify and label environmental objects to automatically generate semantic maps. However, these methods have significant drawbacks: First, their semantic recognition capabilities are entirely limited by the dataset used by the large model during the pre-training phase. They cannot effectively identify and label objects that the model has never seen before or that are unique to a specific scene. Second, this approach is essentially a "top-down" labeling process that relies on predefined object categories. It is difficult to discover and summarize new or unknown abstract semantic concepts in the environment that are jointly defined by multiple perceptual information (such as temperature, sound, and odor), such as a "device area" defined by specific machine noise, high temperature, and visual characteristics. This results in the robot's understanding of the environment remaining superficial and pre-defined, unable to truly adapt to complex and ever-changing unknown environments, and unable to respond to complex commands based on deep environmental semantics, thus limiting the robot's intelligence level and human-computer interaction capabilities. Summary of the Invention

[0004] The purpose of this invention is to provide a semantic map creation method and system based on multimodal perception, so as to solve the problem that the semantic map construction methods in the prior art rely on predefined categories and cannot discover unknown semantic concepts, thereby improving the robot's environmental understanding and human-computer interaction capabilities in unknown environments.

[0005] To achieve the above objectives, this invention provides a method for creating and using a semantic map based on multimodal perception, comprising: collecting multimodal data from the environment by a mobile robot and attaching corresponding spatial location information to the multimodal data, wherein the multimodal data includes at least image data and sound data; processing the multimodal data to extract corresponding multimodal features; clustering the multimodal features using an unsupervised clustering algorithm to form at least one data cluster; generating natural language semantic labels for the data clusters using a pre-trained multimodal large model; constructing a multidimensional semantic map based on the spatial location information of the data clusters and the natural language semantic labels; and responding to a natural language-based query request by performing a search within the multidimensional semantic map to provide feedback. The multimodal data may also include sensor data such as temperature, gas, and lidar.

[0006] Optionally, the method may further include, before or during the acquisition of the multimodal data: establishing a geometric map of the environment using a real-time localization and mapping algorithm; and adding spatial location information to the acquired multimodal data based on the geometric map.

[0007] Optionally, the extraction of corresponding multimodal features includes: performing panoramic segmentation on the image data to extract visual features; and performing sound event detection on the sound data to extract audio features.

[0008] Optionally, to further improve the feature representation ability and the accuracy of subsequent clustering, before using the unsupervised clustering algorithm for clustering, the method further includes: employing a multimodal fusion model based on a transformer architecture to fuse features from different modalities into a unified feature vector for subsequent clustering analysis. This step aims to transform and fuse the raw, heterogeneous multimodal sensor data into a unified feature vector containing deep semantic information. Specific encoders include dinov2 for images, a Time Series Transformer Encoder for multidimensional data, and an Audio Transformer or CNN for other multimodal data such as audio.

[0009] Optionally, the unsupervised clustering algorithm includes at least one of the K-Means algorithm, DBSCAN algorithm, or Gaussian mixture model.

[0010] Optionally, the step of generating natural language semantic labels for the data cluster using a pre-trained multimodal large model includes: selecting one or more representative multimodal data samples from the data cluster; and inputting the representative multimodal data samples into the multimodal large model to obtain the natural language semantic labels.

[0011] Optionally, the multidimensional semantic map is constructed based on a graph database; wherein the natural language semantic labels serve as nodes in the graph database, and connections are established between nodes to represent semantic relationships.

[0012] To achieve the above objectives, the present invention also provides a semantic map creation and usage system based on multimodal perception, comprising: a data acquisition module for collecting multimodal data from the environment by a mobile robot and attaching corresponding spatial location information to the multimodal data, wherein the multimodal data includes at least image data and sound data; a feature extraction module for processing the multimodal data to extract corresponding multimodal features; an unsupervised clustering module for clustering the multimodal features using an unsupervised clustering algorithm to form at least one data cluster; a semantic label generation module for generating natural language semantic labels for the data clusters using a pre-trained multimodal large model; a map construction module for constructing a multidimensional semantic map based on the spatial location information of the data clusters and the natural language semantic labels; and a human-computer interaction module for responding to natural language-based query requests and providing feedback by performing searches in the multidimensional semantic map.

[0013] Furthermore, the present invention also provides a mobile robot, including the system described above.

[0014] Furthermore, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to claim 1.

[0015] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects: 1. Automatic discovery and learning of unknown semantics. This invention, through the core process of "unsupervised clustering followed by large model labeling," can automatically discover and summarize regions or sets of objects with similar multimodal features from raw perceptual data, and assign them semantics. This enables robots to learn and understand unknown or novel semantic concepts, greatly enhancing their adaptability in unstructured environments.

[0016] 2. Enhanced the depth and naturalness of human-computer interaction. The semantic map constructed by this invention integrates multi-dimensional information such as temperature and sound, enabling the robot to understand abstract environmental states such as "cool" and "quiet." Therefore, users can issue complex commands based on deep environmental semantics, such as "take me to the coolest place," and the system can respond by querying the map, significantly improving the intelligence level of human-computer interaction.

[0017] 3. Reduced map construction costs. By using unsupervised clustering and automatic labeling of large models, this invention eliminates the reliance on extensive manual annotation in traditional semantic map construction, achieving automated semantic map construction, reducing labor costs, and improving mapping efficiency.

[0018] 4. Enhanced environmental monitoring and analysis capabilities. In scenarios such as industrial inspection, this invention can spatially correlate and aggregate abnormal data from multiple sensors, quickly locate problem points, and generate advanced semantic alarms such as "equipment malfunction," thereby improving the efficiency and accuracy of fault diagnosis. Attached Figure Description

[0019] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the architecture of a semantic map creation and usage system based on multimodal awareness, provided in an embodiment of the present invention. Figure 2 A flowchart illustrating a method for creating and using a semantic map based on multimodal perception, provided as an embodiment of the present invention; Figure 3 A signaling interaction timing diagram of a method implementation process provided in an embodiment of the present invention; Figure 4 This is a conceptual diagram illustrating the construction of a multidimensional semantic map in an embodiment of the present invention.

[0020] Figure labeling: 10-Mobile robot; 11-Visual sensor; 12-Sound sensor; 13-External sensor interface; 20-Robot brain; 21-Data acquisition module; 22-Data fusion module; 221-Feature extraction unit; 222-Unsupervised clustering unit; 223-Semantic label generation unit; 23-Map generation module; 24-Human-computer interaction module; 30-Multidimensional semantic map database; 401-Mobile robot; 402-Human; 403-Machine; 411-Sound information; 412-Infrared information; 413-Temperature information; 421-Semantic label "Human activity area"; 422-Semantic label "Equipment operation area". Detailed Implementation

[0021] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0022] A preferred implementation: Feature vectorization based on a multimodal fusion model In a preferred embodiment of the present invention, in order to improve the representational ability of features and the accuracy of subsequent clustering, this method introduces a multimodal fusion model based on a Transformer architecture before performing unsupervised clustering. This process includes two core steps: modal feature extraction and multimodal feature fusion.

[0023] Step 1: Modal Feature Extraction For sensor data from different sources, this method uses state-of-the-art deep learning models to vectorize the data in order to capture its deep semantic features.

[0024] Image data feature extraction: Self-supervised learning visual models (such as DINOv2) are used to process the acquired environmental images. This model can learn rich visual representations from images without manual annotation. For example, inputting a 224x224 pixel image into the DINOv2 model can yield a 1024-dimensional feature vector, which can encode high-level visual information such as objects, textures, and spatial layouts in the image.

[0025] Sound data feature extraction: First, the original sound waveform is converted into a Mel spectrogram, a two-dimensional image that visually displays the energy distribution of the sound signal at different frequencies. Then, a convolutional neural network (CNN) or audio transformer model, similar to those used for image processing, is employed to extract feature vectors from the Mel spectrogram. For example, a 2-second audio clip can be converted into a 512-dimensional feature vector, capturing the acoustic patterns of sound events such as device noises or human conversations.

[0026] Feature extraction from multidimensional time-series data (e.g., temperature, humidity, gas concentration, etc.): For scalar data that varies over time (such as temperature sensor readings) or multidimensional vector data (such as triaxial accelerometer readings), this method employs a transformer-based time series encoder. This model, through its self-attention mechanism, effectively captures long-term dependencies and periodic patterns in time series data. For example, by inputting temperature, humidity, and CO2 concentration data collected every minute over the past 60 minutes as a sequence, a 256-dimensional feature vector characterizing recent environmental changes can be generated.

[0027] Step 2: Multimodal Feature Fusion After obtaining the independent feature vectors of each modality, they need to be merged into a unified feature vector that can comprehensively describe the semantics of a specific spatiotemporal point.

[0028] • This method employs a multimodal fusion transformer. Specifically, feature vectors extracted from image, sound, and time-series data (e.g., a 1024-dimensional image vector, a 512-dimensional sound vector, and a 256-dimensional environment vector) are concatenated together, and a special learnable [CLS] token is added at the beginning of the vector sequence.

[0029] • Input this concatenated vector sequence into the encoder layer of the fusion transformer. Through the multi-head self-attention mechanism, the model can calculate the interrelation weights between different modal features. For example, the model can learn that when the feature of "crowd gathering" appears in the image, the weights of the sound feature of "noisy human voices" and the environmental feature of "increased CO2 concentration" will increase accordingly.

[0030] Finally, the output vector corresponding to the [CLS] token after processing by the transformer encoder is taken as the fused unified feature vector. This vector contains deep semantic information across modalities and has stronger representational power compared to simple feature concatenation.

[0031] Compatibility with large industrial models: The feature extraction method described in this invention has good scalability and can be compatible with existing or future multimodal industrial large-scale models. These large-scale models are usually pre-trained with powerful encoders capable of processing data from various industrial sensors (such as vibration, pressure, current, and infrared imaging). This method can directly call these pre-trained encoders as feature extractors for specific modalities, thereby transferring the knowledge learned by the large models on a wide range of industrial data to the semantic map construction task of this method, greatly improving the system's adaptability and performance in specific industrial scenarios.

[0032] Example 1 This embodiment provides a method for creating and using semantic maps based on multimodal perception in an office setting. This embodiment aims to elaborate on the basic implementation process of the technical solution of this invention, covering the complete process from data acquisition, processing, map construction to natural language interaction.

[0033] Please see Figure 1This diagram illustrates the architecture of a semantic map creation and usage system based on multimodal perception, according to an embodiment of the present invention. The system can be deployed on a mobile robot 10, or it can be implemented collaboratively by the mobile robot 10 and a remote robot brain 20 (e.g., a cloud server or edge computing node). The mobile robot 10, as the entity responsible for environmental perception and physical interaction, is equipped with at least a visual sensor 11 (such as an RGB-D depth camera) for acquiring image data and a sound sensor 12 (such as a microphone array) for acquiring sound data. Furthermore, the mobile robot 10 can connect to other types of sensors via an external sensor interface 13, including but not limited to lidar, inertial measurement units, or temperature sensors.

[0034] The robot brain 20, as the computing core of the system, is responsible for processing the data collected by the mobile robot 10 and implementing various core functions. In this embodiment, the robot brain 20 can be a high-performance onboard computer or server, which integrates multiple functional modules. Specifically, the data acquisition module 21 is responsible for communicating with various sensors on the mobile robot 10 to obtain raw multimodal data streams. The data fusion module 22 is a key part of realizing the core technical idea of ​​this invention, and it is further divided into three units: the first is a feature extraction unit 221, which is used to extract features with representational significance from the raw data; the second is an unsupervised clustering unit 222, which automatically groups the data according to the similarity of features without preset categories; and the third is a semantic label generation unit 223, which uses a large-scale artificial intelligence model to give natural language descriptions to the clustered data groups. The map generation module 23 is responsible for integrating spatial information, feature information, and semantic labels to construct and maintain a structured multidimensional semantic map database 30. Finally, the human-computer interaction module 24 provides a corresponding interface to receive the user's natural language commands, parse their intentions, perform retrieval in the multidimensional semantic map database 30, and ultimately provide feedback to the user or control the mobile robot 10 to perform corresponding tasks.

[0035] Please combine Figure 2 This figure is a flowchart of a method for creating and using a semantic map based on multimodal perception, provided in an embodiment of the present invention. The following will be combined with... Figure 1 The system architecture shown describes in detail the specific steps for executing this process in an office setting.

[0036] First, in step S101, the system collects multimodal data and location information. In this embodiment, the mobile robot 10 is placed in an unknown office environment, which may include areas with different functions such as office areas, meeting rooms, and tea rooms. The robot initiates an autonomous exploration task, during which the data acquisition module 21 runs continuously.

[0037] As a preferred implementation, the system utilizes real-time localization and mapping (RTD) algorithms to create a geometric map of the environment simultaneously with or before data acquisition. For example, a LIO-SAM algorithm based on tight coupling of LiDAR and inertial measurement unit (IMU) can be employed. The LiDAR on the mobile robot 10 (accessed via external sensor interface 13) scans the environment to acquire point cloud data, while the IMU provides high-frequency attitude information. After fusing these two types of data, a high-precision two-dimensional or three-dimensional geometric map of the environment (such as a grid map or point cloud map) can be constructed in real time, accurately estimating the robot's own position and attitude within the map, i.e., pose. This process provides a unified spatial coordinate system for all subsequently acquired data.

[0038] Accordingly, the data acquisition module 21 simultaneously records data streams from different sensors. Specifically, the vision sensor 11 acquires color and depth images at a resolution of 640x480 at a frequency of 10 frames per second; the sound sensor 12 (a four-channel microphone array) continuously records ambient sound at a sampling rate of 48kHz. Whenever a data sample is acquired (e.g., a frame of image or a 1-second audio clip), the data acquisition module 21 immediately obtains the robot's pose information (including x, y coordinates and orientation angle θ) from the real-time localization and mapping system, and packages this pose information as spatial location information along with the data sample and a precise timestamp. In this way, all acquired multimodal data is given a spatiotemporal label, laying the foundation for subsequent data fusion and analysis in the spatial dimension. Multimodal data also includes sensor data such as temperature, gas, and LiDAR.

[0039] Next, in step S102, the acquired multimodal data is processed to extract the corresponding multimodal features. This step is performed by the feature extraction unit 221 in the data fusion module 22. For visual data, this embodiment employs an advanced panoramic segmentation model, such as the SAM2 model. For each frame of input color image, this model can automatically identify and segment all objects in the image, regardless of whether these objects belong to common categories. For each segmented object mask, such as a desk, a computer monitor, or a water glass, the feature extraction unit 221 further uses a pre-trained visual encoder (such as the visual part of the CLIP model) to extract the visual feature vector of the object. This vector can be a 512-dimensional floating-point array, which contains the visual semantic information of the object.

[0040] For audio data, feature extraction unit 221 first segments the continuous audio stream into 1-second segments, and then processes each segment using a sound event detection model. This model can identify sound events contained in the audio, such as "keyboard typing," "human conversation," and "printer operation," and extracts an audio feature vector for each identified event. This feature vector can also be a fixed-dimensional array to represent the type and characteristics of the sound. Through this processing, the raw, unstructured image and sound data is transformed into structured, computationally usable feature vectors.

[0041] Subsequently, the process proceeds to step S103, where feature fusion and unsupervised clustering are performed. After the feature extraction unit 221 completes its processing, multiple modal features collected at the same spatial location (or within a very small neighborhood) with similar timestamps need to be correlated and fused. As an optional implementation, a multimodal fusion model based on a transformer architecture can be adopted. This model receives a set of visual feature vectors and audio feature vectors from the same spatiotemporal point, and learns the intrinsic correlation between different modal features through its internal self-attention and cross-attention mechanisms, ultimately outputting a unified multimodal feature vector. For example, the model can learn that the visual features of a "computer monitor" and the audio features of "keyboard typing" often occur simultaneously, and fuse them into a unified feature vector that can represent the scenario of "working on a computer".

[0042] After the mobile robot 10 completes its exploration of the entire office, the system has accumulated tens of thousands of multimodal feature vectors with spatial location information. At this point, the unsupervised clustering unit 222 begins operation, employing an unsupervised clustering algorithm to cluster all the collected multimodal feature vectors. In this embodiment, the K-Means clustering algorithm is used. This algorithm aims to divide the data into K clusters, such that each data point belongs to the cluster corresponding to its nearest mean (i.e., cluster center). Before applying this algorithm, the number of clusters K needs to be preset; this value can be set empirically or dynamically determined using evaluation metrics such as the silhouette coefficient. After the clustering process, the originally scattered feature vectors are automatically divided into several data clusters. Ideally, each data cluster corresponds to a set of regions or objects in the environment with similar multimodal attributes. For example, all data points collected near office workstations that include visual features such as "desk," "computer," and "chair" and audio features such as "keyboard typing sounds" are highly likely to be clustered into one cluster; similarly, data points collected in the break room that include visual features such as "water dispenser" and "microwave oven" and audio features such as "water flow sounds" may be clustered into another cluster. Understandably, the core value of this step lies in the fact that it does not rely on any predefined labels, but rather discovers potential functional areas in the environment entirely based on the distribution and similarity of the data itself.

[0043] Next, in step S104, natural language semantic labels are generated for the clusters using a pre-trained multimodal large model. This step is performed by the semantic label generation unit 223. For each data cluster generated in the previous step, the system needs to assign it a semantic name that humans can understand. The method used in this embodiment is to select one or more representative multimodal data samples from each data cluster. A simple selection strategy is to select the data point closest to the cluster center of the cluster, which contains the original image, sound, and extracted feature vector.

[0044] The system then inputs this representative sample into a pre-trained multimodal vision-language large-scale model, such as the locally deployed Qwen-VL model. The input to the model can be a representative image accompanied by a text prompt describing other modal information, such as: "This image was taken in an environment and recorded 'keyboard typing sounds.' Please describe the semantics of this scene in one sentence." Upon receiving this input, the multimodal large-scale model leverages its powerful world knowledge and image-text understanding capabilities to generate a natural language description. For example, for a sample containing a desk, computer, and keyboard sounds, the model might output: "An office area where people are working"; for a sample containing a water dispenser and running water sounds, the model might output: "A tea room area providing drinking water." This natural language description generated by the model is used as the semantic label for the corresponding data cluster. This process achieves a leap from unsupervised pattern discovery to meaningful semantic annotation.

[0045] Subsequently, in step S105, a multidimensional semantic map is constructed or updated. The map generation module 23 is responsible for integrating all the information generated in the preceding steps and storing it in the multidimensional semantic map database 30. This database can be a structured file (such as a JSON or XML file) or a professional database system. For each successfully labeled semantic cluster, the map generation module 23 creates an entry. This entry contains at least: a unique ID; the spatial location information corresponding to the cluster (which can be the set of spatial locations of all data points in the cluster, or the coordinates of its geometric center); and a natural language semantic label generated by the large model (such as "office area where work is taking place"). Optionally, the entry can also store richer information such as the cluster center feature vector, the number of data points within the cluster, and the index of the original data samples.

[0046] Please see Figure 4which conceptually illustrates such a multi-dimensional semantic map. This map not only includes the mobile robot 401 and physical objects in the environment (such as the person 402 and the machine 403), but more importantly, it closely associates spatial regions (such as spatial regions A and B) with multi-modal perception information (such as sound information 411, infrared information 412, and temperature information 413) as well as high-level semantic labels (such as the semantic label "human activity area" 421 and the semantic label "equipment operation area" 422).

[0047] Finally, the technical solution of the present invention further includes the use of the map, that is, the human-robot interaction link. Please refer to the combination Figure 2 and Figure 3 for understanding, in which Figure 3 shows the signaling interaction among the user, the robot system and the environment. After the construction of the semantic map is completed, the system can respond to the user's natural language query. In step S201, the human-robot interaction module 24 receives the user's natural language query through a speech recognition or text input interface. For example, the user asks the robot through a microphone: "Where are people working?".

[0048] In step S202, the human-robot interaction module 24 parses the query and retrieves the map. It first uses natural language understanding technology to extract the core query intent and keywords from the user's question, which is "working" in this example. Then, it performs retrieval in the multi-dimensional semantic map database 30. One retrieval method is direct semantic label matching, that is, searching for labels containing the keyword "working". In this embodiment, the system will successfully match the previously generated semantic label "working office area". Once the matching is successful, the system extracts the spatial position information associated with this label.

[0049] In step S203, the system outputs a result or guides the robot to perform a task. The human-robot interaction module 24 converts the retrieved position information into user-understandable feedback, for example, answering through a speech synthesis module: "There is a working office area in the area on your left front." Meanwhile, the corresponding area on the map can be highlighted on the control interface of the robot or an augmented reality device. If the instruction is "Take me to the nearest office area", the system will further call the navigation module to plan a path from the current position to the target area, and control the mobile robot 10 to perform the movement task.

[0050] Through the above steps, this embodiment successfully automatically discovers and marks semantic regions with different functions in an unknown office environment, constructs a multi-dimensional semantic map, and realizes natural language interaction based on the map, thereby significantly improving the robot's environmental understanding capability and intelligence level.

[0051] Embodiment 2 This embodiment will demonstrate the application of the technical solution of the present invention in an industrial inspection scenario, and will focus on how to integrate sensor data unique to the industrial environment (such as temperature and gas concentration) to detect environmental anomalies, and then automatically generate alarm information with clear semantics. The system architecture of this embodiment and... Figure 1 The configurations are basically the same, but the sensor configuration of the mobile robot 10 is different.

[0052] In this embodiment, the mobile robot 10 used for inspection is equipped with a vision sensor 11 and a sound sensor 12, and also integrates a thermal imaging camera and a sensor for detecting specific toxic or flammable gases (e.g., a methane sensor) through an external sensor interface 13. The robot is deployed in a workshop of a chemical plant and performs periodic inspections along a preset path.

[0053] In step S101 (acquiring multimodal data and location information), as the mobile robot 10 moves, the data acquisition module 21 synchronously records data from all sensors. This data includes: color images captured by a standard camera for identifying the device's appearance; thermal infrared images captured by a thermal imaging camera for monitoring the device's temperature; audio data collected by a sound sensor for detecting abnormal noises during device operation; and concentration readings output in real time by a gas sensor. Similarly, all this data is appended with high-precision spatial location information and timestamps provided by the real-time localization and mapping system.

[0054] In step S102 (extracting multimodal features), the feature extraction unit 221 processes these newly added modal data. For thermal imaging images, the system extracts their temperature distribution features, such as calculating the highest temperature and average temperature in the image, or extracting feature vectors describing the shape and size of high-temperature areas. For gas sensor readings, since they are structured numerical values, they can be directly encoded, for example, by normalizing them as a feature dimension.

[0055] Step S103 (feature fusion and unsupervised clustering) is the key to this embodiment. In industrial inspection scenarios, abnormal events are usually low-probability events, which manifest as outliers in the data distribution. Therefore, density-based clustering algorithms (such as DBSCAN) are more suitable than the K-Means algorithm. The DBSCAN algorithm can discover clusters of arbitrary shapes and effectively identify noise points (i.e., outliers).

[0056] During the inspection process, the robot's "brain" 20 can perform DBSCAN clustering on the collected multimodal feature vectors in near real-time. Suppose that during an inspection, when the robot passes near pump number 3, several data points collected consecutively exhibit similar anomalous characteristics: visually identified as "pump number 3," thermal imaging shows its surface temperature reaches 85 degrees Celsius (far exceeding the normal operating threshold of 50 degrees Celsius), high-frequency whistling is detected in the sound, and trace amounts of methane at a concentration of 100 ppm are detected in the air. These anomalous data points that are close to each other in the feature space will be automatically identified and clustered into a high-density data cluster by the DBSCAN algorithm. Data points representing normal operating conditions will be grouped into one or more larger clusters or considered noise.

[0057] In step S104 (generating semantic labels for clusters using a large model), the semantic label generation unit 223 processes this newly discovered anomalous data cluster. It selects a representative sample from the cluster, which includes a color image of pump number 3, a thermal imaging image showing the high-temperature area, a spectrogram of the abnormal sound, and a methane concentration reading. To help the multimodal large model better understand this information, the system can convert non-visual information into text descriptions. For example, it can construct the following input prompt: "This is an industrial device. Its thermal imaging shows an abnormal increase in local temperature. Simultaneously, methane gas has been detected in the environment, and a high-frequency abnormal noise emitted by the device has been recorded. Please generate a short semantic alarm label based on this information." This multimodal input, containing both image and text descriptions, is fed into a large multimodal model fine-tuned with industry-specific knowledge. After analysis, the model may generate a highly specific and information-rich semantic label, such as: "Pump No. 3 is suspected of experiencing media leakage due to seal damage, accompanied by bearing overheating." It's worth noting that this label is far more intelligent and intuitive than traditional single-value alarms such as "temperature exceeding limits" or "gas concentration alarm."

[0058] In step S105 (building / updating the multidimensional semantic map), the newly generated abnormal semantic tag, along with its precise spatial location and multimodal data snapshots (such as images, sound clips, temperature values, etc.) when the alarm is triggered, is immediately recorded in the multidimensional semantic map database 30 and can automatically trigger the factory's central monitoring and alarm system.

[0059] In the interactive usage phase (steps S201-S203), when the plant's operations engineer receives an alarm, he can issue query commands through the human-machine interface module 24, such as "Show all high-temperature alarm points" or "Check events related to 'leakage'." The system will parse these commands, retrieve matching tags or associated feature data in the semantic map, and immediately highlight the specific location of pump No. 3 on the plant's electronic map, while simultaneously displaying relevant images, sounds, and data curves in a pop-up window. This greatly helps engineers quickly locate problems and make decisions, thereby significantly improving the efficiency and safety of inspections and maintenance.

[0060] Example 3 This embodiment aims to demonstrate the flexibility and universality of the technical solution of the present invention. By applying different combinations of algorithm models in open public scenes such as parks, it is also possible to achieve the goal of building rich semantic maps and supporting complex interactions.

[0061] In this embodiment, the mobile robot 10 roams freely within the park, its task being to construct a semantic map that helps visitors find comfortable rest areas. The data collected by the robot mainly includes: park scene images captured by the visual sensor 11, background sounds recorded by the sound sensor 12 (such as birdsong, wind sounds, and distant human voices), and air quality index readings provided by an air quality sensor connected through the external sensor interface 13.

[0062] In the data processing flow, this embodiment employs some technical choices that differ from Embodiment 1. In the feature fusion stage (before step S103), to reduce computational complexity, the system does not use a complex transformer-based fusion model, but instead chooses a simpler and more direct approach. After the feature extraction unit 221 extracts the visual feature vector (e.g., 256-dimensional), audio feature vector (e.g., 128-dimensional), and encoded air quality features (e.g., 1-dimensional), the system simply concatenates these feature vectors from different modalities to form a unified feature vector with a total dimension of 385. Subsequently, dimensionality reduction algorithms such as principal component analysis can be used to reduce the dimensionality of the concatenated high-dimensional vector to remove redundant information and retain the main components, for example, reducing it to 128 dimensions. Although this method may lose some subtle correlations between modalities, it still works effectively in many scenarios and has lower computational overhead.

[0063] In the unsupervised clustering step (step S103), this embodiment uses a Gaussian mixture model instead of K-Means or DBSCAN. A Gaussian mixture model is a probability-based clustering method that assumes all data points are generated by a mixture of several Gaussian distributions. Unlike K-Means, which rigidly assigns each point to a cluster, the Gaussian mixture model can calculate the probability of each data point belonging to each cluster. This "soft clustering" characteristic is well-suited for describing situations where functional area boundaries are blurred, such as parks. For example, an area might resemble both a "rest area" and a "viewing area," and the Gaussian mixture model can provide the probability that it belongs to both areas, rather than forcing an either-or choice.

[0064] In the semantic label generation step (step S104), this embodiment selects another multimodal large model from the BLIP series. After the unsupervised clustering unit 222 discovers a data cluster using the Gaussian mixture model algorithm, the semantic label generation unit 223 selects one or more data samples from that cluster with the highest probability of belonging to that cluster as representatives. Assuming a representative sample includes an image of a bench, low-decibel human voices, and good air quality readings, the system inputs this information into the BLIP model, and after model analysis, it may generate the semantic label: "A quiet rest area suitable for relaxation".

[0065] Ultimately, in the multidimensional semantic map database 30 constructed by the map generation module 23, each semantic region not only has associated labels and locations, but also stores the semantic attribute probability distribution calculated by the Gaussian mixture model, making the map description more refined and flexible.

[0066] In the human-computer interaction phase (steps S201-S203), this refined map can support more complex combined condition queries. For example, a tourist asks through the park's smart navigation application (integrating the human-computer interaction module 24 of this invention): "I want to find a quiet place with good air quality to sit down and read a book." The human-computer interaction module 24 parses the query and extracts three semantic intents: "good air quality," "quiet," and "can sit." Subsequently, it executes a compound query in the multidimensional semantic map database 30: First, it filters out all areas whose semantic tags contain keywords such as "rest area," "bench," and "seat." Then, among these candidate areas, it further filters out those areas whose air quality feature value is higher than a certain threshold (corresponding to "good air quality") and whose noise level in the audio feature is lower than a certain threshold (corresponding to "quiet"). The system can even sort the results based on the probability provided by the Gaussian mixture model, prioritizing the areas that best meet all the conditions. Finally, the system pushes a message to the tourist's mobile phone: "The bench area by the park lake is currently the place that best meets your requirements, with excellent air quality and is very quiet," along with navigation routes. This proves that the core idea of ​​this invention does not depend on a specific algorithm implementation and has strong versatility and adaptability.

[0067] Example 4 This embodiment introduces an advanced implementation of a multidimensional semantic map data structure, namely, using a graph database to construct the semantic map to support more complex semantic relationship reasoning and interaction. The scenario setting for this embodiment remains an office environment, and its core processes of data collection, feature extraction, clustering, and labeling (steps S101 to S104) are similar to those in Embodiment 1. The focus of this embodiment is on the implementation of step S105 (constructing / updating the multidimensional semantic map) and the interaction process.

[0068] In step S105, the map generation module 23 no longer stores the map data as a flat JSON file or a relational database table, but instead constructs it as an attribute graph and stores it in a dedicated graph database. In this structure, semantic concepts in the environment are abstracted as nodes in the graph, while the relationships between concepts are abstracted as edges.

[0069] The specific construction process is as follows: 1. Create nodes: Each natural language semantic tag generated by the semantic tag generation unit 223 will create a corresponding node in the graph database. For example, a node with the tag ":SemanticConcept" will be created, which has a name attribute with values ​​such as "office area", "water dispenser", "meeting room", etc. In addition, specific entities in the environment, such as "employee A" and "printer P1", can also be created as nodes (e.g., with the tag ":Entity"). 2. Add attributes to nodes: Each node can have rich attributes to describe its features. For the ":SemanticConcept" node, its attributes may include: location (storing the spatial coordinates or range of the semantic region), avg_temperature (average temperature of the region), noise_level (noise level in decibels), feature_vector (multimodal feature vector of the cluster centers of the region), etc. 3. Create relationships (edges): Nodes are connected by edges with direction and type to represent the semantic relationships between them. For example: * An Entity node named "Office Desk A1" can point to a :SemanticConcept node named "Office Area" via an edge named [:IS_PART_OF] (is part of...). * An Entity node named "Employee A" can point to the "Office Desk A1" node via an edge named [:USES] (uses). * A :SemanticConcept node named "Water Dispenser" can point to a :SemanticConcept node named "Corridor" via an edge named [:IS_LOCATED_IN] (located in).

[0070] In this way, previously isolated semantic information is organized into an interconnected knowledge network, which makes complex reasoning possible.

[0071] In the human-computer interaction phase (steps S201-S203), when a user initiates a complex natural language query, the human-computer interaction module 24 needs to convert it into a graph query language (such as Cypher) to query the graph database. This greatly enhances the system's question-and-answer capabilities.

[0072] For example, a user asks: "Where is the nearest water dispenser to employee A?" The processing flow of the human-computer interaction module 24 is as follows: 1. Natural Language Understanding: The module first parses the question, identifies the entities "Employee A" and "Water Dispenser", and the relationship between them as "closest". 2. Graph Query Generation: The module converts the parsed results into a graph query statement. Taking Cypher language as an example, the statement may be as follows: cypherMATCH (p:Entity{name: 'Employee A'})-[:USES]->(d:Entity)-[:IS_PART_OF]->(loc:SemanticConcept)WITHloc.location AS userLocationMATCH (wm:SemanticConcept {name: 'Water Dispenser'})WITHuserLocation, wm, distance(point(userLocation), point(wm.location)) ASdistRETURN wm.name, wm.location, distORDER BY dist ASCLIMIT 1 3. Query Execution and Result Return: The graph database executes the query. The query first finds the node named "Employee A" and then locates its office location (userLocation) along the relationship. Next, it finds all nodes named "Water Dispenser". Then, it uses the database's built-in spatial function `distance` to calculate the Euclidean distance `dist` between the user's location and each water dispenser location. Finally, it sorts the locations by distance `dist` in ascending order and returns the location information of the nearest water dispenser. 4. Result Presentation: The human-computer interaction module 24 formats the query results into natural language and answers the user: "The nearest water dispenser to employee A is located at the end of the east corridor on the third floor, at coordinates (15.2, 30.5)." Understandably, by employing a semantic map with a graph structure, the system can not only answer simple questions about "what" and "where," but also handle complex reasoning problems involving multi-level relationships between objects (e.g., employee -> desk -> office area), spatial relationships (e.g., distance, orientation), and attribute comparisons. This results in a qualitative leap in the capability and depth of human-computer interaction, fully demonstrating the enormous potential of the technical solution of this invention in building truly intelligent robots with environmental awareness.

[0073] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0074] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features of the present invention can be arbitrarily combined with each other.

Claims

1. A method for creating and using a semantic map, characterized in that, include: A mobile robot collects multimodal data from the environment and adds corresponding spatial location information to the multimodal data, which includes at least image data and sound data. The multimodal data is processed to extract the corresponding multimodal features; An unsupervised clustering algorithm is used to cluster the multimodal features to form at least one data cluster; Natural language semantic labels are generated for the data cluster using a pre-trained multimodal large model; Based on the spatial location information of the data cluster and the natural language semantic tags, a multidimensional semantic map is constructed; In response to natural language-based query requests, a search is performed in the multidimensional semantic map to provide feedback.

2. The method according to claim 1, characterized in that, The method further includes, before or during the acquisition of the multimodal data: A geometric map of the environment is created using the Simultaneous Localization and Mapping (SLAM) algorithm. And based on the geometric map, spatial location information is added to the collected multimodal data.

3. The method according to claim 1 or 2, characterized in that, The extraction of the corresponding multimodal features includes: The image data is subjected to panoramic segmentation to extract visual features; And perform sound event detection on the sound data to extract audio features.

4. The method according to any one of claims 1 to 3, characterized in that, Before performing clustering using the unsupervised clustering algorithm, the method further includes: A multimodal fusion model based on a converter architecture is adopted to fuse features from different modalities into a unified feature vector for subsequent clustering analysis.

5. The method according to any one of claims 1 to 4, characterized in that, The unsupervised clustering algorithm includes at least one of the K-Means algorithm, DBSCAN algorithm, or Gaussian mixture model.

6. The method according to any one of claims 1 to 5, characterized in that, The step of generating natural language semantic labels for the data cluster using a pre-trained multimodal large model includes: Select one or more representative multimodal data samples from the data cluster; The representative multimodal data samples are input into the multimodal large model to obtain the natural language semantic labels.

7. The method according to any one of claims 1 to 6, characterized in that, The multidimensional semantic map is constructed based on a graph database; In this system, the natural language semantic labels serve as nodes in the graph database, and edges are established between the nodes to represent semantic relationships.

8. A semantic map creation and usage system, characterized in that, include: A data acquisition module is used to collect multimodal data from the environment by a mobile robot and to attach corresponding spatial location information to the multimodal data. The multimodal data includes at least image data and sound data. The feature extraction module is used to process the multimodal data to extract the corresponding multimodal features; An unsupervised clustering module is used to cluster the multimodal features using an unsupervised clustering algorithm to form at least one data cluster; The semantic label generation module is used to generate natural language semantic labels for the data cluster using a pre-trained multimodal large model; The map building module is used to construct a multidimensional semantic map based on the spatial location information of the data cluster and the natural language semantic tags; The human-computer interaction module is used to respond to natural language-based query requests and to perform retrievals in the multidimensional semantic map to provide feedback.

9. A mobile robot, characterized in that, Includes the system described in claim 8.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the method of claim 1.

Images