Semantic map construction method, autonomous mobile device control method and related device

Abstract

The invention discloses a semantic map construction method, an autonomous mobile device control method and a related device. The method comprises the following steps: acquiring a target image with depth information in a target scene; obtaining an object mask in the target image by using the label extraction network, the detection frame extraction network and the mask extraction network; extracting spatial information from the target image; fusing the object mask and the spatial information to form a semantic point cloud; and carrying out downsampling on the semantic point cloud, and projecting the semantic point cloud into a grid map to form a semantic map. Through the mode, a more accurate and more comprehensive semantic map is constructed, so that the navigation and positioning accuracy of subsequent automatic mobile equipment is improved.

Description

Technical Field

[0001] The present application relates to the technical field of automatic mobile device control, and in particular to the construction of a semantic map and a control method and related apparatus for an autonomous mobile device. Background Art

[0002] Navigation is one of the basic capabilities that robots (autonomous mobile devices) can use to move and perform tasks autonomously in physical space. Whether it is a handling robot used in industrial automation, a food delivery robot in the service industry, or a cleaning robot in the home, the capability of the navigation system directly determines the practicality and functional diversity of the robot. When performing natural language navigation tasks, the robot must understand human intentions based on language instructions and navigate to the correct location. The modular approach dissects the entire navigation process into three parts: perception, decision-making, and execution: the perception module builds a semantic map through sensor data, the decision module plans the path based on instructions and maps, and the execution module converts decisions into actual actions. This approach enables each module to be optimized and improved independently, improving the flexibility and adaptability of the overall system.

[0003] Although this approach has been successful, previous studies have been limited by the performance of the model. The construction of the semantic map was not accurate or comprehensive enough, resulting in problems with the robot finding its way around or not finding its way around during navigation. Summary of the invention

[0004] In order to solve the above problems, the semantic map construction method, autonomous mobile device control method, autonomous mobile device, and readable storage medium provided in the present application can construct a more accurate and comprehensive semantic map, thereby improving the navigation and positioning accuracy of subsequent automatic mobile devices and solving the problem of inaccurate or non-locating autonomous mobile devices during the navigation process.

[0005] To solve the above technical problems, a technical solution adopted in the present application is: to provide a method for constructing a semantic map, the method comprising: obtaining a target image with depth information in a target scene; obtaining an object mask in the target image using a label extraction network, a detection box extraction network and a mask extraction network; extracting spatial information from the target image; fusing the object mask and the spatial information to form a semantic point cloud; downsampling the semantic point cloud and projecting it into a grid map to form a semantic map.

[0006] Among them, the object mask in the target image is obtained by using a label extraction network, a detection frame extraction network and a mask extraction network, including: using a label extraction network to obtain the object label in the target image; using a detection frame extraction network to extract the detection frame of the target image with label information, and determine the detection frame of the object represented by each label; using a mask extraction network to extract the mask of the object with the detection frame, and obtain the object mask corresponding to each object in the target image.

[0007] Among them, the semantic point cloud is downsampled and projected into a grid map to form a semantic map, including: downsampling the semantic point cloud and projecting it into a grid map to obtain a grid map to be processed; determining the label confidence of each semantic point based on the label information corresponding to each semantic point in each grid in the grid map to be processed; determining the target semantic point based on the label confidence to form a semantic map.

[0008] Among them, the target semantic point is determined according to the label confidence to form a semantic map, including: grouping all semantic points in each grid according to the average height of the semantic points; retaining the label with the highest built-in confidence in each group; and constructing a top-down semantic map based on the height of all labels in each group.

[0009] Among them, the identifier of the label with the highest height in each grid is embedded into the semantic map.

[0010] The method comprises: downsampling the semantic point cloud and projecting it into the grid map to obtain the grid map to be processed, comprising: downsampling the semantic point cloud and projecting it into the grid map; filtering the labels in the grid map according to the frequency and score of each label to obtain the grid map to be processed.

[0011] To solve the above technical problems, another technical solution adopted in the present application is: to provide a control method for an autonomous mobile device, the method comprising: receiving natural language instructions; parsing the natural language instructions using a large language model, and determining the destination of the autonomous mobile device in combination with a semantic map; wherein the semantic map is constructed using the method provided by any of the above technical solutions; and planning the mobile tasks of the autonomous mobile device based on the destination.

[0012] Among them, a large language model is used to parse natural language instructions, and combined with a semantic map, the destination of the autonomous mobile device is determined, including: using a large language model to parse natural language instructions, and obtaining an API interface corresponding to the parsed natural language instructions from an API library; determining the destination of the autonomous mobile device from a semantic map according to the parsed natural language instructions; planning the mobile task of the autonomous mobile device based on the destination, including: planning the mobile task of the autonomous mobile device based on the destination and the API interface.

[0013] In order to solve the above technical problems, another technical solution adopted in the present application is: to provide an autonomous mobile device, which includes: a processor and a memory and an execution mechanism coupled to the processor; wherein the memory is used to store computer programs, and the processor is used to execute computer programs to implement a method provided by any of the above technical solutions.

[0014] To solve the above technical problems, another technical solution adopted in the present application is: to provide a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, it is used to implement the method provided by any of the above technical solutions.

[0015] The beneficial effects of the embodiments of the present application are as follows: different from the prior art, the semantic map construction method, the control method of the autonomous mobile device, the autonomous mobile device, and the readable storage medium provided by the present application obtain the object mask in the target image with depth information in the target scene by utilizing the label extraction network, the detection box extraction network, and the mask extraction network; and extract the spatial information from the target image; fuse the object mask and the spatial information to form a semantic point cloud; downsample the semantic point cloud and project it into a grid map to form a semantic map, thereby constructing a more accurate and comprehensive semantic map, thereby improving the navigation and positioning accuracy of subsequent automatic mobile devices and solving the problem of inaccurate and non-found positioning of autonomous mobile devices during navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0017] Figure 1 It is a flowchart of an embodiment of a method for constructing a semantic map provided by the present application;

[0018] Figure 2 This is a flowchart of another embodiment of the method for constructing a semantic map provided by the present application.

[0019] Figure 3 yes Figure 2 A schematic diagram of a flow chart of an embodiment of step 22;

[0020] Figure 4 yes Figure 2 A schematic diagram of the process of an embodiment of step 25;

[0021] Figure 5 yes Figure 4 A flow chart of an embodiment of step 253;

[0022] Figure 6 It is a flowchart of an embodiment of a control method of an autonomous mobile device provided by the present application;

[0023] Figure 7 is a flow chart of another embodiment of the control method of an autonomous mobile device provided by the present application;

[0024] Figure 8 It is a structural schematic diagram of an embodiment of an autonomous mobile device provided by the present application;

[0025] Fig. 9 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0027] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0028] Navigation is one of the basic capabilities that enables robots to move autonomously and perform tasks in physical space. Whether it is a handling robot used in industrial automation, a food delivery robot in the service industry, or a cleaning robot in the home, the capability of the navigation system directly determines the practicality and functional diversity of the robot. When performing natural language navigation tasks, the robot must understand human intentions based on language instructions and navigate to the correct location. The modular approach dissects the entire navigation process into three parts: perception, decision-making, and execution: the perception module builds a semantic map through sensor data, the decision module plans the path according to instructions and maps, and the execution module converts decisions into actual actions. This approach enables each module to be optimized and improved independently, improving the flexibility and adaptability of the overall system.

[0029] Although this approach has been successful, previous studies have been limited by the performance of the model. The construction of the semantic map was not accurate or comprehensive enough, resulting in problems with the robot finding its way around or not finding its way around during navigation.

[0030] Based on this, the present application obtains the object mask in the target image with depth information in the target scene by using the label extraction network, the detection frame extraction network and the mask extraction network; extracts the spatial information from the target image; fuses the object mask and the spatial information to form a semantic point cloud; downsamples the semantic point cloud and projects it into a grid map to form a semantic map, thereby constructing a more accurate and comprehensive semantic map, thereby improving the navigation and positioning accuracy of subsequent automatic mobile devices, and solving the problem of inaccurate or unavailable positioning of autonomous mobile devices during navigation. Please refer to any of the following embodiments for details.

[0031] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of a method for constructing a semantic map provided by the present application. The method comprises:

[0032] Step 11: Obtain a target image with depth information in the target scene.

[0033] In some embodiments, an image acquisition component with a depth image acquisition function on an autonomous mobile device may be used in advance to acquire a target image with depth information in a target scene.

[0034] In some embodiments, the target image with depth information may be presented in the form of an image stream.

[0035] In some embodiments, the target image with depth information may be a color image, such as an RGB-D image, where RGB represents color and D represents depth.

[0036] In some embodiments, the target image with depth information may be a grayscale image.

[0037] In some embodiments, the target image with depth information in the target scene can be acquired by other image acquisition components with depth image acquisition function.

[0038] Step 12: Use the visual model to obtain the object mask in the target image.

[0039] In some embodiments, the visual model can be constructed based on a neural network model. After the visual model is trained, the visual model is used to obtain an object mask in the target image. In some embodiments, when there are multiple objects in the target image, each object corresponds to a target mask. The object mask is composed of multiple target masks. The target mask can reflect the area size of the target and the corresponding pixel points.

[0040] In some embodiments, when there is an object in the target image, the object corresponds to a target mask, and the object mask is composed of the target mask.

[0041] In image processing, masking is a technique used to specify specific areas or pixels in an image. A mask is usually a matrix of the same size as the original image, in which the element values ​​can be used to identify whether the pixels at the corresponding position should participate in a specific processing operation. Through the mask, the area of ​​image processing can be precisely controlled to achieve refined operations on the image.

[0042] Step 13: Extract spatial information from the target image.

[0043] In some embodiments, the spatial information of the world coordinates corresponding to each pixel can be extracted from the target image in combination with the conversion relationship between the camera coordinate system of the image acquisition component and the world coordinate system.

[0044] Step 14: Fusion of object mask and spatial information to form semantic point cloud.

[0045] In some embodiments, by fusing the object mask and the spatial information, each pixel corresponding to the mask has the spatial information and the label, thereby forming a point cloud with semantics. The semantics in the semantic point cloud may correspond to the label of the point. For example, if the label of the point is a chair, the semantics of the point is a chair. The object mask may reflect the area size of the object and the corresponding pixel points, while the spatial information may reflect the position of each pixel point in space, and each object may form a corresponding semantic point cloud through the spatial information and the object mask.

[0046] Step 15: Downsample the semantic point cloud and project it into the grid map to form a semantic map.

[0047] In some embodiments, there is a lot of redundant data in the semantic point cloud, and downsampling can be used to effectively reduce the data and reduce the subsequent calculation amount.

[0048] In some embodiments, the semantic point cloud may be downsampled by at least one of random downsampling, uniform downsampling, voxel downsampling, farthest point downsampling, moving least squares downsampling, and normal space sampling.

[0049] In some embodiments, the semantic point cloud is downsampled and projected into a grid map to form a semantic map. The specific method of forming the semantic map can be found in any of the following embodiments, which will not be described in detail here.

[0050] In this embodiment, a target image with depth information in the target scene is obtained, the object mask in the target image is obtained by using a visual model, spatial information is extracted from the target image, the object mask and spatial information are fused to form a semantic point cloud, the semantic point cloud is downsampled, and projected into a grid map to form a semantic map. In this way, a more accurate and comprehensive semantic map is constructed, thereby improving the navigation and positioning accuracy of subsequent automatic mobile devices and solving the problem of inaccurate or non-found positioning of autonomous mobile devices during navigation.

[0051] See also Figure 2 , Figure 2 This is a flow chart of another embodiment of constructing a semantic map provided by the present application. The method includes:

[0052] Step 21: Obtain a target image with depth information in the target scene.

[0053] Step 22: Obtain the object mask in the target image using the label extraction network, the detection box extraction network and the mask extraction network.

[0054] The above-mentioned visual model includes a label extraction network, a detection box extraction network, and a mask extraction network. Figure 3 , step 22 may be the following process:

[0055] Step 221: Obtain object labels in the target image using a label extraction network.

[0056] In some embodiments, the label extraction network can be constructed based on the RAM model. The label extraction network can obtain the label of any object in the target image. For example, if there are objects such as chairs, tables, and stools in the target image, the label extraction network obtains the object labels of these chairs, tables, and stools in the target image. That is, after the target image is input into the label extraction network, the label extraction network will output all the object labels in the target image.

[0057] In some embodiments, the RAM model has zero-shot label recognition capability.

[0058] RAM model, as a powerful image recognition model, uses the principle of zero-shot learning, and can understand and complete new tasks with no or very few examples, showing strong zero-shot and few-shot learning capabilities. This means that RAM model can not only process the data encountered in training, but also recognize and classify the data that has not been seen. This ability is the core of zero-shot learning.

[0059] In addition, during the design and training process of the RAM model, large-scale image text data was used for pre-training. These data sets contain a variety of unannotated image labels. Through a complex data processing process, including automatic text semantic parsing, label data cleaning and optimization, the RAM model is able to improve the quality of annotations and ensure clearer and more accurate recognition. This capability enables the RAM model to perform well in tasks such as classification, detection, and segmentation, especially when dealing with unseen data, that is, open set recognition capabilities, demonstrating the strong potential and application value of the RAM model.

[0060] In summary, the RAM model achieves zero-shot label recognition capability through its unique training method and architecture, which is a revolutionary technological advancement and brings new possibilities to the field of image recognition.

[0061] In some embodiments, the label extraction network only extracts labels for the target image.

[0062] Step 222: Utilize the detection frame extraction network to extract the detection frame of the target image with label information, and determine the detection frame of the object represented by each label.

[0063] In some embodiments, the detection frame extraction network can be constructed based on the Grounding DINO model. The detection frame extraction network performs detection frame extraction on the target image with label information to determine the detection frame of the object represented by each label. For example, if there are objects such as chairs, tables, and stools in the target image, the label extraction network obtains the object labels of these chairs, tables, and stools in the target image. After that, the detection frame extraction network is used to perform detection frame extraction on the target image with object labels of chairs, tables, and stools, thereby extracting the detection frames of the objects represented by the chairs, tables, and stools. That is, the pixels within the detection frame can represent the corresponding objects, such as chairs, tables, or stools.

[0064] The Grounding DINO model is an open-set detection model that is based on the Transformer pre-training model and solves the open-set detection task by introducing language information.

[0065] The core method of the Grounding DINO model is to solve the open set detection task by combining language and visual modality features. In order to effectively integrate language and visual modality features, the model sets three stages and constructs feature enhancer, language-guided query selection and cross-modality decoder for cross-modal fusion. This method is different from previous studies. Previous studies basically use models to evaluate open datasets, while GroundingDINO evaluates open set data through tags. Its performance on the three datasets has reached the most advanced level.

[0066] In addition, the Grounding DINO model also solves the hallucination problem of multimodal large models, and enhances AI's understanding of the relationship between visual content and text by accurately matching phrases in long text descriptions with specific objects or scenes in images. Grounding DINO excels in semantic understanding of images and text, and can quickly and accurately detect and identify target objects in images based on language cues.

[0067] In general, the Grounding DINO model provides a new solution for open set detection tasks by combining language information and visual features, as well as optimizing for different application scenarios, especially showing great potential in edge deployment and embodied intelligence.

[0068] In some embodiments, the detection box extraction network only extracts the detection box of the target image.

[0069] Step 223: Use the mask extraction network to extract the mask of the object with the detection frame to obtain the object mask corresponding to each object in the target image.

[0070] In some embodiments, the mask extraction network can be constructed based on the SAM model. The mask extraction network performs mask extraction on objects with detection frames to obtain object masks corresponding to each object in the target image. For example, after the detection frame extraction network extracts the detection frames of objects represented by chairs, tables, and stools, the mask extraction network determines the masks corresponding to the chairs, tables, and stools from each detection frame.

[0071] In some embodiments, the SAM model has zero-shot localization capability.

[0072] The main features and functions of the SAM model include:

[0073] Image segmentation without annotation: SAM can segment any object in any image without any annotation, which greatly reduces the demand for professional annotated data and simplifies the image segmentation process.

[0074] Wide range of application scenarios: The SAM model has a wide range of application scenarios and can be used in any scenario that requires image recognition and segmentation. For example, in the AR / VR field, SAM can upgrade the object the user is looking at to 3D; for content creators, SAM can extract images for collage or video editing.

[0075] Powerful training dataset: The SA-1B database used by SAM includes 11 million high-resolution images and 1.1 billion high-quality segmentation masks, which is one of the largest segmentation datasets currently available, ensuring the accuracy and generalization ability of the model.

[0076] Zero-shot learning and multimodal processing: Through hint engineering, SAM can quickly adapt to new tasks and data distributions to achieve zero-shot learning. This enables SAM to handle a variety of tasks, including edge detection, object proposal generation, instance segmentation, and preliminary text-to-mask prediction.

[0077] In some embodiments, the mask extraction network only performs object mask extraction on the target image.

[0078] Step 23: Extract spatial information from the target image.

[0079] In some embodiments, the spatial information of the world coordinates corresponding to each pixel can be extracted from the target image in combination with the conversion relationship between the camera coordinate system of the image acquisition component and the world coordinate system.

[0080] Step 24: Fusion of object mask and spatial information to form semantic point cloud.

[0081] By fusing the object mask and spatial information, each pixel corresponding to the mask has spatial information and a label, thereby forming a point cloud with semantics. The semantics in the semantic point cloud can correspond to the label of the point. For example, if the label of the point is chair, then the semantics of the point is chair.

[0082] Step 25: Downsample the semantic point cloud and project it into the grid map to form a semantic map.

[0083] In some embodiments, see Figure 4 , step 25 may be the following process:

[0084] Step 251: downsample the semantic point cloud and project it into a grid map to obtain a grid map to be processed.

[0085] In some embodiments, the semantic point cloud is downsampled and projected into a grid map; the tags in the grid map are filtered according to the frequency and score of each tag to obtain a grid map to be processed.

[0086] In an application scenario, during the semantic segmentation of the target image in the target scene, each label will appear at least once, so the frequency and score of each label can be counted. The score can be the confidence of the label. Further, the score can be an average score, for example, each time the label appears, there is a corresponding score, so all the scores of the label are averaged to obtain the average score.

[0087] The tags with a frequency lower than a threshold or a low score, or the tags with a frequency lower than a threshold and a low score are filtered out to obtain a grid map to be processed.

[0088] When the score is an average score, the tags with a frequency lower than the threshold or a low average score, or the tags with a frequency lower than the threshold and a low average score are filtered to obtain a grid map to be processed.

[0089] Step 252: Determine the label confidence of each semantic point based on the label information corresponding to each semantic point in each grid in the grid map to be processed.

[0090] Confidence is a statistical concept used to describe the probability of the relationship between the measured value and the true value. That is, confidence is used to indicate the probability that a semantic point belongs to a certain label. The higher the confidence, the greater the probability that the semantic point belongs to a certain label.

[0091] In some embodiments, there may be multiple semantic points in each grid. Based on this, each semantic point has a corresponding label and a score corresponding to the label. Based on this, the label confidence of each semantic point can be determined using the number of labels and the score.

[0092] Furthermore, the confidence of each label in each grid can also be calculated. For example, the confidence of each label can be calculated using the data of the label in each grid and the score of each label, and the total score.

[0093] Step 253: Form a semantic map based on the label confidence.

[0094] In some embodiments, see Figure 5 , step 253 may be the following process:

[0095] Step 51: Group all labels in each grid according to the average height of the labels.

[0096] In the same grid, there are different objects with different heights, and the objects have corresponding labels, so each label also corresponds to a different height. In order to distinguish them, all labels in each grid are grouped according to the average height of the label, so as to distinguish different objects.

[0097] Step 52: Keep the label with the highest built-in confidence in each group.

[0098] Since the labels in each group most likely correspond to the same object, the label with the highest confidence is retained.

[0099] Step 53: Construct a top-down semantic map based on the heights of all labels in each group.

[0100] That is, a top-down semantic map is constructed based on the descending order of the height of the remaining labels in each grid.

[0101] For example, if the remaining labels in the grid W include label a, label b, and label c, then a top-down semantic map is constructed based on the descending order of the heights of label a, label b, and label c. If the height corresponding to label a is greater than label b and less than label c, the top-down order is label c, label a, and label b.

[0102] In some embodiments, the identifier of the highest-height label in each grid is embedded into the semantic map. The identifier may be an identity tag (ID).

[0103] In this embodiment, a label extraction network, a detection box extraction network and a mask extraction network are used to obtain an object mask in a target image with depth information in a target scene; spatial information is extracted from the target image; the object mask and spatial information are fused to form a semantic point cloud; the semantic point cloud is downsampled and projected into a grid map to form a semantic map, thereby constructing a more accurate and comprehensive semantic map, thereby improving the navigation and positioning accuracy of subsequent automatic mobile devices and solving the problem of inaccurate or non-found positioning of autonomous mobile devices during navigation.

[0104] In all scenarios, first, a large vision model is used to obtain the segmentation masks of objects from the RGB image, and the point clouds of these objects are derived from the depth image. Specifically, a semantic point cloud is created by merging these masks with the corresponding point clouds. The semantic point cloud is then downsampled and projected down onto the ground to achieve the desired semantic map. Specifically, the RAM model, the Grounding DINO model, and the SAM model are combined to obtain the semantic mask of the image.

[0105] In an application scenario, define a semantic map M∈R H×WIn , H and W represent the height and width of the grid map from a top-down perspective, respectively. Each grid contains a unique identifier i, which represents an object label. The robot (autonomous mobile device) is equipped with RGB and depth cameras. Let the robot explore the environment comprehensively, collecting RGB-D frames and recording its current posture information during the process. The total number of data sequences is denoted by N. Then, spatial and semantic information is extracted from the collected data. In order to accurately identify and locate the target in the image, RAM, Ground Dino and SAM are used to extract the object mask in the RGB frame. And use a matrix A of the same size as the image to record the IDi and score s corresponding to the object in the mask, as shown below:

[0106] A(x f ,y f )=(i,s);

[0107] Among them, x f ,y f represents the position of the pixel in the image, x f ∈(0,W ′ ), y f ∈(0,H ′ ), W ′ and H ′ Indicates the height and width of the image.

[0108] At the same time, all depth images in the depth map d = (x f ,y f ) are converted to point P in the current camera coordinate system k Then, the point P in the world coordinate system is calculated by the following transformation matrix: w =(x,y,h):

[0109]

[0110] in, K is the intrinsic parameter matrix of the depth camera, and Q(d)∈R is the depth value of pixel d. Wk is the transformation from the world coordinate system to the kth camera coordinate system.

[0111] After obtaining semantic information from the RGB frame and spatial information from the depth map, the two are closely combined. For example, the object mask is matched with the depth map pixel by pixel. For each pixel at the same position, A is matched with P corresponding to d. w Integrate and get the semantic point P′ w Then P′ w Mapped to the corresponding grid on the grid map M, we get P g , which can be expressed as:

[0112]

[0113] in, and They represent the positions on the grid map corresponding to x and y in the world coordinate system, respectively, and a is the size of a grid on the map.

[0114] Furthermore, when processing RGB-D streams, multiple semantic points are often projected to the same grid position on the map. In order to ensure that the most accurate labels are retained, the frequency of label occurrence and their corresponding scores are considered for filtering. Specifically, the total frequency N_i and the average score S_i of each label during the semantic segmentation of all RGB frames are first counted. In this process, labels with extremely low frequencies throughout the perception process, or those with both low frequencies and low average scores can be considered unreliable. These labels are recorded in the list i∈D as follows:

[0115]

[0116] Among them, α, β and γ represent threshold values.

[0117] Then the labels in the grid are filtered according to D. After filtering the labels as a whole, the labels of each grid are further filtered. When processing the data stream, the data of Pg is also accumulated and represented as g. For example, when the label c is located at the grid position hour:

[0118]

[0119] in, and Respectively indicate that label c is in the same grid The score and height of the kth occurrence in the grid. n, s, and h represent the total occurrence, cumulative score, and total height of a label in the grid, respectively.

[0120] After processing all frames, calculate the average score for each label in the grid and average height Assume that a label detected multiple times at the same position is considered reliable if the average score is high. Therefore, the confidence p of each label in the grid is calculated using the following formula:

[0121]

[0122] Among them, w1 and w2 represent weights.

[0123] Sometimes, you will have different objects of different heights in the same grid, such as a "stool" under a "table". To layer objects, you can layer them according to their average height. Group the labels in the grid with similar The labels are divided into a group. In each group, only the label with the highest confidence p is retained. Then, these selected labels are sorted in descending order according to their height. The label with IDi at the top of this hierarchy is embedded into the grid map M to construct a top-down semantic grid map, represented as

[0124] After the semantic map is constructed, the semantic map is imported into the autonomous mobile device, and the autonomous mobile device can receive natural language instructions according to the semantic map and move.

[0125] Furthermore, in order to effectively identify and locate objects (targets or objects) in the semantic map, a semantic dictionary L = label:i is constructed to record the names of all objects and their corresponding unique identifiers i. When searching for the location of a specific object, its unique identifier i is first found in the dictionary L. Subsequently, all grids equal to i are searched in the semantic map M, and these grids are regarded as potential locations of the object. These grids are grouped into clusters using a density-based clustering algorithm, and the center coordinates p of the object on the map are calculated. center .

[0126] Secondly, to address situations where the query object name is similar but not exactly the same as the semantic label recorded in the dictionary, such as "TV" vs. "television" and "couch" vs. "sofa", LLMs (Large Language Models) are integrated into the system of autonomous mobile devices. Large language models excel in interpreting and associating semantics, and therefore help determine the closest matching dictionary label for any given query. If no semantics that fully matches the query object is found in the dictionary, the large language model will be queried to see which object in the dictionary best matches the target term.

[0127] Specifically, some pre-written executable codes are prepared and encapsulated in a local API library for the robot (autonomous mobile device) to call. The functions of each API are explained with a promt (prompt) and paired with examples of natural language instructions. During the robot's task execution, new natural language instructions are embedded in this prompt as input to the LLMs. Then, the LLMs generate corresponding codes for these instructions. Subsequently, the robot completes the task by executing these generated codes. For the specific execution, the global plus local planning method is used, and the specific linear velocity v and angular velocity ω of the robot are calculated in each execution cycle.

[0128] See also Figure 6 , Figure 6 1 is a flow chart of an embodiment of a control method for an autonomous mobile device provided by the present application. The method comprises:

[0129] Step 61: Receive natural language instructions.

[0130] In some embodiments, the natural language instruction can be collected by a microphone on an autonomous mobile device. For example, when a user speaks and generates corresponding audio (which can be used as a natural language instruction), the microphone can collect the audio.

[0131] Step 62: parse the natural language instruction using the large language model and combine it with the semantic map to determine the destination of the mobile device.

[0132] Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and are an important path to artificial intelligence. Currently, large language models use a Transformer architecture and pre-training objectives (such as Language Modeling) similar to small models. The difference from small models is the increase in model size, training data, and computing resources.

[0133] In some embodiments, the semantic map can be obtained in advance. For example, an autonomous mobile device is used to collect corresponding images with depth information in the target scene in advance, and the image is recognized to construct a semantic map corresponding to the target scene. The specific method of constructing the semantic map can be found in any of the above embodiments, which will not be described here.

[0134] In some embodiments, the semantic map corresponds to the semantics of the corresponding target and the location information of the target. For example, the semantic map corresponds to target A, target B, and target C. When the user outputs a natural language instruction of "go to target A", the autonomous mobile device receives the natural language instruction and parses it to go to target A. This indicates that the destination of the autonomous mobile device is "target A".

[0135] Step 63: Plan a movement task of the autonomous mobile device based on the destination.

[0136] When the destination of the autonomous mobile device is "target A", the location information of target A is found from the semantic map. Then the mobile task of the autonomous mobile device is planned according to the current location information of the autonomous mobile device and the location information of target A. For example, the initial moving trajectory is determined by combining the current location information and the location information of target A, and then the initial moving trajectory is adjusted according to other targets along the path of the initial moving trajectory to obtain the final moving trajectory. Then the autonomous mobile device moves to the destination according to the final moving trajectory.

[0137] In this embodiment, a large language model is used to understand natural language instructions, and combined with a constructed semantic map, the destination of the autonomous mobile device's navigation is located, so that the optimal path is planned and actions are executed to complete the task. Based on this, integrating the language model into navigation improves the ability of autonomous mobile devices to perform more complex and diverse navigation tasks.

[0138] See also Figure 7 , Figure 7 1 is a flow chart of another embodiment of the control method of the autonomous mobile device provided by the present application. The method comprises:

[0139] Step 71: Receive natural language instructions.

[0140] Step 72: parse the natural language instruction using the large language model, and obtain the API interface corresponding to the parsed natural language instruction from the API library.

[0141] In some embodiments, the API library may pre-set API interfaces or executable codes corresponding to corresponding functions. These functions may correspond to corresponding natural language instructions. For example, the forward function, backward function, turn function, acceleration function and other functions of the autonomous mobile device all correspond to corresponding API interfaces or executable codes. For example, if the natural language instruction is parsed and presented as "accelerate and move forward 10 meters", the API interface or executable code corresponding to the forward function and the acceleration function can be obtained from the API library.

[0142] Step 73: Determine the destination of the autonomous mobile device from the semantic map according to the parsed natural language instruction.

[0143] Step 74: Plan the mobile task of the autonomous mobile device based on the destination and the API interface.

[0144] In some embodiments, the moving task may be reflected in calling an API interface to execute the corresponding function and then move to the target location.

[0145] In this embodiment, a large language model is used to understand natural language instructions, and combined with a constructed semantic map, the destination of the autonomous mobile device navigation is located, so that the optimal path is planned and then the action is executed to complete the task. Based on this, the language model is integrated into navigation to improve the ability of autonomous mobile devices to perform more complex and diverse navigation tasks. Autonomous mobile devices no longer just follow fixed programs for navigation, but can follow human instructions to perform more complex and diverse navigation tasks, making them more efficient and intelligent in practical applications.

[0146] See also Figure 8 , Figure 81 is a schematic diagram of the structure of an embodiment of an autonomous mobile device provided by the present application. The autonomous mobile device 80 includes a processor 81, a memory 82 coupled to the processor 81, and an execution mechanism 83; wherein the memory 82 is used to store a computer program, and the processor 81 is used to execute the computer program to implement the following method:

[0147] Obtain a target image with depth information in the target scene; use the label extraction network, the detection box extraction network and the mask extraction network to obtain the object mask in the target image; extract spatial information from the target image; fuse the object mask and spatial information to form a semantic point cloud; downsample the semantic point cloud and project it into a grid map to form a semantic map;

[0148] Or, receive natural language instructions; use a large language model to parse the natural language instructions, and combine with a semantic map to determine the destination of the autonomous mobile device; plan the mobile task of the autonomous mobile device based on the destination.

[0149] It can be understood that the processor 81 is also used to execute computer programs to implement the method of any of the above embodiments, which will not be elaborated here.

[0150] See also Fig. 9 , Fig. 9 1 is a schematic diagram of a computer readable storage medium according to an embodiment of the present application. The computer readable storage medium 90 is used to store a computer program 91. When the computer program 91 is executed by a processor, it is used to implement the following method:

[0151] Obtain a target image with depth information in the target scene; use the label extraction network, the detection box extraction network and the mask extraction network to obtain the object mask in the target image; extract spatial information from the target image; fuse the object mask and spatial information to form a semantic point cloud; downsample the semantic point cloud and project it into a grid map to form a semantic map;

[0152] Or, receive natural language instructions; use a large language model to parse the natural language instructions, and combine with a semantic map to determine the destination of the autonomous mobile device; plan the mobile task of the autonomous mobile device based on the destination.

[0153] It can be understood that when the computer program 91 is executed by the processor, it is also used to implement the method of any of the above embodiments, which will not be elaborated here.

[0154] In summary, the semantic map construction method, autonomous mobile device control method, autonomous mobile device, and readable storage medium provided by the present application obtain object masks in target images with depth information in target scenes by utilizing label extraction networks, detection box extraction networks, and mask extraction networks; and extract spatial information from the target images; fuse the object masks and spatial information to form semantic point clouds; downsample the semantic point clouds and project them onto a grid map to form a semantic map, thereby constructing a more accurate and comprehensive semantic map, thereby improving the navigation and positioning accuracy of subsequent automatic mobile devices and solving the problem of inaccurate or non-found locations of autonomous mobile devices during navigation.

[0155] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of the circuit or unit is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0156] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0157] In addition, each functional unit in each embodiment of the present application may be integrated into a first processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0158] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made according to the description and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for constructing a semantic map, characterized in that: The method comprises: Acquire a target image with depth information in a target scene; Obtaining an object mask in the target image using a label extraction network, a detection box extraction network, and a mask extraction network; extracting spatial information from the target image; fusing the object mask and the spatial information to form a semantic point cloud; The semantic point cloud is downsampled and projected into a grid map to form the semantic map.

2. The method according to claim 1, characterized in that The method of obtaining the object mask in the target image by using the label extraction network, the detection frame extraction network and the mask extraction network includes: Using the label extraction network to obtain the object label in the target image; Using the detection frame extraction network to extract the detection frame of the target image with label information, and determine the detection frame of the object represented by each label; The mask extraction network is used to extract the mask of the object with the detection frame to obtain the object mask corresponding to each object in the target image.

3. The method according to claim 1, characterized in that The downsampling of the semantic point cloud and projecting it into a grid map to form the semantic map includes: Downsampling the semantic point cloud and projecting it into a grid map to obtain a grid map to be processed; Determining the label confidence of each semantic point according to the label information corresponding to each semantic point in each grid in the grid map to be processed; The semantic map is formed according to the label confidence.

4. The method according to claim 3, characterized in that The forming of the semantic map according to the label confidence comprises: Group all labels in each grid according to their average height; Keep each set of labels with the highest built-in confidence; According to the heights of all the labels in each group, a top-down semantic map is constructed.

5. The method according to claim 4, characterized in that The identifier of the highest-height label in each grid is embedded into the semantic map.

6. The method according to claim 3, characterized in that The downsampling of the semantic point cloud and projecting it into a grid map to obtain a grid map to be processed includes: Downsampling the semantic point cloud and projecting it into a grid map; The tags in the grid map are filtered according to the frequency and score of each tag to obtain the grid map to be processed.

7. A control method for an autonomous mobile device, characterized in that: The method comprises: receiving natural language instructions; Parsing the natural language instruction using a large language model and combining it with a semantic map to determine the destination of the autonomous mobile device; wherein the semantic map is constructed using the method according to any one of claims 1 to 6; A movement task of the autonomous mobile device is planned based on the destination.

8. The method according to claim 7, characterized in that The using of a large language model to parse the natural language instruction and combining it with a semantic map to determine the destination of the autonomous mobile device includes: Parsing the natural language instruction using a large language model, and acquiring an API interface corresponding to the parsed natural language instruction from an API library; determining a destination of the autonomous mobile device from the semantic map according to the parsed natural language instruction; The step of planning a movement task of the autonomous mobile device based on the destination includes: A movement task of the autonomous mobile device is planned based on the destination and the API interface.

9. An autonomous mobile device, characterized in that: The autonomous mobile device comprises: a processor and a memory and an execution mechanism coupled to the processor; wherein the memory is used to store a computer program, and the processor is used to execute the computer program to implement the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the computer program is used to implement the method according to any one of claims 1 to 8.

Images