Robot positioning method and system based on generative model
By introducing a generative model-based method in robot positioning technology and using multimodal large models for environmental map matching and positioning, the problems of positioning accuracy and robustness of the existing technology in large-scale environments are solved, and high-precision and flexible robot positioning are achieved.
Patent Information
- Application Number
- CN202411654224.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-19
- Publication Date
- 2025-03-25
AI Technical Summary
Existing robot positioning technology is difficult to achieve real-time high-precision positioning in larger map environments, and is easily disturbed by factors such as lighting and occlusion, resulting in large positioning errors.
Using a robot positioning method based on a generative model, one to multiple candidate poses of the current local environment map by inputting the 2D raster environment map, local environment map and problem prompt text to trained generative models, using visual feature encoding network, vectorization module, feature map size adjustment layer, splicing layer and language model, one to multiple candidate poses of the current local environment map are output, and robot motion correction is used to obtain the best poses.
It significantly improves the accuracy and robustness of robot positioning, and can achieve fast matching positioning in various scenarios to meet the needs of efficient autonomous navigation in large-scale environments.
Smart Images
Figure CN119693601A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of robot navigation technology, and in particular to a robot positioning method and system based on a generative model. Background Art
[0002] Robot positioning means that when an environmental map has been established, the robot obtains environmental information about the surrounding environment based on its own measurement equipment, and can accurately know its position and posture by comparing it with the environmental map information. Precise robot positioning is the key to achieving efficient autonomous navigation and task execution.
[0003] In the past 20 years, robot positioning technology has been a research hotspot in the field of machine vision and artificial intelligence, and many classic algorithms have emerged, such as the Monte Carlo positioning algorithm, the adaptive Monte Carlo positioning algorithm, the extended Kalman filter algorithm, and the cartographer-based positioning algorithm. These algorithms are mainly based on line laser radar to perceive the surrounding environment information, and then find the best position and posture through feature matching with the existing 2D grid environment map to complete the self-positioning operation. These positioning methods can achieve very good positioning in a smaller environment, but in a larger map environment, these methods cannot perform real-time high-precision positioning due to the large amount of calculation required. At the same time, these methods are easily disturbed by factors such as lighting and occlusion, resulting in large positioning errors. Summary of the invention
[0004] Based on the technical problems existing in the background technology, the present invention proposes a robot positioning method and system based on a generative model, which greatly improves the accuracy and robustness of robot positioning.
[0005] The present invention proposes a robot positioning method based on a generative model, comprising the following steps:
[0006] Send any 2D grid environment map, the local environment map under the 2D grid environment map, and the question prompt text into a trained generative model, wherein the generative model includes a visual feature encoding network, a vectorization module, a feature map size adjustment layer, a splicing layer, and a language model;
[0007] The 2D grid environment map and the local environment map are sequentially passed through the visual feature encoding network and the feature map resizing layer to output the global feature vector and the local feature vector.
[0008] The question prompt text outputs a text feature vector through a vectorization module;
[0009] The global feature vector, local feature vector and text feature vector are concatenated through the concatenation layer and input into the language model to output one or more candidate poses of the current local environment map;
[0010] The motion-based robot corrects all candidate poses to obtain the best robot pose.
[0011] Furthermore, the resolution of the local environment map is smaller than the resolution of the 2D grid environment map.
[0012] Furthermore, the training process of the generative model is as follows:
[0013] Constructing a set of training sample data pairs and a set of question prompt texts, wherein one training sample data pair includes a 2D grid environment map and a local map under the 2D grid environment map;
[0014] Set the loss function of the generative model to be consistent with the loss function of the selected language model;
[0015] Select a training sample data pair from the training sample data pair set, and select a question prompt text from the question prompt text set to form a pair of joint training data;
[0016] Freeze the visual feature encoding network and vectorization module, and train the large language model;
[0017] Freeze the vectorization module and the language model to train the visual feature encoding network;
[0018] The training of the language large model and the visual feature encoding network is performed alternately to achieve the training goal of the generative model.
[0019] Furthermore, the process of obtaining the local map under the 2D grid environment map is as follows:
[0020] Choose an obstacle-free posture at random, use the current posture as the reference origin to emit rays in all directions, and record the first intersection posture of each ray and the obstacle in the 2D grid environment map;
[0021] Collect all the ray intersection pose information, and obtain the minimum bounding rectangle of all the ray intersection poses according to the standard of containing 95% of the ray intersections;
[0022] The 2D grid environment map is clipped according to the minimum enclosing rectangle. The local map obtained is the tentative local map of the robot at the current position.
[0023] 5% of the random obstacle poses are erased and added to the provisional local environment map to obtain a local map of the current pose under the 2D grid environment map.
[0024] Furthermore, the supervisory label of the current posture is obtained as the supervisory label of the training sample data pair. The process of obtaining the supervisory label is as follows: the reference origin posture coordinates are obtained and normalized, and the normalized reference origin posture coordinates are used as the supervisory label of the current posture, wherein the reference origin posture coordinates are the coordinates with the current posture as the reference origin.
[0025] A robot positioning system based on a generative model, comprising a generative model and a correction module, wherein the generative model comprises a visual feature encoding network, a vectorization module, a feature map size adjustment layer, a splicing layer and a language large model;
[0026] The visual feature encoding network is used to extract deep features from 2D grid environment maps and local environment maps;
[0027] The feature map size adjustment layer is used to adjust the size of the deep features output by the visual feature encoding network to output global feature vectors and local feature vectors;
[0028] The vectorization module is used to convert the question prompt text into a multi-dimensional feature vector of fixed length, thereby outputting a text feature vector;
[0029] The concatenation layer is used to concatenate the global feature vector, the local feature vector, and the text feature vector;
[0030] The language model outputs one or more candidate poses of the current local environment map based on the concatenated feature vector output by the concatenated layer;
[0031] The correction module is used to correct all candidate poses based on the motion of the robot to obtain the best robot pose.
[0032] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the robot positioning method as described above is implemented when the processor executes the computer program.
[0033] A computer-readable storage medium stores a plurality of classification programs, and the plurality of classification programs are used to be called by a processor and execute the robot positioning method as described above.
[0034] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.
[0035] The advantages of the generative model-based robot positioning method and system provided by the present invention are: the generative model-based robot positioning method and system provided in the structure of the present invention directly realizes the rapid matching and positioning of the local environment map in the global environment map through the generative capability of the multimodal large model, and can realize the rapid matching and positioning of the local environment map in the global environment map end-to-end, greatly improving the accuracy and robustness of the robot positioning, and can meet the robot positioning needs in various scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 It is a structural schematic diagram of the present invention;
[0037] Figure 2 It is the structural diagram of the generative model;
[0038] Figure 3 This is an example of a 2D grid environment map;
[0039] The letters and numbers next to each module graphic represent the output feature map size of the current module, that is, the number of feature map channels × feature map width × feature map height. DETAILED DESCRIPTION
[0040] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.
[0041] Recently, the large-scale multimodal generative models of vision and language have made significant progress. With their ultra-large parameters, the large-scale multimodal generative models of vision and language can simultaneously focus on data in both visual (such as images and videos) and language (such as text) modalities, and complete multiple complex tasks such as image description generation, visual question answering, and text-to-image generation.
[0042] like Figures 1 to 3 As shown, a robot positioning method based on a generative model proposed in the present invention comprises the following steps:
[0043] Send any 2D grid environment map, the local environment map under the 2D grid environment map, and the question prompt text into a trained generative model, wherein the generative model includes a visual feature encoding network, a vectorization module, a feature map size adjustment layer, a splicing layer, and a language model;
[0044] The 2D grid environment map and the local environment map are sequentially passed through the visual feature encoding network and the feature map resizing layer to output the global feature vector and the local feature vector.
[0045] The question prompt text outputs a text feature vector through a vectorization module;
[0046] The global feature vector, local feature vector and text feature vector are concatenated through the concatenation layer and input into the language model to output one or more candidate poses of the current local environment map;
[0047] The motion-based robot corrects all candidate poses to obtain the best robot pose.
[0048] This embodiment proposes a robot positioning method, which directly realizes the rapid matching and positioning of the local environment map in the global environment map through the generative capability of the multimodal large model. It can realize the rapid matching and positioning of the local environment map in the global environment map end-to-end, greatly improve the accuracy and robustness of the robot positioning, and can meet the robot positioning needs in various scenarios.
[0049] This embodiment is applicable to the scenario where the robot needs to be accurately positioned in any form of 2D grid environment map. For the convenience of description, this embodiment takes a 2D laser radar robot and a 2D grid environment map as an example. Figure 3 shown.
[0050] To facilitate the description of this embodiment, some terms are defined: feature map resolution refers to feature map width×feature map height, and feature map size refers to feature map channel number×feature map width×feature map height.
[0051] like Figure 1 As shown, a robot positioning method based on a generative model includes the following steps:
[0052] S1. Design a generative model. The generative model is the core of generative artificial intelligence technology. Its main function is to directly generate the expected answer based on the input text prompts and visual prompts with the help of a large language model. The network structure of the generative model designed in this embodiment is as follows: Figure 2As shown, the globalmap layer represents a pre-built 2D environment grid map; the localmap layer represents the local environment map of the current posture collected by the robot; the globalmap layer and the localmap layer are both single-channel grayscale images, which can be images of any input resolution. Generally, the image resolution of the localmap layer is much smaller than that of the globalmap layer. For the convenience of description, the resolution of the input image is collectively referred to as WxH in this embodiment; the text prompt represents the problem prompt text, which is mainly used to describe the problem to be solved by the current input image; the image encoder module is a visual feature encoding network for extracting deep features of the input image. The image encoder module can be any commonly used network structure, such as the resnet network structure, the visual encoder of the multimodal encoder CLIP (Learning Transferable Visual Models From Natural Language Supervision), and it only needs to ensure that its output feature map is a 16-fold downsampled feature map of the input feature map; the word The embedding module is a vectorization module for inputting question prompt text. It mainly converts the input question prompt text into a multi-dimensional feature vector of fixed length to obtain a text feature vector. The wordembedding module can be any commonly used text vectorization module, such as the vectorization module used by the text encoder in CLIP. Tt represents the dimension or sequence length of the text vector; the reshape layer is a feature map size adjustment layer, which is used to adjust the input feature map size; the concat layer is a splicing layer, which is used to merge multiple input feature maps (global feature vectors, local feature vectors, and text feature vectors) into a spliced feature vector according to the channel dimension as the output of the concat layer; the LLM module is a language large model, which can be any commonly used language large model structure, such as Llama network, Qwen network, etc.; Robotpose represents the robot pose generated by the LLM module; Fl represents the localmap layer input image after image The local feature vector output by the encoder module and the reshape layer; Fg represents the global feature vector output by the globalmap layer input image after the imageencoder module and the reshape layer operation. For the convenience of description, the feature map size of Fl is 512xTlx1, and the feature map size of Fg is 512xTgx1, where Tg and Tl both represent the dimension or sequence length of the visual cue vector.
[0053] S2. Training the generative model mainly involves optimizing the network model parameters of the generative model through a large amount of training sample data of paired environment map images with known pose information and question prompt text, so that the generative model can accurately obtain the optimal pose of the local environment map perceived by the robot in the global environment map. The specific steps are as follows:
[0054] S21, constructing a set of training sample data pairs, mainly collecting various environmental maps and local environmental maps of various positions under the environmental maps, and forming training data pairs. The specific steps are as follows:
[0055] S211. Collect environmental maps, mainly collecting 2D grid environmental maps in various scenarios.
[0056] S212. Obtain a local map. The main method is to arbitrarily select a posture without obstacles in the 2D grid environment map, and use the posture as the reference origin to emit rays in all directions, record the first intersection posture of each ray and the obstacle in the grid map, and collect the intersection posture information of all rays. Then, according to the standard that 95% of the ray intersections can be included, the minimum bounding rectangle of all ray intersection postures is obtained. Then, the grid map is cropped according to the minimum bounding rectangle. The obtained local map is the provisional local map of the robot in the current posture. Finally, 5% of the random obstacle postures are erased and added to the provisional local map to obtain the local map of the current posture under the 2D grid environment map.
[0057] S213. Obtain the supervisory label of the training data. First, obtain the coordinates of the reference origin pose in step S212. Then, normalize the coordinates of the reference origin pose. That is, divide the x-coordinate and y-coordinate of the reference origin pose by the width and height of the globalmap layer input image, respectively, and then multiply by 1000. Finally, use the newly obtained normalized coordinates as the supervisory label of the current pose.
[0058] S214, obtaining a training sample data pair, mainly combining the image data obtained in step S211, step S212, and step S213 as a training sample data pair.
[0059] S22. Obtain question prompt text. The question prompt text is mainly used to describe the problem to be solved in the current input image pair. The description content of the question prompt text can be diverse, as long as it can accurately represent the current problem. In this embodiment, the question prompt texts used include: "Output the robot's pose coordinates", "Predict the robot's pose coordinates", "Determine the robot's pose coordinates", "Please tell the robot's current pose coordinates", "Where is the robot", etc.
[0060] S23. Design a loss function for the generative model. The loss function used by the generative model designed in this embodiment is consistent with the loss function of the selected LLM model.
[0061] S24, training the generative model, mainly involves sending the training sample data pairs and the corresponding question prompt text into the defined generative model to learn the relevant model parameters. The specific training strategy is as follows:
[0062] S241, obtaining joint training data, mainly randomly selecting a training sample data pair from the training sample data pair set, and selecting a question prompt text from the predefined question prompt text set, to form a pair of joint training data.
[0063] S242, fine-tuning the language model module. The language model module has a huge number of parameters and cannot be trained from scratch. This embodiment adopts a fine-tuning training mode based on LORA (LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS; low-rank adaptation of large language models) technology, and fine-tunes the existing language model base to adapt to this embodiment. The specific fine-tuning method is to freeze the visual feature encoding network and the vectorization module and train the language model.
[0064] S243. Fine-tune the visual feature encoding network, freeze the vectorization module and the language large model, and train the visual feature encoding network.
[0065] S244, joint training, mainly alternately performs the training of step S242 and step S243. In this embodiment, the training steps are switched after every 10 algorithm iterations.
[0066] S3. Use the generative model to feed any 2D grid environment map and the local environment map under the map into the trained generative model, and directly output the best candidate pose corresponding to the local environment map. The specific prediction process is as follows:
[0067] S31. Obtaining the output feature vector of the global environment map mainly involves sending the 2D raster environment map to the visual feature encoding network of the generative model to obtain the global feature vector. Since the global environment map is fixed during the use of the robot and the global environment map is relatively large and requires a large amount of computation, this embodiment only needs to calculate the output feature vector once for any global environment map.
[0068] S32, obtaining the output feature vector of the question prompt text, mainly selecting a question prompt text, sending it to the vectorization module of the generative model, and obtaining the text feature vector. When the present invention performs generative model reasoning, for all robot positioning tasks, the same question prompt text such as "output the robot's posture coordinates" is used, so the output feature vector of the question prompt text only needs to be calculated once.
[0069] S33. Obtain an output feature vector of the local environment map, and send the local environment map to the visual feature encoding network of the generative model to obtain a local feature vector.
[0070] S34, obtaining candidate poses, sending the feature vectors obtained in steps S31, S32, and S33 to the LLM module, and directly outputting the candidate poses of the current local environment map.
[0071] S35, obtaining the best robot posture. Since similar postures may exist in the 2D grid environment map, the candidate posture obtained in step S34 may not be unique. In this case, the robot needs to move, and the candidate posture is finely corrected to finally obtain the best robot posture.
[0072] The robot positioning method proposed in this embodiment can realize end-to-end rapid matching and positioning of the local environment map in the global environment map. The positioning method has strong applicability, high accuracy and high robustness, and can meet the robot positioning needs in various scenarios.
[0073] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A robot positioning method based on a generative model, characterized in that: The steps include: Send any 2D grid environment map, the local environment map under the 2D grid environment map, and the question prompt text into a trained generative model, wherein the generative model includes a visual feature encoding network, a vectorization module, a feature map size adjustment layer, a splicing layer, and a language model; The 2D grid environment map and the local environment map are sequentially passed through the visual feature encoding network and the feature map resizing layer to output the global feature vector and the local feature vector. The question prompt text outputs a text feature vector through a vectorization module; The global feature vector, local feature vector and text feature vector are concatenated through the concatenation layer and input into the language model to output one or more candidate poses of the current local environment map; The motion-based robot corrects all candidate poses to obtain the best robot pose.
2. The robot positioning method based on generative model according to claim 1, characterized in that: The resolution of the local environment map is smaller than the resolution of the 2D grid environment map.
3. The robot positioning method based on generative model according to claim 1, characterized in that: The training process of the generative model is as follows: Constructing a set of training sample data pairs and a set of question prompt texts, wherein one training sample data pair includes a 2D grid environment map and a local map under the 2D grid environment map; Set the loss function of the generative model to be consistent with the loss function of the selected language model; Select a training sample data pair from the training sample data pair set, and select a question prompt text from the question prompt text set to form a pair of joint training data; Freeze the visual feature encoding network and vectorization module, and train the large language model; Freeze the vectorization module and the language model to train the visual feature encoding network; The training of the language large model and the visual feature encoding network is performed alternately to achieve the training goal of the generative model.
4. The robot positioning method based on generative model according to claim 3, characterized in that: The process of obtaining a local map under a 2D grid environment map is as follows: Choose an obstacle-free posture at random, use the current posture as the reference origin to emit rays in all directions, and record the first intersection posture of each ray and the obstacle in the 2D grid environment map; Collect all the ray intersection pose information, and obtain the minimum bounding rectangle of all the ray intersection poses according to the standard of containing 95% of the ray intersections; The 2D grid environment map is clipped according to the minimum bounding rectangle. The local map obtained is the tentative local map of the robot at the current position. 5% of the random obstacle poses are erased and added to the provisional local environment map to obtain a local map of the current pose under the 2D grid environment map.
5. The robot positioning method based on generative model according to claim 4, characterized in that: Obtain the supervision label of the current posture as the supervision label of the training sample data pair. The process of obtaining the supervision label is as follows: obtain the reference origin posture coordinates and normalize them, and use the normalized reference origin posture coordinates as the supervision label of the current posture, where the reference origin posture coordinates are the coordinates with the current posture as the reference origin.
6. A robot positioning system based on a generative model, characterized in that: The method comprises a generative model and a correction module, wherein the generative model comprises a visual feature encoding network, a vectorization module, a feature map size adjustment layer, a splicing layer and a language large model; The visual feature encoding network is used to extract deep features from 2D grid environment maps and local environment maps; The feature map size adjustment layer is used to adjust the size of the deep features output by the visual feature encoding network to output global feature vectors and local feature vectors; The vectorization module is used to convert the question prompt text into a multi-dimensional feature vector of fixed length, thereby outputting a text feature vector; The concatenation layer is used to concatenate the global feature vector, the local feature vector, and the text feature vector; The language model outputs one or more candidate poses of the current local environment map based on the concatenated feature vector output by the concatenated layer; The correction module is used to correct all candidate poses based on the motion of the robot to obtain the best robot pose.
7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the robot positioning method according to any one of claims 1 to 5 is implemented.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of classification programs, and the plurality of classification programs are used to be called by a processor and execute the robot positioning method according to any one of claims 1 to 5.