A robot local navigation method based on rgb pictures

Abstract

The application discloses a robot local navigation method based on an RGB picture and relates to the technical fields of intelligent identification and local navigation. The application proposes the local navigation method based on the RGB, can improve the generalization ability of the local navigation model, is favorable for the migration of navigation experience and the deployment of the robot. Meanwhile, through the training of the robot in different scenes, the robustness and adaptability of the robot to environmental changes are improved.

Description

Technical Field

[0001] This invention relates to the fields of intelligent recognition and local navigation technology, specifically a robot local navigation method based on RGB images. Background Technology

[0002] After obtaining destination information, the robot first plans a roughly feasible route through global path planning, and then calls the local path planner to plan the robot's specific action strategy in the local area based on the information of this route and obstacles.

[0003] Local navigation techniques can generally be divided into two categories: traditional methods and reinforcement learning.

[0004] Traditional navigation methods (such as dynamic windows, time elastic bands, model predictive control, etc.) all rely on prior exploration of the environment. However, in indoor public spaces, warehouses, or construction sites, the environment changes frequently, and the dynamic adaptability of current robot navigation technologies is difficult to meet.

[0005] Meanwhile, the effectiveness of existing reinforcement learning methods in transferring local navigation experience depends on the similarity between the old and new environments. In new environments with significant differences, the transferred experience may not be able to effectively cope with entirely new obstacle layouts, terrains, or dynamic factors, and may even cause the robot to travel on inappropriate paths, affecting the accuracy and safety of navigation.

[0006] Meanwhile, when robots enter unfamiliar environments, they need to be remapped or retrained, which increases the cost of training and deploying robots.

[0007] Therefore, a new solution is needed to address the above problems. Summary of the Invention

[0008] The purpose of this invention is to provide a robot local navigation method based on RGB images to solve the technical problems mentioned in the background art.

[0009] To achieve the above objectives, the present invention provides the following technical solution: a robot local navigation method based on RGB images, comprising at least the following steps:

[0010] S1: Collect room data, process it, and generate twin room data;

[0011] S2: After selecting a room, randomly generate a collision-free robot model on the floor and control the robot camera to randomly take pictures of the room; use a large visual model to analyze and segment the images, select objects far from the center of the images as candidate images, and map the centroid of the mask onto the 3D dataset.

[0012] S3: After filtering and merging centroids, starting from the robot and using the projection points of candidate centroids onto the horizontal plane where the camera is located as the endpoints, a path planning algorithm is used to obtain a feasible path; the path is fitted using the robot's optional action set and the sampling points of the path are obtained, and the camera images on the sampling points are retained; the action set and the camera images of the sampling points are merged into a navigation dataset, and in particular, the coordinates of the centroids are marked in the starting and ending images;

[0013] S4: Based on the Transformer model, imitation learning is used to learn on the robot navigation dataset to obtain a trained robot navigation model;

[0014] S5: Transfer the trained model to a real robot, and the robot navigates autonomously after the images are manually labeled.

[0015] Furthermore, S1 includes at least the following steps:

[0016] Collect 3D data of a real or virtual single room, segment and label each object in the room, and search for candidate labels that are similar to the labels of each object in the digital assets;

[0017] Based on the similarity between the snapshot of the candidate label and the snapshot of the object, retain the most relevant and closest label;

[0018] Based on the size of each type of object, the digital assets corresponding to the closest labels of the objects are scaled, rotated, aligned, and replaced with the original objects to form twin room data.

[0019] Furthermore, by considering the set of objects in the room O = (o1,…,o…), i ,…,o N N is the number of objects in the room;

[0020] After embedding features into the object labels, the candidate label set is retrieved. At the same time, the DINOv2 model is used to extract features from the candidate label set snapshot and the object label snapshot. The cosine similarity is calculated using the following formula (1), and the label with the highest similarity is taken as the closest label.

[0021]

[0022] Where A and B are feature vectors, representing the feature vector of the object label and the feature vector of the candidate label snapshot, respectively;

[0023] Find the axial bounding box of the object, scale the 3D model corresponding to the nearest label according to the bounding width, and rotate and translate the 3D model to the room where the object is located after removal to form twin room data.

[0024] Furthermore, in S2, the Grounded-SAM model is used to perform semantic segmentation on the camera image, select the object whose center of the object mask is closest to the center of the image, and calculate the centroid of the object mask.

[0025] When the homogeneous coordinates of the centroid of the object mask are Depth The transformation matrix between the robot and the camera is The coordinates of the image in the robot's coordinate system are K. -1 z c p obj .

[0026] Furthermore, in S3, the set of discrete actions that the robot can choose is set as P = (p1, ..., p...). i ,…,p M M represents the number of actions;

[0027] Then, a single data point in the navigation dataset is represented as:

[0028] {a1,…,a t ,…a T ;I1,…,I t ,…I T} (2)

[0029] Where a t and I t Let T be the action taken by the robot at the t-th discrete point and the image captured, respectively, where T is a fixed constant.

[0030] Example of a discrete action set: move forward, turn left 45 degrees, turn right 45 degrees, rotate the camera up, rotate the camera down.

[0031] Furthermore, S4 includes at least the following steps:

[0032] Given training samples on the path: {a1,…,a t ,…a T ;I1,…,I t ,…I T After grouping the data, it is fed into the Transformer model, and the output action set satisfies formula (3):

[0033] a′ t-J:t =Trans(MLP(a t-J:t ),Conv(I t-J:t (3)

[0034] a′ t-J:tAfter passing through different MLPs (Multi-Layer Perceptrons), the correlation distance between action policy classification and prediction can be obtained;

[0035] Where MLP(a) t-J:t This refers to processing action sequences using an MLP.

[0036] Where Conv(I) t-J:t This refers to processing image sequences using Conv (convolutional layer);

[0037] Using the Transformer model, the robot can generate action a′ in the future. t That is, a′ t Actions generated for the model;

[0038] The loss function during training is given by formula (4):

[0039] L = L act +L track (4)

[0040] Where L act =MSE(π) θ (a′ t ),a t ), π θ For policy functions;

[0041] L track =MSE(s) t ,d θ (s t )), s t Let d be the centroid of the object mask. θ For s t Relative distances between different images.

[0042] Compared with the prior art, the beneficial effects of the present invention are:

[0043] This invention proposes an RGB-based local navigation method, which improves the generalization ability of local navigation models and facilitates the transfer of navigation experience and robot deployment. Simultaneously, by training the robot in different scenarios, its robustness and adaptability to environmental changes are enhanced. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of the original room data and twin data of the present invention;

[0046] Figure 2 This is a schematic diagram of the network framework of the present invention;

[0047] Figure 3 This is a schematic diagram of the local navigation of the first robot of the present invention;

[0048] Figure 4 This is a schematic diagram of the partial navigation of the second robot according to the present invention;

[0049] Figure 5 This is a schematic diagram of the local navigation of the third robot of the present invention. Detailed Implementation

[0050] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0051] Please see Figures 1-5 A robot local navigation method based on RGB images includes at least the following steps:

[0052] S1: Collect room data, process it, and generate twin room data;

[0053] S2: After selecting a room, randomly generate a collision-free robot model on the floor and control the robot camera to randomly take pictures of the room; use a large visual model to analyze and segment the images, select objects far from the center of the images as candidate images, and map the centroid of the mask onto the 3D dataset.

[0054] S3: After filtering and merging centroids, starting from the robot and using the projection points of candidate centroids onto the horizontal plane where the camera is located as the endpoints, a path planning algorithm is used to obtain a feasible path; the path is fitted using the robot's optional action set and the sampling points of the path are obtained, and the camera images on the sampling points are retained; the action set and the camera images of the sampling points are merged into a navigation dataset, and in particular, the coordinates of the centroids are marked in the starting and ending images;

[0055] S4: Based on the Transformer model, imitation learning is used to learn on the robot navigation dataset to obtain a trained robot navigation model;

[0056] S5: Transfer the trained model to a real robot, and the robot navigates autonomously after the images are manually labeled.

[0057] S1 includes at least the following steps:

[0058] Collect 3D data of a real or virtual single room, segment and label each object in the room, and search for candidate labels that are similar to the labels of each object in the digital assets;

[0059] Based on the similarity between the snapshot of the candidate label and the snapshot of the object, retain the most relevant and closest label;

[0060] Based on the size of each type of object, the digital assets corresponding to the closest labels of the objects are scaled, rotated, aligned, and replaced with the original objects to form twin room data.

[0061] By analyzing the set of objects in the room O = (o1, ..., o2) i ,…,o N N is the number of objects in the room;

[0062] After embedding features into the object labels, the candidate label set is retrieved. At the same time, the DINOv2 model is used to extract features from the candidate label set snapshot and the object label snapshot. The cosine similarity is calculated using the following formula (1), and the label with the highest similarity is taken as the closest label.

[0063]

[0064] Where A and B are feature vectors, representing the feature vector of the object label and the feature vector of the candidate label snapshot, respectively;

[0065] Find the axial bounding box of the object, scale the 3D model corresponding to the nearest label according to the bounding width, and rotate and translate the 3D model to the room where the object is located after removal to form twin room data.

[0066] S2 uses the Grounded-SAM model to perform semantic segmentation on camera images, selects the object whose mask center is closest to the image center, and calculates the centroid of the object mask.

[0067] When the homogeneous coordinates of the centroid of the object mask are Depth The transformation matrix between the robot and the camera is The coordinates of the image in the robot's coordinate system are K. -1 z c p obj .

[0068] In S3, the set of discrete actions that the robot can choose is set as P = (p1, ..., p...). i ,…,p M M represents the number of actions;

[0069] Then, a single data point in the navigation dataset is represented as:

[0070] {a1,…,at ,…a T ;I1,…,I t ,…I T} (2)

[0071] Where a t and I t Let T be the action taken by the robot at the t-th discrete point and the image captured, respectively, where T is a fixed constant.

[0072] Example of a discrete action set: move forward, turn left 45 degrees, turn right 45 degrees, rotate the camera up, rotate the camera down.

[0073] S4 includes at least the following steps:

[0074] Given training samples on the path: {a1,…,a t ,…a T ;I1,…,I t ,…I T After grouping the data, it is fed into the Transformer model, and the output action set satisfies formula (3):

[0075] a′ t-J:t =Trans(MLP(a t-J:t ),Conv(I t-J:t (3)

[0076] a′ t-J:t After passing through different MLPs (Multi-Layer Perceptrons), the correlation distance between action policy classification and prediction can be obtained;

[0077] Where MLP(a) t-J:t This refers to processing action sequences using an MLP.

[0078] Where Conv(I) t-J:t This refers to processing image sequences using Conv (convolutional layer);

[0079] Using the Transformer model, the robot can generate action a′ in the future. t That is, a′ t Actions generated for the model;

[0080] The loss function during training is given by formula (4):

[0081] L = L act +L track (4)

[0082] Where L act =MSE(π) θ (a′t ),a t ), π θ For policy functions;

[0083] L track =MSE(s) t ,d θ (s t )), s t Let d be the centroid of the object mask. θ For s t Relative distances between different images.

[0084] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for robot local navigation based on RGB pictures, characterized in that: At least comprising the following steps: S1: Collecting data of a room and processing to generate twin room data; S2: After selecting a room, randomly generating a non-collision robot model on the floor and controlling the robot camera to randomly take pictures in the room; using a visual large model to analyze and segment the picture, selecting the object farthest from the picture center as the candidate picture, and mapping the mask center to the three-dimensional data set; S3: After screening and merging the center of mass, using a path planning algorithm to obtain a feasible path from the robot as the starting point and the projection point of the candidate center of mass on the camera plane as the ending point; using the robot action set to fit the path and obtain the sampling points of the path, and retaining the camera pictures on the sampling points; the action set, sampling point camera picture is combined as a navigation data set, and the particle coordinates are marked in the starting point and ending point pictures; S4: Based on the Transformer model, using imitation learning to learn on the robot navigation data set to obtain a trained robot navigation model; S5: Migrate the trained model to a real robot, and after manually marking the pictures, the robot autonomously navigates.

2. The method of claim 1, wherein: The S1 at least comprises the following steps: Collecting three-dimensional data of a real or virtual single room, segmenting and labeling each object in the room, and searching for candidate labels similar to each object label in digital assets; According to the similarity of the candidate label snapshot and the object snapshot, the most relevant closest label is retained; According to the size of each type of object, the digital assets corresponding to the closest label of the object are scaled, rotated and aligned, and the original object is replaced to form twin room data.

3. The method of claim 2, wherein: By collecting objects in a room N is the number of objects in the room; After feature embedding of the object label, the candidate label set is retrieved, and the DINOv2 model is used to extract features of the candidate label set snapshot and the object label snapshot, and the cosine similarity is calculated using formula (1) as follows: （1） Where A and B are feature vectors, representing the feature vectors of the object label and the feature vectors of the candidate label snapshot, respectively; The axial bounding box of the object is obtained, the three-dimensional model corresponding to the closest label is scaled according to the bounding box width, and the three-dimensional model is rotated and translated to the room after removing the object to form twin room data.

4. The method of claim 3, wherein: The Grounded-SAM model is used for semantic segmentation of the camera picture in S2, the object closest to the picture center is selected, and the center of mass of the object mask is obtained; When the homogeneous coordinates of the object mask centroid are , the depth is , and the transformation matrix of the robot to the camera is , then the picture in the coordinates of the robot coordinate system is .

5. The method of claim 4, wherein: The S3 sets the optional discrete action set of the robot as , M is the number of actions; A piece of data in the navigation data set is represented as: （2） wherein and are the action taken and the image taken by the robot at the tth discrete point, respectively, and T is taken as a fixed constant.

6. The method of claim 5, wherein: The S4 at least comprises the following steps: Training samples on a given path: After grouping its data, the data is sent into the Transformer model, and the output action set satisfies formula (3): （3） After going through different MLPs (Multi-Layer Perceptrons), the action policy classification and the relevant distances of prediction are obtained. wherein to process the sequence of actions by the MLP; wherein for employing processing a sequence of images; With the Transformer model, the robot is able to generate actions at future time steps i.e. actions generated by the model; The loss function during training is formula (4): （4） wherein , is a policy function; , For object mask centroid, For Correlation distance between different pictures.

Images