An active exploration and real-time mapping method for semantic completeness

Abstract

The present invention proposes an active exploration and real-time mapping method for semantic completeness, which relates to the technical field of active synchronous positioning and mapping. The front-end odometer is used to provide the robot with real-time posture conditions, and the back-end receives the posture conditions provided by the front-end odometer at different times, optimizes the posture conditions, and combines semantic extraction to output a three-dimensional semantic point cloud to obtain a global semantic map and the current posture of the robot, and then performs a two-dimensional projection on the global semantic map to form a two-dimensional semantic map group; finally, based on the current posture of the robot and the two-dimensional semantic map group, an active exploration strategy is adopted to complete the exploration of a specified step length, and the optimal path for semantic completeness exploration in the current scene and the constructed semantic map are obtained. The method proposed by the present invention has excellent active exploration performance, can perform efficient and accurate active semantic mapping work for semantic-related indicators, and guide real-time mapping exploration through semantic information.

Description

Technical Field

[0001] The present invention relates to the technical field of active synchronous positioning and mapping, and more specifically, to an active exploration and real-time mapping method oriented to semantic completeness. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) is mainly used to solve the positioning and mapping problems of robots and other intelligent bodies moving in unknown environments. Based on its own sensors, it observes the surrounding environment and collects data, and it iteratively updates its own posture and builds a map of the surrounding environment during a period of movement; the posture includes the current coordinates and direction of the robot, and the SLAM with the main sensor being a camera is called visual SLAM.

[0003] Traditional visual SLAM focuses on the collective reconstruction of the environment and lacks high-level environmental perception at the content level. For this reason, the academic community has proposed embedding a semantic extraction module in the visual SLAM system to obtain environmental semantic information and use semantic information to efficiently complete pose estimation, positioning, and loop detection, and the final output three-dimensional point cloud map also contains semantic information. For example, the prior art discloses a mobile robot indoor navigation method based on semantic information, which uses a visual SLAM algorithm based on graph optimization to redraw the indoor environment in three dimensions through the kinect sensor to generate a three-dimensional prior map with clear and discernible environmental details, and then through semantic information annotation, realizes the construction of a grid-topology-semantic hierarchical map. A top-down navigation method is adopted in the hierarchical semantic map. First, the semantic layer and the topological layer generate a target location sequence according to the task goal. Based on the target location sequence, the A* smooth path algorithm is used in the grid layer to realize the mobile robot indoor navigation based on semantic information. In addition, some semantic SLAM work uses neural networks instead of traditional corner point extraction methods to extract more accurate image feature information, that is, using neural networks to assist in the extraction of landmarks in SLAM. Semantic-based matching can establish more stable pose constraints than image corners and edges, allowing the visual SLAM system to have higher pose and mapping accuracy under changes in lighting, but at the same time it also places higher performance requirements on the hardware.

[0004] On the one hand, visual SLAM and semantic SLAM problems are usually described as a state estimation problem of a robot under manual control, that is, the robot passively performs positioning and mapping processes, rather than actively exploring the environment and collecting valid data on demand, and has poor autonomy. On the other hand, current semantic SLAM work follows the traditional visual SLAM process, first collecting data by manually controlling the robot, and then performing semantic mapping based on the collected data. When collecting data, this type of SLAM framework that collects data first and then maps may cause the semantic segmentation model to output incorrect semantic prediction results due to inappropriate observation angles and positions, resulting in poor quality of collected data. In addition, since the path of collected data is not oriented to semantic SLAM planning, there is a problem of redundant collected data. Summary of the invention

[0005] In order to solve the problems of poor data quality and data redundancy in the traditional semantic simultaneous positioning and mapping method, the present invention proposes an active exploration and real-time mapping method for semantic completeness. The active exploration has good performance and guides the real-time mapping exploration through semantic information. The real-time mapping exploration further improves the completeness and accuracy of the collected semantic information, and realizes the robot's autonomous exploration of semantic completeness without prior environmental knowledge, and finally outputs the optimal exploration path and semantic mapping results under the specified step size.

[0006] In order to achieve the above technical effects, the technical solution of the present invention is as follows:

[0007] A method for active exploration and real-time mapping for semantic completeness includes the following steps:

[0008] S1. The robot observes the environment in real time;

[0009] S2. Use the front-end odometer to provide the robot with real-time posture information. The back-end receives the posture information provided by the front-end odometer at different times, optimizes the posture information, and outputs a three-dimensional semantic point cloud in combination with semantic extraction to obtain a global semantic map and the current posture information of the robot.

[0010] S3. Performing a two-dimensional projection on the global semantic map to form a two-dimensional semantic map group;

[0011] S4. Based on the current posture of the robot and the two-dimensional semantic map group, an active exploration strategy is adopted to complete the exploration of the specified step length, and the optimal path for semantic completeness exploration in the current scene and the constructed semantic map are obtained.

[0012] Preferably, in step S4, the process of the active exploration strategy includes:

[0013] S41. Assume the robot's current number of explorations is S curr , the specified number of explorations is S update, the maximum number of explorations is S max , initialize the current number of explorations S curr is 0;

[0014] S42. The current two-dimensional semantic map group result M curr and the current posture P curr Input to the global target generation network N global , output the global target G through the global target generation network globa1 ;

[0015] S43. Use the fast marching method to calculate the optimal global path Traj from the robot's current position to the global target position best ;

[0016] S44.Robot along Traj best Move forward one unit and update the current pose P curr ;

[0017] S45. Get the robot's current position P curr Observation under O curr , based on O curr Update the current map M using the semantic SLAM method curr ;

[0018] S46. Repeat steps S44 to S45 update times, and record the current number of explorations S curr The value of

[0019] S47. Determine the current number of explorations S curr Whether the value reaches the maximum number of explorations S max If so, use Nglobal to update the global target Gglobal and return to S42; otherwise, end the exploration and obtain the optimal path L actually explored by the robot. best , the current maintenance map result M curr The two-dimensional map of the 2nd to 16th channels is used as the semantic mapping result M sem , the semantic map result M sem and the semantic map truth value M gt The same part as the semantic integrity SI.

[0020] This solution demonstrates excellent active exploration performance and can perform efficient and accurate active semantic mapping for semantic-related indicators, maximizing the semantic information and area information explored by the robot within a specified step size.

[0021] Preferably, the global target generation network described in step S42 includes a seven-layer neural network, wherein the first five layers are sequentially connected convolutional layers conv1~conv5, the last two layers are sequentially connected fully connected layers FC1~FC2, and also include embedding layers Eembed1~Eembed2, wherein the input of the convolutional layer conv1 is the robot initialization position and the semantic mapping result, the input of the embedding layer Eembed1 is the semantic map truth value, the input of the embedding layer Eembed2 is the semantic category, and the semantic map truth value and the semantic category are prior information; the output end of the embedding layer Eembed1, the output end of the embedding layer Eembed2, and the output end of the convolutional layer conv5 are all connected to the fully connected layer FC1, and the output end of the fully connected layer FC2 outputs the global target.

[0022] Preferably, in the process of generating the global goal, the PPO algorithm of reinforcement learning is used for training. During the training process, the semantic completeness observed by the robot within a specified step size is used as a reward, and the richness of the semantic information is proportional to the reward value; the intermediate results of the training process include: two-dimensional semantic mapping results, cumulative results of global goal generation, cumulative trajectory of the robot and corresponding reward values, and the two-dimensional semantic mapping results, cumulative results of global goal generation, cumulative trajectory of the robot and corresponding reward values ​​are all stored as experience data for subsequent training to update the network parameters of the global goal generation network.

[0023] Here, the reinforcement learning method is used to achieve the generation of global goals. By learning information such as the semantic richness of regions in indoor environments, the relative arrangement relationship between semantic objects and indoor areas, the robot is guided to generate the optimal global goal through current observations and maintained maps.

[0024] Preferably, during the training process, the number of exploration steps under the same global goal is limited, and the global target point and path planning are recalculated every n steps. The training process is:

[0025] S421. Use parameter set θ to initialize habitat environment and parameter set θ n Initialize the global target generation network;

[0026] S422. Initialize the robot at random coordinates, and use the previous module to explore the target point with a specified step length in the current environment;

[0027] S423. The robot stops the exploration after moving forward for a specified step length, and ends the current training round; the reward value is calculated based on the finally generated semantic map and the corresponding true value, and the relevant observation image and reward value are stored in the experience buffer pool;

[0028] S424. Sampling a specified amount of experience data from the experience buffer pool and updating network parameters in the global target generation network;

[0029] S425. Determine whether the current number of training times is greater than the set maximum number of training iterations. If so, terminate the training and save the current training model and related parameter information; otherwise, return to step S422.

[0030] This scheme limits the number of exploration steps under the same global goal, reducing the exploration time in reinforcement learning and the complexity caused by excessively long exploration time.

[0031] Preferably, in step S43, the optimal global path Traj from the current position of the robot to the global target position is calculated by using the fast marching method. best The process is:

[0032] S431. Establish the eikonal equation: Where (x, y) is the two-dimensional coordinate of the target point in the map, T(x, y) is the time to reach the target point, V(x, y) is the forward speed of the robot, Δx and Δy are the set extension distances respectively; for the target point (x, y), the four adjacent points in the positive and negative directions of the x and y axes whose distance is the extension distance constitute its adjacent point set X, X = {(x+Δx, y), (x-Δx, y), (x, y+Δy), (x, y-Δy)};

[0033] S432. Construct an equation based on the distance relationship between the adjacent point set and the target point:

[0034] T 1 =min(T (x-Δx,y) ,T (x+Δx,y) )

[0035] T 2 =min(T (α,y-Δy) ,T (x,y+Δy) )

[0036]

[0037]

[0038]

[0039] Solve the equations together to get the time T(x,y) required to reach the target point;

[0040] S433. Iteratively use the fast marching method to solve the time information of the neighboring points and the next neighboring points of the target point, and finally obtain the time value required for all map points to reach the target point, and record it as a two-dimensional time map corresponding to all map points;

[0041] S434. Based on the two-dimensional time graph, the idea of ​​gradient descent method is applied to continuously select the lowest time value, so as to obtain the optimal global path to reach the target point from any position on the two-dimensional grid map.

[0042] Here, based on the robot's current coordinates, the global target position coordinates, and the currently maintained semantic map, this paper uses the traditional fast marching method to solve the shortest path between the two coordinates in all traversable areas. Compared with the learning-based method, the fast marching method can achieve similar results while reducing the amount of calculation.

[0043] Preferably, in step S44, the robot moves along Traj best The behavior decision when moving forward is divided into multiple discrete points according to the specified distance l; the discrete point closest to the robot on the path is set as the local target G local , which is the position that the robot needs to reach at the next moment. The robot's posture and local target coordinates are input into the behavior decision maker. The posture includes the position coordinates and direction, and the action that the robot needs to take at the current moment is output. The robot kinematic model adopts the definition of the robot operating system ROS, and adopts the arc model in the four-wheel differential model. The wheels on the same side have the same speed, and the wheels on the opposite side are driven independently.

[0044] The robot's motion process under two close coordinates is described as k ,y k ) to coordinate (x k+1 ,y k+1 ), the total angle difference is δθ k The non-linear motion, the dead reckoning model is:

[0045]

[0046] Among them, x k ,y k ,x k+1 ,y k+1 is the x and y coordinates at time k and k+1, S k is the displacement at time k, θ k is the direction angle at time k; δ represents the differential, and Δ represents the difference;

[0047] Preferably, for dynamic obstacles, a safety domain method in the SLAM field is used to avoid obstacles.

[0048] Preferably, during the active exploration process, the robot receives visual image frames and uses the semantic segmentation model M of the semantic extraction module to extract the image frames. seg For semantic segmentation, let the image frame sequence of the exploration process be The total number of semantic pixels is N semgt , the two-dimensional semantic coverage threshold is λcov , the two-dimensional semantic accuracy threshold is λ cer , let the length of the image frame Ik be H, the width be W, the total number of pixels be HxW, and the total number of pixels containing semantic information be N semgt ,Semantic,Semantic Segmentation Model M seg The number of pixels containing semantic information in the prediction result N sem ; Assume that the two-dimensional semantic coverage of a single frame is R cov , the two-dimensional semantic accuracy is R cer ,but:

[0049]

[0050] Using the semantic segmentation model M seg Predict the number of pixels containing semantic information in the current image frame I, denoted as N sem ; Based on the frame length H, frame width W, and the total number of semantic pixels N semgt , predicted number N sem Calculate the semantic coverage and accuracy of the current frame, denoted as R cov and R cer ; if R cov ≥λ cov or R cer ≤λ cer , then update for Add the current frame I to For each image frame at each moment in the active exploration process, all image frames whose single-frame 2D semantic coverage exceeds the threshold or whose accuracy is lower than the threshold are actively collected and saved, and the saved content includes the current frame, semantic prediction status and corresponding true value.

[0051] Preferably, the active exploration process also includes fine-tuning the semantic segmentation model using the actively collected data, including the following steps:

[0052] The current task dataset is formed based on the actively collected data, and the original semantic segmentation model is tested on the current dataset to obtain the unoptimized recognition result;

[0053] Based on the size of the current task dataset and its similarity to the original semantic segmentation model training set, determine the number of frozen layers of the model, retain only the last n layers or the last softmax layer for fine-tuning, and keep the network parameters of the remaining layers unchanged;

[0054] Preprocess the data set based on Detectron rules and complete the registration to verify the correctness of data loading;

[0055] The preprocessed data set is divided into three parts: training set, validation set and test set;

[0056] Retrain the last n layers of the network based on the test set, using a smaller learning rate than the original training to avoid mutations in the weights of the original model;

[0057] Train until convergence, fine-tune, and test the model fine-tuning results based on the test set.

[0058] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0059] The present invention proposes an active exploration and real-time mapping method for semantic completeness. The robot observes the environment in real time, uses the front-end odometer to provide the robot with real-time posture information, and the back-end receives the posture information provided by the front-end odometer at different times, optimizes the posture information, and combines semantic extraction to output a three-dimensional semantic point cloud to obtain a global semantic map and the current posture information of the robot. Then, the global semantic map is projected in two dimensions to form a two-dimensional semantic map group. Finally, based on the current posture information of the robot and the two-dimensional semantic map group, an active exploration strategy is adopted to complete the exploration of the specified step length, and the optimal path for semantic completeness exploration in the current scene and the constructed semantic map are obtained. The method proposed in the present invention has excellent active exploration performance, can perform efficient and accurate active semantic mapping work for semantic-related indicators, guides real-time mapping exploration through semantic information, and reversely further improves the completeness and accuracy of the collected semantic information, realizes the robot's autonomous exploration of semantic completeness without prior environmental knowledge, and finally outputs the optimal exploration path and semantic mapping results under the specified step length. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A schematic diagram showing a flow chart of the active exploration and real-time mapping method for semantic completeness proposed in Embodiment 1 of the present invention;

[0061] Figure 2 A structural block diagram showing the overall implementation of the active exploration and real-time mapping method for semantic integrity proposed in Embodiment 1 of the present invention;

[0062] Figure 3 A structural diagram showing a global target generation network proposed in Embodiment 2 of the present invention;

[0063] Figure 4 A schematic diagram showing obstacle avoidance in a robot safety zone proposed in Embodiment 3 of the present invention;

[0064] Figure 5 The figure shows the effect of actually implementing the proposed method in Example 4 of the present invention. DETAILED DESCRIPTION

[0065] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0066] In order to better illustrate the present embodiment, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the actual size;

[0067] It is understandable to those skilled in the art that descriptions of certain well-known contents in the drawings may be omitted.

[0068] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0069] The positional relationships described in the drawings are only for illustrative purposes and should not be construed as limitations on this patent.

[0070] Example 1

[0071] like Figure 1 As shown, this embodiment proposes an active exploration and real-time mapping method for semantic completeness, including the following steps:

[0072] S1. The robot observes the environment in real time;

[0073] S2. Use the front-end odometer to provide the robot with real-time posture information. The back-end receives the posture information provided by the front-end odometer at different times, optimizes the posture information, and outputs a three-dimensional semantic point cloud in combination with semantic extraction to obtain a global semantic map and the current posture information of the robot.

[0074] S3. Performing a two-dimensional projection on the global semantic map to form a two-dimensional semantic map group;

[0075] S4. Based on the current posture of the robot and the two-dimensional semantic map group, an active exploration strategy is adopted to complete the exploration of the specified step length, and the optimal path for semantic completeness exploration in the current scene and the constructed semantic map are obtained.

[0076] The overall implementation structure diagram can be found in Figure 2 The method proposed in this embodiment realizes the robot's autonomous exploration of semantic completeness without prior environmental knowledge, and finally outputs the optimal exploration path and semantic mapping results under the specified step size. Figure 2 The overall system includes four parts: semantic SLAM framework, active exploration strategy AESSI, global state, and semantic model fine-tuning. Among them, semantic SLAM adopts the method based on ORBSLAM2. Then, the three-dimensional semantic point cloud output by the semantic SLAM framework is two-dimensionally rasterized to reduce the calculation cost of subsequent modules; the active exploration strategy AESSI includes optimal target generation, global path planning, and behavior decision maker; the global semantic map is collected in the global state, and a two-dimensional semantic map group is formed through two-dimensional projection; the semantic model fine-tuning includes two parts: active collection of fine-tuning data and model fine-tuning method.

[0077] This embodiment is a two-way promotion scheme that guides real-time map exploration through semantic information, and real-time exploration and observation further improves the completeness and accuracy of the collected semantic information. It has excellent active exploration performance, and can perform efficient and accurate active semantic mapping for semantic-related indicators. While achieving similar performance in the exploration area coverage indicator and the current cutting-edge methods in the active SLAM field, it is far superior to other methods in terms of semantic coverage and accuracy.

[0078] Example 2

[0079] In this embodiment, the process of the active exploration strategy AESSI (Active Exploration Strategies for Semantic Integrity) includes:

[0080] S41. Assume the robot's current number of explorations is S curr , the specified number of explorations is S update , the maximum number of explorations is S max , initialize the current number of explorations S curr is 0;

[0081] S42. The current two-dimensional semantic map group result M curr and the current posture P curr Input to the global target generation network N global , output the global target G through the global target generation network global ;

[0082] S43. Use the fast marching method to calculate the optimal global path Traj from the robot's current position to the global target position best ;

[0083] S44.Robot along Traj best Move forward one unit and update the current pose P curr ;

[0084] S45. Get the robot's current position P curr Observation under O curr , based on O curr Update the current map M using the semantic SLAM method curr ;

[0085] S46. Repeat steps S44 to S45 update times, and record the current number of explorations S curr The value of

[0086] S47. Determine the current number of explorations S curr Whether the value reaches the maximum number of explorations S maxIf so, use Nglobal to update the global target Gglobal and return to S42; otherwise, end the exploration and obtain the optimal path L actually explored by the robot. best , the current maintenance map result M curr The two-dimensional map of the 2nd to 16th channels is used as the semantic mapping result M sem , the semantic map result M sem and the semantic map truth value M gt The same part as the semantic integrity SI.

[0087] This solution demonstrates excellent active exploration performance, and can perform efficient and accurate active semantic mapping for semantic-related indicators, maximizing the semantic information and area information explored by the robot within a specified step size.

[0088] The core of AESSI guides the robot to generate the optimal global goal so that it can observe the most complete and accurate semantic information. This invention adopts reinforcement learning method to achieve global goal generation. By learning information such as the semantic richness of the region in the indoor environment, the relative arrangement relationship between semantic objects and indoor areas, the robot is guided to generate the optimal global goal through the current observation and the maintained map.

[0089] In this embodiment, see Figure 3 The global target generation network consists of a seven-layer neural network, in which the first five layers are sequentially connected convolutional layers conv1~conv5, the last two layers are sequentially connected fully connected layers FC1~FC2, and also include embedding layers Eembed1~Eembed2, in which the input of the convolutional layer conv1 is the robot initialization position and the semantic mapping result, the input of the embedding layer Eembed1 is the true value of the semantic map, the input of the embedding layer Eembed2 is the semantic category, and the true value of the semantic map and the semantic category are prior information; the output of the embedding layer Eembed1, the output of the embedding layer Eembed2, and the output of the convolutional layer conv5 are all connected to the fully connected layer FC1, and the output of the fully connected layer FC2 outputs the global target.

[0090] In the process of global goal generation, the PPO algorithm of reinforcement learning is used for training. During the training process, the semantic completeness observed by the robot within a specified step size is used as a reward, and the richness of semantic information is proportional to the reward value; the intermediate results of the training process include: two-dimensional semantic mapping results, cumulative results of global goal generation, cumulative trajectory of the robot and corresponding reward values. The two-dimensional semantic mapping results, cumulative results of global goal generation, cumulative trajectory of the robot and corresponding reward values ​​are all stored as experience data for subsequent training to update the network parameters of the global goal generation network.

[0091] At the same time, in order to reduce the exploration time in reinforcement learning and the sample complexity caused by long exploration time, the present invention limits the number of exploration steps under the same global goal, and recalculates the global target point and path planning every n steps. The entire model training process is completed based on the habitat simulation environment. During the training process, the number of exploration steps under the same global goal is limited, and the global target point and path planning are recalculated every n steps. The training process is:

[0092] S421. Use parameter set θ to initialize habitat environment and parameter set θ n Initialize the global target generation network;

[0093] S422. Initialize the robot at random coordinates, and use the previous module to explore the target point with a specified step length in the current environment;

[0094] S423. The robot stops the exploration after moving forward for a specified step length, and ends the current training round; the reward value is calculated based on the finally generated semantic map and the corresponding true value, and the relevant observation image and reward value are stored in the experience buffer pool;

[0095] S424. Sampling a specified amount of experience data from the experience buffer pool and updating network parameters in the global target generation network;

[0096] S425. Determine whether the current number of training times is greater than the set maximum number of training iterations. If so, terminate the training and save the current training model and related parameter information; otherwise, return to step S422.

[0097] Example 3

[0098] Based on the robot's current coordinates, the global target position coordinates, and the currently maintained semantic map, this paper uses the traditional fast marching method to solve the shortest path between two coordinates in all traversable areas. Compared with the learning-based method, the fast marching method can achieve similar results while reducing the amount of calculation.

[0099] The fast marching method is an efficient numerical algorithm proposed to solve the Eikonal equation. The mathematical form of the Eikonal equation is It is a nonlinear partial differential equation and can be considered as an approximate wave equation. In solving the propagation problem in a two-dimensional plane, the fast marching method can be used to solve the shortest propagation path in a discrete state, with high computational efficiency; its core idea is similar to the graph algorithm Dijkstra algorithm, both of which iteratively search for the current optimal path and finally obtain the global optimal path.

[0100] In this embodiment, the optimal global path Traj from the current position of the robot to the global target position is calculated using the fast marching method.best The process is:

[0101] S431. Establish the eikonal equation: Where (x, y) is the two-dimensional coordinate of the target point in the map, T(x, y) is the time to reach the target point, V(x, y) is the forward speed of the robot, Δx and Δy are the set extension distances respectively; for the target point (x, y), the four adjacent points in the positive and negative directions of the x and y axes whose distance is the extension distance constitute its adjacent point set X, X = {(x+Δx, y), (x-Δx, y), (x, y+Δy), (x, y-Δy)};

[0102] S432. Construct an equation based on the distance relationship between the adjacent point set and the target point:

[0103] T 1 =min(T (x-Δx,y) ,T (x+Δx,y) )

[0104] T 2 =min(T (α,y-Δy) ,T (x,y+Δy) )

[0105]

[0106]

[0107]

[0108] Solve the equations together to get the time T(x,y) required to reach the target point;

[0109] S433. Iteratively use the fast marching method to solve the time information of the neighboring points and the next neighboring points of the target point, and finally obtain the time value required for all map points to reach the target point, and record it as a two-dimensional time map corresponding to all map points;

[0110] S434. Based on the two-dimensional time graph, the idea of ​​gradient descent method is applied to continuously select the lowest time value, so as to obtain the optimal global path to reach the target point from any position on the two-dimensional grid map.

[0111] Robots along Traj best The behavior decision when moving forward is divided into multiple discrete points according to the specified distance l; the discrete point closest to the robot on the path is set as the local target G local, which is the position that the robot needs to reach at the next moment. The robot's posture and local target coordinates are input into the behavior decision maker. The posture includes the position coordinates and direction, and the action that the robot needs to take at the current moment is output. The robot kinematic model adopts the definition of the robot operating system ROS, and adopts the arc model in the four-wheel differential model. The wheels on the same side have the same speed, and the wheels on the opposite side are driven independently.

[0112] The robot's motion process under two close coordinates is described as k ,y k ) to coordinate (x k+1 ,y k+1 ), the total angle difference is δθ k The non-linear motion, the dead reckoning model is:

[0113]

[0114] Among them, x k ,y k ,x k+1 ,y k+1 is the x and y coordinates at time k and k+1, S k is the displacement at time k, θ k is the direction angle at time k; δ represents the differential, and Δ represents the difference;

[0115] In addition, this embodiment calculates the optimal global path based on the generated obstacle map, so the existence of static obstacles has been considered and avoided during path calculation. For possible dynamic obstacles, a common safety domain method in the SLAM field is used for simple obstacle avoidance. For a schematic diagram of obstacle avoidance safety domain, see Figure 4 , the steps are as follows:

[0116] 1) With the current position of the robot as the center, a circular range with a radius of 2r is set as the robot's safety zone during movement. The inner circular range with a radius of r is the robot's emergency avoidance zone, and the outer circular range from r to 2r is the deceleration zone.

[0117] 2) When the robot detects an obstacle in the deceleration zone based on the camera, it reduces its travel speed to half of the original speed;

[0118] 3) When an obstacle is observed in the emergency escape zone, the speed will be reduced to 0 and the vehicle will stop on the spot until the obstacle is observed to leave the emergency escape zone and then the speed will be restored.

[0119] During the active exploration process, the robot receives visual image frames and uses the semantic segmentation model M based on the semantic extraction module. segWhen performing semantic segmentation, the semantic situations observed at different times are also different. This embodiment uses the two-dimensional semantic coverage and two-dimensional semantic accuracy similar to the semantic completeness index in this article to quantify the description. Suppose the image frame sequence of the exploration process is The total number of semantic pixels is N semgt , the two-dimensional semantic coverage threshold is λ cov , the two-dimensional semantic accuracy threshold is λ cer , let the length of the image frame Ik be H, the width be W, the total number of pixels be HxW, and the total number of pixels containing semantic information be N semgt ,Semantic,Semantic Segmentation Model M seg The number of pixels containing semantic information in the prediction result N sem ; Assume that the two-dimensional semantic coverage of a single frame is R cov , the two-dimensional semantic accuracy is R cer ,but:

[0120]

[0121] Using the semantic segmentation model M seg Predict the number of pixels containing semantic information in the current image frame I, denoted as N sem ; Based on the frame length H, frame width W, and the total number of semantic pixels N semgt , predicted number N sem Calculate the semantic coverage and accuracy of the current frame, denoted as R cov and R cer ; if R cov ≥λ cov or R cer ≤λ cer , then update for Add the current frame I to For each image frame at each moment in the active exploration process, all image frames whose single-frame 2D semantic coverage exceeds the threshold or whose accuracy is lower than the threshold are actively collected and saved, and the saved content includes the current frame, semantic prediction status and corresponding true value.

[0122] A pre-trained model refers to a model that has been trained based on a large dataset and has excellent shallow basic features and deep abstract feature extraction capabilities. The Mask RCNN pre-trained model used in this article has excellent performance on the training set MS-COCO. In order to ensure that it still has excellent performance in the experimental environment of this article, we fine-tune it based on the dataset collected by the method described above. Fine-tuning refers to adjusting the pre-trained network through small-scale training so that it can also have excellent performance in work scenarios similar to the original training set without the need for retraining.

[0123] The active exploration process also includes fine-tuning the semantic segmentation model using actively collected data, including the following steps:

[0124] The current task dataset is formed based on the actively collected data, and the original semantic segmentation model is tested on the current dataset to obtain the unoptimized recognition result;

[0125] Based on the size of the current task dataset and its similarity to the original semantic segmentation model training set, determine the number of frozen layers of the model, retain only the last n layers or the last softmax layer for fine-tuning, and keep the network parameters of the remaining layers unchanged;

[0126] Preprocess the data set based on Detectron rules and complete the registration to verify the correctness of data loading;

[0127] The preprocessed data set is divided into three parts: training set, validation set and test set. The ratio in this embodiment is 7:1:2.

[0128] Retrain the last n layers of the network based on the test set, using a smaller learning rate than the original training to avoid mutations in the weights of the original model;

[0129] Train until convergence, fine-tune, and test the model fine-tuning results based on the test set.

[0130] Example 4

[0131] This embodiment is based on the embodiments 1 to 3, and simulates the robot's exploration composition indoors. The simulation results are as follows: Figure 5 As shown, the robot step length can be divided into 0, 33, 100, 133, 142, Figure 5 The visual input image and map exploration are shown in Figure 2. It can be seen that the method proposed in the present invention has better performance in both semantic coverage and accuracy.

[0132] Obviously, the above embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. An active exploration and real-time mapping method for semantic completeness, It is characterized in that The following steps are involved: S1. The robot observes the environment in real time; S2. Use the front-end odometer to provide the robot with real-time posture information. The back-end receives the posture information provided by the front-end odometer at different times, optimizes the posture information, and outputs a three-dimensional semantic point cloud in combination with semantic extraction to obtain a global semantic map and the current posture information of the robot. S3. Performing a two-dimensional projection on the global semantic map to form a two-dimensional semantic map group; S4. Based on the current posture of the robot and the two-dimensional semantic map group, an active exploration strategy is adopted to complete the exploration of the specified step length, and the optimal path for semantic integrity exploration in the current scene and the constructed semantic map are obtained; in step S4, the process of the active exploration strategy includes: S41. Assume the robot's current number of explorations is S curr , the specified number of explorations is S update , the maximum number of explorations is S max , initialize the current number of explorations S curr is 0; S42. The current two-dimensional semantic map group result M curr and the current posture P curr Input to the global target generation network N global , output the global target G through the global target generation network global ; S43. Use the fast marching method to calculate the optimal global path Traj from the robot's current position to the global target position best ; S44.Robot along Traj best Move forward one unit and update the current pose P curr ; S45. Get the robot's current position P curr The observation semantic information O curr , based on O curr Update the current map M using the semantic SLAM method curr ; S46. Repeat steps S44 to S45 update times, and record the current number of explorations S curr The value of S47. Determine the current number of explorations S curr Whether the value reaches the maximum number of explorations S max If yes, use N global Update the global target G global , and return to S42; otherwise, end the exploration and obtain the optimal path L actually explored by the robot best , the current maintenance map result M curr The two-dimensional map of the 2nd to 16th channels is used as the semantic mapping result M sem , the semantic map result M sem and the semantic map truth value M gt The same part as the semantic integrity SI.

2. The method for active exploration and real-time mapping for semantic completeness according to claim 1, It is characterized in that The global target generation network described in step S42 includes a seven-layer neural network, wherein the first five layers are sequentially connected convolutional layers conv1~conv5, the last two layers are sequentially connected fully connected layers FC1~FC2, and also include embedding layers Eembed1~Eembed2, wherein the input of the convolutional layer conv1 is the robot initialization position and the semantic mapping result, the input of the embedding layer Eembed1 is the semantic map true value, the input of the embedding layer Eembed2 is the semantic category, and the semantic map true value and the semantic category are prior information; the output end of the embedding layer Eembed1, the output end of the embedding layer Eembed2, and the output end of the convolutional layer conv5 are all connected to the fully connected layer FC1, and the output end of the fully connected layer FC2 outputs the global target.

3. The method for active exploration and real-time mapping for semantic completeness according to claim 2, It is characterized in that In the process of global goal generation, the PPO algorithm of reinforcement learning is used for training. During the training process, the semantic completeness observed by the robot within a specified step size is used as a reward, and the richness of semantic information is proportional to the reward value; the intermediate results of the training process include: two-dimensional semantic mapping results, cumulative results of global goal generation, cumulative trajectory of the robot and corresponding reward values. The two-dimensional semantic mapping results, cumulative results of global goal generation, cumulative trajectory of the robot and corresponding reward values ​​are all stored as experience data for subsequent training to update the network parameters of the global goal generation network.

4. The method for active exploration and real-time mapping for semantic completeness according to claim 3, It is characterized in that During the training process, the number of exploration steps under the same global goal is limited, and the global target point and path planning are recalculated every n steps. The training process is: S421. Using parameter set θ h Initialize habitat environment and parameter set θ n Initialize the global target generation network; S422. Initialize the robot at random coordinates, and use the previous module to explore the target point with a specified step length in the current environment; S423. The robot stops the exploration after moving forward by a specified step length, and ends the current training round; Calculate the reward value based on the final generated semantic map and the corresponding true value, and store the relevant observation image and reward value in the experience buffer pool; S424. Sampling a specified amount of experience data from the experience buffer pool and updating network parameters in the global target generation network; S425. Determine whether the current number of training times is greater than the set maximum number of training iterations. If so, terminate the training and save the current training model and related parameter information; otherwise, return to step S422.

5. The method for active exploration and real-time mapping for semantic completeness according to claim 1, It is characterized in that Step S43 uses the fast marching method to calculate the optimal global path Traj from the current position of the robot to the global target position. best The process is: S431. Establish the eikonal equation: Where (x, y) is the two-dimensional coordinate of the target point in the map, T(x, y) is the time to reach the target point, V(x, y) is the forward speed of the robot, Δx and Δy are the set extension distances respectively; for the target point (x, y), the four adjacent points in the positive and negative directions of the x and y axes whose distance is the extension distance constitute its adjacent point set X, X = {(x+Δx, y), (x-Δx, y), (x, y+Δy), (x, y-Δy)}; S432. Construct an equation based on the distance relationship between the adjacent point set and the target point: T 1 =min(T (x-Δx,y) ,T (x+Δx,y) ) T 2 =min(T (x,y-Δy) ,T (x,y+Δy) ) Solve the equations together to get the time T(x,y) required to reach the target point; S433. Iteratively use the fast marching method to solve the time information of the neighboring points and the next neighboring points of the target point, and finally obtain the time value required for all map points to reach the target point, and record it as a two-dimensional time map corresponding to all map points; S434. Based on the two-dimensional time graph, the idea of ​​gradient descent method is applied to continuously select the lowest time value, so as to obtain the optimal global path to reach the target point from any position on the two-dimensional grid map.

6. The method for active exploration and real-time mapping for semantic completeness according to claim 1, It is characterized in that In step S44, the robot moves along Traj best The behavior decision when moving forward is divided into multiple discrete points according to the specified distance l; the discrete point closest to the robot on the path is set as the local target G local , which is the position that the robot needs to reach at the next moment. The robot's posture and local target coordinates are input into the behavior decision maker. The posture includes the position coordinates and direction, and the action that the robot needs to take at the current moment is output. The robot kinematic model adopts the definition of the robot operating system ROS, and adopts the arc model in the four-wheel differential model. The wheels on the same side have the same speed, and the wheels on the opposite side are driven independently. The robot's motion process under two close coordinates is described as k ,y k ) to coordinate (x k+1 ,y k+1 ), the total angle difference is δθ k The non-linear motion, the dead reckoning model is: Among them, x k ,y k ,x k+1 ,y k+1 is the x and y coordinates at time k and k+1, S k is the displacement at time k, θ k is the direction angle at time k; δ represents the differential, and Δ represents the difference.

7. The method for active exploration and real-time mapping for semantic completeness according to claim 6, It is characterized in that For dynamic obstacles, the safety domain method in the SLAM field is used to avoid obstacles.

8. The method for active exploration and real-time mapping for semantic completeness according to claim 1, It is characterized in that During the active exploration process, the robot receives visual image frames and uses the semantic segmentation model M based on the semantic extraction module. seg For semantic segmentation, let the image frame sequence of the exploration process be The total number of semantic pixels is N semgt , the two-dimensional semantic coverage threshold is λ cov , the two-dimensional semantic accuracy threshold is λ cer , let the length of the image frame Ik be H, the width be W, the total number of pixels be HxW, and the total number of pixels containing semantic information be N semgt ,Semantic,Semantic Segmentation Model M seg The number of pixels containing semantic information in the prediction result N sem ; Assume that the two-dimensional semantic coverage of a single frame is R cov , the two-dimensional semantic accuracy is R cer ,but: Using the semantic segmentation model M seg Predict the number of pixels containing semantic information in the current image frame I, denoted as N sem ; Based on the frame length H, frame width W, and the total number of semantic pixels N semgt , predicted number N sem Calculate the semantic coverage and accuracy of the current frame, denoted as R cov and R cer ; if R cov ≥λ cov or R cer ≤λ cer , then update for Add the current frame I to For each image frame at each moment in the active exploration process, all image frames whose single-frame 2D semantic coverage exceeds the threshold or whose accuracy is lower than the threshold are actively collected and saved, and the saved content includes the current frame, semantic prediction status and corresponding true value.

9. The method for active exploration and real-time mapping for semantic completeness according to claim 8, It is characterized in that The active exploration process also includes fine-tuning the semantic segmentation model using actively collected data, including the following steps: The current task dataset is formed based on the actively collected data, and the original semantic segmentation model is tested on the current dataset to obtain the unoptimized recognition result; Based on the size of the current task dataset and its similarity to the original semantic segmentation model training set, determine the number of frozen layers of the model, retain only the last n layers or the last softmax layer for fine-tuning, and keep the network parameters of the remaining layers unchanged; Preprocess the data set based on Detectron rules and complete the registration to verify the correctness of data loading; The preprocessed data set is divided into three parts: training set, validation set and test set; Retrain the last n layers of the network based on the test set, using a smaller learning rate than the original training to avoid mutations in the weights of the original model; Train until convergence, fine-tune, and test the model fine-tuning results based on the test set.

Images