A method for optimizing semantic goal navigation tasks based on exploration boundary mask

Abstract

This invention provides a method for optimizing semantic target navigation tasks based on exploration boundary masks. First, a problem model for path planning and endpoint selection in semantic target navigation tasks is established and validated. Second, the boundary between explored and unexplored areas on the map is calculated, referred to as the exploration boundary, which restricts the action space for endpoint selection in reinforcement learning path planning to the exploration boundary. Then, the relationship between different action points on the exploration boundary is used to optimize value calculation and reduce its variance. Finally, the states and actions of training samples are processed, and state-level data augmentation is performed to obtain more training samples and improve the training efficiency of endpoint selection. This invention solves the low performance and low efficiency problems in semantic target navigation tasks, improving the success rate of path length-weighted calculations by 25.4% and reducing simulation time by a factor of three.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of artificial intelligence and robot navigation, specifically relating to a method for optimizing semantic target navigation tasks based on exploring boundary masks. Background Technology

[0002] Semantic object navigation tasks require robots equipped with RGBD cameras to autonomously explore unknown map environments to find objects of specific categories, which is of great significance for intelligent inspection, security, and exploration. Semantic object navigation tasks are quite complex, requiring multiple capabilities to solve, such as scene mapping, analysis of object arrangement relationships, the ability to balance exploration and utilization, and path planning. The lack of any one of these capabilities will significantly impact the final performance of the task. Semantic object navigation tasks are typically measured using a success rate weighted by path length (SPL). An SPL of 0 indicates a failure, while a success SPL is the ratio of the shortest path length to the actual path length.

[0003] In recent years, several methods have been proposed for solving semantic target navigation tasks. End-to-end reinforcement learning methods are representative of map-free approaches, directly learning the mapping from raw sensor data to actions. However, they require billions of training data points, placing significant demands on training time and hardware resources. Furthermore, their performance is limited due to the lack of explicit use of structured information such as maps. In contrast, modular reinforcement learning methods maintain a semantic map by processing RGB and depth images. However, these methods still require tens of millions of training data points, and their action space is limited to every point on the map; in other words, their action space is very large, and their performance is similarly limited.

[0004] This invention provides a modular solution for semantic target navigation tasks. It uses a single-agent Markov decision process to model the endpoint selection of a single path planning step in semantic target navigation tasks, uses exploration boundary masks to reduce the size of the reinforcement learning action space, utilizes the value similarity between different action points on the exploration boundary to reduce the variance of value calculation, and uses state-level data augmentation to obtain more training samples without additional simulation time, thereby solving the problems of low performance and low efficiency in semantic target navigation tasks. Summary of the Invention

[0005] Purpose of the invention: This invention proposes a method for optimizing semantic target navigation tasks based on exploring boundary masks, in order to optimize the modular algorithm for solving semantic target navigation tasks.

[0006] Technical solution: The present invention provides a method for optimizing semantic target navigation tasks based on exploring boundary masks, comprising the following steps:

[0007] (1) Establish and validate a problem model for path planning endpoint selection in semantic target navigation tasks;

[0008] (2) Calculate the boundary line between the explored and unexplored areas in the map, which is called the exploration boundary. This restricts the action space for the endpoint selection of the reinforcement learning path planning to the exploration boundary.

[0009] (3) Optimize value calculation by exploring the relationship between different action points on the boundary and reduce the variance of value calculation;

[0010] (4) Process the state and action of the training samples, perform state-level data augmentation, obtain more training samples, and improve the training efficiency of planning endpoint selection.

[0011] Furthermore, the problem model described in step (1) is represented as a quadruple.<S,A,T,M> The form; in which:

[0012] S represents the state space of the robot and its environment: the state s at each time step t. t ∈S includes first-person RGB images (rgb). t First-person depth image t robot pose information t ;

[0013] A represents the robot's motion space: at each time step t, the robot's actions include moving forward, turning left, turning right, and stopping, i.e., a. t ∈A={forward,left,right,stop};

[0014] T is the state transition function: it describes the relationship between the state at the next time step, the state at the current time step, and the action at the current time step, i.e., s. t+1 =T(s) t ,a t );

[0015] M represents round task information: each round's task information includes the target object category gcat and the target object position gpos.

[0016] Furthermore, the problem model establishment described in step (1) includes training samples of first-person RGB images, first-person depth images, robot pose information, target object category, and target object position.

[0017] Furthermore, the implementation process of step (2) is as follows:

[0018] Determine if target category semantics are present. If target category semantics are present, select the pixel corresponding to the target category semantics as the planning endpoint. Determine if the current planning endpoint is seen. If the current planning endpoint is not reached and not seen, but there is a passable path to the planning endpoint, use the planning endpoint from the previous time step. Otherwise, perform the following steps to reselect the planning endpoint: First, perform Canny edge detection on the exploration area map to obtain the exploration boundary. Then, group and filter the exploration boundary. Finally, use the filtered action mask to restrict the selection of the planning endpoint.

[0019] Furthermore, the implementation process of step (3) is as follows:

[0020] For each filtered exploration boundary Fr, calculate the value corresponding to each action point on it, and obtain the array V. Fr :

[0021] V Fr ={v(p):p∈Fr}

[0022] For array V Fr Sort the data to obtain SV. Fr :

[0023] SV Fr =sort(V Fr );

[0024] The final value of the exploration boundary is obtained by averaging the values ​​of all points whose values ​​fall within the middle third of the boundary:

[0025]

[0026] Furthermore, the implementation process of step (4) is as follows:

[0027] For the state-action reward pair (s) obtained at time step t, t ,a t ,r t ), for a t Randomly and uniformly sample the boundary of the exploration area to obtain in Considering the limitations of the depth sensor's maximum measurement distance, object occlusion, and viewpoint issues in actual task execution, the constructed map will exhibit significant differences in local locations. Therefore, a mixture of the current time step state and the next time step state is used as the augmented state, i.e., s' t =λs t +(1-λ)s t+1 λ controls the occlusion ratio of the newly acquired local map; finally, (s' t ,a' t ,r t Add to the training samples.

[0028] Furthermore, the determination of whether a target category semantic exists is made by calculating whether the number of corresponding semantic pixels in the semantic map layer is 0, that is:

[0029] ngcat t = count({p:p∈SemMap) t AND label(p) = gcat})

[0030] Where label represents the semantic label corresponding to the point, SemMap t This is the semantic map at the current time step; if ngcat t If the value is greater than 0, it is determined that the target category semantics have appeared; otherwise, it is determined that they have not appeared.

[0031] Furthermore, the determination of whether the current planned endpoint is seen is made by calculating the number of pixels in the 10*10 area surrounding the planned endpoint on the exploration area map at the current time step, i.e.:

[0032]

[0033] If nseen t If the value is greater than 80, it is determined that the current planned endpoint has been seen; otherwise, it is determined that it has not been seen.

[0034] Furthermore, the process of grouping and filtering the exploration boundaries is as follows:

[0035] In the exploration area map, the point where the exploration boundary is obtained through Canny edge detection is set to 1, and all other points are set to 0, resulting in a map FrMap that displays the exploration boundary. t ObsMap (obstacle map) t The points corresponding to obstacles in FrMap t The value in is set to 0; for FrMap t A connected component labeling algorithm based on depth-first search, namely FrMap, is used. t =DFSLabel(FrMap) t This allows for different exploration boundaries within the FrMap. t The values ​​are given as different numbers; exploration boundaries less than 5 units from the agent's current position are removed; the length of each exploration boundary is calculated, using the number of pixels on the exploration boundary as its length, i.e., len. t (Fr i = count({p∈FrMap) t :FrMap t (p)=i}); Select the three exploration boundaries with the longest length.

[0036] Furthermore, the step of using the filtered action mask to restrict the selection of the planning endpoint is to set the strategy probability distribution of action points outside the filtered exploration boundary to 0 and the value to infinitesimal to achieve action mask restriction.

[0037] Beneficial effects: Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention provides a modular solution for semantic target navigation, uses exploration boundary masks to reduce the size of the reinforcement learning action space, utilizes the value similarity between different action points on the exploration boundary to reduce the variance of value calculation, and uses state-level data augmentation to obtain more training samples without additional simulation time. Thus, it solves the problems of low performance and low efficiency in semantic target navigation tasks, improves the success rate of path length weighted by 25.4%, and reduces simulation time by 3 times. Attached Figure Description

[0038] Figure 1 It is a visualization of the semantic target navigation task solving process;

[0039] Figure 2 It is a visualization of first-person RGB images and depth images;

[0040] Figure 3 It is a visualization of the state transition function;

[0041] Figure 4 This is a flowchart of a method for using an exploration boundary mask to constrain the endpoint selection space in path planning;

[0042] Figure 5 It is a comparison between obstacle maps and 3D point cloud maps;

[0043] Figure 6 This is an example of one round of a semantic target navigation task. Detailed Implementation

[0044] The present invention will now be described in further detail with reference to the accompanying drawings.

[0045] This invention proposes a method for optimizing semantic target navigation tasks based on exploring boundary masks, specifically including the following steps:

[0046] Step 1: Establish and validate the problem model of path planning endpoint selection in semantic target navigation task: The model establishment data includes first-person RGB image, first-person depth image, robot pose information, target object category, and target object position.

[0047] The path planning endpoint selection problem model is specifically represented by a quadruple.<S,A,T,M> Form: S is the state space representing the robot and its environment information: the state s at each time step t. t∈S includes first-person RGB images (rgb). t First-person depth image t robot pose information t A represents the robot's action space: at each time step t, the robot can perform actions including moving forward, turning left, turning right, and stopping, i.e., a. t ∈A={forward,left,right,stop}; T is the state transition function: describing the relationship between the next time step state, the current time step state, and the current time step action, i.e., s t+1 =T(s) t ,a t M is the round task information: the task information for each round includes the target object category gcat and the target object position gpos.

[0048] exist Figure 1 The solution process of the semantic target navigation task is visualized: each round of the task initializes the agent's birth pose and the target object category. After the semantic target navigation simulator obtains the initial information, it generates the environment state, and then the agent interacts with the simulator. At each time step of the interaction, the simulator provides a first-person RGB image, a first-person depth image, and robot pose information. The agent uses this invention to select the path planning endpoint and submits the action to the simulator to enter the next time step of the interaction. Figure 2 , Figure 3 The image visualizes RGB and depth images, as well as the state transition function. Figure 2 The given set of corresponding RGB images and depth images were both taken by a camera worn on the agent. The RGB images provide the color information currently visible to the agent, and the depth images provide the distance of each object to the agent. Figure 3 An example of a state transition function is given, showing how the RGB and depth images change after the agent turns right. It can be seen that in semantic target navigation tasks, the agent can only rely on its own onboard sensors without a map. The information the agent can understand is only a small part of the environment, and the environmental state is temporal. This makes traditional navigation methods unsuitable, and learning-based methods require enormous amounts of data and time.

[0049] Step 2: Calculate the boundary between explored and unexplored areas on the map, called the exploration boundary. This restricts the action space for endpoint selection in reinforcement learning path planning to the exploration boundary. Specifically, this includes the following steps:

[0050] Figure 4The algorithm flowchart for path planning endpoint selection is given. First, if the target category semantics are present, the pixel corresponding to the target category semantics is selected as the planning endpoint. If the planning endpoint is not reached and not seen, but there is a passable path to the planning endpoint, the planning endpoint of the previous time step is used. Otherwise, the following steps are performed to reselect the planning endpoint: First, Canny edge detection is performed on the exploration area map to obtain the exploration boundary. Then, the exploration boundary is grouped and filtered. Finally, the filtered action mask is used to restrict the selection of the planning endpoint.

[0051] Calculate the number of target semantic pixels in the semantic map at the current time step, i.e., ngcat t = count({p:p∈SemMap) t AND label(p) = gcat}), where label represents the semantic label corresponding to the point, SemMap t This is the semantic map at the current time step; if ngcat t If the value is greater than 0, it is determined that the target category semantics have appeared; otherwise, it is determined that they have not appeared.

[0052] Calculate the number of pixels in the already explored area within a 10x10 region surrounding the planned endpoint on the map at the current time step. If nseen t If the value is greater than 80, it is determined that the current planned endpoint has been seen; otherwise, it is determined that it has not been seen.

[0053] In the exploration area map, the point where the exploration boundary is obtained through Canny edge detection is set to 1, and all other points are set to 0, thus obtaining the map FrMap that displays the exploration boundary. t ObsMap (obstacle map) t The points corresponding to obstacles in FrMap t The value in the range is set to 0, meaning these points are mostly walls and are invalid exploration boundaries. ObsMap t (p)>0,FrMap t (p) = 0; for FrMap t A connected component labeling algorithm based on depth-first search, namely FrMap, is used. t =DFSLabel(FrMap) t Thus, different exploration boundaries are defined in FrMap. t The values ​​are given as different numbers; then, exploration boundaries less than 5 units from the agent's current position are removed, as these points are located at the agent's feet and cannot be observed; the length of each exploration boundary is calculated, and the number of pixels on the exploration boundary is used as its length, len. t (Fr i = count({p∈FrMap)t :FrMap t (p)=i}); Finally, select the three exploration boundaries with the longest length.

[0054] The action mask constraint is implemented by setting the probability distribution of the strategy for exploring action points outside the boundary after filtering to 0 and the value to infinitesimal.

[0055] Figure 5 The comparison between the obstacle map and the 3D point cloud map at the same time step is shown, and further explanation is provided here: In the left image, black represents free or unexplored areas, gray represents obstacles, and white represents the selected exploration boundary. The small squares in the exploration boundary represent the selected planning endpoints. The right image is the corresponding 3D point cloud map of the left image, showing which areas have been explored. The point next to the cross in the lower right area of ​​the right image cannot be a good planning endpoint because its surrounding area has already been explored, and the task route from the current agent position to any unexplored area does not need to pass through the cross. In addition, using either of the two lower left area squares as the planning endpoint has the same effect on semantic target navigation tasks because the environmental information gain obtained by the agent on the way to either point is basically the same.

[0056] Step 3: Optimize value calculation by exploring the relationships between different action points on the boundary and reducing the variance of value calculation.

[0057] For each filtered exploration boundary Fr, calculate the value corresponding to each action point on it, and obtain the array V. Fr :

[0058] V Fr ={v(p):p∈Fr}

[0059] For array V Fr Sort the data to obtain SV. Fr :

[0060] SV Fr =sort(V Fr )

[0061] The final value of the exploration boundary is obtained by averaging the values ​​of all points whose values ​​fall within the middle third of the boundary:

[0062]

[0063] by Figure 5For example, the value of an action point in a semantic target navigation task means the increased probability of success brought about by visiting a point. Therefore, the value of choosing a specific square on the exploration boundary or other action points located on the same exploration boundary is similar. The average value of the middle third of the values ​​of all action points on the exploration boundary where the square is located is taken as the final value of the exploration boundary.

[0064] Step 4: Process the state and action of the training samples, perform state-level data augmentation, and obtain more training samples without additional simulation time.

[0065] For the state-action reward pair (s) obtained at time step t, t ,a t ,r t ), for a t Randomly and uniformly sample the boundary of the exploration area to obtain in Considering the limitations of depth sensor's maximum measurement distance, object occlusion, and viewpoint in actual task execution, map construction often exhibits significant differences in local locations. Therefore, a mixture of the current time step state and the next time step state is used as the augmented state, i.e., s' t =λs t +(1-λ)s t+1 λ controls the occlusion ratio of the newly acquired local map; finally, (s' t ,a' t ,r t Add it to the training dataset.

[0066] like Figure 5 As shown, by replacing other action points located in the same exploration boundary as the block, and using state mixing as the state of the new action point, a new state-action pair is obtained from the state-action pair represented by the block without additional simulation time.

[0067] Figure 6 An example of a round is given, in which the target category is a television. In the first map, the agent chooses a location different from the current direction as the planning endpoint, which is mostly in unexplored areas. In the second and third maps, the agent uses the exploration boundary of the living room, which is usually equipped with a television, as the planning endpoint instead of other rooms. In the fourth map, the agent observes and confirms that it is a television, and then approaches the target with the shortest path.

[0068] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. A method for optimizing semantic target navigation tasks based on exploring boundary masks, characterized in that, Includes the following steps: (1) Establish and validate a problem model for path planning endpoint selection in semantic target navigation tasks; (2) Calculate the boundary line between the explored and unexplored areas in the map, which is called the exploration boundary. This restricts the action space for the endpoint selection of the reinforcement learning path planning to the exploration boundary. (3) Optimize value calculation by exploring the relationship between different action points on the boundary and reduce the variance of value calculation; (4) Process the state and action of the training samples, perform state-level data augmentation, obtain more training samples, and improve the training efficiency of planning the endpoint selection; The implementation process of step (3) is as follows: For each filtered exploration boundary Calculate the value corresponding to each action point to obtain an array. : For arrays Sort and get : ； The final value of the exploration boundary is obtained by averaging the values ​​of all points whose values ​​fall within the middle third of the boundary: 。 2. The method for semantic target navigation task optimization based on exploration boundary mask according to claim 1, characterized in that, The problem model described in step (1) is represented as a quadruple. The form; in which: The state space representing information about the robot and its environment: each time step The state below Including first-person RGB images First-person depth image Robot pose information ; For the robot's motion space: each time step The robot's actions include moving forward, turning left, turning right, and stopping. ; It is a state transition function: it describes the relationship between the state of the next time step, the state of the current time step, and the action of the current time step, i.e. ; Round-based mission information: Each round's mission information includes the target object category. Target object position .

3. The method for semantic target navigation task optimization based on exploration boundary mask according to claim 1, characterized in that, The problem model establishment in step (1) includes training samples of first-person RGB image, first-person depth image, robot pose information, target object category and target object position.

4. The method for optimizing semantic target navigation tasks based on exploration boundary masks according to claim 1, characterized in that, The implementation process of step (2) is as follows: Determine whether the target category semantics appear. If the target category semantics appear, select the pixel corresponding to the target category semantics as the planning endpoint. Determine whether the current planning endpoint is seen. If the current planning endpoint is not reached and the planning endpoint is not seen, but there is a passable path to the planning endpoint, then use the planning endpoint of the previous time step. Otherwise, perform the following steps to reselect the planning endpoint: First, perform Canny edge detection on the exploration area map to obtain the exploration boundary, then group and filter the exploration boundary, and finally use the filtered action mask to restrict the selection of the planning endpoint.

5. The method for optimizing semantic target navigation tasks based on exploration boundary masks according to claim 1, characterized in that, The implementation process of step (4) is as follows: For time step The state action reward obtained is ,right Randomly and uniformly sample the boundary of the exploration area to obtain ,in Considering the limitations of the depth sensor's maximum measurement distance, object occlusion, and viewpoint issues in actual task execution, the constructed map will exhibit significant differences in local locations. Therefore, a mixture of the current time step state and the next time step state is used as the enhanced state. , The occlusion ratio of the newly acquired local map was controlled; finally, Add training samples.

6. The method for optimizing semantic target navigation tasks based on exploratory boundary masks according to claim 4, characterized in that, The determination of whether a target category semantic exists is made by calculating whether the number of corresponding semantic pixels in the semantic map layer is 0, that is: in, This represents the semantic label corresponding to the point. For the semantic map at the current time step; if If the target category semantics have appeared, then it is determined that they have appeared; otherwise, it is determined that they have not appeared.

7. The method for optimizing semantic target navigation tasks based on exploration boundary masks according to claim 4, characterized in that, The determination of whether the current planned endpoint is seen is made by calculating the number of pixels in the 10*10 area surrounding the planned endpoint on the exploration area map at the current time step, i.e.: like If the current destination has been seen, it is determined that it has not been seen; otherwise, it is determined that it has not been seen.

8. The method for optimizing semantic target navigation tasks based on exploratory boundary masks according to claim 4, characterized in that, The process of grouping and filtering the exploration boundaries is as follows: In the exploration area map, the point where the exploration boundary is obtained through Canny edge detection is set to 1, and all other points are set to 0, resulting in a map that displays the exploration boundary. Obstacle map The point corresponding to the obstacle is in The value in is set to 0; for A connected component labeling algorithm based on depth-first search is used, i.e. This allows different exploration boundaries to be explored. The values ​​are given as different numbers; exploration boundaries less than 5 units from the agent's current position are removed; the length of each exploration boundary is calculated, using the number of pixels on the exploration boundary as its length, i.e. Select the three exploration boundaries with the longest length.

9. The method for optimizing semantic target navigation tasks based on exploratory boundary masks according to claim 4, characterized in that, The method of using the filtered action mask to restrict the selection of the planning endpoint is to set the strategy probability distribution of action points outside the filtered exploration boundary to 0 and the value to infinitesimal to achieve action mask restriction.

Images