Service robot navigation method combining active target detection and dynamic obstacle avoidance

Abstract

The invention discloses a service robot navigation method combining active target detection and dynamic obstacle avoidance. The method comprises the steps that firstly, a priori knowledge graph is constructed, visual features, target word embedding features and knowledge graph features are extracted, the target word embedding features are fused with the visual features and the knowledge graph features through a cross attention mechanism, the features obtained through fusion are connected, and target flow interaction features are obtained; then, acquiring and rasterizing an environment point cloud, modeling the robot state and the dynamic obstacle state, and obtaining robot aggregation local observation features and dynamic obstacle aggregation local observation features; a robot-dynamic obstacle interaction graph is constructed, and obstacle logistics interaction features are obtained through GCN; and finally, fusing the target flow interaction feature with the obstacle logistics interaction feature to obtain a target flow-obstacle logistics interaction feature. And inputting the target flow-obstacle logistics interaction characteristics into the A3C model, and outputting an action. According to the method, efficient fusion of active target detection and dynamic obstacle avoidance is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot navigation, and in particular is a service robot navigation method combining active target detection and dynamic obstacle avoidance. Background Art

[0002] When home service robots perform complex service tasks, the placement of target objects is often disorganized. However, when the robots perform tasks such as "picking something" or "operating something", they must ensure that the target objects are within the robot's operable range. Therefore, active target detection is a prerequisite for home service robots to perform complex tasks.

[0003] Currently, most methods use end-to-end models to achieve active target detection, reasoning and outputting corresponding actions based on the visual images observed by the robot. However, existing end-to-end models only consider how to improve the accuracy and efficiency of the robot's active target detection in the absence of dynamic obstacle collisions, and do not consider the dynamic obstacles (such as pedestrians) encountered during the active target detection process. In the field of crowd navigation, there are technologies that model dynamic pedestrians, predict their movements, and form features that robots can learn to avoid pedestrians. However, they do not achieve both active target detection and dynamic pedestrian obstacle avoidance simultaneously. Summary of the Invention

[0004] In view of the deficiencies in the prior art, the technical problem to be solved by the present invention is to provide a service robot navigation method that combines active target detection and dynamic obstacle avoidance.

[0005] The present invention solves the technical problem by adopting the following technical solutions:

[0006] A service robot navigation method combining active target detection and dynamic obstacle avoidance comprises the following steps:

[0007] Step 1: Extract target flow features;

[0008] The environment image is input into a pre-trained convolutional neural network to extract visual features, and the target word is input into the pre-trained GloVe model for word embedding to obtain the target word embedding feature. A large language model is used to construct a priori knowledge graph that reflects the co-occurrence relationship of targets in the environment. The node features and adjacency matrix of the prior knowledge graph are input into the graph convolutional network for feature extraction. The extracted features are dimensionally adjusted to obtain the knowledge graph feature. Using the cross-attention mechanism, the target word embedding feature is fused with the visual features and the knowledge graph features respectively to obtain the visual-target interaction feature and the knowledge graph-target interaction feature. The visual-target interaction feature and the knowledge graph-target interaction feature are connected to obtain the target flow interaction feature.

[0009] Step 2: Extract obstacle flow characteristics;

[0010] Obtain an environmental point cloud, project it onto a plane, and rasterize it. Obtain a robot-centered observation sequence and an observation sequence centered on each dynamic obstacle within the observation time step. Split the robot-centered observation at a certain moment into two parts: static obstacles and dynamic obstacles. Extract features from each part using a convolutional neural network. Flatten the two features and then connect them to obtain the robot's local observation features at that moment. Similarly, obtain the robot's local observation features at other moments. Connect the robot's local observation features at all moments and input them into a long short-term memory network for aggregation to obtain the robot's aggregated local observation features. Similarly, obtain the aggregated local observation features of each dynamic obstacle.

[0011] A robot-dynamic obstacle interaction graph is constructed based on the aggregated local observation features of the robot and the aggregated local observation features of each dynamic obstacle. The node features and adjacency matrix of the robot-dynamic obstacle interaction graph are input into a graph convolutional neural network for feature extraction. The extracted features are dimensionally adjusted to obtain the obstacle logistics interaction features.

[0012] Step 3: The target flow interaction features and obstacle flow interaction features are fused using the cross-attention mechanism to obtain the target flow-obstacle flow interaction features. The target flow-obstacle flow interaction features are dimensionally adjusted and input into the A3C model. The robot selects the action at the current moment from the discrete action space, and performs the navigation task based on this action.

[0013] Repeat steps 1 to 3 until the robot navigates to the target.

[0014] Furthermore, in the first step, the current environment image is input into the pre-trained convolutional neural network for target detection to obtain the target features of the current environment; the current environment knowledge graph is constructed based on the target features of the current environment, and the similarity between each node in the current environment knowledge graph and each node in the prior knowledge graph is calculated. If the similarity exceeds the similarity threshold, the corresponding node in the current environment knowledge graph and its associated edges are added to the prior knowledge graph, and the prior knowledge graph is updated.

[0015] Furthermore, in the first step, the convolutional neural network for extracting visual features uses ResNet50, and the input features of the last pooling layer of ResNet50 are successively reduced in dimension and reshaped to obtain visual features.

[0016] Furthermore, for the construction of the robot-dynamic obstacle interaction graph, the robot and each dynamic obstacle are taken as nodes. If there is a dynamic obstacle in the observation centered on the robot, it is considered that there is a direct connection between the robot and the dynamic obstacle. If there are other dynamic obstacles in the observation centered on the dynamic obstacle, it is considered that there is a direct connection between the two dynamic obstacles, and there is an edge between the two directly connected nodes.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] 1. In previous studies, active target detection tasks were mostly implemented using end-to-end models, while dynamic obstacle avoidance tasks were usually implemented using traditional obstacle avoidance algorithms. Furthermore, these two tasks were usually performed separately and not integrated with the real environment. Therefore, dynamic obstacle avoidance could not be achieved during active target detection. This application, however, uses an end-to-end model to directly map the robot's current visual image to navigation actions. The model's input is the robot's current visual image and the environmental point cloud acquired by the lidar, and its output is the robot's actions. This allows the robot to effectively avoid dynamic obstacles while gradually approaching the target, achieving an efficient fusion of active target detection and dynamic obstacle avoidance. This effectively addresses the issue of task separation and lack of integration with the real environment in previous studies.

[0019] 2. The core of this method is to divide the navigation task into two parts: target flow and obstacle flow. In the target flow part, the target word embedding features are fused with the prior knowledge graph features and visual features through a cross-attention mechanism to form target flow interaction features. In the obstacle flow part, the environmental point cloud is rasterized, and the robot and dynamic obstacles are modeled separately. The robot's aggregated local observation features and the aggregated local observation features of each dynamic obstacle are extracted to generate a robot-dynamic obstacle interaction graph. Graph Convolutional Networks (GCN) are used to extract features of the robot-dynamic obstacle interaction graph to obtain obstacle flow features. The target flow interaction features and obstacle flow interaction features are deeply fused through the cross-attention mechanism to form target flow-obstacle flow interaction features. The target flow-obstacle flow interaction features are mapped to robot actions through the A3C reinforcement learning algorithm.

[0020] 3. Prior knowledge is crucial for home service robots to achieve efficient active target search. However, different home environments vary, and the same item may appear in different locations in different environments. Therefore, to improve adaptability to different home environments, we leverage the potential of large language models for textual reasoning to model commonly used household items and generate a universal prior knowledge graph for robot navigation. However, this prior knowledge graph does not reflect the actual environment. Therefore, during the robot's navigation task, the prior knowledge graph is dynamically updated in real time based on the current environment image, generating a personalized prior knowledge graph that adapts to different home environments. This approach not only improves navigation accuracy and efficiency, but also enhances the robot's adaptability in unknown environments.

[0021] 3. A graph-based obstacle feature extraction method is proposed for robot navigation in dynamic environments. This method first rasterizes the environment point cloud and uses a convolutional neural network (CNN) to extract the robot's local observation features from the robot-centered observation image, and the dynamic obstacle's local observation features from the dynamic obstacle-centered observation image. Then, a long short-term memory (LSTM) network is used to aggregate the local observation features at each moment to generate the robot's aggregated local observation features and the dynamic obstacle's aggregated local observation features. Finally, a graph convolutional network is introduced to fuse the robot's aggregated local observation features with the dynamic obstacle's aggregated local observation features to obtain the obstacle flow interaction features, which are used to guide the robot to effectively avoid dynamic obstacles during navigation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is the overall flow chart of the present invention;

[0023] Figure 2 The rasterized image of the environment point cloud of the present invention;

[0024] Figure 3 This is a framework diagram for the observation feature aggregation of the present invention. DETAILED DESCRIPTION

[0025] Specific embodiments are given below in conjunction with the accompanying drawings. The specific embodiments are only used to introduce the technical solutions of the present invention in detail and are not intended to limit the scope of protection of the present application.

[0026] The present invention provides a service robot navigation method combining active target detection and dynamic obstacle avoidance, comprising the following steps:

[0027] Step 1: Extract target flow features;

[0028] An environment image is obtained and input into a pre-trained ResNet50 network to extract visual features. The pre-trained ResNet50 network is obtained by pre-training the ResNet50 network on the ImageNet dataset. The dimension of the input feature X of the last pooling layer of the ResNet50 network is 2048×7×7. The feature X is reduced to a dimension of 256×7×7 by a 1×1 convolution. This feature is then reshaped to obtain a visual feature X' with a dimension of 256×49.

[0029] In order to process the semantic information of the target (such as tables, water cups and other items), the target words are input into the pre-trained GloVe model for word embedding, and the target semantic information is converted into a word embedding feature S with a dimension of 1×300; the word embedding feature S is linearly transformed using a fully connected layer to obtain the target word embedding feature S′ with a dimension of 1×256.

[0030] For the construction of the prior knowledge graph, the text reasoning ability of the large language model (deepseek) is used to build a prior knowledge graph for robot navigation. The design quality of the prompt word will directly affect the quality of the prior knowledge graph generated by the large language model. Therefore, in order to guide the large language model to generate a prior knowledge graph suitable for service robot navigation, the prompt word is designed as follows: "Suppose you need to design a service robot to perform a navigation task. The robot needs to navigate to the target point based on the semantic information provided by the user, and its perception information is limited to the current environment image. In order to improve navigation performance, please design a knowledge graph for the co-occurrence probability between items. The knowledge graph should contain nodes and edges, reflecting the common co-occurrence relationship of items in the home environment. Please output knowledge in the form of nodes, edges and adjacency matrix. Based on the prompt word, the large language model generates a prior knowledge graph G(V,E), where V represents a set of nodes and E represents a set of edges. A node represents an item in the home environment, and an edge represents the relationship between nodes. In this embodiment, the prior knowledge graph contains 41 types of nodes, such as refrigerators, microwave ovens, and ovens. The adjacency matrix of each node is listed in the form of a symmetric matrix based on co-occurrence probability, reflecting the relationship between nodes. Co-occurrence probabilities are divided into three categories: high probability (0.7-1.0) indicates strong spatial associations (such as kitchen equipment combinations); medium probability (0.4-0.6) indicates cross-regional or functional associations (such as dining table-kitchen); and low probability (0.1-0.3) indicates implicit or indirect associations (such as socket-shower room).

[0031] Although the robot will refer to the prior knowledge graph when performing navigation tasks, the prior knowledge graph cannot fully reflect the actual situation of target co-occurrence in the environment. Therefore, the prior knowledge graph is dynamically updated according to the environment to improve the accuracy of navigation. For the current environment image, the ResNet50 network pre-trained on the ImageNet dataset is used to detect the target (referring to the object in the image) and obtain the target feature F of the current environment. v ={CS,OH,X c ,Y c ,a,f v}, where CS represents the confidence of the target, OH represents the unique hot encoding of the target, (X c ,Y c ) represents the center coordinates of the bounding box, a represents the area of ​​the bounding box, and f v Represents the visual features extracted by the ResNet50 network, which is a 512-dimensional feature vector. The target feature F of the current environment v The dimension is n×517, where n represents the number of targets;

[0032] Construct the current environment knowledge graph G′(V′,E′) based on the target features of the current environment, calculate the similarity between each node in the current environment knowledge graph and each node in the prior knowledge graph, and if the similarity exceeds the similarity threshold, indicating that there is a correlation between the two nodes, then add the corresponding node and its associated edges in the current environment knowledge graph to the prior knowledge graph and update the prior knowledge graph; input the node features and adjacency matrix of the updated prior knowledge graph into the graph convolutional network for feature extraction to obtain feature K; use the fully connected layer to perform linear transformation on feature K to obtain the knowledge graph feature K′ with a dimension of 1×256;

[0033] Using the cross attention mechanism, the target word embedding feature S′ is fused with the visual feature V′ and the knowledge graph feature K′ respectively to obtain the visual-target interaction feature V with a dimension of 49×256. T and the knowledge graph-target interaction feature K with dimension 1×256 T ; In the fusion process, visual features and knowledge graph features are used as query vectors, and target word embedding features are used as key vectors and value vectors; the visual-target interaction feature V T and knowledge graph-target interaction feature K T Connect and get the target flow interaction feature F T .

[0034] Step 2: Extract obstacle flow characteristics;

[0035] When a robot performs active target detection tasks, it needs to avoid obstacles in the environment in real time. The movement trend of dynamic objects (including robots and other dynamic obstacles) is significantly affected by the surrounding dynamic and static objects, which will indirectly affect the robot's navigation performance. Therefore, aggregated local observation features are extracted for each dynamic object. Specifically, a lidar is used to scan the surrounding environment to generate an environmental point cloud. The environmental point cloud is projected onto a two-dimensional plane and rasterized. Each grid corresponds to a small area in the real scene, and a raster image is obtained. The raster image is annotated according to the detection results of the real scene. For example, a five-pointed star represents the robot's position, a square box represents a static obstacle, and a triangle represents a dynamic obstacle. Figure 2 As shown;

[0036] The local observation of the robot at time t includes the observation centered on the robot and the set of observations centered on the dynamic obstacle represents the observation centered on the i-th dynamic obstacle at time t, and m represents the number of dynamic obstacles. In order to learn the historical motion trajectory of the dynamic object, it is also necessary to record the observations before time t to obtain the observation sequence centered on the robot within the observation time step. and the observation sequence centered on the dynamic obstacle NT represents the observation time step length;

[0037] like Figure 3 As shown in the figure, the observations are divided into robot channels and dynamic obstacle channels for feature aggregation; taking the robot channel as an example, for the observation centered on the robot at time t Will The robot is divided into two parts: static obstacles and dynamic obstacles. The features of these two parts are extracted by convolutional neural network (CNN). The two features are flattened and then connected to obtain the local observation features of the robot at time t. Similarly, the local observation features of the robot at other moments within the observation time step are obtained. After connecting the local observation features of the robot at all moments, they are input into the long short-term memory network (LSTM) for aggregation to obtain the robot aggregated local observation features z. r ;

[0038] The dynamic obstacle channel uses the same operation as the robot channel to process the observation centered on the dynamic obstacle at each moment in the observation time step, and obtains the dynamic obstacle aggregated local observation feature, and then obtains the dynamic obstacle aggregated local observation feature set {z 1 ,z 2 ,...,z m};

[0039] In order to directly learn the interaction between the robot's aggregated local observation features and the dynamic obstacle's aggregated local observation features, a graph structure is introduced to describe the interaction between these features. The robot's aggregated local observation features and the dynamic obstacle's aggregated local observation features are used as node features. If there is a dynamic obstacle in the robot-centered observation, it is considered that there is a direct connection between the robot and the dynamic obstacle, and there is an edge between the two. If there are other dynamic obstacles in the dynamic obstacle-centered observation, there is a direct connection between the two dynamic obstacles, and there is an edge between the two. Thus, a robot-dynamic obstacle interaction graph is constructed. The node features and adjacency matrix of the robot-dynamic obstacle interaction graph are input into the graph convolutional network (GCN) for feature extraction to obtain the interaction graph feature, whose dimension is m+1×512. In order to facilitate the fusion with the target flow feature through the cross-attention mechanism, the fully connected layer is used to adjust the dimension of the relationship graph feature to obtain the obstacle flow interaction feature F. O .

[0040] Step 3: The target flow interaction features are fused with the obstacle flow interaction features using a cross-attention mechanism. The target flow interaction features are used as the query vector, and the obstacle flow interaction features are used as the key and value vectors to obtain the target flow-obstacle flow interaction features, whose dimension is 50×256. The fully connected layer is used to reduce the dimension of the target flow-obstacle flow interaction features. The reduced target flow-obstacle flow interaction features (with a dimension of 8×256) are input into the A3C model. The appropriate action is selected from the discrete action space {MoveAhead, RotateLeft, RotateRight, LookUp, LookDown, Done} to obtain the current action. The robot performs the navigation task based on this action. MoveAhead means the robot moves forward 0.25m, TurnLeft and TurnRight mean the robot rotates 45 degrees to the right / left, respectively, LookUp and LookDown mean the robot tilts its view up / down 30 degrees, and Done means the robot terminates the task.

[0041] Repeat steps 1 to 3 until the robot navigates to the target, achieving navigation that combines active target detection with dynamic obstacle avoidance.

[0042] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A service robot navigation method combining active target detection and dynamic obstacle avoidance, characterized in that: The following steps are involved: Step 1: Extract target flow features; Input the environment image into the pre-trained convolutional neural network to extract visual features, and input the target word into the pre-trained GloVe model for word embedding to obtain the target word embedding feature; A large language model is used to construct a priori knowledge graph that reflects the co-occurrence relationship of targets in the environment. The node features and adjacency matrix of the priori knowledge graph are input into the graph convolutional network for feature extraction. The extracted features are dimensionally adjusted to obtain knowledge graph features. A cross-attention mechanism is used to fuse the target word embedding features with the visual features and knowledge graph features to obtain vision-target interaction features and knowledge graph-target interaction features. The vision-target interaction features and knowledge graph-target interaction features are then concatenated to obtain target flow interaction features. Step 2: Extract obstacle flow characteristics; Obtain an environmental point cloud, project it onto a plane, and rasterize it. Obtain a robot-centered observation sequence and an observation sequence centered on each dynamic obstacle within the observation time step. Split the robot-centered observation at a certain moment into two parts: static obstacles and dynamic obstacles. Extract features from each part using a convolutional neural network. Flatten the two features and then connect them to obtain the robot's local observation features at that moment. Similarly, obtain the robot's local observation features at other moments. Connect the robot's local observation features at all moments and input them into a long short-term memory network for aggregation to obtain the robot's aggregated local observation features. Similarly, obtain the aggregated local observation features of each dynamic obstacle. A robot-dynamic obstacle interaction graph is constructed based on the aggregated local observation features of the robot and the aggregated local observation features of each dynamic obstacle. The node features and adjacency matrix of the robot-dynamic obstacle interaction graph are input into a graph convolutional neural network for feature extraction. The extracted features are dimensionally adjusted to obtain the obstacle logistics interaction features. Step 3: Use the cross-attention mechanism to fuse the target flow interaction features with the obstacle flow interaction features to obtain the target flow-obstacle flow interaction features; The target flow-obstacle flow interaction features are dimensionally adjusted and then input into the A3C model. The robot selects the action at the current moment from the discrete action space, and performs the navigation task based on this action. Repeat steps 1 to 3 until the robot navigates to the target.

2. The service robot navigation method combining active target detection and dynamic obstacle avoidance according to claim 1, characterized in that: In the first step, the current environment image is input into the pre-trained convolutional neural network for target detection to obtain the target features of the current environment; The current environment knowledge graph is constructed according to the target characteristics of the current environment, and the similarity between each node in the current environment knowledge graph and each node in the prior knowledge graph is calculated. If the similarity exceeds the similarity threshold, the corresponding node and its associated edges in the current environment knowledge graph are added to the prior knowledge graph to update the prior knowledge graph.

3. The service robot navigation method combining active target detection and dynamic obstacle avoidance according to claim 1 or 2, characterized in that: In the first step, the convolutional neural network ResNet50 is used to extract visual features. The input features of the last pooling layer of ResNet50 are successively reduced in dimension and reshaped to obtain visual features.

4. The service robot navigation method combining active target detection and dynamic obstacle avoidance according to claim 1, characterized in that: For the construction of the robot-dynamic obstacle interaction graph, the robot and each dynamic obstacle are taken as nodes. If there is a dynamic obstacle in the observation centered on the robot, it is considered that there is a direct connection between the robot and the dynamic obstacle. If there are other dynamic obstacles in the observation centered on the dynamic obstacle, it is considered that there is a direct connection between the two dynamic obstacles. There is an edge between the two directly connected nodes.

Images