A robot-based real-time semantic map generation method and device

Abstract

The present invention discloses a real-time semantic map generation method and device based on a robot, the method comprising: combining a topological map generation method with deep learning feature extraction, the two are operated in parallel with each other when the robot traverses the environment, and are associated and data fused during feature aggregation, and when the robot completes traversal of the environment, a topological map with semantic information is output in real time. The device comprises: a processor and a memory. The present invention combines two topological map construction methods to generate a new semantic map, and proposes a new image feature aggregation based on the combination of deep learning and topological images, which improves the accuracy of path planning, thereby improving the service quality of the robot.

Description

Technical Field

[0001] The present invention relates to the field of robots, and in particular to a robot-based real-time semantic map generation method and device. Background Art

[0002] In today's society, intelligent robots have been popularized and applied to a certain extent. For example: restaurant or hotel robots delivering food and sanitary items. Garbage cleaning robots at home. Front desk robots for banks and shopping malls. Outdoor express delivery robots, some dancing and entertainment robots. Including the autonomous driving technology that has been vigorously studied at home and abroad recently. In essence, it is to treat the car as an intelligent machine device to complete the driving task for humans. It needs to extract environmental semantic information and understand the open environment. These technologies can be applied to robots in different fields. The algorithm research of semantic maps can promote robots to become more intelligent in the future development to a certain extent, and be able to complete more challenging tasks.

[0003] The operation of indoor service robots is inseparable from the construction of indoor maps. Only effective maps can support robots to complete various instructions from humans. The 2D grid map generated by the simultaneous localization and mapping algorithm (SLAM) does not have semantic information. It only contains the relative coordinate system of the map and some simple positional relationships. When a human issues an instruction to get a book from the living room, the robot cannot understand what the living room is and what a book is. Therefore, some algorithms are needed to make the map rich in semantic information. Therefore, semantic map construction algorithms based on deep learning and visual information have emerged, but the semantic maps constructed in this way still have certain defects. First, the storage data volume of this type of map is image storage, and the real-time image construction process of the robot places too high requirements on the image processor of the embedded device. In addition, although ordinary semantic maps have good visualization effects, path planning is difficult in the process of positioning and navigation. Summary of the invention

[0004] The present invention provides a real-time semantic map generation method and device based on a robot. The present invention combines two topological map construction methods to generate a new semantic map, and proposes a new image feature aggregation based on deep learning and topological image combination, which improves the accuracy of path planning and thus improves the service quality of the robot. See the following description for details:

[0005] In the first aspect, a real-time semantic map generation method based on a robot is provided. The method combines a topological map generation method with deep learning feature extraction. When the robot traverses the environment, the two are operated in parallel and associated and data fused during feature aggregation. When the robot completes traversal of the environment, a topological map with semantic information is output in real time. The specific steps include:

[0006] The residual neural network pre-trained by ImageNet is used as the image feature extractor. k Determine the topological node information, obtain the coordinate information connection between the topological node and the image, and use the Euclidean distance between the topological node and the image to determine whether to establish the topological node and the subordinate relationship between the topological node and the image;

[0007] If there is no node information in the map or the Euclidean distance between the image and all nodes is greater than the threshold, it means that a new node information needs to be opened up. The coordinates and angles of the new node are the coordinate angles when the image information is obtained. The semantic information nodes of the topological nodes are filled, and the image visual information is aggregated in the nodes. The visual information is used to obtain the scene location through scene recognition by feature fusion, and the feature target is obtained through target detection.

[0008] The node semantic information is enriched through the feature result information of scene location and feature target.

[0009] Wherein, the residual neural network is:

[0010] The input layer is an image with coordinate information obtained by the RGB sensor of the robot device. The convolution kernel is used in the middle to obtain the initial feature map, and the residual layer is used to extract features from the initial feature map to obtain the final feature map. Finally, average pooling is used to output an image feature vector of a certain dimension, and the obtained image scene representation is used to aggregate and superimpose it into the topological node.

[0011] Furthermore, the scene location is obtained by scene recognition through feature fusion of visual information as follows:

[0012] Vector F k is the current detection image feature, used to update the feature information of the aggregated node, representing the cumulative vector of the jth node, from the initial value Start calculation, indicating that the current node aggregates the feature vector F of the first RGB image 1 , k means that the input image sequence belonging to the same node is the kth one, represents the cumulative fusion features of k images, It means The accumulated fusion features before fusing the k-th image are specifically fused as follows:

[0013]

[0014] in, is the cumulative feature fusion vector of the previous sequence, and the feature vector of the current fusion The linear superposition of the feature vector of the previous sequence or the feature vector of the previous sequence on the image feature vector F k and The product of the difference and the hyperparameter μ.

[0015] In a second aspect, a robot-based real-time semantic map generation device is provided, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to enable the device to execute any one of the method steps described in the first aspect.

[0016] In a third aspect, a computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes any one of the method steps described in the first aspect.

[0017] The beneficial effects of the technical solution provided by the present invention are:

[0018] 1. This invention describes the generation process of two types of semantic topological maps and combines the semantic map creation algorithm to ensure the high accuracy of the map and the real-time performance of the algorithm;

[0019] 2. The present invention proposes a new deep learning network architecture, which combines neural networks with topological structures, can construct feature aggregation of topological nodes in real time, and improves the generation accuracy of semantic maps. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Schematic diagram of the creation process of the semantic topology graph of the same mileage;

[0021] Figure 2 It is a schematic diagram of the framework of the topological node classification algorithm based on image feature aggregation;

[0022] Figure 3 This is a diagram showing the comparison of average accuracy in the Robot@Home dataset;

[0023] Figure 4 A schematic diagram of the structure of a robot-based real-time semantic map generation device. DETAILED DESCRIPTION

[0024] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0025] In order to solve the problems existing in the background technology, the embodiment of the present invention utilizes deep learning and visual information, combined with ordinary semantic maps, to generate a topological map with semantic information. The map with a graph structure is used to facilitate the robot to perform autonomous navigation and path planning functions, and to better implement human instructions.

[0026] Among them, robot semantics can be summarized into three different levels. The first is to extract semantic information from static images obtained by robot vision sensors. Common methods include: image classification, target detection, image segmentation, image retrieval, etc. The second is to further build a semantic map of the robot environment. Including: 2D map, 3D point cloud, etc. The last is to combine semantics to enhance the robot's environmental interaction and human-computer interaction capabilities. Including robot movement, path planning, environmental understanding, etc.

[0027] The embodiment of the present invention mainly focuses on the generation algorithm of 2D topological maps and the research on the aggregation of topological node features. In the process of generating topological maps, the maps are enriched with environmental semantic information as much as possible. The semantic content of the perceived environment. Therefore, the focus of the embodiment of the present invention is the generation algorithm of two types of topological maps and the aggregation of high-level features. In addition, the embodiment of the present invention mainly studies and tests the scene recognition part in the process of semantic map generation. Finally, it is compared in multiple large public data sets and has good performance.

[0028] The first method of building a topological map based on mileage distance

[0029] The first topological map construction method uses the robot's mileage data to save a node information at the same distance, the distance segment is set to T, and the image information obtained by the robot device is I k , the coordinates of the image are Generate a node N every T distance according to the mileage distance j = {N j ,j=1…m}, m represents the total number of generated nodes, and j represents the sequence number of the node. The coordinates are represented as The way to define that an image belongs to a node is by calculating the Euclidean distance between the node and the image. The Euclidean distance is calculated as follows:

[0030]

[0031] Among them, D(I k ,N j ) represents the Euclidean distance between the node and the image.

[0032] The embodiment of the present invention determines which type of node an image belongs to by calculating the minimum Euclidean distance between the image and the node. j (I k ) indicates I k Belong to N j The index value of the node. k The node that belongs to this index. The specific formula is as follows:

[0033] N j (I k ) = arg min j{D(I k ,N j )} (2)

[0034] The embodiment of the present invention sets a distance threshold When the minimum distance is greater than the threshold It means that the image does not belong to all nodes, and the image is discarded. The specific formula is as follows:

[0035]

[0036] The above shows the construction method of the topological map. In this way, a topological map of the graph structure is generated during the robot mapping process, and then semantic information is added to the graph nodes to generate a topological map rich in indoor semantic information. The specific process is as follows Figure 1 shown.

[0037] First, a semantic map is generated by a neural deep learning scene recognition algorithm with feature fusion. k The prediction result is obtained by inputting the neural network, expressed as p(y k |I k ), and a corresponding n-dimensional 01 array can be obtained through the key target detection algorithm Where V j represents the key target of the jth node, which is an important part of scene semantics. Because a node is composed of multiple images, the information of multiple images needs to be fused into one node. The final result of scene recognition is written as C(N j ), indicating the final scene category of the node, whether it belongs to the kitchen, living room or other. The specific method is to obtain the result p(y k |I k ) dimensions are added together and the maximum value among all dimensions indicates that the node belongs to this scene.

[0038] The results of the key target detection process are obtained using V j To fill in the representation, the object value obtained by target detection for each image is The embodiment of the present invention uses a threshold α to indicate whether the target exists. When the confidence is greater than the threshold α, the dimension is set to 1, otherwise it is set to 0. Finally, information superposition is performed. The superposition method is the same as V j Perform an OR operation to get the final result.

[0039] In addition, the robot motion process includes node coordinates And the posture angle θ, so the semantic information of the entire node contains correspond Figure 1 Node=(x,y,θ,Object,Label).

[0040] The second method of building topological maps based on image aggregation

[0041] This method is based on the image I k = {I k , k=1…n} to determine the topological node information. The coordinate information of the topological node and the image is expressed by formula (1)(2)(3). The Euclidean distance between the topological node and the image is used to determine whether to establish the node and the subordinate relationship between the node and the image. If there is no node information in the map or when the Euclidean distance between the image and all the nodes is greater than the threshold, that is, This means that a new node information needs to be opened. The coordinates and angles of the new node are the coordinates and angles when the image information is obtained, that is, θ. The semantic information node filling of the topological node can be done in the same way as in method 1, using the image visual information aggregated in the node, and the visual information is used through the scene recognition algorithm of feature fusion to obtain the scene location, which can be expressed as p(y k |I k ), and at the same time, the characteristic target is obtained through the target detection algorithm, which can be expressed as p(V k |I k ).

[0042] The semantic information of nodes is enriched by the feature result information of both. The advantage of this method is that it can ensure that the distance between nodes is within a reasonable range, not too large to cause image information loss, nor too small to cause repeated consumption of image information. Real-time construction is achieved during the construction of the robot map.

[0043] Topological node aggregation method construction based on deep learning

[0044] This section proposes a novel topological node preservation algorithm based on image feature aggregation based on the second topological graph construction method. The algorithm framework is as follows Figure 2 As shown in the figure, the framework mainly consists of two modules that run synchronously in real time. The upper module in the figure mainly uses the residual neural network pre-trained by ImageNet as the image feature extractor, and the input layer is the coordinate information obtained by the RGB sensor of the robot device. The 3×224×224 image is obtained by using 64 7×7 convolution kernels in the middle to obtain a 64×112×112 feature map. Then, four residual layers are used to extract features. Only one layer is shown in the figure for simplification, and a 2048×7×7 feature map is obtained. Finally, average pooling is used to output a 2048-dimensional image feature vector F. k The image scene representation obtained by the neural network is used to aggregate and overlay the topological nodes of the lower module in the figure. The lower module mainly uses the image coordinate information The Euclidean distance between the coordinates and the topological nodes is used to determine whether the image generates a node or belongs to a node with the closest Euclidean distance. The specific topological node generation or attribution algorithm is the topological map construction method based on image aggregation proposed in the previous part of this article. The topological map is created in real time during the running calculation process. After the two real-time modules are finished running, the acquired feature vectors are associated and aggregated with the topological map and saved in the same node. The specific node feature fusion algorithm is described next.

[0045] The feature fusion algorithm proposed in this paper is aimed at the fusion process of the current detection image and the accumulated features of the corresponding topological points. k is the current detected image feature, which is used to update the feature information of the aggregated node, taking the generation of the jth node as an example. j Represents the cumulative vector of the jth node, starting from the initial value Start calculation, indicating that the current node aggregates the feature vector F of the first RGB image 1 , k means that the input image sequence belonging to the same node is the kth one, represents the cumulative fusion features of k images, It means The accumulated fusion features before fusing the kth image. The specific fusion algorithm is shown in formula (4).

[0046]

[0047] is the cumulative feature fusion vector of the previous sequence, and the feature vector of the current fusion It is the feature vector of the previous sequence or the linear superposition of the feature vector of the previous sequence on the image feature vector F k and The product of the difference between and the hyperparameter μ, the range of the hyperparameter μ is between 0 and 1. The specific selection conditions are Indicates whether the Euclidean distance between the input vector and the superimposed vector of the previous sequence is greater than the threshold If it is greater than, it means that the two vectors have a small similarity, and vector fusion can be performed. If it is less than the threshold, it means that the similarity is large and no fusion is performed. The latest fusion result of the topological point is updated through the fusion algorithm, and these key semantic information s are saved in the node. This approach eliminates the influence of the similarity vector on the result weight. It is beneficial to reduce the impact of image noise disturbance and promote the result performance.

[0048] The result of the algorithm can be used to obtain a fused feature vector semantic topology map, which can be used for a variety of tasks. jIt is possible to perform scene classification tasks and key object classification tasks in the j-node area, as well as complete some other deep learning-related work tasks according to specific tasks. This article is mainly used to complete the scene classification and key object classification of aggregation nodes. For key object classification, ResNet can be retrained specifically. The embodiment of the present invention directly uses the ResNet classification network pre-trained with ImageNet data, and removes the Softmax layer after the last linear fully connected layer. For the scene classification task, the embodiment of the present invention adds two fully connected layers after the output vector, uses ReLU (Rectified Linear Unit) as the activation function, and the output vector is determined according to the number of scene categories that need to be detected. This article uses k-fold cross validation for experimental testing, and trains the last two fully connected layer data through data, and finally obtains the scene classification result.

[0049] The topological node aggregation method algorithm based on deep learning proposed in the embodiment of the present invention ( Figure 2 The algorithm has the advantages of real-time and end-to-end, combining the topological map generation method with deep learning feature extraction. When the robot traverses the environment, the two are operated in parallel and associated and data fused when the features are aggregated. When the robot completes traversing the environment, a topological map with semantic information can be output in real time. The algorithm process runs fully automatically and can be implemented end-to-end.

[0050] Experiment 1:

[0051] The embodiment of the present invention compares the first type of topological image generation method and the second type of topological image generation method in the Auto scene of the Robot@Home dataset. The performance of the second type of method is slightly higher than that of the first type of method. The method based on topological images promotes the improvement of accuracy. Because it aggregates image features, it is similar to voting counts to determine the specific scene category, so the accuracy is higher.

[0052] Table 1. Accuracy of Auto dataset under topological map generation algorithm based on mileage distance

[0053] Scenario total Detecting the correct amount Accuracy Toilet 3 3 100％ bedroom 4 2 50％ corridor 12 7 58.3％ living room 7 6 85.7％ kitchen 4 2 50％ average 30 20 66.7％

[0054] Table 2 Accuracy of Auto dataset under topological map generation algorithm based on image aggregation

[0055] Scenario total Detecting the correct amount Accuracy Toilet 2 2 100％ bedroom 3 2 66.7％ corridor 9 5 55.5％ living room 7 6 85.7％ kitchen 4 2 50％ average 25 17 68％

[0056] In addition, the accuracy of the five scenarios in Robot@Home was compared, and the classification network based on EfficientNet-B2 and the feature aggregation network method based on EfficientNet-B2 combined with Yolo detection were added. The two types of topological node aggregation methods proposed in the embodiment of the present invention were compared. The results are shown in Figure 2. Figure 3 As shown. It can be very intuitively shown that the node accuracy of the two semantic topological graphs of this patent is significantly greater than that of the traditional neural network method. The main reason for this is that the classification model is a scene detection method based on RGB images, and tens of thousands of RGB images are used as test data for detection, so there is a certain false detection rate. However, the method of this article is to aggregate all RGB images within the Euclidean distance to which the node belongs, and select the highest probability as the final scene category. Therefore, as long as most images can detect the correct category, the result will be the correct node type. So the accuracy will be high.

[0057] Experiment 2: Experimental comparison of topological node aggregation based on deep learning

[0058] The COsy Localization Database dataset was used for comparison. This dataset is a sequence dataset generated based on a mobile platform and is used to evaluate vision-based positioning systems. This dataset uses laser sensors and infrared sensors to obtain information such as continuous RGB images, relative positioning coordinate system data, scene categories, and mileage data. This dataset mainly includes three sub-datasets, namely COLD-Freiburg, COLD-Ljubljana, and COLD-Saarbrücken. The following experimental part uses F, L, and S to represent. They are sampled from three different laboratories in Europe, and each type of sub-data contains different path planning in the same scene and data under different weather conditions. It contains a total of 76 data packets, each of which is a sequence selection data under different weather conditions. The data sequence of each data packet exceeds 1,000 RGB images. Therefore, this dataset is very suitable as an experimental platform for scene recognition and robot path testing.

[0059] The sub-datasets used are the partA scene in Freiburg, the partA scene in Ljubljana, and the partA and B scenes in Saarbrücken, which correspond to F-partA, L-partA, S-partA, and S-partB in the table, respectively. Each scene contains two different paths, path1 and path2. The test scenes are the ten scenes in the leftmost column of the table, in which the single-person office and the double-person office are combined into person office. In some scenes, only one type of data exists, and it cannot be tested in the four-fold cross validation, because one type of data has two types, path1 and path2. Therefore, when testing such scenes, path1 and path2 are used as the training set and test set respectively, and the average value is taken after the results are obtained. The floating range of accuracy is after the positive and negative signs.

[0060] This method is based on the topological node aggregation method of deep learning to build experiments, and the image-based method is the image-based aggregation calculation method proposed above. From the experimental data, it can be concluded that the aggregation method based on deep learning is higher in accuracy than the image-based aggregation method.

[0061] Table 3. Comparison experiment of the accuracy of the deep learning-based aggregation method and the image-based aggregation method in the COLD dataset.

[0062]

[0063] A robot-based real-time semantic map generation device, see Figure 4 The device combines the topological map generation method with the deep learning feature extraction. When the robot traverses the environment, the two are operated in parallel and associated and data fused when the features are aggregated. When the robot traverses the environment, a topological map with semantic information is output in real time. The device includes: a processor 1 and a memory 2. The memory 2 stores program instructions. The processor 1 calls the program instructions stored in the memory 2 to enable the device to perform the following method steps:

[0064] The residual neural network pre-trained by ImageNet is used as the image feature extractor. k Determine the topological node information, obtain the coordinate information connection between the topological node and the image, and use the Euclidean distance between the topological node and the image to determine whether to establish the topological node and the subordinate relationship between the topological node and the image;

[0065] If there is no node information in the map or the Euclidean distance between the image and all nodes is greater than the threshold, it means that a new node information needs to be opened up. The coordinates and angles of the new node are the coordinate angles when the image information is obtained. The semantic information nodes of the topological nodes are filled, and the image visual information is aggregated in the nodes. The visual information is used to obtain the scene location through scene recognition by feature fusion, and the feature target is obtained through target detection.

[0066] The node semantic information is enriched through the feature result information of scene location and feature target.

[0067] Among them, the residual neural network is:

[0068] The input layer is an image with coordinate information obtained by the RGB sensor of the robot device. The convolution kernel is used in the middle to obtain the initial feature map, and the residual layer is used to extract features from the initial feature map to obtain the final feature map. Finally, average pooling is used to output an image feature vector of a certain dimension, and the obtained image scene representation is used to aggregate and superimpose it into the topological node.

[0069] The scene location is obtained by scene recognition through feature fusion of visual information:

[0070] Vector F k is the current detection image feature, used to update the feature information of the aggregated node, representing the cumulative vector of the jth node, from the initial value Start calculation, indicating that the current node aggregates the feature vector F of the first RGB image 1 , k means that the input image sequence belonging to the same node is the kth one, represents the cumulative fusion features of k images, It means The accumulated fusion features before fusing the k-th image are specifically fused as follows:

[0071]

[0072] in, is the cumulative feature fusion vector of the previous sequence, and the feature vector of the current fusion The linear superposition of the feature vector of the previous sequence or the feature vector of the previous sequence on the image feature vector F k and The product of the difference and the hyperparameter μ.

[0073] It should be pointed out here that the device description in the above embodiment corresponds to the method description in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0074] The execution subjects of the above-mentioned processor 1 and memory 2 can be devices with computing functions such as computers, single-chip microcomputers, and microcontrollers. In specific implementation, the embodiments of the present invention do not limit the execution subjects and are selected according to the needs of actual applications.

[0075] The data signal is transmitted between the memory 2 and the processor 1 via the bus 3, which will not be described in detail in the embodiment of the present invention.

[0076] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, the storage medium includes a stored program, and when the program is running, the device where the storage medium is located is controlled to execute the method steps in the above embodiment.

[0077] The computer-readable storage medium includes but is not limited to a flash memory, a hard disk, a solid-state drive, and the like.

[0078] It should be pointed out here that the description of the readable storage medium in the above embodiment corresponds to the description of the method in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0079] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present invention are generated.

[0080] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be accessed by the computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium or a semiconductor medium, etc.

[0081] References

[0082] [1] C.Jin, A.Elibol, P.Zhu and NYChong, "Semantic Mapping Based onImage Feature Fusion in Indoor Environments," 2021 21st International Conference on Control, Automation and Systems (ICCAS), 2021, pp.693-698, doi:10.23919 / ICCAS52745.2021.9650062.

[0083] Unless otherwise specified, the models of the components in the embodiments of the present invention are not limited, and any device that can perform the above functions may be used.

[0084] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0085] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A robot-based real-time semantic map generation method, characterized in that: The method combines the topological map generation method with deep learning feature extraction. When the robot traverses the environment, the two are operated in parallel and associated and data fused during feature aggregation. When the robot completes traversal of the environment, a topological map with semantic information is output in real time. The specific steps include: The residual neural network pre-trained by ImageNet is used as the image feature extractor. k Determine the topological node information, obtain the coordinate information connection between the topological node and the image, and use the Euclidean distance between the topological node and the image to determine whether to establish the topological node and the subordinate relationship between the topological node and the image; If there is no node information in the map or the Euclidean distance between the image and all nodes is greater than the threshold, it means that a new node information needs to be opened up. The coordinates and angles of the new node are the coordinate angles when the image information is obtained. The semantic information nodes of the topological nodes are filled, and the image visual information is aggregated in the nodes. The visual information is used to obtain the scene location through scene recognition by feature fusion, and the feature target is obtained through target detection. Enrich node semantic information through feature result information of scene location and feature target; The method of obtaining the scene location by scene recognition through feature fusion of visual information is specifically as follows: Vector F k is the current detection image feature, used to update the feature information of the aggregated node, representing the cumulative vector of the jth node, from the initial value Start calculation, indicating that the current node aggregates the feature vector F of the first RGB image 1 , k means that the input image sequence belonging to the same node is the kth one, represents the cumulative fusion features of k images, It means The accumulated fusion features before fusing the k-th image are specifically fused as follows: Among them, the current fused feature vector The linear superposition of the feature vector of the previous sequence or the feature vector of the previous sequence on the image feature vector F k and The product of the difference and the hyperparameter μ.

2. A robot-based real-time semantic map generation method according to claim 1, characterized in that: The residual neural network is: The input layer is an image with coordinate information obtained by the RGB sensor of the robot device. The convolution kernel is used in the middle to obtain the initial feature map, and the residual layer is used to extract features from the initial feature map to obtain the final feature map. Finally, average pooling is used to output an image feature vector of a certain dimension, and the obtained image scene representation is used to aggregate and superimpose it into the topological node.

3. A robot-based real-time semantic map generation device, characterized in that: The device comprises: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute the method according to any one of claims 1 to 2.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is enabled to perform the method according to any one of claims 1 to 2.

Images