Mapping method for removing dynamic obstacles by means of intelligent robot with body and storage medium

Abstract

The invention relates to a mapping method for removing a dynamic obstacle by an intelligent robot with a body, and the method comprises the steps: inputting a plurality of frames of pictures in the collected visual image information of the same scene into a trained neural network when the visual image information of a visual SLAM map is collected, carrying out the recognition of feature points, and carrying out the semantic segmentation of a dynamic object in the pictures; the ratio of the feature points on the segmented dynamic object in each frame of image of the visual image information of the same scene to other feature points is calculated, and the ratio of the dynamic degree to the static degree of each frame of image in the visual image information of the same scene is obtained; selecting the picture with the maximum dynamic and static degree ratio as a key frame of the scene; after feature points of dynamic objects of key frames in each scene are removed, construction is carried out to obtain a visual SLAM map. And repeated mapping and comparison processes are not needed, so that human resources are greatly saved.

Description

Technical Field

[0001] This application relates to the field of SLAM mapping technology, specifically to a mapping method and storage medium for an embodied intelligent robot to remove dynamic obstacles. Background Technology

[0002] Mapping is an essential part of SLAM, and good mapping can effectively improve the robot's localization accuracy. During mapping, dynamic obstacles often have a significant impact on the map, easily leading to noise, trailing, and other issues.

[0003] In current common robot mapping algorithms, the removal of moving objects often employs the following operations:

[0004] Option 1: Repeatedly build multiple maps and determine whether the part is a static part based on the matching of multiple maps (parts that are the same in multiple map building processes are often non-dynamic obstacles).

[0005] Option 2: Use a detection model to select some objects and remove the corresponding feature points during the mapping process.

[0006] However, in Solution 1, repeatedly building multiple maps is often only suitable for smaller scenarios. In large scenarios like airports, map building can be time-consuming even with a single attempt, and repeated map building is extremely resource-intensive. In Solution 2, using bounding boxes as the basis for judgment often results in many feature points being mistakenly identified as dynamic obstacles (because the bounding box is often larger than the object). Both Solution 1 and Solution 2, in practical use, often resemble unfinished products, merely simple logical processing to optimize dynamic obstacles without accurately identifying "dynamic points." Summary of the Invention

[0007] In view of the above problems, this application provides a mapping method and storage medium for an embodied intelligent robot to remove dynamic obstacles, thereby solving the problems of existing technical solutions.

[0008] To achieve the above objectives, the inventors provide a mapping method for embodied intelligent robots to remove dynamic obstacles, comprising:

[0009] When collecting visual image information to build a visual SLAM map, multiple frames of images from the same scene are input into the trained neural network to identify feature points and perform semantic segmentation on dynamic objects in the images.

[0010] The ratio of feature points on dynamic objects to other feature points in each frame of the visual image information of the same scene is used to obtain the ratio of the degree of motion and stillness in each frame of the visual image information of the same scene.

[0011] Select the image with the highest motion-to-stillness ratio as the keyframe for that scene;

[0012] A visual SLAM map is constructed by removing feature points of dynamic objects from keyframes in each scene.

[0013] In some embodiments, the structure of the neural network is a neural residual neural network ResNet50;

[0014] The first 5 layers of the ResNet50 neural residual neural network form the backbone network.

[0015] For the segmentation task, layers 3, 4, and 5 of the backbone network are used as feature layers, namely f3, f4, and f5. An upsampling operation is used to restore layers 4 and 5 to the size of the feature layer 3. The number of channels corresponding to layers 3, 4, and 5 is compressed to 1, resulting in the corresponding channels being divided into A3, A4, and A5.

[0016] Invert channels A3, A4, and A5, and multiply them with feature layer f3 as inverse attention to obtain features fa3, fa4, and fa5 containing attention.

[0017] The features fa3, fa4, and fa5 are then compressed into single-channel features, and the loss is calculated with the actual segmentation result to obtain the segmentation result.

[0018] For the feature point extraction task, a new branch is derived from feature fa3 to output the location information of the feature point. The data format of the location information of the feature point is (H, W, 1), where H is the location of the feature point and W is the corresponding confidence level.

[0019] In some embodiments, selecting the image with the highest motion-to-stillness ratio as the keyframe of the scene specifically includes the following steps:

[0020] The first frame of the visual image information of the same scene is taken as the keyframe, and the ratio of motion to stillness of the keyframe is taken as the standard ratio of motion to stillness.

[0021] The visual image information is slid through a preset window. The motion-to-static ratio of each frame is obtained and compared with the standard motion-to-static ratio of the keyframe. If the calculated motion-to-static ratio is greater than the standard motion-to-static ratio, the image corresponding to the motion-to-static ratio is taken as the keyframe and the window is reset. This step is repeated until the SLAM mapping of the current scene is completed.

[0022] In some embodiments, the neural network is trained as follows:

[0023] The COCO dataset is used as the original dataset, and its segmentation results are used as the real segmentation map to train the segmentation head of the neural network.

[0024] Randomly generate n homography matrices, and perform perspective transformation on each of the original images in the original dataset to obtain n transformed images corresponding to each original image;

[0025] For each original image, including the original image and the transformed n+1 images, ORB feature points are generated.

[0026] All feature points extracted from n+1 images are integrated into a single-channel image with pixel values ​​ranging from 0 to 1. This image is then used as the feature map for training the neural network's feature point extraction head.

[0027] In some embodiments, the step of performing ORB feature point analysis on each original image (including the original image and the transformed n+1 images) further includes the following steps:

[0028] The feature points extracted from the transformed image are inversely transformed using the corresponding homography matrix to restore them to the original image.

[0029] Another technical solution is also provided: a storage medium storing a computer program, which, when executed by a processor, performs the following steps:

[0030] When collecting visual image information to build a visual SLAM map, multiple frames of images from the same scene are input into the trained neural network to identify feature points and perform semantic segmentation on dynamic objects in the images.

[0031] The ratio of feature points on dynamic objects to other feature points in each frame of the visual image information of the same scene is used to obtain the ratio of the degree of motion and stillness in each frame of the visual image information of the same scene.

[0032] Select the image with the highest motion-to-stillness ratio as the keyframe for that scene;

[0033] A visual SLAM map is constructed by removing feature points of dynamic objects from keyframes in each scene.

[0034] In some embodiments, the structure of the neural network is a neural residual neural network ResNet50;

[0035] The first 5 layers of the ResNet50 neural residual neural network form the backbone network.

[0036] For the segmentation task, layers 3, 4, and 5 of the backbone network are used as feature layers, namely f3, f4, and f5. An upsampling operation is used to restore layers 4 and 5 to the size of the feature layer 3. The number of channels corresponding to layers 3, 4, and 5 is compressed to 1, resulting in the corresponding channels being divided into A3, A4, and A5.

[0037] Invert channels A3, A4, and A5, and multiply them with feature layer f3 as inverse attention to obtain features fa3, fa4, and fa5 containing attention.

[0038] The features fa3, fa4, and fa5 are then compressed into single-channel features, and the loss is calculated with the actual segmentation result to obtain the segmentation result.

[0039] For the feature point extraction task, a new branch is derived from feature fa3 to output the location information of the feature point. The data format of the location information of the feature point is (H, W, 1), where H is the location of the feature point and W is the corresponding confidence level.

[0040] In some embodiments, selecting the image with the highest motion-to-stillness ratio as the keyframe of the scene specifically includes the following steps:

[0041] The first frame of the visual image information of the same scene is taken as the keyframe, and the ratio of motion to stillness of the keyframe is taken as the standard ratio of motion to stillness.

[0042] The visual image information is slid through a preset window. The motion-to-static ratio of each frame is obtained and compared with the standard motion-to-static ratio of the keyframe. If the calculated motion-to-static ratio is greater than the standard motion-to-static ratio, the image corresponding to the motion-to-static ratio is taken as the keyframe and the window is reset. This step is repeated until the SLAM mapping of the current scene is completed.

[0043] In some embodiments, the neural network is trained as follows:

[0044] The COCO dataset is used as the original dataset, and its segmentation results are used as the real segmentation map to train the segmentation head of the neural network.

[0045] Randomly generate n homography matrices, and perform perspective transformation on each of the original images in the original dataset to obtain n transformed images corresponding to each original image;

[0046] For each original image, including the original image and the transformed n+1 images, ORB feature points are generated.

[0047] All feature points extracted from n+1 images are integrated into a single-channel image with pixel values ​​ranging from 0 to 1. This image is then used as the feature map for training the neural network's feature point extraction head.

[0048] In some embodiments, the step of performing ORB feature point analysis on each original image (including the original image and the transformed n+1 images) further includes the following steps:

[0049] The feature points extracted from the transformed image are inversely transformed using the corresponding homography matrix to restore them to the original image.

[0050] Unlike existing technologies, the above-mentioned solution, when constructing a visual SLAM map, inputs each frame of visual image information from the same scene into a trained neural network to identify feature points and segment dynamic objects in the images. The motion-to-static ratio of each image is calculated based on the ratio of feature points on the segmented dynamic objects to other feature points. The image with the highest motion-to-static ratio is then selected as the keyframe for the scene. By removing feature points of dynamic objects from the keyframes, a visual SLAM map with dynamic objects removed can be constructed. This eliminates the need for repeated map construction and comparison processes, significantly saving human resources.

[0051] The above description of the invention is merely an overview of the technical solution of this application. In order to enable those skilled in the art to better understand the technical solution of this application and to implement it based on the description and drawings, and to make the above-mentioned objectives and other objectives, features and advantages of this application easier to understand, the following description is provided in conjunction with the specific embodiments and drawings of this application. Attached Figure Description

[0052] The accompanying drawings are only used to illustrate the principles, implementation methods, applications, features, and effects of specific embodiments of this application and other related content, and should not be considered as limitations on this application.

[0053] In the accompanying drawings of the instruction manual:

[0054] Figure 1 This is a schematic flowchart illustrating a mapping method for removing dynamic obstacles using an embodied intelligent robot, as described in a specific implementation.

[0055] Figure 2 This is a flowchart illustrating step S130 of a specific implementation method;

[0056] Figure 3 This is a schematic diagram of the structure of the storage medium described in a specific embodiment.

[0057] The reference numerals used in the above figures are explained as follows:

[0058] 310. Storage medium,

[0059] 320. Processor. Detailed Implementation

[0060] To illustrate the possible application scenarios, technical principles, implementable specific solutions, and achievable objectives and effects of this application in detail, the following description, in conjunction with the listed specific embodiments and accompanying drawings, provides a detailed explanation. The embodiments described herein are merely illustrative of the technical solutions of this application and are therefore intended to limit the scope of protection of this application.

[0061] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The term "embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment, nor does it specifically limit its independence or connection with other embodiments. In principle, in this application, as long as there are no technical contradictions or conflicts, the technical features mentioned in each embodiment can be combined in any way to form corresponding implementable technical solutions.

[0062] Unless otherwise defined, the technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the use of related terms herein is merely for the purpose of describing particular embodiments and is not intended to limit this application.

[0063] In the description of this application, the term "and / or" is used to describe the logical relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A exists, B exists, and A and B exist simultaneously. Additionally, the character " / " in this document generally indicates that the preceding and following objects have an "or" logical relationship.

[0064] In this application, terms such as “first” and “second” are used only to distinguish one entity or operation from another, and do not necessarily require or imply any actual quantity, hierarchy or order relationship between these entities or operations.

[0065] Unless otherwise specified, the use of terms such as “comprising,” “including,” “having,” or other similar expressions in this application is intended to cover non-exclusive inclusion, which does not exclude the presence of additional elements in a process, method, or product that includes the stated elements, such that a process, method, or product that includes a list of elements may include not only those defined elements but also other elements not expressly listed, or elements inherent to such a process, method, or product.

[0066] As understood in the Examination Guidelines, in this application, expressions such as "greater than," "less than," and "exceeding" are understood to exclude the stated number; expressions such as "above," "below," and "within" are understood to include the stated number. Furthermore, in the description of the embodiments in this application, "multiple" means two or more (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups" and "multiple times," unless otherwise explicitly specified.

[0067] In the description of the embodiments of this application, the space-related expressions used, such as "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "vertical," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential," indicate the orientation or positional relationship based on the orientation or positional relationship shown in the specific embodiments or drawings. They are only for the purpose of describing the specific embodiments of this application or for the reader's understanding, and do not indicate or imply that the device or component referred to must have a specific position, a specific orientation, or be constructed or operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0068] Unless otherwise expressly specified or limited, the terms "installation," "connection," "linking," "fixing," and "setting," as used in the description of the embodiments of this application, should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two components or the interaction between two components. For those skilled in the art to which this application pertains, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0069] Please see Figure 1 This embodiment provides a mapping method for embodied intelligent robots to remove dynamic obstacles, including:

[0070] Step S110: When collecting visual image information for building a visual SLAM map, multiple frames of images from the collected visual image information of the same scene are input into the trained neural network to identify feature points and perform semantic segmentation on dynamic objects in the images.

[0071] Step S120: The ratio of feature points on the segmented dynamic objects to other feature points in each frame of the visual image information of the same scene is used to obtain the ratio of the degree of motion and stillness in each frame of the visual image information of the same scene.

[0072] Step S130: Select the image with the largest motion-to-stillness ratio as the keyframe for the scene;

[0073] Step S140: After removing the feature points of dynamic objects in the keyframes of each scene, a visual SLAM map is constructed.

[0074] When constructing a visual SLAM map, each frame of visual images of the same scene is input into a trained neural network to identify feature points and segment dynamic objects in the images. The motion-to-static ratio of each image is calculated based on the ratio of feature points on the segmented dynamic objects to other feature points. The image with the highest motion-to-static ratio is then selected as the keyframe for that scene. By removing feature points of dynamic objects from the keyframes, a visual SLAM map with removed dynamic objects can be constructed. This eliminates the need for repeated map construction and comparison processes, significantly saving manpower.

[0075] In some embodiments, the structure of the neural network is a neural residual neural network ResNet50;

[0076] The first 5 layers of the ResNet50 neural residual neural network form the backbone network.

[0077] For the segmentation task, layers 3, 4, and 5 of the backbone network are used as feature layers, namely f3, f4, and f5. An upsampling operation is used to restore layers 4 and 5 to the size of the feature layer 3. The number of channels corresponding to layers 3, 4, and 5 is compressed to 1, resulting in the corresponding channels being divided into A3, A4, and A5.

[0078] Invert channels A3, A4, and A5, and multiply them with feature layer f3 as inverse attention to obtain features fa3, fa4, and fa5 containing attention.

[0079] The features fa3, fa4, and fa5 are then compressed into single-channel features, and the loss is calculated with the actual segmentation result to obtain the segmentation result.

[0080] For the feature point extraction task, a new branch is derived from feature fa3 to output the location information of the feature point. The data format of the location information of the feature point is (H, W, 1), where H is the location of the feature point and W is the corresponding confidence level.

[0081] A multi-branch network structure was designed, and the feature extraction network was optimized twice using the segmentation results. Post-processing was then applied to further refine the feature points, resulting in diluted, non-dynamic feature points. The multi-branch network structure is as follows:

[0082] A. The first 5 layers of ResNet50 are used as the backbone network.

[0083] B. For the segmentation task, layers 3, 4, and 5 of ResNet50 are used as feature layers, denoted as f3, f4, and f5. These layers are then upsampled to the size of the features from layer 3. The corresponding channel numbers are then compressed to 1, denoted as A3, A4, and A5.

[0084] C. Simultaneously, invert A3, A4, and A5, and multiply them with f3 as inverse attention to obtain the features fa3, fa4, and fa5 containing the attention.

[0085] D. Then, compress fa3, fa4, and fa5 into single-channel features and calculate the loss with the actual segmentation result. This yields the detection result.

[0086] E. For the feature point extraction part, a new branch is introduced from fa3;

[0087] F. The original approach of outputting (N, 2) feature point locations was abandoned. Instead, the feature point location information was changed to a (H, W, 1) data format, representing the feature point location and the confidence level, respectively. This is used as the point feature.

[0088] By using a model based on segmentation and feature point extraction, the interference from dynamic objects during the mapping process is effectively reduced.

[0089] Please see Figure 2 In some embodiments, selecting the image with the highest motion-to-stillness ratio as the keyframe of the scene specifically includes the following steps:

[0090] Step S210: Take the first frame of the visual image information of the same scene as the keyframe, and take the motion-to-stillness ratio of the keyframe as the standard motion-to-stillness ratio.

[0091] Step S220: Slide the visual image information through a preset window, obtain the motion-to-static ratio for each frame of the image, and compare it with the standard motion-to-static ratio of the key frame. If the calculated motion-to-static ratio is greater than the standard motion-to-static ratio, then the image corresponding to the motion-to-static ratio is taken as the key frame and the window is reset. Repeat this step until the SLAM mapping of the current scene is completed.

[0092] In the keyframe selection algorithm, the original method of "judging by a certain interval and the matching score with the previous frame" was abandoned as the main criterion. The following scheme was adopted:

[0093] A. Define the first frame as a keyframe and calculate the ratio R0 of the number of dynamic feature points to the number of static feature points in the first frame. R0 will be used as the standard for subsequent keyframes.

[0094] B. It is important to note that after starting the map, you should aim at a wider area to make R0 more representative.

[0095] C. In the subsequent mapping process, slide within a certain window and calculate the ratio of dynamic and static feature points Rn for each frame. If Rn>R0 or is greater than a certain window, it is taken as a new keyframe and the window is reset.

[0096] D. Repeat steps B and C until the scene mapping is complete.

[0097] E. After completing the above steps, you will obtain a view that has eliminated most of the dynamic feature points.

[0098] In the actual mapping process, feature points and segmentation results can be obtained simultaneously through neural networks. Feature points located in the segmented dynamic objects (such as pedestrians, vehicles, and other objects that are not part of the background) are called "dynamic feature points," while the rest are called "static feature points."

[0099] In some embodiments, the neural network is trained as follows:

[0100] The COCO dataset is used as the original dataset, and its segmentation results are used as the real segmentation map to train the segmentation head of the neural network.

[0101] Randomly generate n homography matrices, and perform perspective transformation on each of the original images in the original dataset to obtain n transformed images corresponding to each original image;

[0102] For each original image, including the original image and the transformed n+1 images, ORB feature points are generated.

[0103] All feature points extracted from n+1 images are integrated into a single-channel image with pixel values ​​ranging from 0 to 1. This image is then used as the feature map for training the neural network's feature point extraction head.

[0104] The image enhancement module for the corresponding network is designed as follows:

[0105] A. Use the COCO dataset as the original dataset, and use its segmentation results as the real segmentation map to train the segmentation head.

[0106] B. Randomly generate n reasonable homography matrices H, and perform perspective transformation on each of them and the original image. Each image will generate a total of n+1 images containing the original image. ORB feature point extraction is then performed on each of the n+1 images.

[0107] C. Combine all feature points from each image into a single-channel image with values ​​ranging from 0 to 1. Pixels 1 and 0 in the image represent whether the location is a feature point, respectively. Use this image as the point feature extraction head for training.

[0108] The above steps complete the construction and training process of the neural network. After training, the ONNX model is quantized using the INT8 format, and the corresponding deployment is completed (the network consists of simple convolutional combinations, which are well adapted to various compilation and computing resources).

[0109] In some embodiments, the step of performing ORB feature point analysis on each original image (including the original image and the transformed n+1 images) further includes the following steps:

[0110] The feature points extracted from the transformed image are inversely transformed using the corresponding homography matrix to restore them to the original image.

[0111] After extracting n+1 images, the feature point positions are inversely transformed using the corresponding homography matrix H, and then restored to the original images. This yields feature point positions from multiple angles, which is beneficial for subsequent camera position transformation calculations. During the camera position transformation process, all static feature points are treated as normal feature points and imported into the normal calculation flow to obtain more accurate camera position transformation information.

[0112] The mapping method for removing dynamic obstacles by an embodied intelligent robot provided in the above embodiments has the following advantages:

[0113] Advantage 1: By designing a model based on segmentation and feature point extraction, the interference of dynamic objects during the mapping process is effectively reduced.

[0114] Advantage 2: The mapping method designed effectively reduces the need for repeated mapping and comparison processes in large-scale scene mapping, such as "pedestrian following," saving a significant amount of manpower.

[0115] Advantage 3: By utilizing relatively useless dynamic feature points and designing them into a ratio R that can represent the degree of motion and stillness in the scene, the generalization ability of the solution is improved.

[0116] Please see Figure 3 In another embodiment, a storage medium 310 stores a computer program, which is executed by a processor 320 to perform the following steps:

[0117] When collecting visual image information to build a visual SLAM map, multiple frames of images from the same scene are input into the trained neural network to identify feature points and perform semantic segmentation on dynamic objects in the images.

[0118] The ratio of feature points on dynamic objects to other feature points in each frame of the visual image information of the same scene is used to obtain the ratio of the degree of motion and stillness in each frame of the visual image information of the same scene.

[0119] Select the image with the highest motion-to-stillness ratio as the keyframe for that scene;

[0120] A visual SLAM map is constructed by removing feature points of dynamic objects from keyframes in each scene.

[0121] When constructing a visual SLAM map, each frame of visual images of the same scene is input into a trained neural network to identify feature points and segment dynamic objects in the images. The motion-to-static ratio of each image is calculated based on the ratio of feature points on the segmented dynamic objects to other feature points. The image with the highest motion-to-static ratio is then selected as the keyframe for that scene. By removing feature points of dynamic objects from the keyframes, a visual SLAM map with removed dynamic objects can be constructed. This eliminates the need for repeated map construction and comparison processes, significantly saving manpower.

[0122] In some embodiments, the structure of the neural network is a neural residual neural network ResNet50;

[0123] The first 5 layers of the ResNet50 neural residual neural network form the backbone network.

[0124] For the segmentation task, layers 3, 4, and 5 of the backbone network are used as feature layers, namely f3, f4, and f5. An upsampling operation is used to restore layers 4 and 5 to the size of the feature layer 3. The number of channels corresponding to layers 3, 4, and 5 is compressed to 1, resulting in the corresponding channels being divided into A3, A4, and A5.

[0125] Invert channels A3, A4, and A5, and multiply them with feature layer f3 as inverse attention to obtain features fa3, fa4, and fa5 containing attention.

[0126] The features fa3, fa4, and fa5 are then compressed into single-channel features, and the loss is calculated with the actual segmentation result to obtain the segmentation result.

[0127] For the feature point extraction task, a new branch is derived from feature fa3 to output the location information of the feature point. The data format of the location information of the feature point is (H, W, 1), where H is the location of the feature point and W is the corresponding confidence level.

[0128] A multi-branch network structure was designed, and the feature extraction network was optimized twice using the segmentation results. Post-processing was then applied to further refine the feature points, resulting in diluted, non-dynamic feature points. The multi-branch network structure is as follows:

[0129] A. The first 5 layers of ResNet50 are used as the backbone network.

[0130] B. For the segmentation task, layers 3, 4, and 5 of ResNet50 are used as feature layers, denoted as f3, f4, and f5. These layers are then upsampled to the size of the features from layer 3. The corresponding channel numbers are then compressed to 1, denoted as A3, A4, and A5.

[0131] C. Simultaneously, invert A3, A4, and A5, and multiply them with f3 as inverse attention to obtain the features fa3, fa4, and fa5 containing the attention.

[0132] D. Then, compress fa3, fa4, and fa5 into single-channel features and calculate the loss with the actual segmentation result. This yields the detection result.

[0133] E. For the feature point extraction part, a new branch is introduced from fa3;

[0134] F. The original approach of outputting (N, 2) feature point locations was abandoned. Instead, the feature point location information was changed to a (H, W, 1) data format, representing the feature point location and the confidence level, respectively. This is used as the point feature.

[0135] By using a model based on segmentation and feature point extraction, the interference from dynamic objects during the mapping process is effectively reduced.

[0136] In some embodiments, selecting the image with the highest motion-to-stillness ratio as the keyframe of the scene specifically includes the following steps:

[0137] The first frame of the visual image information of the same scene is taken as the keyframe, and the ratio of motion to stillness of the keyframe is taken as the standard ratio of motion to stillness.

[0138] The visual image information is slid through a preset window. The motion-to-static ratio of each frame is obtained and compared with the standard motion-to-static ratio of the keyframe. If the calculated motion-to-static ratio is greater than the standard motion-to-static ratio, the image corresponding to the motion-to-static ratio is taken as the keyframe and the window is reset. This step is repeated until the SLAM mapping of the current scene is completed.

[0139] In the keyframe selection algorithm, the original method of "judging by a certain interval and the matching score with the previous frame" was abandoned as the main criterion. The following scheme was adopted:

[0140] A. Define the first frame as a keyframe and calculate the ratio R0 of the number of dynamic feature points to the number of static feature points in the first frame. R0 will be used as the standard for subsequent keyframes.

[0141] B. It is important to note that after starting the map, you should aim at a wider area to make R0 more representative.

[0142] C. In the subsequent mapping process, slide within a certain window and calculate the ratio of dynamic and static feature points Rn for each frame. If Rn>R0 or is greater than a certain window, it is taken as a new keyframe and the window is reset.

[0143] D. Repeat steps B and C until the scene mapping is complete.

[0144] E. After completing the above steps, you will obtain a view that has eliminated most of the dynamic feature points.

[0145] In the actual mapping process, feature points and segmentation results can be obtained simultaneously through neural networks. Feature points located in the segmented dynamic objects (such as pedestrians, vehicles, and other objects that are not part of the background) are called "dynamic feature points," while the rest are called "static feature points."

[0146] In some embodiments, the neural network is trained as follows:

[0147] The COCO dataset is used as the original dataset, and its segmentation results are used as the real segmentation map to train the segmentation head of the neural network.

[0148] Randomly generate n homography matrices, and perform perspective transformation on each of the original images in the original dataset to obtain n transformed images corresponding to each original image;

[0149] For each original image, including the original image and the transformed n+1 images, ORB feature points are generated.

[0150] All feature points extracted from n+1 images are integrated into a single-channel image with pixel values ​​ranging from 0 to 1. This image is then used as the feature map for training the neural network's feature point extraction head.

[0151] The image enhancement module for the corresponding network is designed as follows:

[0152] A. Use the COCO dataset as the original dataset, and use its segmentation results as the real segmentation map to train the segmentation head.

[0153] B. Randomly generate n reasonable homography matrices H, and perform perspective transformation on each of them and the original image. Each image will generate a total of n+1 images containing the original image. ORB feature point extraction is then performed on each of the n+1 images.

[0154] C. Combine all feature points from each image into a single-channel image with values ​​ranging from 0 to 1. Pixels 1 and 0 in the image represent whether the location is a feature point, respectively. Use this image as the point feature extraction head for training.

[0155] The above steps complete the construction and training process of the neural network. After training, the ONNX model is quantized using the INT8 format, and the corresponding deployment is completed (the network consists of simple convolutional combinations, which are well adapted to various compilation and computing resources).

[0156] In some embodiments, the step of performing ORB feature point analysis on each original image (including the original image and the transformed n+1 images) further includes the following steps:

[0157] The feature points extracted from the transformed image are inversely transformed using the corresponding homography matrix to restore them to the original image.

[0158] After extracting n+1 images, the feature point positions are inversely transformed using the corresponding homography matrix H, and then restored to the original images. This yields feature point positions from multiple angles, which is beneficial for subsequent camera position transformation calculations. During the camera position transformation process, all static feature points are treated as normal feature points and imported into the normal calculation flow to obtain more accurate camera position transformation information.

[0159] Finally, it should be noted that although the above embodiments have been described in the text and drawings of this application, this should not limit the scope of patent protection of this application. Any technical solutions that are based on the essential concept of this application and utilize the content described in the text and drawings of this application, resulting in equivalent structural or procedural substitutions or modifications, as well as the direct or indirect application of the technical solutions of the above embodiments to other related technical fields, are all included within the scope of patent protection of this application.

Claims

1. A mapping method for removing dynamic obstacles using an embodied intelligent robot, characterized in that, include: When collecting visual image information to build a visual SLAM map, multiple frames of images from the same scene are input into the trained neural network to identify feature points and perform semantic segmentation on dynamic objects in the images. The ratio of feature points on dynamic objects to other feature points in each frame of the visual image information of the same scene is used to obtain the ratio of the degree of motion and stillness in each frame of the visual image information of the same scene. Select the image with the highest motion-to-stillness ratio as the keyframe for that scene; A visual SLAM map is constructed by removing feature points of dynamic objects from keyframes in each scene.

2. The mapping method for removing dynamic obstacles by an embodied intelligent robot according to claim 1, characterized in that, The structure of the neural network is a neural residual neural network ResNet50; The first 5 layers of the ResNet50 neural residual neural network form the backbone network. For the segmentation task, layers 3, 4, and 5 of the backbone network are used as feature layers, namely f3, f4, and f5. An upsampling operation is used to restore layers 4 and 5 to the size of the feature layer 3. The number of channels corresponding to layers 3, 4, and 5 is compressed to 1, resulting in the corresponding channels being divided into A3, A4, and A5. Invert channels A3, A4, and A5, and multiply them with feature layer f3 as inverse attention to obtain features fa3, fa4, and fa5 containing attention. The features fa3, fa4, and fa5 are then compressed into single-channel features, and the loss is calculated with the actual segmentation result to obtain the segmentation result. For the feature point extraction task, a new branch is derived from feature fa3 to output the location information of the feature point. The data format of the location information of the feature point is (H, W, 1), where H is the location of the feature point and W is the corresponding confidence level.

3. The mapping method for removing dynamic obstacles by an embodied intelligent robot according to claim 1, characterized in that, Selecting the image with the highest motion-to-stillness ratio as the keyframe for the scene specifically includes the following steps: The first frame of the visual image information of the same scene is taken as the keyframe, and the ratio of motion to stillness of the keyframe is taken as the standard ratio of motion to stillness. The visual image information is slid through a preset window. The motion-to-static ratio of each frame is obtained and compared with the standard motion-to-static ratio of the keyframe. If the calculated motion-to-static ratio is greater than the standard motion-to-static ratio, the image corresponding to the motion-to-static ratio is taken as the keyframe and the window is reset. This step is repeated until the SLAM mapping of the current scene is completed.

4. The mapping method for removing dynamic obstacles by an embodied intelligent robot according to claim 1, characterized in that, The training method for the neural network is as follows: The COCO dataset is used as the original dataset, and its segmentation results are used as the real segmentation map to train the segmentation head of the neural network. Randomly generate n homography matrices, and perform perspective transformation on each of the original images in the original dataset to obtain n transformed images corresponding to each original image; For each original image, including the original image and the transformed n+1 images, ORB feature points are generated. All feature points extracted from n+1 images are integrated into a single-channel image with pixel values ​​ranging from 0 to 1. This image is then used as the feature map for training the neural network's feature point extraction head.

5. The mapping method for removing dynamic obstacles by an embodied intelligent robot according to claim 4, characterized in that, The step of performing ORB feature point analysis on each original image, which includes the original image and the transformed n+1 images, further includes the following steps: The feature points extracted from the transformed image are inversely transformed using the corresponding homography matrix to restore them to the original image.

6. A storage medium storing a computer program, characterized in that, The computer program, when executed by the processor, performs the following steps: When collecting visual image information to build a visual SLAM map, multiple frames of images from the same scene are input into the trained neural network to identify feature points and perform semantic segmentation on dynamic objects in the images. The ratio of feature points on dynamic objects to other feature points in each frame of the visual image information of the same scene is used to obtain the ratio of the degree of motion and stillness in each frame of the visual image information of the same scene. Select the image with the highest motion-to-stillness ratio as the keyframe for that scene; A visual SLAM map is constructed by removing feature points of dynamic objects from keyframes in each scene.

7. The storage medium according to claim 6, characterized in that, The structure of the neural network is a neural residual neural network ResNet50; The first 5 layers of the ResNet50 neural residual neural network form the backbone network. For the segmentation task, layers 3, 4, and 5 of the backbone network are used as feature layers, namely f3, f4, and f5. An upsampling operation is used to restore layers 4 and 5 to the size of the feature layer 3. The number of channels corresponding to layers 3, 4, and 5 is compressed to 1, resulting in the corresponding channels being divided into A3, A4, and A5. Invert channels A3, A4, and A5, and multiply them with feature layer f3 as inverse attention to obtain features fa3, fa4, and fa5 containing attention. The features fa3, fa4, and fa5 are then compressed into single-channel features, and the loss is calculated with the actual segmentation result to obtain the segmentation result. For the feature point extraction task, a new branch is derived from feature fa3 to output the location information of the feature point. The data format of the location information of the feature point is (H, W, 1), where H is the location of the feature point and W is the corresponding confidence level.

8. The storage medium according to claim 7, characterized in that, Selecting the image with the highest motion-to-stillness ratio as the keyframe for the scene specifically includes the following steps: The first frame of the visual image information of the same scene is taken as the keyframe, and the ratio of motion to stillness of the keyframe is taken as the standard ratio of motion to stillness. The visual image information is slid through a preset window. The motion-to-static ratio of each frame is obtained and compared with the standard motion-to-static ratio of the keyframe. If the calculated motion-to-static ratio is greater than the standard motion-to-static ratio, the image corresponding to the motion-to-static ratio is taken as the keyframe and the window is reset. This step is repeated until the SLAM mapping of the current scene is completed.

9. The storage medium according to claim 6, characterized in that, The training method for the neural network is as follows: The COCO dataset is used as the original dataset, and its segmentation results are used as the real segmentation map to train the segmentation head of the neural network. Randomly generate n homography matrices, and perform perspective transformation on each of the original images in the original dataset to obtain n transformed images corresponding to each original image; For each original image, including the original image and the transformed n+1 images, ORB feature points are generated. All feature points extracted from n+1 images are integrated into a single-channel image with pixel values ​​ranging from 0 to 1. This image is then used as the feature map for training the neural network's feature point extraction head.

10. The storage medium according to claim 9, characterized in that, The step of performing ORB feature point analysis on each original image, which includes the original image and the transformed n+1 images, further includes the following steps: The feature points extracted from the transformed image are inversely transformed using the corresponding homography matrix to restore them to the original image.

Images