A robot algorithm for depth constraint in a dynamic scene, a storage medium and equipment

Abstract

The application belongs to the technical field of simultaneous localization and mapping, and specifically discloses a robot algorithm for depth constraint in a dynamic scene, which comprises the following steps: step S1, collecting image information and performing complete segmentation on dynamic objects by using an image segmentation network; step S2, analyzing the motion state of the objects by a depth constraint method, and removing the accurately recognized dynamic regions; and step S3, using a time domain bidirectional pixel point cloud background repair model, utilizing static information in adjacent previous and subsequent frames to complete the repair of the background static key frame where the dynamic objects are located, construct a complete point cloud map, and calculate the pose. p The application solves the technical problems in the prior art that it is difficult to accurately segment potential dynamic objects in a dynamic scene, there are limitations and deviations in judging the dynamic objects, the dynamic objects cannot be accurately removed, and thus the self pose cannot be accurately estimated.

Description

Technical Field

[0001] This invention belongs to the field of Simultaneous Location And Mapping (SLAM) technology, specifically relating to a robot algorithm, storage medium, and device for depth constraints in dynamic scenes. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a core supporting technology for robot autonomous navigation and environmental interaction. It can estimate a robot's pose and build a local map without prior environmental information, providing necessary support for downstream perception and decision-making. Visual SLAM combines deep learning, multi-view geometry, and feature extraction methods such as points and lines to achieve environmental perception and structural representation. Benefiting from the rapid development of computer vision, visual SLAM can improve its ability to capture environmental semantic data with the assistance of low-cost cameras, exhibiting excellent scene adaptability and thus possessing broad development potential and significant economic application value. However, in practical applications, problems such as inaccurate segmentation, poor segmentation accuracy, or even inability to segment dynamic objects due to camera movement or rapid object movement persist. In the segmentation of potential dynamic objects, although existing segmentation networks can correctly segment objects and maintain good accuracy, the processing of semantic information about the boundaries of dynamic objects remains insufficient, failing to guarantee segmentation accuracy. In the dynamic object identification stage, relying solely on multi-view geometry has inherent limitations. When the motion vector is collinear with the camera trajectory or exhibits axial motion, the identification of dynamic objects in keyframes is prone to deviation, leading to inaccurate removal of dynamic objects in keyframes. Furthermore, existing technologies are insufficient in restoring the static background of the current frame image after dynamic region removal. They fail to project the image's color and depth information onto the current frame, and the size of the occluded area varies between frames, failing to match optimal weights. This results in poor static background restoration at the location of dynamic objects, consequently affecting the dense point cloud mapping of static environmental features and impacting the accuracy of pose estimation. Summary of the Invention

[0003] The purpose of this invention is to provide a robot algorithm for depth constraints in dynamic scenes, which solves the technical problems in the prior art that it is difficult to accurately segment potential dynamic objects in dynamic scenes, the judgment of dynamic objects has limitations and biases, and it is impossible to accurately eliminate dynamic objects, thus making it impossible to accurately estimate the pose of the robot.

[0004] The robot algorithm for depth constraints in dynamic scenes includes the following steps.

[0005] Step S1: Acquire image information and use an image segmentation network to perform complete segmentation of the dynamic object.

[0006] Step S2: Analyze the motion state of the object using the depth constraint method, and remove the precisely identified dynamic regions.

[0007] Step S3: Use the temporal bidirectional pixel point cloud background restoration model, utilizing adjacent front and back pixels... p The static information in the frame is used to repair the static keyframes of the background where the dynamic object is located, construct a complete point cloud map, and calculate the pose.

[0008] In step S2, the camera projection transformation mapping operator is used to obtain the pixel points, 3D spatial coordinates, and depth, and the spatial position of the object after the mapping transformation is analyzed; a system is established. The mapping function is used to calculate the coordinates of the projected image in the current frame; calculate the depth error between a series of consecutive frames; construct a motion fraction formula to describe the influence of depth error and epipolar error, compare the motion fraction with the corresponding threshold of static points, and identify the motion state of spatial points; after completing the dynamic judgment of feature points, the accurately identified dynamic regions are removed.

[0009] Preferably, the formula for calculating the fraction of motion is:

[0010] ,

[0011] in, For the score of the movement, epipolar error Its corresponding epipolar threshold Compare; Depth error Its corresponding depth error threshold Comparison; when the epipolar error Or depth error When the threshold is exceeded, the threshold for static points is: ,when > When the time is right, it indicates that the point in space is in motion; otherwise, it is at rest.

[0012] Preferred, The mapping function is: ,in, and These are the space vector parameters and the direction vector parameters. This is a high-dimensional projection transformation; This is represented as the inverse operation of a high-dimensional projection transformation. For the transition between two frames, This is an indexed set of feature points.

[0013] Depth error The formula for calculation is: ,in, The mean-finding function is based on the corresponding mapping function. Calculate the coordinates of the projected image of the current frame. Represented as a reference frame Depth map corresponding to pixels at coordinates. This is the depth map corresponding to the pixel point in the current frame's projected image coordinates. The depth map describes the depth of the projected point by taking the average depth within a 2x2 grid centered on the projected coordinates.

[0014] Preferably, the backbone network of the image segmentation network incorporates a residual structure to enhance its performance, forming a residual-feature pyramid network. Feature maps with high-dimensional information features in the backbone network are integrated into a two-dimensional collaborative attention module, adaptively weighting the pixels in dynamic object regions to enrich image feature information while enhancing the feature response of dynamic object regions. Feature maps with low-dimensional information features in the backbone network are integrated into a multi-scale hierarchical enhanced feature fusion module to enhance the features of these low-dimensional information features, strengthening the weaker semantic information in these feature maps. The outputs of the two-dimensional collaborative attention module and the multi-scale hierarchical enhanced feature fusion module are combined to generate the image segmentation result.

[0015] Preferably, in the multi-scale hierarchical enhanced feature fusion module, a multi-layer symmetrical encoder and decoder are used. The encoder first segments the image into feature maps of various sizes, and the results are then processed by a Block layer. The Block layer consists of two parts connected in series: a multi-head spatial dimensionality reduction attention module and a hybrid feedforward neural network. Two branches are designed for each image size: one branch outputs to the next layer to perform image fusion for downsampling and reducing resolution; the other branch performs feature fusion to deepen the repair of high-level features and edge information. Next, the feature maps generated by the two branches of different layers are merged in a 1x1 dimensional channel convolution, and then processed by the decoder layer by layer upsampling. Subsequently, the restored image is linearly transformed by linear projection to generate the probability output of the corresponding category.

[0016] The preferred algorithm for the multi-head spatial dimensionality reduction attention module is shown in the following equation:

[0017] ,

[0018] The algorithm for the hybrid feedforward neural network is shown in the following equation:

[0019] ,

[0020] in, Represents the normalization function. The function represents downsampling, where Q, K, and V represent the input query, key, and value, respectively. For single attention head feature dimensions, For negative perturbations of a fixed value; Indicates the input feature map, This indicates that layer normalization is performed on the feature map. Attention This indicates the processing of the multi-head spatial dimensionality reduction attention module. This represents two fully connected layers, with GeLU as the activation function. For activation function, This represents the output of the Block layer; the output of the Block layer. First, upsampling is performed, then the result is superimposed on the feature image output from the previous encoder and fused. After another upsampling process, the result is output to the decoder.

[0021] Preferably, after the feature map is input into the two-dimensional collaborative attention module, global attention average pooling and max pooling operations are performed on the feature map. The corresponding features are then enhanced through multiple perceptual layers to strengthen the connection between channels, and the weights of each channel are redistributed to obtain a weighted channel attention map. Subsequently, the channel attention map is input into the channel-preserving convolution kernel to obtain three different scales of horizontal and vertical features. These features are then fused element-wise to obtain spatial fusion features. Finally, the dimensional correlation of the spatial fusion features is adjusted by channel mixing to obtain the output. The output of the two-dimensional collaborative attention module is fused with the channel attention map and the probability output of the multi-scale hierarchical enhanced feature fusion module through element-wise accumulation. After multi-layer upsampling, the segmentation result of the image segmentation network is output.

[0022] Preferably, in step S3, a temporal bidirectional pixel point cloud background restoration model is used, taking the current frame to be restored as the time starting point, and using a time-weighted method based on the previous time domain... p After frames and time domain p The frame performs background inpainting using a pixel grid model, mapping dynamic object regions onto the pixel grid. The image's color and depth information are projected onto the current frame based on feature point matching relationships. Different weights are assigned to regions occluded by dynamic objects between different frames to obtain the inpainted current frame. The calculation formula is:

[0023] ,

[0024] in, This is the repaired current frame. It is the current frame that has not been repaired. Indicates the time domain before the current framep frame, Indicates the time domain of the current frame p frame, and Associative weights, images closer to the current frame have higher weights.

[0025] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a robot algorithm for depth constraints in dynamic scenes as described above.

[0026] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to implement the steps of a robot algorithm for depth constraints in dynamic scenes as described above.

[0027] The present invention has the following advantages:

[0028] 1. It not only considers the projection transformation relationship between all 3D spatial points and pixels, but also uses the projection coordinates of the current frame to calculate the depth error. Furthermore, it incorporates epipolar error. With threshold The difference between the epipolar and depth errors is used to determine whether an object is in motion. When dealing with objects whose motion vectors are collinear with the camera trajectory or exhibit relative axial motion, the epipolar constraint, by using a weighted approach combined with depth error, generates a comprehensive motion score that is compared to a motion error threshold. If the score exceeds the threshold, the object is identified as a dynamic feature point and removed. This innovative technology comprehensively considers both epipolar and depth errors, enabling more accurate identification and segmentation of dynamic object regions. This allows for more precise removal of dynamic regions, overcoming the shortcomings of existing technologies where inaccurate dynamic region segmentation and removal hinders robot pose estimation.

[0029] 2. To address the issue of restoring static keyframes in the region after potential dynamic object removal, thus preventing disruption to the construction of a complete static point cloud map, this invention proposes a temporal bidirectional pixel mesh model restoration method. This method uses the current frame to be restored... Starting from the time frame, a time-weighted approach is used to perform background restoration using a pixel grid model based on the 10 frames before and after the time frame. The color and depth information of the image are projected onto the current frame according to the feature point matching relationship. Since the size of the area occluded by dynamic objects varies between different frames, a pixel grid model is used to match different weights to the pixels of the acquired images from previous and subsequent frames. By mapping and restoring the pixels from previous and subsequent frames, the static background of the RGB image of the current frame after dynamic region removal can be effectively restored, thereby improving the mapping performance of the SLAM algorithm in dynamic environments.

[0030] 3. This invention uses ResNet101 and FPN to form an image segmentation network, adding a two-dimensional collaborative attention module and a multi-scale hierarchical enhanced feature fusion module. By inputting information of different dimensions into the corresponding modules, the low-dimensional boundary information-rich feature map is input into the multi-scale hierarchical enhanced feature fusion module. Through a three-layer symmetrical encoder and decoder structure, image segmentation is performed. Feature maps of different sizes are downsampled to reduce resolution and extract latent features. The decoder module performs multi-layer upsampling to recover the image features of different sizes after convolution. This design helps the network to reduce the loss of details of low-resolution features and improve segmentation accuracy even when objects are occluded by strengthening boundary semantic information. This solves the technical problem of difficulty in accurately segmenting dynamic objects that may present multiple poses or have irregular parts that are occluded during movement.

[0031] 4. The high-dimensional feature map is input into the two-dimensional collaborative attention module. After different pooling strategies such as max pooling and average pooling, the network can better acquire rich feature representations of the image and enhance its understanding of dynamic object boundary scenes. This design can enhance the feature response of the target region, guide the network to focus on the region where dynamic objects are located, and weaken the interference from the background region. Attached Figure Description

[0032] Figure 1 This is a basic flowchart of a robot algorithm for depth constraints in dynamic scenarios according to the present invention.

[0033] Figure 2 A flowchart illustrating the SLAM system using this invention.

[0034] Figure 3 This is a schematic diagram of the image segmentation network in this invention.

[0035] Figure 4 This is a schematic diagram of the structure of the multi-scale hierarchical enhancement feature fusion module in this invention.

[0036] Figure 5 This is a schematic diagram of the MSRA module in this invention.

[0037] Figure 6 This is a flowchart of capturing spatial relationships in the channel attention diagram in this invention.

[0038] Figure 7 This is a schematic diagram illustrating the principle of determining the motion of potential dynamic objects in this invention.

[0039] Figure 8 This is a schematic diagram of the temporal domain bidirectional pixel grid background restoration process in this invention.

[0040] Figure 9This is a comparison chart of the object segmentation effect between the present invention and existing technologies such as YOLOv12n and Yolact.

[0041] Figure 10 This is a comparison chart showing the experimental results of dynamic object identification between the present invention and existing algorithms such as ORB-SLAM3 and DynaSLAM.

[0042] Figure 11 This is a comparison of the dense point cloud mapping effects of the present invention and the ORB-SLAM3 algorithm.

[0043] Figure 12 This is a comparison diagram of the trajectories obtained by the present invention and algorithms such as ORB-SLAM3, DS-SLAM, and DynaSLAM in SLAM experiments.

[0044] Figure 13 To verify the feasibility of this invention, a real-world scene image of dynamic object occlusion was provided.

[0045] Figure 14 This is a comparison diagram of the trajectories obtained by the present invention and the ORB-SLAM3 algorithm in a real-world experiment. Detailed Implementation

[0046] The following detailed description of the embodiments, with reference to the accompanying drawings, will further illustrate the specific implementation of the present invention, in order to help those skilled in the art to have a more complete, accurate, and thorough understanding of the inventive concept and technical solutions of the present invention.

[0047] Example 1.

[0048] like Figures 1-8 As shown, the present invention provides a robot algorithm for depth constraints in dynamic scenes, including the following steps.

[0049] Step S1: Acquire image information and use an image segmentation network to perform complete segmentation of the dynamic object.

[0050] In this step, image information is first acquired using the robot's built-in RGB-D camera. The acquired image information is then input into an image segmentation network for segmentation. The backbone of the image segmentation network consists of ResNet101 and FPN, and a residual structure is introduced to enhance the performance of the backbone network, forming a residual-feature pyramid network. The image segmentation network also incorporates the following technical improvements based on the aforementioned backbone network.

[0051] 1) Integrate the feature maps P4 and P5, which have high-dimensional information features in the backbone network, into the two-dimensional collaborative attention module (DSAM) to enrich the image feature information and enhance the feature response of dynamic object regions.

[0052] The role of the two-dimensional collaborative attention module is to assign corresponding weights to each channel in the feature map. After inputting the H×W×C feature map F into the two-dimensional collaborative attention module, global attention average pooling and max pooling operations are performed on the feature map to obtain information for each channel of the feature map. The corresponding features obtained through average pooling and max pooling are... and After strengthening the connections between channels through multiple perception layers (HP) and redistributing the weights of each channel, a channel attention graph with weighted allocation is obtained. Channel Note Diagram The algorithm is as follows: HP stands for Multi-Layer Perception Layer. This represents the Sigmoid function.

[0053] Then, pay attention to the passage diagram. The input is fed into a D, 5x5 channel convolution kernel to obtain horizontal and vertical features at three different scales, thereby enhancing the ability to capture spatial relationships. These features are then fused element-wise to obtain the spatial fusion feature. This allows the features of each spatial location to simultaneously carry contextual information in both horizontal and vertical directions, resulting in spatially fused features. The algorithm is as follows: , Indicates horizontal features, Vertical features are represented by channel-preserving convolution kernel attention maps. Processed, N The scales for the corresponding features are 7, 11, and 21 in this embodiment.

[0054] Finally, the spatial fusion characteristics are adjusted through channel mixing. The dimensional correlation is output. The corresponding algorithm is as follows: ,in Represents the ReLU activation function. W and b These are learnable parameters.

[0055] 2) The feature map P3 with low-dimensional information features in the backbone network is integrated into the multi-scale hierarchical enhanced feature fusion module. It is used to input the feature map P3 with rich boundary semantic information and perform feature enhancement processing, aiming to strengthen the weak semantic information in these feature maps.

[0056] In the multi-scale hierarchical enhanced feature fusion module, a three-layer symmetrical encoder and decoder are used. Two branches are designed for each image size: one branch outputs to the next layer to perform image fusion, which is used to downsample and reduce the resolution to adjust the number of channels to form a hierarchical design, reducing the amount of computation without losing feature information; the other branch performs feature fusion to deepen the repair of high-level features and edge information, reducing the loss of details in low-resolution features. Next, the feature maps generated by the two branches of different layers are merged in a 1x1 dimensional channel convolution, and then processed by the decoder layer by layer to enhance the stability and non-linear expressive power of the features. Finally, the restored image is linearly transformed by linear projection to generate the probability output of the corresponding category.

[0057] Specifically, after feature map P3 is input, this module first segments the image into feature maps of various sizes to more effectively capture local image details; for example, image segmentation is used to divide the feature map into fixed 1 / 4 sizes, forming a hierarchical design, and multiple downsampling reduces the resolution, which is beneficial for the encoder to extract potential image features. The resulting image—such as low resolution… Feature information is then processed through the Block layer (MSRA Transformer Block). The Block layer consists of two parts connected in series: a Multi-Head Spatial Dimensionality Reduction Attention (MSRA) module and a hybrid feedforward neural network. In the Block layer, pixel weights are first normalized based on the distance between pixels, and then input into the hybrid feedforward neural network.

[0058] The attention mechanism of the Multi-Head Spatial Dimensionality Reduction Attention Module (MSRA) uses dropkey to implicitly assign an adaptive operator to each attention head. This helps constrain the attention distribution by weakening regions with high attention values, making the model focus more on dynamic regions and capturing robust global features. First, convolution is performed on the key (K) and value (V) to extract local features or adjust channel dimensions. Then, the query (Q) is multiplied by the adjusted key (K) using a matrix multiplication. After assigning the adaptive operator, scaling and normalization are performed to avoid gradient vanishing. Finally, a matrix multiplication is performed with the value (V) to extract relevant features. The purpose of this attention mechanism is to introduce local context information and parameter sharing, enhancing the model's ability to model local structures while controlling computational complexity. The algorithm of the Multi-Head Spatial Dimensionality Reduction Attention Module (MSRA attention mechanism) is shown in the following equation:

[0059] ,

[0060] in, Represents the normalization function. The function represents downsampling, where Q, K, and V represent the input query, key, and value, respectively. For single attention head feature dimensions, For negative perturbations of a fixed value. Attention This indicates the processing of the multi-head spatial dimensionality reduction attention module.

[0061] In a hybrid feedforward neural network, the input is processed sequentially through a fully connected layer (FC), a depthwise convolutional layer (DW Conv), an activation function (GeLU), and another fully connected layer (FC), before being superimposed to output the final result. The fully connected layer expands the edge feature dimension of the input image, capturing dynamic object features and matching system dimensions in the output. The depthwise convolutional layer (DW Conv) performs independent convolution operations on each input channel, enhancing the ability to capture local information. The activation function uses a Gaussian error linear unit (GeLU), introducing non-linearity into the network and providing smoother operation compared to the traditional ReLU function. The algorithm for the hybrid feedforward neural network is shown in the following equation:

[0062] ,

[0063] in, Indicates the input feature map, This indicates that layer normalization is performed on the feature map, and the result serves as the input to the multi-head spatial dimensionality reduction attention module. Attention This indicates the processing of the multi-head spatial dimensionality reduction attention module. This represents two fully connected layers, with GeLU as the activation function. For activation function, This represents the output of the Block layer; the output of the Block layer. First, upsampling is performed, then the result is superimposed on the feature image output from the previous encoder and fused. After another upsampling process, the result is output to the decoder.

[0064] The structure of this hybrid feedforward neural network makes the gradient more stable during the training process, increases the semantic relevance of dynamic object boundaries, and enables the network to pay attention to both local details and global features, thereby improving the overall feature extraction capability.

[0065] Combining the aforementioned technical improvements 1) and 2), the high-dimensional and low-dimensional feature maps extracted from the backbone network are input into the dual-dimensional collaborative attention module and the multi-scale hierarchical enhanced feature fusion module, respectively. The dual-dimensional collaborative attention module can adaptively adjust the pixel weights of dynamic object regions, while the multi-scale hierarchical enhanced feature fusion module can strengthen weak boundary semantic information, thereby improving the accuracy of dynamic object recognition. The output of the dual-dimensional collaborative attention module... With regard to the channel diagram The probability output of the multi-scale hierarchical enhanced feature fusion module is fused element by element, and after multi-layer upsampling, the segmentation result of the image segmentation network is output.

[0066] Step S2: Analyze the motion state of the object using the depth constraint method, and remove the precisely identified dynamic regions.

[0067] This step uses a camera projection transformation mapping operator to obtain pixel coordinates, 3D spatial coordinates, and depth, and then analyzes the spatial position of the object after the mapping transformation. (For 3D spatial points...) and projection point The projection relationship between the two is expressed as follows:

[0068] ,

[0069] in, This represents the camera projection transformation mapping operator. , These are the focal length parameters of the camera in the X and Y directions. , These are the pixel coordinates in the X and Y directions. , and These represent the X and Y coordinates and depth of a point in space, respectively. These are depth-stable parameters.

[0070] To calculate the coordinates of the projected image in the current frame, this step establishes a... The mapping function is used to calculate the coordinates of the projected image in the current frame, and is then used in the calculation of the depth error formula. and These are the space vector parameters and the direction vector parameters. For high-dimensional projection transformation; corresponding mapping function Represented as:

[0071] ,

[0072] in, This is represented as the inverse operation of a high-dimensional projection transformation. For the transition between two frames, This is an indexed set of feature points.

[0073] To more accurately determine whether a feature point belongs to a dynamic object, this step also incorporates depth calculation and judgment. For each spatial pixel mapping... This step calculates its depth error across a series of consecutive frames. Depth error The formula for calculation is:

[0074] ,

[0075] in, The mean-finding function is based on the corresponding mapping function. Calculate the coordinates of the projected image of the current frame. Represented as a reference frame Depth map corresponding to pixels at coordinates. This is the depth map corresponding to the pixel point in the current frame's projected image coordinates. The depth map describes the depth of the projected point by taking the average depth within a 2x2 grid centered on the projected coordinates.

[0076] After obtaining the depth error, in order to better determine whether the feature points are in a dynamic state, a motion fraction formula is constructed to describe the influence of depth error and epipolar error. The calculation formula for the motion fraction formula is as follows:

[0077] ,

[0078] in, For the score of the movement, epipolar error Its corresponding epipolar threshold Compare; Depth error Its corresponding depth error threshold Comparison; when the epipolar error Or depth error When the threshold is exceeded, the threshold for static points is: ,when > When the time is right, it indicates that the point in space is in motion; otherwise, it is at rest.

[0079] After completing the dynamic judgment of feature points, the accurately identified dynamic regions are removed. Thus, this algorithm, after completely segmenting the occluded object and accurately determining that the object is a dynamic object, uses depth constraints to achieve accurate removal of dynamic objects.

[0080] Step S3: Use the temporal bidirectional pixel point cloud background restoration model, utilizing adjacent front and back pixels... p The static information in the frame is used to repair the static keyframes of the background where the dynamic object is located, construct a complete point cloud map, and calculate the pose.

[0081] In this step, a temporal bidirectional pixel point cloud background restoration model is used. Taking the current frame to be restored as the time starting point, a time-weighted approach is employed, based on the preceding time frame... p After frames and time domain p Frame-based background restoration of pixel mesh models (e.g.) p=10), mapping the dynamic object region onto a pixel grid, projecting the image's color and depth information onto the current frame based on feature point matching relationships, and assigning different weights to the occluded areas of dynamic objects in different frames to obtain the repaired current frame. Repaired current frame The calculation formula is:

[0082] ,

[0083] in, This is the repaired current frame. It is the current frame that has not been repaired. Indicates the time domain before the current frame p frame, Indicates the time domain of the current frame p frame, and To establish association weights, images closer to the current frame have higher weights. The correlation between image pixels and color information is also adjusted according to the weights, and the reliability of feature information from images closer to the current frame is higher.

[0084] Based on the rotation and translation vectors obtained from estimating the camera position, the RGB-D data is converted into point cloud data. Then, multiple point cloud data are aggregated to obtain a dense point cloud map. The transformation process is calculated as follows:

[0085] ,

[0086] in It is a 3x3 three-dimensional rotation matrix. It is a 3x1 three-dimensional displacement vector. It is a 3x1 three-dimensional scaling vector; inputting it will yield the transformed matrix. .

[0087] Through these steps, the algorithm can effectively recover the static background after dynamic region removal, obtain the feature point information of the static background, and then construct a complete dense point cloud map, thereby improving the performance of the SLAM algorithm in dynamic environments.

[0088] The following section uses specific experiments to illustrate the technical effects of the aforementioned robot algorithm for depth constraints in dynamic scenarios.

[0089] like Figure 9As shown, to verify the segmentation performance of the improved segmentation network algorithm, a comprehensive validation was conducted on the dynamic subset of the TUM dataset under two different occlusion conditions. The red boxes in the figure serve as auxiliary markers. The first and second rows in the figure can be used to verify the segmentation ability of dynamic objects under different occlusion conditions. Through segmentation comparison, it was found that in the first row with low occlusion, the human feature information is obvious, but because YOLOv12n and Yolact lack the ability to process semantic information of the target boundary region, they cannot complete the segmentation of the occluded object boundary. In the second row with high occlusion, the human feature information is not obvious, and YOLOv12n and Yolact also cannot complete the segmentation of dynamic objects. Therefore, the algorithm in this paper integrates a two-dimensional collaborative attention module into the high-dimensional features of the backbone network to enrich image features and enhance the dynamic region response; and adds a multi-scale hierarchical enhanced feature fusion module to the low-dimensional features to strengthen weak semantic information in the feature layer with rich boundary semantics. After the high-dimensional and low-dimensional feature maps are fed into the corresponding modules, the former adapts to the dynamic region pixel weights, and the latter strengthens the weak boundary semantics, ultimately improving the accuracy of dynamic object recognition.

[0090] like Figure 10 As shown, subsets of four dynamic sequences from the TUM dataset were selected for feature point evaluation. The first and second rows represent feature point evaluations by the ORB-SLAM3 and Dyna-SLAM algorithms, respectively, while the third row represents the feature point evaluations by this algorithm. It can be seen that the ORB-SLAM3 algorithm classifies feature points on both dynamic and static objects as static feature points (human figures are dynamic targets; green indicates static feature points, and red indicates dynamic feature points). Dyna-SLAM's feature point evaluation performance is significantly better than the former, but it may misclassify features when the object is rotating. This algorithm, however, is more accurate in identifying feature points compared to the previous two algorithms.

[0091] Figure 11 Among them Figure 11 (a)~ Figure 11 (d) is the original RGB image containing the animated figure. Figure 11 (e)~ Figure 11 (h) is the mapping of the repaired dense point cloud. For example... Figure 11 As shown, the background in the occluded area of ​​the dynamic figure was effectively repaired. The figure in the rpy sequence was stationary for a certain period of time and therefore was not removed, which fully demonstrates the accuracy of the dynamic object motion judgment. The dense point cloud mapping effect is good, showing the SLAM system's good localization and mapping capabilities.

[0092] like Figure 12As shown, ORB-SLAM3 exhibits poor robustness because it cannot distinguish between dynamic and static objects. Dyna-SLAM employs instance segmentation technology to mark the regions containing dynamic objects and removes them from the environment to reduce the impact of dynamic objects on the global structure. Figure 1 While the traditional epipolar constraint can mitigate the impact of motion, in highly occluded environments or with excessively large rotation angles, it is difficult to accurately identify potential dynamic objects and determine their motion state, leading to deviations in pose estimation and consequently affecting the algorithm's localization accuracy. This algorithm utilizes the AETNET segmentation network and depth constraints to overcome the inherent limitations of traditional epipolar constraints, accurately determining the object's motion state. It not only better addresses the influence of dynamic feature points but also repairs the static background information of dynamic regions, increasing the number of keyframes used for dense point cloud mapping.

[0093] like Figure 13 As shown, to verify the feasibility of this method, a real-world SLAM experiment was conducted using a Husky wheeled robot. The platform's specifications were: i7-10875H CPU, 8 GB of RAM, GTX1080 GPU, and Ubuntu 18.04 operating system. A dataset was collected from a real-world scene. The experimental results are as follows: Figure 14 As shown, the ORB-SLAM3 algorithm suffers from poor trajectory accuracy due to its inability to identify feature points. This method, however, incorporates an improved segmentation network and accurately identifies dynamic feature points, thereby enhancing its trajectory accuracy.

[0094] Example 2.

[0095] Corresponding to Embodiment 1 of the present invention, Embodiment 2 of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps in accordance with the method of Embodiment 1.

[0096] Step S1: Acquire image information and use an image segmentation network to perform complete segmentation of the dynamic object.

[0097] Step S2: Analyze the motion state of the object using the object motion estimation method, and remove the precisely identified dynamic regions.

[0098] Step S3: Use the temporal bidirectional pixel point cloud background restoration model, utilizing adjacent front and back pixels... p The static information in the frame is used to repair the static keyframes of the background where the dynamic object is located, construct a complete point cloud map, and calculate the pose.

[0099] The aforementioned storage media include: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), optical discs, and other media capable of storing program code.

[0100] The specific limitations regarding the steps implemented after program execution in a computer-readable storage medium can be found in Embodiment 1, and will not be described in detail here.

[0101] Example 3.

[0102] Corresponding to Embodiment 1 of the present invention, Embodiment 3 of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, it performs the following steps in accordance with the method of Embodiment 1.

[0103] Step S1: Acquire image information and use an image segmentation network to perform complete segmentation of the dynamic object.

[0104] Step S2: Analyze the motion state of the object using the object motion estimation method, and remove the precisely identified dynamic regions.

[0105] Step S3: Use the temporal bidirectional pixel point cloud background restoration model, utilizing adjacent front and back pixels... p The static information in the frame is used to repair the static keyframes of the background where the dynamic object is located, construct a complete point cloud map, and calculate the pose.

[0106] The specific limitations regarding the implementation steps of the computer device mentioned above can be found in Embodiment 1, and will not be described in detail here.

[0107] It should be noted that each block in the block diagrams and / or flowcharts in the accompanying drawings of this invention, as well as combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified function or action, or by a combination of dedicated hardware and machine instructions.

[0108] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other occasions without modification, are all within the protection scope of the present invention.

Claims

1. A robot algorithm for depth constraints in dynamic scenes, characterized in that: Includes the following steps: Step S1: Acquire image information and use an image segmentation network to perform complete segmentation of the dynamic object; Step S2: Analyze the motion state of the object using the depth constraint method, and remove the precisely identified dynamic regions. Step S3: Use the temporal bidirectional pixel point cloud background restoration model, utilizing adjacent front and back pixels... p The static information in the frame is used to repair the static keyframes of the background where the dynamic object is located, construct a complete point cloud map, and calculate the pose. In step S2, the camera projection transformation mapping operator is used to obtain the pixel points, 3D spatial coordinates, and depth, and the spatial position of the object after the mapping transformation is analyzed; a system is established. The mapping function is used to calculate the coordinates of the projected image of the current frame; calculate the depth error between a series of consecutive frames; construct a motion score formula to describe the influence of depth error and epipolar error, compare the motion score with the corresponding threshold of static points, and identify the motion state of spatial points; after completing the dynamic judgment of feature points, the accurately identified dynamic regions are removed. The formula for calculating the fraction of motion is: ， in, For the score of the movement, epipolar error Its corresponding epipolar threshold Compare; Depth error Its corresponding depth error threshold Comparison; when the epipolar error Or depth error When the threshold is exceeded, the threshold for static points is: ,when > When the time is 0, it indicates that the spatial point is in motion; otherwise, it is at rest. The mapping function is: ,in, and These are the space vector parameters and the direction vector parameters. This is a high-dimensional projection transformation; This is represented as the inverse operation of a high-dimensional projection transformation. For the transition between two frames, The depth map is an indexed set of feature points; the depth of the projected points is described by taking the average depth within a 2x2 grid centered on the projected coordinates. In step S3, a temporal bidirectional pixel point cloud background restoration model is used. Taking the current frame to be restored as the time starting point, a time-weighted method is used, based on the previous time domain... p After frames and time domain p The frame performs background inpainting using a pixel grid model, mapping dynamic object regions onto the pixel grid. The image's color and depth information are projected onto the current frame based on feature point matching relationships. Different weights are assigned to regions occluded by dynamic objects between different frames to obtain the inpainted current frame. The calculation formula is: ， in, This is the repaired current frame. It is the current frame that has not been repaired. Indicates the time domain before the current frame p frame, Indicates the time domain of the current frame p frame, and Associative weights, images closer to the current frame have higher weights.

2. The robot algorithm for depth constraints in dynamic scenes according to claim 1, characterized in that: Depth error The formula for calculation is: ,in, The mean-finding function is based on the corresponding mapping function. Calculate the coordinates of the projected image of the current frame. Represented as a reference frame Depth map corresponding to pixels at coordinates. This is the depth map corresponding to the pixel points in the projected image coordinates of the current frame.

3. The robot algorithm for depth constraints in dynamic scenes according to claim 1, characterized in that: The image segmentation network incorporates a residual structure to enhance its performance, forming a residual-feature pyramid network. High-dimensional feature maps from the backbone are integrated into a two-dimensional collaborative attention module, which adaptively weights the pixels in dynamic object regions, enriching image feature information while enhancing the feature response of these regions. Low-dimensional feature maps from the backbone are integrated into a multi-scale hierarchical enhanced feature fusion module to enhance the weaker semantic information in these low-dimensional feature maps. The outputs of the two-dimensional collaborative attention module and the multi-scale hierarchical enhanced feature fusion module are combined to generate the image segmentation result.

4. The robot algorithm for depth constraints in dynamic scenes according to claim 3, characterized in that: In the multi-scale hierarchical enhanced feature fusion module, a encoder and decoder with several layers of symmetric structure are used. The encoder first segments the image into feature maps of various sizes, and the results are then processed by the Block layer. The Block layer consists of two parts connected in series: a multi-head spatial dimensionality reduction attention module and a hybrid feedforward neural network. Two branches are designed for each image size: one branch outputs to the next layer to perform image fusion, which is used for downsampling to reduce resolution; the other branch performs feature fusion, which is used to deepen the repair of high-level features and edge information. Next, the feature maps generated by the two branches of different layers are merged on a 1x1 dimensional channel convolution, and then processed by the decoder layer by layer. Subsequently, the restored image is linearly transformed using linear projection to generate the probability output for the corresponding category.

5. The robot algorithm for depth constraints in dynamic scenes according to claim 4, characterized in that: The algorithm for the multi-head spatial dimensionality reduction attention module is shown in the following equation: ， The algorithm for the hybrid feedforward neural network is shown in the following equation: ， in, Represents the normalization function. The function represents downsampling, where Q, K, and V represent the input query, key, and value, respectively. For single attention head feature dimensions, For negative perturbations of a fixed value; Indicates the input feature map, This indicates that layer normalization is performed on the feature map. Attention This indicates the processing of the multi-head spatial dimensionality reduction attention module. This represents two fully connected layers, with GeLU as the activation function. For activation function, This represents the output of the Block layer; the output of the Block layer. First, upsampling is performed, then the result is superimposed on the feature image output from the previous encoder and fused. After another upsampling process, the result is output to the decoder.

6. The robot algorithm for depth constraints in dynamic scenes according to claim 5, characterized in that: After the feature map is input into the two-dimensional collaborative attention module, global attention average pooling and max pooling operations are performed on the feature map. The corresponding features are then processed through multiple perceptual layers to strengthen the connection between channels and the weights of each channel are redistributed to obtain a channel attention map with weighted distribution. Subsequently, the channel attention map is input into the channel-preserving convolution kernel to obtain three horizontal and vertical features at different scales. These features are then fused element-wise to obtain the spatial fusion feature. Finally, the dimensional correlation of the spatial fusion features is adjusted by channel mixing to obtain the output; the output of the two-dimensional collaborative attention module is fused with the probability output of the channel attention map and the multi-scale hierarchical enhancement feature fusion module through element-wise accumulation, and the segmentation result of the image segmentation network is output after multi-layer upsampling.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of a robot algorithm for depth constraints in dynamic scenes as described in any one of claims 1-6.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, it implements the steps of a robot algorithm for depth constraints in dynamic scenes as described in any one of claims 1-6.

Images