VSLAM method based on multi-target tracking network and instance segmentation network

Abstract

The application discloses a VSLAM method based on a multi-target tracking network and an instance segmentation network, and belongs to the technical field of visual simultaneous localization and mapping in a dynamic environment. The method comprises the following steps: if the current frame is a segmentation frame, inputting the current frame into an improved segmentation network to generate a mask of the current frame; if the current frame is a non-segmentation frame, generating the mask of the current frame through constant speed motion model fast forward mapping; inputting the mask of the current frame into the front end to generate a preliminary camera pose of the current frame; in response to the current frame satisfying a key frame generation condition, generating a key frame mask and inputting the key frame mask into the rear end; inputting the preliminary camera pose of the current frame into a multi-target tracking module to output a detection box after spatial compensation; inputting the mask of the current frame into the multi-target tracking module to construct a real motion mask of the current frame; and inputting the real motion mask of the current frame into the rear end to generate an optimized camera pose, which significantly improves the pose estimation accuracy and map robustness in a feature sparse scene.

Description

Technical Field

[0001] This invention belongs to the field of visual simultaneous localization and mapping technology in dynamic environments, specifically involving a VSLAM method based on multi-target tracking networks and instance segmentation networks. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a core technology in fields such as robot autonomous navigation and self-driving. Among them, vision-based SLAM systems are more popular among researchers due to their low cost. Classical VSLAM systems rely on the assumption of a "static environment," estimating camera pose and constructing an environment map by identifying environmental features. However, the presence of many dynamic objects in real-world applications often violates the "static environment assumption," leading to errors in the system's camera pose estimation. Therefore, the system needs to be updated to better suit real-world application scenarios.

[0003] With the demands of real-world road conditions and the development of deep learning technology, introducing deep learning into VSLAM systems has become mainstream. Dynamic Visual Simultaneous Localization and Mapping (Dynamic VSLAM) has emerged as a result. Currently, most existing algorithms identify dynamic regions through instance segmentation, achieving accurate mapping by eliminating the influence of potential moving regions on camera pose estimation. However, existing technologies have the following problems: (1) Existing technologies cannot distinguish between the ambiguity of objects being static and dynamic, resulting in a lack of map construction features; Existing high-precision dynamic VSLAM systems employ a "one-size-fits-all" elimination strategy for features in potential dynamic regions: as long as the instance segmentation network identifies an object as belonging to a pre-defined "potential dynamic category" such as "person" or "vehicle," visual feature points in that region are forcibly eliminated regardless of whether the object is currently stationary (e.g., a vehicle parked on the roadside). However, this elimination strategy often incorrectly discards a large number of high-value static background features, which can easily lead to tracking loss or trajectory drift in complex road conditions due to insufficient available feature points.

[0004] (2) The existing technology suffers from computational redundancy, insufficient real-time performance, and map pollution problems caused by the separation of multi-target tracking and VSLAM modules; Existing technologies typically stack instance segmentation, multi-object tracking, and VSLAM as independent modules, which has two drawbacks: First, traditional segmentation networks have a large number of parameters, making it impossible to generate masks for each frame of the video through network inference, which is difficult to meet real-time requirements; second, multi-object tracking networks and VSLAM systems calculate camera motion independently, resulting in a waste of computing power. Summary of the Invention

[0005] To address the aforementioned shortcomings in existing technologies, the VSLAM method based on multi-target tracking network and instance segmentation network provided by this invention solves the problems of existing technologies being unable to distinguish between static and dynamic objects, as well as the computational redundancy and insufficient real-time performance caused by the separation of multi-target tracking and VSLAM modules.

[0006] To achieve the aforementioned objectives, the technical solution adopted by this invention is: a VSLAM method based on a multi-target tracking network and an instance segmentation network, comprising the following steps: S1. Receive the image stream and divide the image frames into segmented frames and non-segmented frames. If the current frame is a segmented frame, input the current frame into the improved segmentation network to generate the mask of the current frame. If the current frame is a non-segmented frame, generate the mask of the current frame by quickly forward mapping through the constant velocity motion model. S2. Input the mask of the current frame into the VSLAM front end to generate the preliminary camera pose of the current frame. In response to the current frame meeting the key frame generation conditions, input the current frame into the improved segmentation network to generate the key frame mask, and input it into the VSLAM back end. S3. Input the preliminary camera pose of the current frame into the multi-target tracking module, use the camera pose of two consecutive frames to calculate the camera's self-space motion, combine the depth information to back-project the trajectory state vector to the three-dimensional space, and output the detection box after spatial compensation; if the current frame is a non-segmented frame, then propagate the historical mask to the mask of the current frame. S4. Input the mask of the current frame into the multi-target tracking module, identify real moving objects and briefly stationary objects based on the detection box, and construct the real motion mask of the current frame. S5. Input the real motion mask and preliminary camera pose of the current frame into the VSLAM backend for global optimization and mapping to generate the optimized camera pose.

[0007] Furthermore: In S1, the improved segmentation network includes a lightweight backbone network, a neck-decoupled CBAM attention module, and a SparseInst segmentation head connected in sequence; The lightweight backbone network takes the current frame as its input and adopts the ResNet-18 backbone network. It includes four residual extraction stages, and the four stages output feature maps with spatial resolutions of 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input image, respectively. The number of channels increases sequentially to 64, 128, 256 and 512. The outputs of the four stages are aligned with the multi-scale features c2, c3, c4 and c5 required by the Sparseinst native framework. Multi-scale features c2, c3, c4, and c5 are input into the neck decoupling CBAM attention module to obtain the output features of the neck decoupling CBAM attention module. The specific workflow of the neck decoupling CBAM attention module is as follows: Let the multi-scale features of the output of the lightweight backbone network serve as the input feature tensor of the neck-decoupled CBAM attention module. The corresponding dimension satisfies: number of channels ,high ,width , The number of channels in the output feature map. The height of the input feature tensor. The width of the input feature tensor. Calculate the channel attention map to determine the stride of the convolution operation. : In the formula, It is the sigmoid activation function. For multilayer perceptrons, For average pooling layer, This is a max pooling layer; Calculate the intermediate feature tensor after scaling by channel attention weights. : In the formula, For element-wise multiplication; Calculate the output feature tensor refined by the spatial attention module, and use it as the output feature of the neck decoupling CBAM attention module. The output feature tensor refined by the spatial attention module... The specific expression is: In the formula, Spatial attention module based on intermediate feature tensor Calculate the generated two-dimensional spatial attention weight map; The multi-scale features c2, c3, c4, and The multi-scale feature set is input into the instance context encoder to perform multi-scale feature fusion to fully capture the contextual information of targets at different scales. The fused multi-scale features are input into the Sparselnst segmentation head, which performs instance class prediction and instance mask generation operations and outputs the mask of the current frame.

[0008] Furthermore: In S1, the method for generating the mask of the current frame by rapidly forward mapping using a constant velocity motion model is as follows: A1. Access the historical pose buffer in memory and extract the optimized global camera pose from the previous frame. and the camera global pose of the first two frames Calculate the relative motion transformation matrix between adjacent historical frames. : In the formula, Indicates the index of the frame, where the first... The frame is the previous frame, where the first frame is... The frame refers to the first two frames; A2. Assuming the camera's speed and direction remain constant within a very short time interval between adjacent frames, a constant-speed motion model is constructed to predict the relative motion transformation matrix from the camera coordinate system of the previous frame to the camera coordinate system of the current frame. This allows us to derive the rotation matrix between two consecutive camera frames. Translation vector ; A3. Extract the final high-precision propagation mask set of the previous frame from memory. Local 3D propagation is performed on the pixels within the mask set to obtain the corresponding 3D spatial points; Wherein, for any pixel The coordinates of the corresponding three-dimensional point are obtained by back-projecting it onto the three-dimensional physical space in the camera coordinate system of the previous frame. : In the formula, Let x be the x-coordinate of the projection point. The ordinate of the projection point. The true physical depth of the projection point. This is the final high-precision propagation mask set for the previous frame. For pixels Horizontal coordinates, For pixels Vertical coordinates, Let x be the x-coordinate of the optical center. Let be the ordinate of the optical center. This refers to the focal length along the X-axis of the image physical plane, which is a camera intrinsic parameter. The focal length, defined in the camera's intrinsic parameters, is along the Y-axis of the image's physical plane. l For pixel index; A4. Perform a rigid body 3D transformation based on the 3D spatial points to obtain the intermediate 3D points in the current frame's camera coordinate system. : Using camera intrinsic matrix intermediate three-dimensional points Reprojecting the image back to the 2D image plane of the current frame using perspective, we calculate the prior mask pixels that only compensate for the camera's rigid body motion. Its expression is: In the formula, This represents the new depth value of the intermediate 3D point in the current camera coordinate system. For pixels Horizontal coordinates, For pixels Vertical coordinate; A5. Traverse all pixels in the mask set, obtain the prior mask pixels that were successfully projected, and generate the mask for the current frame.

[0009] Furthermore: In S2, the VSLAM front-end workflow is as follows: S21. Based on the mask of the current frame, perform soft weight reduction processing on feature points belonging to the potential dynamic region, extract features based on the high confidence background region, and generate a weighted feature point set. The set of feature points extracted in the current frame is: , For the first i Feature points are assigned confidence weights based on the mask of the current frame. Let be the total number of feature points, and be the th feature point. i Confidence weights of feature points The specific expression is: In the formula, The preset potential dynamic region weight decay coefficient, , The mask for the current frame; S22. Based on the weighted feature point set, the VSLAM front-end constructs a weighted projection error objective function to solve for the preliminary camera pose of the current frame. : In the formula, For the first i The coordinates of the feature point in the three-dimensional space of the world coordinate system. These are the two-dimensional observation coordinates of the point on the current frame image. Let be the projection function of the binocular camera. This is a robust kernel function used to further suppress outlier interference. To measure the covariance matrix, The global pose of the camera position in the current frame to be optimized. The minimum value corresponds to function; S23. After generating the initial camera pose of the current frame, in response to the current frame meeting the keyframe generation conditions, the current frame is used as a keyframe and segmented. The keyframe is input into the improved segmentation network to generate the keyframe mask.

[0010] Furthermore: In S3, the multi-target tracking module works as follows: S31. Camera global pose optimized based on the previous frame. and the initial camera pose of the current frame The relative three-dimensional rigid body motion is calculated using the homogeneous transformation matrix. Thus, the rotation matrix between two consecutive frames of the camera can be obtained. Translation vector ; S32. Suppose that the Kalman filter of the multi-target tracking module maintains the target's state vector in the previous frame. The specific expression is: In the formula, Let be the horizontal coordinates of the center pixel of the bounding box. The vertical coordinates of the center point of the bounding box (pixels). The width of the bounding box. The height of the bounding box. The pixel velocity of the bounding box center point in the horizontal direction of the image. The pixel velocity of the bounding box center point along the vertical axis of the image. The rate of change of the bounding box width. The rate of change of the bounding box height. It is the transpose symbol; Extract the pixel coordinates of the bounding box center point The corresponding binocular physical depth Combined with camera intrinsic parameter matrix Projecting it back onto three-dimensional space, the rotation matrix Translation vector By forcibly injecting the coordinates, a 3D predicted point after camera motion compensation is obtained. : In the formula, For three-dimensional prediction points x-coordinate For three-dimensional prediction points The ordinate, For three-dimensional prediction points The physical depth; S33, 3D prediction points based on camera motion compensation Calculate the pixel coordinates of the center point of the equivalent bounding box after physical alignment. : In the formula, The coordinates of the center pixel of the equivalent bounding box are: The vertical coordinates of the center point pixels of the equivalent bounding box. The pixel coordinates of the principal point of the camera intrinsics in the horizontal direction of the image. Here are the pixel coordinates of the principal point in the camera's intrinsic parameters along the vertical axis of the image. The equivalent focal length of the camera in the horizontal direction. The equivalent focal length of the camera in the vertical direction; Based on the depth variation rate of the equivalent bounding box center point in 3D space, the scale drift caused by the depth motion of the equivalent bounding box center point in 3D space is eliminated, and the width of the equivalent bounding box is calculated. and equivalent bounding box height : Based on rotation matrix Equivalent rotational component in the two-dimensional plane Correct the direction and magnitude of the pixel velocity vector, and calculate the pixel motion velocity of the center point of the equivalent bounding box in the horizontal direction of the image. The pixel motion velocity of the center point of the equivalent bounding box in the vertical direction of the image. : S34. Use the method in S33 to convert the state vector Mapped to compensation state vector The operation is encapsulated as a camera motion compensation function, the complete mathematical expression of which is: In the formula, This is a camera motion compensation function that includes operations for position compensation and scale compensation. Generate detection boxes, which include projection boxes from static theory and prediction boxes from temporal motion; Among them, the projection frame of static theory ; The specific method for generating prediction boxes for temporal motion is as follows: Using the Kalman native linear uniform motion state transition matrix and compensation state vector Calculate the final prior predicted state vector : In the formula, The horizontal coordinates of the center pixel of the predicted bounding box. The vertical coordinates of the center pixel of the predicted bounding box. The width of the prediction box, For the height of the predicted bounding box, To predict the pixel velocity of the center point of the bounding box in the horizontal direction of the image, To predict the pixel velocity of the center point of the bounding box along the vertical axis of the image, To predict the rate of change of the box width, To predict the rate of change of the bounding box height; Prediction boxes for temporal motion , where the prediction box It incorporates the target's own motion priors and is specifically designed for multi-target tracking modules to perform cross-frame data association and trajectory updates in the current frame.

[0011] Furthermore: In S3, the method for propagating the historical mask to the mask of the current frame is as follows: B1. Predicting the state vector from the final prior knowledge Extract the pixel coordinates of the center point of the prediction bounding box that contain the physical characteristics of the target's own motion. Calculate its pixel coordinates relative to the center point of the equivalent bounding box. Physical residuals between : In the formula, The horizontal residual of the center pixel. The vertical residual of the center pixel. The horizontal coordinates of the center pixel of the predicted bounding box. The vertical coordinates of the center pixel of the predicted bounding box; B2. Based on the physical assumption that the target object in the short time window is approximately a rigid body translation on the image plane, the physical residual is uniformly superimposed on the pixel coordinates of the prior mask pixels to update the mask pixels of the current frame. All pixels in the mask of the current frame are traversed to obtain the set of successfully updated pixels. Discrete hole filling and edge smoothing are performed through morphological operations to complete the propagation of the historical mask to the mask of the current frame. Specifically, the expression for updating the mask pixels of the current frame by uniformly superimposing the physical residuals with the pixel coordinates of the prior mask pixels is as follows: In the formula, The horizontal coordinates of the mask pixels in the current frame after the update. The vertical coordinates of the mask pixels in the current frame after the update.

[0012] Furthermore: S4 specifically refers to: If the current frame is a segmentation frame, then let the center pixel coordinates of the target bounding box actually observed by the improved segmentation network in the current frame image be... Based on the pixel coordinates of the center point of the equivalent bounding box Calculate the Euclidean distance on the two-dimensional plane to obtain the two-dimensional projection residual. : In the formula, The horizontal coordinates of the center pixel of the target bounding box. The vertical coordinate of the center pixel of the target bounding box; Combined with 3D prediction points physical depth With camera focal length Construct a depth-adaptive decision threshold : In the formula, The preset reference physical displacement tolerance threshold; Determine whether the preset threshold is met. If so, the object is determined to have undergone significant independent physical displacement, and is given the "real motion" status label. The mask region of the object in the mask of the current frame is retained and added to the set of real motion masks of the current frame. If not, the object is determined to be in a relatively static state, and a "briefly static" state label is assigned to it. The object is then removed from the mask of the current frame, and its mask area is released. If the current frame is a non-segmented frame, the state label assigned to the object in the previous segmented frame is obtained. For objects assigned the "real motion" state label, the mask region of the object in the mask of the current frame is retained and added to the set of real motion masks of the current frame. For objects assigned the "brief stillness" state label, the object is removed from the mask of the current frame and its mask region is released.

[0013] Furthermore: In S5, the VSLAM backend workflow is as follows: Determine whether the current frame meets the keyframe generation conditions; If not, the preliminary camera pose of the current frame will be used as the optimized camera pose of the current frame. If so, then based on the keyframe mask. and the true motion mask of the current frame Calculate the differential mask : Based on differential mask Perform weighted inverse compensation on the feature point set extracted from the current frame to calculate the confidence weights in the reconstructed backend map. : The feature point set after weighted inverse compensation is combined with the preliminary camera pose of the current frame. A back-end beam adjustment objective function is constructed and jointly optimized. After nonlinear optimization, the optimized camera pose for the current frame is obtained. With the updated 3D spatial point cloud coordinates.

[0014] The beneficial effects of this invention are as follows: (1) This invention introduces a constant-velocity motion model for multi-target tracking and a three-dimensional temporal prior (three-dimensional temporal geometric constraints) based on Kalman filtering. It then superimposes a secondary verification of the physical motion state on top of instance segmentation, achieving accurate discrimination of the "dynamic / static" attributes of objects. This ensures effective filtering of real dynamic interference while maximizing the reintegration of high-value static target features into the VSLAM backend optimization, significantly improving pose estimation accuracy and map robustness in feature-sparse scenarios. It solves the problem of existing technologies being unable to discern the ambiguity of object motion / staticity, leading to a lack of features for map construction.

[0015] (2) This invention proposes a serialized computing power reuse and elastic asynchronous computing strategy: on the one hand, it abandons the redundant appearance re-identification (ReID) module and the native camera motion compensation module (CMC) in the multi-target tracking network, and directly reuses the high-precision three-dimensional pose calculation results of VSLAM into the Kalman filter; on the other hand, it adopts an elastic inference strategy that combines fixed interval and key frame strong triggering, constructs a front-end mask coarse propagation based on a constant velocity motion model for non-segmented frames, and a back-end fine state inference propagation mechanism that combines camera pose and Kalman filter prediction (constructing a front-end lightweight coarse propagation) to prevent deadlock, and a back-end dual motion fine propagation to preserve the map propagation mechanism. This mechanism not only uses three-dimensional inverse projection to cancel the camera's self-motion, but also integrates the two-dimensional pixel offset of the Kalman filter to achieve high-precision physical propagation including the independent motion of the target. This dual optimization scheme not only significantly reduces the activation frequency of the neural network and the overall computing power load, realizing the efficient real-time operation of the system, but also blocks the "map pollution" problem caused by the mis-entry of dynamic features from the source through the key frame strong triggering mechanism. This addresses the problems of computational redundancy, insufficient real-time performance, and map pollution caused by the fragmentation of multi-target tracking and VSLAM modules in existing technologies. Attached Figure Description

[0016] Figure 1 This is a flowchart of the VSLAM method based on multi-target tracking network and instance segmentation network of the present invention.

[0017] Figure 2 This is a schematic diagram of the improved segmentation network of the present invention. Detailed Implementation

[0018] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0019] like Figure 1 As shown, in one embodiment of the present invention, the VSLAM method based on a multi-target tracking network and an instance segmentation network includes the following steps: S1. Receive the image stream and divide the image frames into segmented frames and non-segmented frames. If the current frame is a segmented frame, input the current frame into the improved segmentation network to generate the mask of the current frame. If the current frame is a non-segmented frame, generate the mask of the current frame by quickly forward mapping through the constant velocity motion model. S2. Input the mask of the current frame into the VSLAM front end to generate the preliminary camera pose of the current frame. In response to the current frame meeting the key frame generation conditions, input the current frame into the improved segmentation network to generate the key frame mask, and input it into the VSLAM back end. S3. Input the preliminary camera pose of the current frame into the multi-target tracking module, use the camera pose of two consecutive frames to calculate the camera's self-space motion, combine the depth information to back-project the trajectory state vector to the three-dimensional space, and output the detection box after spatial compensation; if the current frame is a non-segmented frame, then propagate the historical mask to the mask of the current frame. S4. Input the mask of the current frame into the multi-target tracking module, identify real moving objects and temporarily stationary objects based on the detection box, and construct the real motion mask of the current frame. S5. Input the real motion mask and preliminary camera pose of the current frame into the VSLAM backend for global optimization and mapping to generate the optimized camera pose.

[0020] In S1, the method for dividing the image frame into segmented frames and non-segmented frames is as follows: A global timer is constructed to determine whether the interval threshold set by the global timer has been reached. If it has, the frame is a segmented frame; otherwise, the frame is a non-segmented frame. Based on the segmentation method of the frame, the present invention adopts two differentiated processing strategies. In S1, the improved segmentation network consists of a lightweight backbone network, a neck-decoupled CBAM attention module, and a SparseInst segmentation head connected in sequence. The improved segmentation network is as follows: Figure 2 As shown; The lightweight backbone network takes the current frame as its input and adopts the ResNet-18 backbone network. It includes four residual extraction stages, and the four stages output feature maps with spatial resolutions of 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input image, respectively. The number of channels increases sequentially to 64, 128, 256 and 512. The outputs of the four stages are aligned with the multi-scale features c2, c3, c4 and c5 required by the native Sparseinst framework, thereby constructing a complete multi-scale feature flow from shallow high-resolution geometric details to deep low-resolution high-bit semantics. Each residual extraction stage consists of two BasicBlocks. The main calculation process in the BasicBlock is as follows: Calculate preliminary features : In the formula, It is a non-linear activation function. For batch normalization layer, This is a two-dimensional convolution operation with a kernel size of 3×3. For the input feature tensor of BasicBlock, This is the weight parameter matrix of the first 3×3 convolution layer. For the real number space, The number of channels in the output feature map. The height (spatial resolution) of the input feature tensor. The width (spatial resolution) of the input feature tensor. This represents the stride of the convolution operation; Calculate quadratic features : In the formula, This is the weight matrix for the second layer of 3×3 convolution; Calculate output features : In the formula, For the output tensor of the shortcut connection, when the output has the same spatial dimension and number of channels as the input ( =1), Identity mapping, when downsampling is required and the number of channels changes ( >1), typically a 1×1 convolution is needed for alignment, i.e.: In the formula, This is a two-dimensional convolution operation with a kernel size of 1×1. This is the weight parameter matrix for residual fusion.

[0021] To mitigate the potential decrease in semantic expressiveness due to the reduced depth and channel capacity resulting from backbone network replacement, a Convolutional Block Attention Module (CBAM) is embedded at the end of the backbone. This module employs a channel-spatial sequential attention structure to effectively suppress redundant noise features in complex and occluded backgrounds. Thanks to its modular "neck" design, CBAM is structurally decoupled from the ResNet-18 backbone, maximizing the integrity of the backbone and allowing direct loading of pre-trained weights from official large-scale datasets. This eliminates the need to train the backbone from scratch, accelerating convergence while effectively supplementing global contextual information.

[0022] Multi-scale features c2, c3, c4, and c5 are input into the neck decoupling CBAM attention module to obtain the output features of the neck decoupling CBAM attention module. The specific workflow of the neck decoupling CBAM attention module is as follows: Let the multi-scale features of the output of the lightweight backbone network serve as the input feature tensor of the neck-decoupled CBAM attention module. The corresponding dimension satisfies: number of channels ,high ,width Calculate the channel attention map : In the formula, It is the sigmoid activation function. For multilayer perceptrons, For average pooling layer, This is a max pooling layer; Calculate the intermediate feature tensor after scaling by channel attention weights. : In the formula, For element-wise multiplication; Calculate the output feature tensor refined by the spatial attention module, and use it as the output feature of the neck decoupling CBAM attention module. The output feature tensor refined by the spatial attention module... The specific expression is: In the formula, Spatial attention module based on intermediate feature tensor Calculate the generated two-dimensional spatial attention weight map; Thus, the final combined output of the backbone network and attention details (ResNet-18+CBAM) is not a single feature tensor, but rather the features from the original c5 are purified and replaced by CBAM. The multi-scale feature set consists of the input and the shallow multi-scale features c2, c3, and c4 that have not been processed by CBAM. .

[0023] The multi-scale features c2, c3, c4, and The multi-scale feature set is input into the instance context encoder to perform multi-scale feature fusion to fully capture the contextual information of targets at different scales. The fused multi-scale features are input into the Sparselnst segmentation head, which performs instance class prediction and instance mask generation operations and outputs the mask of the current frame.

[0024] In this embodiment, a morphological dilation operation is performed on the mask of the current frame. Given that the edges of dynamic objects (such as pedestrians and vehicles) are prone to depth discontinuities or motion blur during movement, directly using the original mask can easily lead to the omission of dynamic feature points. The mask dilation operation can effectively expand the safety boundary of potential dynamic regions, thereby significantly improving the stability and fault tolerance of the system in removing dynamic feature points during the initial operation phase. Subsequently, based on the dilated mask region, the system calculates and extracts its minimum bounding box. This bounding box is directly input into the backend feature matching and multi-object tracking network as standard format data, thus solving the data structure alignment problem between pixel-level masks and the input interface of traditional multi-object tracking networks, achieving high-density engineering coupling while ensuring feature-level decoupling.

[0025] In S1, the method for generating the mask of the current frame by rapidly forward mapping using a constant velocity motion model is as follows: A1. Access the historical pose buffer in memory and extract the optimized global camera pose from the previous frame. and the camera global pose of the first two frames Calculate the relative motion transformation matrix between adjacent historical frames. : In the formula, Indicates the index of the frame, where the first... The frame is the previous frame, where the first frame is... The frame refers to the first two frames, and the symbol is... Indicates from the first b Frame camera coordinate system mapped to the first a Rigid body coordinate transformation matrix of the frame camera coordinate system; A2. Assuming the camera's speed and direction remain constant within a very short time interval between adjacent frames, a constant-speed motion model is constructed to predict the relative motion transformation matrix from the camera coordinate system of the previous frame to the camera coordinate system of the current frame. This allows us to derive the rotation matrix between two consecutive camera frames. Translation vector ; In this embodiment, the constant velocity motion model assumes that the motion increment of the current frame is consistent with that of the previous frame, and extrapolates the historical relative motion transformation matrix as the prior motion prediction from the camera coordinate system of the previous frame to the camera coordinate system of the current frame.

[0026] A3. Extract the final high-precision propagation mask set of the previous frame from memory. Local 3D propagation is performed on the pixels within the mask set to obtain the corresponding 3D spatial points; Wherein, for any pixel The coordinates of the corresponding three-dimensional point are obtained by back-projecting it onto the three-dimensional physical space in the camera coordinate system of the previous frame. : In the formula, Let x be the x-coordinate of the projection point. The ordinate of the projection point. The true physical depth of the projection point. This is the final high-precision propagation mask set for the previous frame. For pixels Horizontal coordinates, For pixels Vertical coordinates, Let x be the x-coordinate of the optical center. Let be the ordinate of the optical center. This refers to the focal length along the X-axis of the image physical plane, which is a camera intrinsic parameter. The focal length, defined in the camera's intrinsic parameters, is along the Y-axis of the image's physical plane. l For pixel index; In this embodiment, to reduce the computational cost of propagation, the present invention performs local 3D propagation only on pixels within the historical mask region, rather than performing dense transformation on the entire frame image. This propagation stage introduces a static scene assumption, focusing on estimating pixel temporal drift caused by camera motion. Since the time interval between the unsegmented frame and the historical segmented frame is extremely short, the pixel drift of dynamic targets within the short time window is mainly dominated by the rigid body motion of the camera. Therefore, the approximate propagation based on the rigid body motion of the camera can still provide valuable prior constraints on the dynamic region for the current frame.

[0027] A4. Perform a rigid body 3D transformation based on the 3D spatial points to obtain the intermediate 3D points in the current frame's camera coordinate system. : Using camera intrinsic matrix intermediate three-dimensional points Reprojecting the image back to the 2D image plane of the current frame using perspective, we calculate the prior mask pixels that only compensate for the camera's own rigid body motion. Its expression is: In the formula, This represents the new depth value of the intermediate 3D point in the current camera coordinate system. For pixels Horizontal coordinates, For pixels Vertical coordinates; in this embodiment, depth values ​​are invalid for projections that exceed the current image boundary (e.g., For pixels with values ​​≤0 or that do not meet the effective imaging conditions, the system automatically discards their propagation results.

[0028] A5. Traverse all pixels in the mask set, obtain the prior mask pixels that were successfully projected, and generate the mask for the current frame.

[0029] To eliminate pixel holes and jagged edges caused by discrete pixel stretching or 3D inverse projection, the system performs morphological dilation and closing operations on the prior mask region to establish a safe dynamic edge tolerance boundary. Finally, the mask of the current frame is input to the VSLAM front end, so that before solving the pose of the current frame, the front end performs soft weight reduction on feature points falling within the prior mask region. Thus, without network inference, the real-time closed loop of the front end pose tracking thread is guaranteed with low latency and high robustness.

[0030] In S2, the VSLAM front-end workflow is as follows: S21. Based on the mask of the current frame, perform soft weight reduction processing on feature points belonging to the potential dynamic region, extract features based on the high confidence background region, and generate a weighted feature point set. The set of feature points extracted in the current frame is: , For the first i Feature points are assigned confidence weights based on the mask of the current frame. Let be the total number of feature points, and be the th feature point. i Confidence weights of feature points The specific expression is: In the formula, The preset potential dynamic region weight decay coefficient, , The formula shows that feature points in the static background are given full trust, while feature points falling into the potential dynamic mask are weighted less before participating in subsequent calculations.

[0031] S22. Based on the weighted feature point set, the VSLAM front-end constructs a weighted projection error objective function to solve for the preliminary camera pose of the current frame. : In the formula, For the first i The coordinates of the feature point in the three-dimensional space of the world coordinate system. These are the two-dimensional observation coordinates of the point on the current frame image. Let be the projection function of the binocular camera. This is a robust kernel function used to further suppress outlier interference. To measure the covariance matrix, The global pose of the camera position in the current frame to be optimized. The minimum value corresponds to function; S23. After generating the initial camera pose of the current frame, in response to the current frame meeting the key frame generation conditions, the current frame is used as a key frame and segmented. The key frame is input into the improved segmentation network to generate the key frame mask. In this embodiment, one specific method for generating a keyframe mask is as follows: After calculating the preliminary camera pose of the current frame, the VSLAM frontend executes a keyframe evaluation and asynchronous triggering mechanism: In response to the current frame meeting the keyframe generation conditions, a keyframe trigger request is sent to the global asynchronous task scheduler, and a unique frame sequence number matching the current keyframe is simultaneously packaged. Upon receiving the request, the task scheduler independently schedules the instance segmentation thread to execute the keyframe segmentation task. This architecture achieves memory decoupling through deep image copying, ensuring that the high-computing-power inference of the segmentation network never blocks the real-time pose tracking thread of the frontend. After the asynchronous segmentation task is completed, the generated keyframe mask is input to the VSLAM backend as delayed feedforward data.

[0032] In S3, the multi-target tracking module works as follows: S31. Camera global pose optimized based on the previous frame. and the initial camera pose of the current frame The relative three-dimensional rigid body motion is calculated using the homogeneous transformation matrix. Thus, the rotation matrix between two consecutive frames of the camera can be obtained. Translation vector ; In this embodiment, the extracted rotation matrix The size is 3×3, and the translation vector is... The dimensions are 3×1; S32. Suppose that the Kalman filter of the multi-target tracking module maintains the target's state vector in the previous frame. The specific expression is: In the formula, Let be the horizontal coordinates of the center pixel of the bounding box. The vertical coordinates of the center point of the bounding box (pixels). The width of the bounding box. The height of the bounding box. The pixel velocity of the bounding box center point in the horizontal direction of the image. The pixel velocity of the bounding box center point along the vertical axis of the image. The rate of change of the bounding box width. The rate of change of the bounding box height. It is the transpose symbol; Extract the pixel coordinates of the bounding box center point The corresponding binocular physical depth Combined with camera intrinsic parameter matrix The rotation matrix is ​​then back-projected into three-dimensional space to complete the 3D-2D full-state adaptive temporal compensation. Translation vector By forcibly injecting the coordinates, we obtain the 3D predicted points after camera motion compensation. : In the formula, For three-dimensional prediction points x-coordinate For three-dimensional prediction points The ordinate, For three-dimensional prediction points The physical depth; S33, 3D prediction points based on camera motion compensation Calculate the pixel coordinates of the center point of the equivalent bounding box after physical alignment. : In the formula, The coordinates of the center pixel of the equivalent bounding box are: The vertical coordinates of the center point pixels of the equivalent bounding box. The pixel coordinates of the principal point (or optical center) of the camera intrinsic parameters in the horizontal direction of the image. The pixel coordinates of the principal point (or optical center) in the camera's intrinsic parameters along the vertical axis of the image. The equivalent focal length of the camera in the horizontal direction (in pixels). The equivalent focal length of the camera in the vertical direction; Based on the depth variation rate of the equivalent bounding box center point in 3D space, the scale drift caused by the depth motion of the equivalent bounding box center point in 3D space is eliminated, and the width of the equivalent bounding box is calculated. and equivalent bounding box height : Based on rotation matrix Equivalent rotational component in the two-dimensional plane Correct the direction and magnitude of the pixel velocity vector, and calculate the pixel motion velocity of the center point of the equivalent bounding box in the horizontal direction of the image. And the pixel motion velocity of the center point of the equivalent bounding box in the vertical direction of the image. : S34. Use the method in S33 to convert the state vector Mapped to compensation state vector The operation is encapsulated as a camera motion compensation function, the complete mathematical expression of which is: In the formula, This is a camera motion compensation function that includes position and scale compensation calculations. Since the rate of change of width and height at the bounding box center point has a relatively small impact on perspective caused by the camera's own motion, the rate of change of width and height is relatively small. and The original values ​​are preserved to maintain the alignment of the Kalman filter dimensions and the stability of the operation.

[0033] Generate detection boxes, which include projection boxes from static theory and prediction boxes from temporal motion; Among them, the projection frame of static theory The static theoretical projection frame strictly does not incorporate the target's own motion state; it only represents the theoretical static position after eliminating the camera's own motion. It is directly output to step S4 as a benchmark reference frame for "motion-static ambiguity determination." The residual of the frame is compared with that of the actual detection frame to determine whether the target is actually moving.

[0034] The specific method for generating prediction boxes for temporal motion is as follows: Using the Kalman native linear uniform motion state transition matrix and compensation state vector Calculate the final prior predicted state vector : In the formula, The horizontal coordinates of the center pixel of the predicted bounding box. The vertical coordinates of the center pixel of the predicted bounding box. The width of the prediction box, For the height of the predicted bounding box, To predict the pixel velocity of the center point of the bounding box in the horizontal direction of the image, To predict the pixel velocity of the center point of the bounding box along the vertical axis of the image, To predict the rate of change of the box width, To predict the rate of change of the bounding box height; Prediction boxes for temporal motion , where the prediction box It incorporates the target's own motion priors and is specifically designed for multi-target tracking modules to perform cross-frame data association and trajectory updates in the current frame.

[0035] In S3, when the current frame is a non-segmented frame, since S1 did not output an accurate mask, the multi-target tracking module will take over the mask generation task. It will reuse the internally calculated camera pose rigid body changes and the target independent two-dimensional motion residuals extracted by Kalman filtering, perform double motion alignment on the mask of the previous frame, and propagate the historical mask to the current frame to make up for the mask missing problem in non-segmented frames. The specific method for propagating the historical mask to the mask of the current frame is as follows: B1. Predicting the state vector from the final prior knowledge Extract the pixel coordinates of the center point of the prediction bounding box that contain the physical characteristics of the target's own motion. Calculate its pixel coordinates relative to the center point of the equivalent bounding box. Physical residuals between : In the formula, The horizontal residual of the center pixel. The vertical residual of the center pixel. The horizontal coordinates of the center pixel of the predicted bounding box. The vertical coordinates of the center pixel of the predicted bounding box, and the physical residual. This represents the pure two-dimensional pixel offset of the target object due to its own movement.

[0036] B2. Based on the physical assumption that the target object in the short time window is approximately a rigid body translation on the image plane, the physical residual is uniformly superimposed on the pixel coordinates of the prior mask pixels to update the mask pixels of the current frame. All pixels in the mask of the current frame are traversed to obtain the set of successfully updated pixels. Discrete hole filling and edge smoothing are performed through morphological operations to complete the propagation of the historical mask to the mask of the current frame. Specifically, the expression for updating the mask pixels of the current frame by uniformly superimposing the physical residuals with the pixel coordinates of the prior mask pixels is as follows: In the formula, The horizontal coordinates of the mask pixels in the current frame after the update. To update the vertical coordinates of the mask pixels in the current frame, this embodiment superimposes the physical residual onto the prior mask pixels to achieve dual alignment of "counteracting camera motion + fitting target motion".

[0037] S4 specifically refers to: If the current frame is a segmentation frame, then let the center pixel coordinates of the target bounding box actually observed by the improved segmentation network in the current frame image be... Based on the pixel coordinates of the center point of the equivalent bounding box Calculate the Euclidean distance on the two-dimensional plane to obtain the two-dimensional projection residual. : In the formula, The horizontal coordinates of the center pixel of the target bounding box. The vertical coordinate of the center pixel of the target bounding box; Based on the principle of perspective projection (pinhole imaging model), the pixel offset of the image plane corresponding to the same actual physical displacement is naturally smaller in distant regions. Using a fixed pixel threshold can easily lead to misjudgment of distant dynamic targets. This invention combines three-dimensional prediction points... physical depth With camera focal length Construct a depth-adaptive decision threshold : In the formula, This is the preset reference physical displacement tolerance threshold (representing the lower limit of the reference displacement error that the system can tolerate in three-dimensional space). In this embodiment, for segmented frames: The actual visual observation box and the theoretical predicted box are received. By calculating the spatial residual between the two in the two-dimensional plane and combining it with a depth adaptive threshold, the object in actual motion is accurately determined. It is then determined whether a preset threshold is met. If so, the object is determined to have undergone significant independent physical displacement, and is given the "real motion" status label. The mask region of the object in the mask of the current frame is retained and added to the set of real motion masks of the current frame. If not, the object is determined to be in a relatively static state, and a "briefly static" state label is assigned to it. The object is then removed from the mask of the current frame, and its mask area is released. If the current frame is a non-segmented frame, the state label assigned to the object in the previous segmented frame is obtained. For objects assigned the "real motion" state label, the mask region of the object in the mask of the current frame is retained and added to the set of real motion masks of the current frame. For objects assigned the "brief stillness" state label, the object is removed from the mask of the current frame and its mask region is released. In this embodiment, for non-segmented frames, the multi-target tracking module skips residual calculation and directly inherits the motion verification label of the target object that has been confirmed in the previous key segmentation frame. Finally, the mask is purified into a real motion mask (i.e., retaining the real moving objects and eliminating potential dynamic objects that remain stationary).

[0038] In S5, the VSLAM backend workflow is as follows: Determine whether the current frame meets the keyframe generation conditions; If not, the preliminary camera pose of the current frame will be used as the optimized camera pose of the current frame. In this embodiment, when the current frame is not a key frame, the real motion mask of the current frame calculated by S4 is used as a bypass output, specifically for maintaining the temporal continuity of the target trajectory in the downstream multi-target tracking module, and is used as historical dynamic prior feedback to the front-end tracking thread of the next frame.

[0039] If so, then based on the keyframe mask. and the true motion mask of the current frame Calculate the differential mask : Based on differential mask Perform weighted inverse compensation on the feature point set extracted from the current frame to calculate the confidence weights in the reconstructed backend map. : The above equation shows that the VSLAM backend performs a strict independent splitting process on the two types of input masks: For the input true motion mask (Perform dynamic blocking): Treat it as a definite source of motion interference. Any feature point falling into this masked area has its mapping weight forcibly reduced to 0. This completely blocks it from entering the backend optimization and local mapping stages, fundamentally avoiding map pollution.

[0040] Keyframe mask for input This serves as the physical benchmark for defining the "safe background" and the "suspected area." Feature points not falling within this area are directly assumed to be an absolutely static background. For feature points that fall into this area but are cleared of any real motion suspicion through differential calculation, their weights are reverse-compensated and pulled back to a state of complete trust. ), and then reintegrated into the mapping constraints.

[0041] The feature point set after weighted inverse compensation is combined with the preliminary camera pose of the current frame. A back-end beam adjustment objective function is constructed and jointly optimized. After nonlinear optimization, the optimized camera pose for the current frame is obtained. With the updated 3D spatial point cloud coordinates.

[0042] In this embodiment, the global pose is output as the final camera trajectory of the current frame and simultaneously written to the historical pose buffer in memory. This completes the full processing cycle for the current frame. In the upcoming next processing loop, this buffer... This will be formally converted into the "historical reference pose of the previous frame". The output is seamlessly fed into the multi-target tracking module for use in camera rigid body motion compensation.

[0043] In the description of this invention, the above are merely preferred embodiments and are not intended to limit the scope of protection of this invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A VSLAM method based on multi-object tracking network and instance segmentation network, characterized in that, Includes the following steps: S1. Receive the image stream and divide the image frames into segmented frames and non-segmented frames. If the current frame is a segmented frame, input the current frame into the improved segmentation network to generate the mask of the current frame. If the current frame is a non-segmented frame, generate the mask of the current frame by quickly forward mapping through the constant velocity motion model. S2. Input the mask of the current frame into the VSLAM front end to generate the preliminary camera pose of the current frame. In response to the current frame meeting the key frame generation conditions, input the current frame into the improved segmentation network to generate the key frame mask, and input it into the VSLAM back end. S3. Input the preliminary camera pose of the current frame into the multi-target tracking module, use the camera pose of two consecutive frames to calculate the camera's self-space motion, combine the depth information to back-project the trajectory state vector to the three-dimensional space, and output the detection box after spatial compensation; if the current frame is a non-segmented frame, then propagate the historical mask to the mask of the current frame. S4. Input the mask of the current frame into the multi-target tracking module, identify real moving objects and briefly stationary objects based on the detection box, and construct the real motion mask of the current frame. S5. Input the real motion mask and preliminary camera pose of the current frame into the VSLAM backend for global optimization and mapping to generate the optimized camera pose.

2. The VSLAM method based on a multi-target tracking network and an instance segmentation network according to claim 1, characterized in that, In S1, the improved segmentation network consists of a lightweight backbone network, a neck-decoupled CBAM attention module, and a SparseInst segmentation head connected in sequence. The lightweight backbone network takes the current frame as its input and adopts the ResNet-18 backbone network. It includes four residual extraction stages, and the four stages output feature maps with spatial resolutions of 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input image, respectively. The number of channels increases sequentially to 64, 128, 256 and 512. The outputs of the four stages are aligned with the multi-scale features c2, c3, c4 and c5 required by the Sparseinst native framework. Multi-scale features c2, c3, c4, and c5 are input into the neck decoupling CBAM attention module to obtain the output features of the neck decoupling CBAM attention module. The specific workflow of the neck decoupling CBAM attention module is as follows: Let the multi-scale features of the output of the lightweight backbone network serve as the input feature tensor of the neck-decoupled CBAM attention module. The corresponding dimension satisfies: number of channels ,high ,width , The number of channels in the output feature map. The height of the input feature tensor. The width of the input feature tensor. Calculate the channel attention map to determine the stride of the convolution operation. : In the formula, It is the sigmoid activation function. For multilayer perceptrons, For average pooling layer, This is the max pooling layer; Calculate the intermediate feature tensor after scaling by channel attention weights. : In the formula, For element-wise multiplication; Calculate the output feature tensor refined by the spatial attention module, and use it as the output feature of the neck decoupling CBAM attention module. The output feature tensor refined by the spatial attention module... The specific expression is: In the formula, Spatial attention module based on intermediate feature tensor Calculate the generated two-dimensional spatial attention weight map; The multi-scale features c2, c3, c4, and The multi-scale feature set is input into the instance context encoder to perform multi-scale feature fusion to fully capture the contextual information of targets at different scales. The fused multi-scale features are input into the Sparselnst segmentation head, which performs instance class prediction and instance mask generation operations and outputs the mask of the current frame.

3. The VSLAM method based on a multi-target tracking network and an instance segmentation network according to claim 2, characterized in that, In S1, the method for generating the mask of the current frame by rapidly forward mapping using a constant velocity motion model is as follows: A1. Access the historical pose buffer in memory and extract the optimized global camera pose from the previous frame. and the camera global pose of the first two frames Calculate the relative motion transformation matrix between adjacent historical frames. : In the formula, Indicates the index of the frame, where the first... The frame is the previous frame, where the first frame is... The frame refers to the first two frames; A2. Assuming the camera's speed and direction remain constant within a very short time interval between adjacent frames, a constant-speed motion model is constructed to predict the relative motion transformation matrix from the camera coordinate system of the previous frame to the camera coordinate system of the current frame. This allows us to derive the rotation matrix between two consecutive camera frames. Translation vector ; A3. Extract the final high-precision propagation mask set of the previous frame from memory. Local 3D propagation is performed on the pixels within the mask set to obtain the corresponding 3D spatial points; Wherein, for any pixel The coordinates of the corresponding three-dimensional point are obtained by back-projecting it onto the three-dimensional physical space in the camera coordinate system of the previous frame. : In the formula, Let x be the x-coordinate of the projection point. The ordinate of the projection point. The true physical depth of the projection point. This is the final high-precision propagation mask set for the previous frame. For pixels Horizontal coordinates, For pixels Vertical coordinates, Let x be the x-coordinate of the optical center. Let be the ordinate of the optical center. This refers to the focal length along the X-axis of the image physical plane, which is a camera intrinsic parameter. The focal length, defined in the camera's intrinsic parameters, is along the Y-axis of the image's physical plane. l For pixel index; A4. Perform a rigid body 3D transformation based on the 3D spatial points to obtain the intermediate 3D points in the current frame's camera coordinate system. : Using camera intrinsic matrix intermediate three-dimensional points Reprojecting the image back to the 2D image plane of the current frame using perspective, we calculate the prior mask pixels that only compensate for the camera's rigid body motion. Its expression is: In the formula, This represents the new depth value of the intermediate 3D point in the current camera coordinate system. For pixels Horizontal coordinates, For pixels Vertical coordinate; A5. Traverse all pixels in the mask set, obtain the prior mask pixels that were successfully projected, and generate the mask for the current frame.

4. The VSLAM method based on a multi-target tracking network and an instance segmentation network according to claim 1, characterized in that, In S2, the VSLAM front-end workflow is as follows: S21. Based on the mask of the current frame, perform soft weight reduction processing on feature points belonging to the potential dynamic region, extract features based on the high confidence background region, and generate a weighted feature point set. The set of feature points extracted in the current frame is: , For the first i Feature points are assigned confidence weights based on the mask of the current frame. Let be the total number of feature points, and be the th feature point. i Confidence weights of feature points The specific expression is: In the formula, The preset potential dynamic region weight decay coefficient, , The mask for the current frame; S22. Based on the weighted feature point set, the VSLAM front-end constructs a weighted projection error objective function to solve for the preliminary camera pose of the current frame. : In the formula, For the first i The coordinates of the feature point in the three-dimensional space of the world coordinate system. These are the two-dimensional observation coordinates of the point on the current frame image. Let be the projection function of the binocular camera. This is a robust kernel function used to further suppress outlier interference. To measure the covariance matrix, The global pose of the camera position in the current frame to be optimized. The minimum value corresponds to function; S23. After generating the initial camera pose of the current frame, in response to the current frame meeting the keyframe generation conditions, the current frame is used as a keyframe and segmented. The keyframe is input into the improved segmentation network to generate the keyframe mask.

5. The VSLAM method based on a multi-target tracking network and an instance segmentation network according to claim 1, characterized in that, In S3, the multi-target tracking module works as follows: S31. Camera global pose optimized based on the previous frame. and the initial camera pose of the current frame The relative three-dimensional rigid body motion is calculated using the homogeneous transformation matrix. Thus, the rotation matrix between two consecutive frames of the camera can be obtained. Translation vector ; S32. Suppose that the Kalman filter of the multi-target tracking module maintains the target's state vector in the previous frame. The specific expression is: In the formula, Let be the horizontal coordinates of the center pixel of the bounding box. The vertical coordinates of the center point of the bounding box (pixels). The width of the bounding box. The height of the bounding box. The pixel velocity of the bounding box center point in the horizontal direction of the image. The pixel velocity of the bounding box center point along the vertical axis of the image. The rate of change of the bounding box width. The rate of change of the bounding box height. It is the transpose symbol; Extract the pixel coordinates of the bounding box center point The corresponding binocular physical depth Combined with camera intrinsic parameter matrix Projecting it back onto three-dimensional space, the rotation matrix Translation vector By forcibly injecting the coordinates, a 3D predicted point after camera motion compensation is obtained. : In the formula, For three-dimensional prediction points x-coordinate For three-dimensional prediction points The ordinate, For three-dimensional prediction points The physical depth; S33, 3D prediction points based on camera motion compensation Calculate the pixel coordinates of the center point of the equivalent bounding box after physical alignment. : In the formula, The coordinates of the center pixel of the equivalent bounding box are: The vertical coordinates of the center point pixels of the equivalent bounding box. The pixel coordinates of the principal point of the camera intrinsics in the horizontal direction of the image. Here are the pixel coordinates of the principal point in the camera's intrinsic parameters along the vertical axis of the image. The equivalent focal length of the camera in the horizontal direction. The equivalent focal length of the camera in the vertical direction; Based on the depth variation rate of the equivalent bounding box center point in 3D space, the scale drift caused by the depth motion of the equivalent bounding box center point in 3D space is eliminated, and the width of the equivalent bounding box is calculated. and equivalent bounding box height : Based on rotation matrix Equivalent rotational component in the two-dimensional plane Correct the direction and magnitude of the pixel velocity vector, and calculate the pixel motion velocity of the center point of the equivalent bounding box in the horizontal direction of the image. The pixel motion velocity of the center point of the equivalent bounding box in the vertical direction of the image. : S34. Use the method in S33 to convert the state vector Mapped to compensation state vector The operation is encapsulated as a camera motion compensation function, the complete mathematical expression of which is: In the formula, This is a camera motion compensation function that includes operations for position compensation and scale compensation. Generate detection boxes, which include projection boxes from static theory and prediction boxes from temporal motion; Among them, the projection frame of static theory ; The specific method for generating prediction boxes for temporal motion is as follows: Using the Kalman native linear uniform motion state transition matrix and compensation state vector Calculate the final prior predicted state vector : In the formula, The horizontal coordinates of the center pixel of the predicted bounding box. The vertical coordinates of the center pixel of the predicted bounding box. The width of the prediction box, For the height of the predicted bounding box, To predict the pixel velocity of the center point of the bounding box in the horizontal direction of the image, To predict the pixel velocity of the center point of the bounding box along the vertical axis of the image, To predict the rate of change of the box width, To predict the rate of change of the box height; Prediction boxes for temporal motion , where the prediction box It incorporates the target's own motion priors and is specifically designed for multi-target tracking modules to perform cross-frame data association and trajectory updates in the current frame.

6. The VSLAM method based on a multi-target tracking network and an instance segmentation network according to claim 5, characterized in that, In S3, the method for propagating the historical mask to the mask of the current frame is as follows: B1. Predicting the state vector from the final prior knowledge Extract the pixel coordinates of the center point of the prediction bounding box that contain the physical characteristics of the target's own motion. Calculate its pixel coordinates relative to the center point of the equivalent bounding box. Physical residuals between : In the formula, The horizontal residual of the center pixel. The vertical residual of the center pixel. The horizontal coordinates of the center pixel of the predicted bounding box. The vertical coordinates of the center pixel of the predicted bounding box; B2. Based on the physical assumption that the target object in the short time window is approximately a rigid body translation on the image plane, the physical residual is uniformly superimposed on the pixel coordinates of the prior mask pixels to update the mask pixels of the current frame. All pixels in the mask of the current frame are traversed to obtain the set of successfully updated pixels. Discrete hole filling and edge smoothing are performed through morphological operations to complete the propagation of the historical mask to the mask of the current frame. Specifically, the expression for updating the mask pixels of the current frame by uniformly superimposing the physical residuals with the pixel coordinates of the prior mask pixels is as follows: In the formula, The horizontal coordinates of the mask pixels in the current frame after the update. The vertical coordinates of the mask pixels in the current frame after the update.

7. The VSLAM method based on a multi-target tracking network and an instance segmentation network according to claim 6, characterized in that, S4 specifically refers to: If the current frame is a segmentation frame, then let the center pixel coordinates of the target bounding box actually observed by the improved segmentation network in the current frame image be... Based on the pixel coordinates of the center point of the equivalent bounding box Calculate the Euclidean distance on the two-dimensional plane to obtain the two-dimensional projection residual. : In the formula, The horizontal coordinates of the center pixel of the target bounding box. The vertical coordinate of the center pixel of the target bounding box; Combined with 3D prediction points physical depth With camera focal length Construct a depth-adaptive decision threshold : In the formula, The preset reference physical displacement tolerance threshold; Determine whether the preset threshold is met. If so, the object is determined to have undergone significant independent physical displacement, and is given the "real motion" status label. The mask region of the object in the mask of the current frame is retained and added to the set of real motion masks of the current frame. If not, the object is determined to be in a relatively static state, and it is given a "briefly static" state label. The object is then removed from the mask of the current frame, and its mask area is released. If the current frame is a non-segmented frame, the state label assigned to the object in the previous segmented frame is obtained. For objects assigned the "real motion" state label, the mask region of the object in the mask of the current frame is retained and added to the set of real motion masks of the current frame. For objects assigned the "brief stillness" state label, the object is removed from the mask of the current frame and its mask region is released.

8. The VSLAM method based on a multi-target tracking network and an instance segmentation network according to claim 7, characterized in that, In S5, the VSLAM backend workflow is as follows: Determine whether the current frame meets the keyframe generation conditions; If not, the preliminary camera pose of the current frame will be used as the optimized camera pose of the current frame. If so, then based on the keyframe mask. and the true motion mask of the current frame Calculate the differential mask : Based on differential mask Perform weighted inverse compensation on the feature point set extracted from the current frame to calculate the confidence weights in the reconstructed backend map. : The feature point set after weighted inverse compensation is combined with the preliminary camera pose of the current frame. A back-end beam adjustment objective function is constructed and jointly optimized. After nonlinear optimization, the optimized camera pose for the current frame is obtained. With the updated 3D spatial point cloud coordinates.

Images