Dynamic target removal method and apparatus

Abstract

The application relates to a dynamic target removal method, which comprises the following steps: inputting an original image into a semantic segmentation model to obtain initial masks corresponding to multiple targets in the original image; inputting the original image into a feature extractor to obtain multiple feature points; when the initial masks are missing part of dynamic targets, performing dynamic target missing compensation on the initial masks to obtain target masks containing complete dynamic targets; performing affine constraint on the multiple feature points and the target masks to obtain an initial image sequence from which the dynamic targets are removed; and performing mask boundary refinement on the initial image sequence to obtain a target image sequence. Through the technical scheme, the dynamic targets in the original image are removed, a target image sequence is obtained, a basis is provided for navigation and map construction of SLAM technology, the influence of the dynamic targets on camera pose estimation is reduced, wrong data caused by dynamic target shielding in a dynamic environment is avoided, and the accuracy of the SLAM technology is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically, to a method and apparatus for dynamic target removal. Background Technology

[0002] Visual SLAM (Simultaneous Localization and Mapping) is a technology that enables simultaneous localization and mapping in unknown environments using visual sensors. It is widely used in autonomous navigation, augmented reality, robotics, and virtual reality. Traditionally, navigation and mapping are separate processes, with navigation relying on a pre-built map, while mapping requires known location information. However, for many practical applications, such as drones, unmanned vehicles, and robots, the environment may be unknown or dynamically changing, necessitating real-time localization and mapping—hence the emergence of SLAM technology. Visual SLAM utilizes computer vision technology to process visual input data from a camera, simultaneously estimating its own position (localization) and building an environment map in unknown environments. Based on key technologies such as visual feature extraction, feature matching, pose estimation, and 3D reconstruction, it has several key advantages: 1. Real-time performance: Visual SLAM systems can process sensor data in real time, updating localization information and maps instantly. 2. No prior information required: Compared to traditional methods, visual SLAM systems can autonomously locate and map in unknown environments without relying on prior maps or GPS information. 3. Accuracy and Robustness: Visual SLAM systems can provide high-precision localization and mapping, while also possessing a certain degree of robustness. Even in situations with insufficient feature points or high sensor noise, they can maintain relatively accurate localization and map quality. 4. Applicability: Visual SLAM technology is suitable for different types of sensors and platforms, including cameras, LiDAR, and inertial measurement units. However, dynamic targets can affect navigation and map building. The impact of dynamic targets on camera pose estimation, or the occlusion of dynamic targets in dynamic environments causing erroneous data, can affect the accuracy of SLAM technology. Summary of the Invention

[0003] To address the aforementioned problems in the prior art, the present invention provides a dynamic target removal method and apparatus.

[0004] According to a first aspect of the present invention, a dynamic target removal method is provided, the method comprising:

[0005] The original image is input into a semantic segmentation model to obtain initial masks corresponding to multiple targets in the original image; wherein, the original image includes multiple frames; the multiple targets include dynamic targets and static targets;

[0006] The original image is input into a feature extractor to obtain multiple feature points;

[0007] When the initial mask is missing a portion of the dynamic target, dynamic target missing compensation is performed on the initial mask to obtain a target mask containing the complete dynamic target;

[0008] Affine constraints are applied to the plurality of feature points and the target mask to obtain an initial image sequence in which the dynamic target is removed;

[0009] The initial image sequence is thinned by mask boundary refinement to obtain the target image sequence.

[0010] Optionally, the step of inputting the original image into the semantic segmentation model to obtain an initial mask corresponding to multiple targets in the original image includes:

[0011] The original image is preprocessed to obtain the preprocessed original image;

[0012] The preprocessed original image is input into a pre-trained backbone feature extraction network to obtain a feature map corresponding to the preprocessed original image.

[0013] Multiple ROI candidate boxes are obtained based on the pixels in the feature map;

[0014] The multiple ROI candidate boxes are filtered using binary classification and bounding box regression to obtain multiple filtered ROI candidate boxes;

[0015] Based on the filtered ROI candidate boxes, after region of interest alignment, the binary classification and bounding box regression are used again to obtain the initial mask.

[0016] Optionally, when the initial mask is missing a portion of the dynamic target, performing dynamic target loss compensation on the initial mask to obtain a target mask containing the complete dynamic target includes:

[0017] When the number of dynamic targets in the (k-1)th frame, (k-2)th frame, (k+1)th frame, and (k+2)th frame of the initial mask is greater than the number of dynamic targets in the kth frame, it is determined that the kth frame is missing some dynamic targets.

[0018] Using the first transformation matrix between the (k-1)th frame image and the (k-2)th frame image, a second transformation matrix between the k-th frame image and the (k-1)th frame image is obtained;

[0019] The pose of the k-th frame image is obtained based on the second transformation matrix;

[0020] The second coordinate of the mask boundary point of the dynamic target in the (k-1)th frame image in the world coordinate system is calculated based on the first coordinate of the mask boundary point of the dynamic target in the camera coordinate system and the pose of the (k-1)th frame image.

[0021] The second coordinates are projected onto the k-th frame image based on the pose of the k-th frame image to obtain the third coordinates of the k-th frame image in the camera coordinate system, so as to perform dynamic target missing compensation on the initial mask and obtain a target mask containing complete dynamic targets.

[0022] Optionally, the second coordinate is obtained according to the following formula:

[0023]

[0024] in, P is the i-th mask boundary point of the dynamic target in the (k-1)-th frame image. i The second coordinate in the world coordinate system, T F,k-1 Let d represent the pose of the (k-1)th frame image. Depth measurement in the camera coordinate system, (c u ,c v (f) is the principal image point of the camera. u ,f v (u,v) is the focal point of the camera, and (u,v) is the mask boundary point P. i The first coordinate in the camera coordinate system.

[0025] Optionally, the third coordinate is obtained according to the following formula:

[0026]

[0027] in, P is the i-th mask boundary point of the dynamic target in the k-th frame image. i In the camera coordinate system, the third coordinate is K, which is the intrinsic parameter matrix of the camera, and T is... F,k Let be the pose of the k-th frame image.

[0028] Optionally, the step of applying affine constraints to the plurality of feature points and the target mask to obtain an initial image sequence with the dynamic target removed includes:

[0029] Calculate the pixel distance between the matching point and other feature points in the frame image where the matching point is located; wherein, the matching point is a pair of feature points with the same binary code in two different frame images;

[0030] Select at least one target feature point whose pixel distance is less than a distance threshold from the other feature points;

[0031] Based on the at least one target feature point and the preset conditions, the correct matching point is obtained;

[0032] Based on the correct matching points, remove the target mask containing the dynamic target to obtain an initial image sequence with the dynamic target removed.

[0033] Optionally, the pixel distance is obtained according to the following formula:

[0034] dist(j,l) = ||jl||2;

[0035] Where dist(j,l) is the pixel distance between j and l, j is a feature point in a pair of matching points, and l is another feature point in the frame image where one of the matching points j is located.

[0036] Optionally, the step of refining the mask boundaries of the initial image sequence to obtain the target image sequence includes:

[0037] Calculate the dynamic semantic segmentation probability of each feature point in the boundary region of the target mask;

[0038] Feature points whose dynamic probability of semantic segmentation is less than the probability threshold are determined to be static feature points;

[0039] Feature points whose semantic segmentation dynamic probability is greater than or equal to the probability threshold are determined to be dynamic feature points;

[0040] The dynamic feature points are removed to refine the mask boundaries of the initial image sequence, thereby obtaining the target image sequence.

[0041] Optionally, the dynamic probability of semantic segmentation is obtained according to the following formula:

[0042]

[0043] in, Feature point p representing the boundary region of the target mask i' Dynamic attributes, The feature point p represents i' b represents the probability of a dynamic feature point. t Let t represent the set of pixels in the boundary region of the target mask at time t, and let Zhao Lusi be the influencing factor.

[0044] According to a second aspect of the present invention, a dynamic target removal apparatus is provided, the apparatus comprising:

[0045] An initial mask acquisition module is used to input the original image into a semantic segmentation model to obtain initial masks corresponding to multiple targets in the original image; wherein, the original image includes multiple frames; the multiple targets include dynamic targets and static targets;

[0046] The feature point acquisition module is used to input the original image into the feature extractor to obtain multiple feature points;

[0047] The target mask acquisition module is used to compensate for the missing dynamic targets in the initial mask when the initial mask is missing a portion of the dynamic target, so as to obtain a target mask containing the complete dynamic target.

[0048] An affine constraint module is used to apply affine constraints to the plurality of feature points and the target mask to obtain an initial image sequence with the dynamic target removed.

[0049] The thinning module is used to thin the mask boundaries of the initial image sequence to obtain the target image sequence.

[0050] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0051] In the above technical solution, the original image is input into a semantic segmentation model to obtain an initial mask corresponding to multiple targets in the original image. The original image comprises multiple frames; each frame is the current frame image of multiple targets; these targets include both dynamic and static targets. The original image is input into a feature extractor to obtain multiple feature points. When the initial mask is missing some dynamic targets, dynamic target loss compensation is performed on the initial mask to obtain a target mask containing complete dynamic targets. Affine constraints are applied to the multiple feature points and the target mask to obtain an initial image sequence with dynamic targets removed. The initial image sequence is then refined by mask boundary refinement to obtain a target image sequence. This technical solution removes dynamic targets from the original image, obtaining a target image sequence, which provides a foundation for navigation and map building in SLAM technology. It reduces the impact of dynamic targets on camera pose estimation, avoids erroneous data caused by dynamic target occlusion in dynamic environments, and thus improves the accuracy of SLAM technology.

[0052] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0053] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the following detailed description to explain the invention, but do not constitute a limitation thereof. In the drawings:

[0054] Figure 1 This is a flowchart illustrating a dynamic target removal method according to an exemplary embodiment.

[0055] Figure 2 This is a schematic diagram illustrating dynamic target missing compensation according to an exemplary embodiment.

[0056] Figure 3 This is a schematic diagram illustrating a matching point according to an exemplary embodiment.

[0057] Figure 4 This is a block diagram illustrating a dynamic target removal device according to an exemplary embodiment. Detailed Implementation

[0058] Figure 1 This is a flowchart illustrating a dynamic target removal method according to an exemplary embodiment, such as... Figure 1 As shown, the method includes the following steps.

[0059] S101, input the original image into the semantic segmentation model to obtain the initial mask corresponding to multiple targets in the original image; wherein, the original image includes multiple frames of images; the multiple targets include dynamic targets and static targets.

[0060] Understandably, this semantic segmentation model can employ the Seaformer model. The original image contains both dynamic and static targets. Dynamic targets can negatively impact the localization and map building processes of SLAM, thus requiring the removal of dynamic targets while retaining static ones. Dynamic targets might be pedestrians or other objects, while static targets might be stationary vehicles or other objects. By inputting the original image into the semantic segmentation model, an initial mask corresponding to each target in the original image can be obtained, and the initial masks for dynamic targets can then be removed subsequently.

[0061] S102, input the original image into the feature extractor to obtain multiple feature points.

[0062] Understandably, the feature extractor can be an ORB feature extractor. Inputting the original image into the ORB feature extractor will yield multiple feature points that are rotation-invariant and scale-invariant. These features are widely distributed in the original image and have good repeatability. They can appear stably in different frames of the original image and can more comprehensively represent the content of the original image.

[0063] S103, when the initial mask is missing part of the dynamic target, the initial mask is compensated for the missing dynamic target to obtain a target mask containing the complete dynamic target.

[0064] It is understandable that the initial mask obtained by the semantic segmentation model may not completely contain all targets, and there may be missing targets or other issues. Therefore, when the initial mask is missing some dynamic targets, dynamic target missing compensation is performed on the initial mask to obtain a target mask containing complete dynamic targets, thus providing a basis for subsequent removal of dynamic targets.

[0065] S104, apply affine constraints to multiple feature points and target masks to obtain an initial image sequence with dynamic targets removed.

[0066] It is understandable that the influence of dynamic targets is eliminated by using multiple feature points of the original image obtained by the feature extractor in S102 and the target mask containing the complete dynamic target obtained in S103 to perform affine constraints, thereby obtaining an initial image sequence with dynamic targets removed.

[0067] S105, refine the mask boundary of the initial image sequence to obtain the target image sequence.

[0068] It is understandable that, since the initial image sequence for removing dynamic targets may have missing targets or other issues at the boundaries, it is necessary to refine the boundaries of the initial image sequence by masking and improving the segmentation of feature points at the boundaries after obtaining the initial image sequence, so as to obtain the target image sequence.

[0069] Optionally, S101 may include:

[0070] The original image is preprocessed to obtain the preprocessed original image;

[0071] The preprocessed original image is input into a pre-trained backbone feature extraction network to obtain the feature map corresponding to the preprocessed original image;

[0072] Multiple ROI candidate boxes are obtained based on the pixels in the feature map;

[0073] Multiple ROI candidate boxes are filtered using binary classification and bounding box regression to obtain multiple filtered ROI candidate boxes;

[0074] Based on the filtered ROI candidate boxes, after aligning the regions of interest, binary classification and bounding box regression are used again to obtain the initial mask.

[0075] Understandably, when preprocessing the original image, normalization and scaling can be used. The preprocessed original image is then input into the pre-trained backbone feature extraction network in the semantic segmentation model to obtain a feature map corresponding to the preprocessed original image. A region of interest (ROI) is defined for each point in the feature map, resulting in multiple ROI candidate boxes. These ROI candidate boxes are then filtered using binary classification and bounding box regression to obtain filtered ROI candidate boxes. Finally, ROI alignment is performed based on the filtered ROI candidate boxes, and binary classification and bounding box regression are used again to obtain the initial mask.

[0076] Optionally, S103 may include:

[0077] When the number of dynamic targets in the (k-1)th, (k-2)th, (k+1)th, and (k+2)th frames of the initial mask is greater than the number of dynamic targets in the kth frame, it is determined that the kth frame is missing some dynamic targets.

[0078] Using the first transformation matrix between the (k-1)th frame image and the (k-2)th frame image, we obtain the second transformation matrix between the kth frame image and the (k-1)th frame image.

[0079] The pose of the k-th frame image is obtained based on the second transformation matrix;

[0080] Calculate the second coordinates of the mask boundary point of the dynamic target in the (k-1)th frame in the world coordinate system based on the first coordinates of the mask boundary point of the dynamic target in the camera coordinate system and the pose of the (k-1)th frame image;

[0081] Based on the pose of the k-th frame image, the second coordinate is projected onto the k-th frame image to obtain the third coordinate of the k-th frame image in the camera coordinate system. This is used to perform dynamic target loss compensation on the initial mask, resulting in a target mask containing the complete dynamic target.

[0082] Understandably, before compensation, it's necessary to determine whether the frame image is missing moving targets. For example, if the number of moving targets in frames (k-1), (k-2), (k+1), and (k+2) is greater than the number of moving targets in frame k, then frame k is considered to be missing some moving targets. This invention uses the number of moving targets in the two consecutive frames immediately preceding and following frame k (i.e., the most recent four consecutive frames) to determine whether frame k is missing moving targets. More images can also be used for this determination; this invention does not impose any limitations on this.

[0083] Figure 2 This is a schematic diagram illustrating dynamic target missing compensation according to an exemplary embodiment, such as... Figure 2 As shown, the first transformation matrix between the (k-1)th frame image and the (k-2)th frame image is T. k-1,k-2 Based on the constant velocity model, the first transformation matrix T between the (k-1)th frame image and the (k-2)th frame image is used. k-1,k-2 To compensate for the second transformation matrix T between the k-th frame image and the (k+1)-th frame image k.k-1 That is, using the first transformation matrix T k-1,k-2 Approximately equal to the second transformation matrix T k.k-1 According to the second transformation matrix T k.k-1 The pose T of the k-th frame image is obtained. F,K =T k.k-1 T F,k-1 The second coordinates of the mask boundary points of the dynamic target in the (k-1)th frame in the world coordinate system are calculated based on the first coordinates of the mask boundary points of the dynamic target in the camera coordinate system and the pose of the (k-1)th frame. In one embodiment, the second coordinates are obtained according to the following formula:

[0084]

[0085] in, P is the i-th mask boundary point of the dynamic target in the (k-1)-th frame image. i The second coordinate in the world coordinate system, T F,k-1 Let d represent the pose of the (k-1)th frame image. Depth measurement in the camera coordinate system, (c u ,c v (f) is the principal image point of the camera. u ,f v (u,v) is the focal point of the camera, and (u,v) is the mask boundary point P. i The first coordinate in the camera coordinate system.

[0086] The second coordinates are projected onto the k-th frame image based on its pose to obtain the third coordinates of the k-th frame image in the camera coordinate system. These third coordinates are then used to compensate for dynamic target loss in the initial mask, resulting in a target mask containing the complete dynamic target. In one implementation, the third coordinates are obtained according to the following formula:

[0087]

[0088] in, Let P be the i-th mask boundary point of the dynamic target in the k-th frame image. i The third coordinate in the camera coordinate system, K is the camera's intrinsic parameter matrix, T F,k Let k be the pose of the k-th frame image;

[0089] Additionally, the camera's intrinsic parameter matrix When using a stereo camera, the depth measurement d of the stereo camera can be expressed as: Where f represents the camera's focal length, b is the distance between the two focal points of the stereo camera (i.e., the baseline), and L represents the parallax of the stereo camera. It is worth noting that when the camera depth is 40 times the baseline size, it is considered inaccurate and its contour points are not used.

[0090] Optionally, S104 may include:

[0091] Calculate the pixel distance between the matching point and other feature points in the frame image where the matching point is located; where the matching point is a pair of feature points with the same binary code in two different frame images;

[0092] Select at least one target feature point whose pixel distance is less than a distance threshold from other feature points;

[0093] Based on at least one target feature point and preset conditions, the correct matching point is obtained;

[0094] The target mask containing the dynamic target is removed based on the correct matching points, resulting in an initial image sequence with the dynamic target removed.

[0095] In one implementation, the pixel distance is obtained according to the following formula:

[0096] dist(j,l) = ||jl||2;

[0097] Where dist(j,l) is the pixel distance between j and l, j is a feature point in a pair of matching points, and l is another feature point in the frame image where one of the matching points j is located.

[0098] Select N target feature points from other feature points whose pixel distance is less than a distance threshold, where N = min(0.01N). sm N am ); where min() is the function to find the minimum value, N sm N is the total number of matching points in the two frames of images. am It is centered on the matching point, with a distance range of θ d The number of internal feature points. It's worth noting that if the matching point is the center, the number N of correctly matched points among the selected N target feature points... cm Satisfying N cm If the value is greater than (N-2) / 2, then the matching point is considered a correct matching point.

[0099] To improve the operational efficiency and detection quality of affine constraints, range constraints are used. The method for determining the selection of N target feature points can be referenced as follows: Specifically, for distances less than θ... d The correctness of the matching points is detected by using feature points within a specified range θ.d Feature points are limited to a range. Instead of selecting from the entire image's feature point set, feature points for matching are chosen from within this range. It's worth noting that when there are fewer than N feature points within the range, the range θ is limited. d All feature points within the range are used for affine constraints. Typically, θ can be used as the constraint. d Set to 400 pixels. Effective filtering and restriction of feature points can typically lead to a significant improvement in both efficiency and detection quality.

[0100] Because there are many feature points with the same binary code in two frames of images, i.e., matching points, some of these points may have the same binary code but are actually feature points on a dynamic target, i.e. incorrect matching points. Correspondingly, the correct matching points can be understood as feature points on a static target. Figure 3 This is a schematic diagram illustrating a matching point according to an exemplary embodiment, such as... Figure 3 As shown, m a and m b This represents a pair of matching points on two frames of images, where ma and mb are surrounded by selected θ. d N target feature points within the range, where green indicates points that correctly match in both frames and are related to m in both frames. a or m b There are identical positional relationships (i.e., green represents static targets), and red indicates incorrect matching points in both frames, and that the target is not related to m in either frame. a or m b There are different positional relationships (i.e., red represents dynamic targets). In this figure, N takes the value of 7. cm The formula calculates to be at least 3, while N in the figure... cm The value is 2, therefore the figure shows a pair of incorrect matching points.

[0101] After determining whether each feature point in the target mask is a correct or incorrect match, the corresponding mask can be used to determine whether it contains a dynamic target based on the ratio r0, where r = n. e / n a ; where n e n represents the number of feature points containing the dynamic target within the segmented target mask area. a This represents the total number of feature points within the segmented target mask area. Considering the possibility of errors, when r > 0.1, the segmented mask is considered dynamic. That is, if the number of feature points of a target mask containing a dynamic target is higher than 0.1, the target mask is determined to be the one that needs to be removed. Thus, the target mask containing the dynamic target is removed, resulting in an initial image sequence with the dynamic target removed.

[0102] Optionally, S105 may include:

[0103] Calculate the dynamic semantic segmentation probability of each feature point in the boundary region of the target mask;

[0104] Feature points whose dynamic probability in semantic segmentation is less than the probability threshold are classified as static feature points.

[0105] Feature points whose dynamic probability in semantic segmentation is greater than or equal to a probability threshold are defined as dynamic feature points.

[0106] Dynamic feature points are removed to refine the mask boundaries of the initial image sequence, resulting in the target image sequence.

[0107] In one implementation, the dynamic probability of semantic segmentation is obtained according to the following formula:

[0108]

[0109] in, Feature points p representing the boundary region of the target mask i' Dynamic attributes, Representing feature point p i' b represents the probability of a dynamic feature point. t Let represent the set of pixels in the boundary region of the target mask at time t, where α is the influence factor. Feature points in the target mask with a semantic segmentation dynamic probability less than 0.75 can be considered static feature points.

[0110] Understandably, the above steps refine the mask boundaries of the initial image sequence obtained in S104 after removing dynamic targets. A binomial logistic regression model is used to calculate the semantic segmentation dynamic probability of each feature point in the target mask containing dynamic targets in the image. Feature points with semantic segmentation dynamic probabilities greater than the probability threshold are further identified to obtain an image sequence of feature points after mask boundary refinement.

[0111] In one implementation, after obtaining the target image sequence, it needs to be applied to map optimization. The process can be as follows: input the static feature points in the fully processed target image sequence into the tracking and mapping thread, estimate the camera pose by frame-by-frame correlation, and optimize the global map to improve the accuracy of global map positioning. Select keyframe images from the image queue; add the selected keyframes to the keyframe buffer queue; recover the camera pose using epipolar geometry for each image retrieved from the keyframe buffer queue; add the retrieved keyframe buffer queue to the loop closure detection thread; use the head keyframe of the retrieved keyframe buffer queue as the current loop closure keyframe; calculate the maximum similarity between the current loop closure keyframe and the co-view keyframe; find loop closure candidate keyframes based on the maximum similarity; find a match between the loop closure candidate keyframe group formed by the loop closure candidate keyframes and the loop closure candidate keyframe group existing in the loop variable; maintain the loop variable so that the loop closure candidate keyframe group formed by the loop closure candidate keyframes is used as the loop closure candidate keyframe group before the next frame; optimize the solution of the moving points by minimizing the reprojection error, and obtain the rotation matrix and translation matrix. Use the obtained rotation matrix and translation matrix to optimize the pose of the global map.

[0112] Through the above technical solution, the present invention removes dynamic targets in the scene, which can effectively improve the accuracy and robustness of the visual SLAM system in localization and mapping in real high dynamic scenes, and greatly reduce bad and unstable data associations.

[0113] Figure 4 This is a block diagram illustrating a dynamic target removal device according to an exemplary embodiment, such as... Figure 4 As shown, the device 400 may include:

[0114] The initial mask acquisition module 401 is used to input the original image into the semantic segmentation model to obtain the initial mask corresponding to multiple targets in the original image; wherein, the original image includes multiple frames; each frame is the current frame image of multiple targets; the multiple targets include dynamic targets and static targets;

[0115] The feature point acquisition module 402 is used to input the original image into the feature extractor to obtain multiple feature points;

[0116] The target mask acquisition module 403 is used to compensate for the missing dynamic targets in the initial mask when the initial mask is missing a part of the dynamic target, so as to obtain a target mask containing the complete dynamic target.

[0117] Affine constraint module 404 is used to perform affine constraints on multiple feature points and target masks to obtain an initial image sequence with dynamic targets removed.

[0118] The thinning module 405 is used to thin the mask boundaries of the initial image sequence to obtain the target image sequence.

[0119] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0120] The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the specific details of the above embodiments. Within the scope of the technical concept of the present invention, various simple modifications can be made to the technical solution of the present invention, and these simple modifications all fall within the protection scope of the present invention.

[0121] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not describe the various possible combinations separately.

[0122] Furthermore, various different embodiments of the present invention can be combined in any way, as long as they do not violate the spirit of the present invention, they should also be regarded as the content disclosed by the present invention.

Claims

1. A dynamic target removal method, characterized in that, The method includes: The original image is input into a semantic segmentation model to obtain initial masks corresponding to multiple targets in the original image; wherein, the original image includes multiple frames; the multiple targets include dynamic targets and static targets; The original image is input into a feature extractor to obtain multiple feature points; When the initial mask is missing a portion of the dynamic target, dynamic target missing compensation is performed on the initial mask to obtain a target mask containing the complete dynamic target; Affine constraints are applied to the plurality of feature points and the target mask to obtain an initial image sequence in which the dynamic target is removed; The initial image sequence is thinned by mask boundary refinement to obtain the target image sequence; Wherein, when the initial mask is missing a portion of the dynamic target, dynamic target loss compensation is performed on the initial mask to obtain a target mask containing the complete dynamic target, including: When the initial mask is in the first Frame image, the first Frame image, the first Frame image and the first The number of moving targets in each frame image is greater than that in the first frame. When determining the number of moving targets in a frame image, the first... The frame image is missing a dynamic target; Using the first Frame image and the first The first transformation matrix between frames is used to obtain the first... Frame image and the first The second transformation matrix between frames; The second transformation matrix is ​​used to obtain the first... The pose of the frame image; According to the first The first coordinate of the mask boundary point of the dynamic target in the frame image in the camera coordinate system and the second coordinate of the third coordinate. The pose calculation of the first frame image The second coordinate of the mask boundary point of the dynamic target in the frame image in the world coordinate system; According to the first The pose of the frame image projects the second coordinate onto the first... In the frame image, the first... The third coordinate of the frame image in the camera coordinate system is used to perform dynamic target loss compensation on the initial mask to obtain a target mask containing the complete dynamic target; The second coordinate is obtained according to the following formula: ； in, For the first The first dynamic target in the frame image mask boundary points The second coordinate in the world coordinate system For the first The pose of a frame image. express Depth measurement in the camera coordinate system For the principal image point of the camera, It is the focal point of the camera. mask boundary points The first coordinate in the camera coordinate system; The third coordinate is obtained according to the following formula: ； in, For the first The first dynamic target in the frame image mask boundary points The third coordinate in the camera coordinate system Let be the intrinsic parameter matrix of the camera. For the first The pose of the frame image.

2. The dynamic target removal method according to claim 1, characterized in that, The step of inputting the original image into the semantic segmentation model to obtain an initial mask corresponding to multiple targets in the original image includes: The original image is preprocessed to obtain the preprocessed original image; The preprocessed original image is input into a pre-trained backbone feature extraction network to obtain a feature map corresponding to the preprocessed original image. Multiple ROI candidate boxes are obtained based on the pixels in the feature map; The multiple ROI candidate boxes are filtered using binary classification and bounding box regression to obtain multiple filtered ROI candidate boxes; Based on the filtered ROI candidate boxes, after region of interest alignment, the binary classification and bounding box regression are used again to obtain the initial mask.

3. The dynamic target removal method according to claim 1, characterized in that, The step of applying affine constraints to the plurality of feature points and the target mask to obtain an initial image sequence with the dynamic target removed includes: Calculate the pixel distance between the matching point and other feature points in the frame image where the matching point is located; wherein, the matching point is a pair of feature points with the same binary code in two different frame images; Select at least one target feature point whose pixel distance is less than a distance threshold from the other feature points; Based on the at least one target feature point and the preset conditions, the correct matching point is obtained; Based on the correct matching points, remove the target mask containing the dynamic target to obtain an initial image sequence with the dynamic target removed.

4. The dynamic target removal method according to claim 3, characterized in that, The pixel distance is obtained according to the following formula: ； in, yes and pixel distance between For a feature point in a pair of matching points, One of the matching points Other feature points in the same frame image.

5. The dynamic target removal method according to claim 1, characterized in that, The step of refining the mask boundaries of the initial image sequence to obtain the target image sequence includes: Calculate the dynamic semantic segmentation probability of each feature point in the boundary region of the target mask; Feature points whose dynamic probability of semantic segmentation is less than the probability threshold are determined to be static feature points; Feature points whose semantic segmentation dynamic probability is greater than or equal to the probability threshold are determined to be dynamic feature points; The dynamic feature points are removed to refine the mask boundaries of the initial image sequence, thereby obtaining the target image sequence.

6. The dynamic target removal method according to claim 5, characterized in that, The semantic segmentation dynamic probability is obtained according to the following formula: ； in, Feature points representing the boundary region of the target mask Dynamic attributes, Representing the feature point The probability of dynamic feature points. express The set of pixels in the boundary region of the target mask at any given time. It is the impact factor.

7. A dynamic target removal device, characterized in that, The device includes: An initial mask acquisition module is used to input the original image into a semantic segmentation model to obtain initial masks corresponding to multiple targets in the original image; wherein, the original image includes multiple frames; the multiple targets include dynamic targets and static targets; The feature point acquisition module is used to input the original image into the feature extractor to obtain multiple feature points; The target mask acquisition module is used to compensate for the missing dynamic targets in the initial mask when the initial mask is missing a portion of the dynamic target, so as to obtain a target mask containing the complete dynamic target. An affine constraint module is used to apply affine constraints to the plurality of feature points and the target mask to obtain an initial image sequence with the dynamic target removed. A thinning module is used to thin the mask boundaries of the initial image sequence to obtain the target image sequence; The target mask acquisition module is further configured to, when the initial mask contains the first... Frame image, the first Frame image, the first Frame image and the first The number of moving targets in each frame image is greater than that in the first frame. When determining the number of moving targets in a frame image, the first... The frame image is missing a dynamic target; Using the first Frame image and the first The first transformation matrix between frames is used to obtain the first... Frame image and the first The second transformation matrix between frames; The second transformation matrix is ​​used to obtain the first... The pose of the frame image; According to the first The first coordinate of the mask boundary point of the dynamic target in the frame image in the camera coordinate system and the second coordinate of the third coordinate. The pose calculation of the first frame image The second coordinate of the mask boundary point of the dynamic target in the frame image in the world coordinate system; According to the first The pose of the frame image projects the second coordinate onto the first... In the frame image, the first... The third coordinate of the frame image in the camera coordinate system is used to perform dynamic target loss compensation on the initial mask to obtain a target mask containing the complete dynamic target; The second coordinate is obtained according to the following formula: ； in, For the first The first dynamic target in the frame image mask boundary points The second coordinate in the world coordinate system For the first The pose of a frame image. express Depth measurement in the camera coordinate system For the principal image point of the camera, It is the focal point of the camera. mask boundary points The first coordinate in the camera coordinate system; The third coordinate is obtained according to the following formula: ； in, For the first The first dynamic target in the frame image mask boundary points The third coordinate in the camera coordinate system Let be the intrinsic parameter matrix of the camera. For the first The pose of the frame image.

Images