A 4D target segmentation method based on a point cloud space-time memory network

Abstract

The application relates to the field of computer vision and discloses a 4D target segmentation method based on a point cloud space-time memory network, which comprises the following steps: S1, constructing a 4D target segmentation task data set; S2, constructing a 4D target segmentation network based on a point cloud space-time memory network and in combination with a non-local / local attention mechanism; S3, training the 4D target segmentation network model, optimizing network parameters by reducing a network loss function until the network converges, and obtaining the 4D target segmentation method based on the point cloud space-time memory network; and S4, segmenting a given target for a new LIDAR video sequence by using the 4D target segmentation method based on the point cloud space-time memory network. The application provides a new data set, designs a fast and stable 4D target segmentation algorithm in combination with a point cloud space-time memory network and an attention mechanism, and the designed algorithm has good robustness for problems such as target occlusion and fast movement in an actual 3D environment.

Description

Technical Field

[0001] This invention relates to the field of computer vision, specifically to a 4D target segmentation method based on point cloud spatiotemporal memory networks. Background Technology

[0002] In recent years, image-based video object segmentation (VOS) has attracted widespread attention due to its extensive applications in video editing and analysis. This task aims to segment objects within consecutive video clips. While recent 2D VOS algorithms have achieved good results, they still face significant challenges when dealing with low-light conditions or textureless objects. In contrast, LiDAR is insensitive to texture and lighting variations, making it a suitable complement to cameras. On the other hand, with the rapid development of 3D LiDAR sensors over the past decade, leveraging point clouds to solve various vision problems has become a hot topic due to the enormous application potential of point clouds in fields such as autonomous driving, motion planning, and robotics. However, LiDAR-based VOS tasks have not yet been clearly defined or explored, primarily due to the lack of suitable datasets.

[0003] 3D video target segmentation, or 4D target segmentation, based on LiDAR is a novel task. It segments specific 3D LiDAR targets (given in the first frame) within a point cloud video. The most relevant tasks for 4D target segmentation are 3D tracking and 3D scene flow. Specifically, 3D tracking provides boundary and orientation information of objects, but its information density is lower than that provided by point-by-point segmentation. Since LiDAR point clouds only exist on the object surface, it is often difficult to generate high-quality target shapes from point clouds within bounding boxes. Furthermore, due to the inaccuracy of the bounding boxes or the close proximity between different objects or backgrounds, the generated target shapes often contain a large number of noise points. 3D scene flow estimates point-by-point motion between two consecutive point cloud frames, providing a low-level and basic understanding of the motion of objects in dynamic scenes. Point-by-point motion information can certainly be used to propagate target masks, just as it is used for 2D optical flow; however, it often includes a lot of unnecessary motion calculations, such as those for the background, and lacks robustness to occlusion and drift. Existing technologies also lack highly accurate and robust 4D target segmentation methods. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes a simple and flexible end-to-end benchmark method for 4D object segmentation. Specifically, this invention utilizes a point cloud spatiotemporal memory network to compute spatiotemporal attention between each point in the current frame and each point in historical frames. Two input transformation modules are designed to map the two types of inputs to a unified feature space. Then, a PointNet++-based Siamese encoder is used to encode the template features and search features. After passing through an attention mechanism module, the matching features are used to predict the final point cloud search segmentation result through a multi-stage decoder based on the PointNet++ segmentation head.

[0005] To achieve the above objectives, this invention provides a 4D target segmentation method based on a point cloud spatiotemporal memory network, comprising the following steps:

[0006] S1. Construct a 4D object segmentation task dataset;

[0007] S2. A 4D target segmentation network is constructed based on a point cloud spatiotemporal memory network and combined with a nonlocal / local attention mechanism.

[0008] S3. Train the 4D target segmentation network model, optimize the network parameters by reducing the network loss function until the network converges, and obtain the 4D target segmentation method based on point cloud spatiotemporal memory network.

[0009] S4. Use the 4D target segmentation method based on point cloud spatiotemporal memory network to segment a given target in a new LIDAR video sequence.

[0010] Preferably, step S1 specifically includes the following steps:

[0011] S11. Count the target IDs that belong to moving targets in the KITTI dataset, and use the target IDs to extract the LIDAR data of each category to obtain a coarse dataset;

[0012] S12. Calculate the length of each LIDAR sequence and the number of point clouds of each target in the first frame. Filter out LIDAR sequences with a length of less than 50 frames and LIDAR sequences with a target point cloud number of less than 50 in the first frame to obtain the final 4D target segmentation dataset.

[0013] Preferably, step S2 specifically includes the following steps:

[0014] S21. Combine the two different types of inputs (i.e., point cloud templates P) R and its target mask M R And point cloud search area P Q The template features E are obtained by mapping the input to a unified feature space through the input transformation module. R and search feature E Q ;

[0015] S22. Using a twin encoder constructed with PointNet++ as the backbone network, the template features E are processed respectively. R and search feature E Q Downsampling yields hierarchical features F R and F Q ;

[0016] S23. Utilize non-local attention mechanism modules and local attention mechanism modules to process the hierarchical feature F. R and F Q Calculate the matching feature O;

[0017] S24. Based on the PointNet++ segmentation head, a multi-stage decoder is constructed. Initially, the matching feature O is used as input, followed by the output of the previous stage and features from the same level of the Siamese encoder as input. Upsampling is performed step by step to obtain the final output M. Q .

[0018] Preferably, the input conversion module is specifically divided into a template conversion module and a search conversion module. Both the template conversion module and the search conversion module consist of a 3-layer fully connected network with the number of neurons in the hidden layers being [64, 128], and each hidden layer is followed by a ReLU layer and a BN layer. The difference between the template conversion module and the search conversion module is their input; the search conversion module uses the point cloud search region P of the LiDAR frame as its input. Q As input, output search feature E Q The template conversion module uses the point cloud template P of the LiDAR frame. R and its target mask M R For input, output template feature E R .

[0019] Preferably, the twin encoder specifically uses the first three stages of PointNet++ as a shared encoder, with each stage performing local aggregation and 2x downsampling on the input, and the number of feature channels of the output being 128, 128 and 256 respectively.

[0020] Preferably, the non-local attention mechanism module specifically involves: first, F... R / F Q Concatenate with Cartesian coordinates, then use two parallel convolutions to map the features to key-value feature pairs (k). Q ,v Q ),(k R ,v R Then, feature matching is performed in a non-local manner to model the relationship between the search point and the template point. During the matching process, the search key feature k is first calculated. Q and template key feature kR The similarity between all point features is used as the weight; then, the weight is multiplied as the affinity matrix by the template value feature v. R And compare the results with the search value feature v Q The concatenation is used as the final matching feature O.

[0021] Preferably, the local attention mechanism module specifically uses two local self-attention blocks to respectively focus on F. Q and F R Feature enhancement is performed, and feature matching is performed with a local cross-attention block that has a similar design to the local self-attention block.

[0022] Preferably, the multi-stage decoder has three stages, with each stage performing a 2x upsampling.

[0023] Preferably, step S3 specifically includes the following steps:

[0024] S31. Utilize the server to execute the training point cloud template generation unit and the point cloud search area generation unit to generate point cloud templates. And point cloud search area

[0025] S32. Utilize the server to execute the input conversion module, converting the two different types of input (i.e., point cloud template P) into... R and its target mask M R And point cloud search area P Q Mapping to a unified feature space yields template features. and search features

[0026] S33. Execute the twin encoder using the server, and input the feature E R and E Q Downsampling is performed to obtain hierarchical features. and

[0027] S34. Execute the non-local attention mechanism module and the local attention mechanism module using the server, based on the hierarchical feature F. R and E Q Obtain matching features O contains specific target information within the search area;

[0028] S35. Execute the multi-stage decoder using the server, inputting matching features O, and outputting the final segmentation result for the search region.

[0029] S36. Network training is performed using a server in an end-to-end manner; the segmentation loss function L is the cross-entropy loss function, and its mathematical expression is:

[0030]

[0031] Where Y represents the true value of the target mask; M represents the predicted result of the target mask; and Ω represents the set of all points in the target mask.

[0032] S37. Utilize the server to optimize the objective function and obtain locally optimal network parameters.

[0033] Preferably, step S31 specifically includes the following steps:

[0034] S311. Extract LiDAR video from the 4D object segmentation task dataset, and then randomly extract 2 LiDAR frames from the LiDAR video: {X t-1 ,X t}, where t is the current time;

[0035] S312, According to LIDAR: {X t-1 ,X t} and its target mask, generate axis-aligned bounding box {B t-1 B t};

[0036] S313. Using a multivariate Gaussian distribution with parameters (1, 1, 5°), generate bounding box B in the X, Y axis directions and rotation direction parameters. t-1 Perform data augmentation, then extract the X values ​​from the augmented bounding box. t-1 The 3D point cloud in the image is normalized and sampled after translation and rotation operations to obtain the point cloud template P. R ;

[0037] S314. Using a multivariate uniform distribution with parameters (-0.3, 0.3, 5°), generate parameters in the X, Y axis directions and rotation direction for the bounding box B. t Data augmentation is performed, followed by translation and scaling of the augmented bounding box to increase the search area. Finally, the X-axis values ​​within the transformed bounding box are extracted. t The 3D point cloud in the image is normalized and sampled after translation and rotation operations to obtain the point cloud search area P. Q .

[0038] Preferably, step S4 specifically includes the following steps:

[0039] S41. For a new LiDAR video sequence, using the first frame and a specific target mask, call the point cloud template generation unit to initialize the point cloud template. Segmentation begins from the second frame of the video sequence;

[0040] S42. For the current LiDAR frame, leveraging temporal continuity, use the target mask from the previous frame to generate an axis-aligned bounding box, then offset and scale it to create the search area for the current LiDAR frame. Extract the point cloud within the search area and perform translation-reduction, rotation-reduction, and normalization to obtain the point cloud search area P of the current LiDAR frame. Q and search area indicator mask;

[0041] S43, transfer the point cloud template P R and its target mask M R And point cloud search area P Q The input is fed into the input conversion module to obtain template features. and search feature E Q ;

[0042] S44, Feature E R and E Q The input is downsampled in the twin encoder to obtain hierarchical features. and F Q ;

[0043] S45, Hierarchical feature F R and F Q The input is fed into the attention mechanism module to obtain the matching feature O;

[0044] S46. Input the matching feature O into the multi-stage decoder and output the segmentation result M of the search region. Q Finally, based on the search area indicator mask, the segmentation result of the search area is expanded into a global LIDAR frame to obtain the final segmentation result M;

[0045] S47. Use the first frame and its target mask, and the current frame and its segmentation results to generate a new point cloud template, and repeat S42-S46 until the segmentation of the entire LiDAR video is completed.

[0046] Compared with the prior art, the beneficial effects of the present invention are:

[0047] The 4D target segmentation method based on point cloud spatiotemporal memory network provided by this invention defines a new task, 4D target segmentation, and constructs a new task dataset, 4D target segmentation dataset, providing a benchmark for this task. By using a multi-level aggregated Siamese encoder to fully encode target information, using non-local and local attention mechanisms to enhance target information, and using a feature pyramid upsampling structure to recover target structure, the system can accurately segment targets in complex 3D environments. It can accurately and quickly segment targets in many difficult real-world scenarios, achieving high-performance results on the constructed dataset. It has good robustness to problems such as target occlusion and rapid movement in real-world 3D environments, with an IoU of 0.33 for car segmentation, 0.39 for truck segmentation, 0.21 for person segmentation, and 0.65 for bicycle segmentation. Attached Figure Description

[0048] Figure 1 This is an algorithmic framework diagram for a 4D target segmentation method based on point cloud memory networks;

[0049] Figure 2 This is a block diagram of the nonlocal / local attention mechanism designed in this invention;

[0050] Figure 3 This is a segmentation result diagram of a 4D target segmentation method based on a point cloud memory network according to the present invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] To address the problems and shortcomings of existing technologies, this invention provides a 4D target segmentation method based on a point cloud spatiotemporal memory network. The method mainly includes four stages: constructing a 4D target segmentation task dataset, designing a 4D target segmentation network that combines a point cloud spatiotemporal memory network and an attention mechanism, model training, and model inference.

[0053] This invention proposes a 4D target segmentation method based on point cloud spatiotemporal memory network, comprising the following steps:

[0054] S1. Construct a 4D object segmentation task dataset;

[0055] S2. A 4D target segmentation network is constructed based on a point cloud spatiotemporal memory network and combined with a nonlocal / local attention mechanism.

[0056] S3. Train the 4D target segmentation network model, optimize the network parameters by reducing the network loss function until the network converges, and obtain the 4D target segmentation method based on point cloud spatiotemporal memory network.

[0057] S4. Use the 4D target segmentation method based on point cloud spatiotemporal memory network to segment a given target in a new LIDAR video sequence.

[0058] The 4D object segmentation task dataset constructed in step S1 is based on the KITTI dataset.

[0059] The following is a detailed explanation of each step.

[0060] Step S1: Construct a 4D object segmentation task dataset; obtain the 4D object segmentation task dataset by performing statistical, extraction, and filtering processes on the KITTI dataset to provide a benchmark for algorithm testing.

[0061] To better study this task, this invention constructs a 4D object segmentation dataset based on KITTI. During construction, commonly encountered objects in traffic, such as cars, people, trucks, and bicycles, are selected as targets. LiDAR videos are generated, and annotations for each target are produced by selecting corresponding target masks from panoramic annotations. This 4D object segmentation dataset can provide a good benchmark for this task.

[0062] The specific implementation process is as follows:

[0063] S11. Count the target IDs of moving targets (cars, people, trucks, bicycles, etc.) in KITTI, and use the target IDs to extract the LIDAR data of each category to obtain a coarse dataset;

[0064] S12. Calculate the length of each LiDAR sequence and the number of point clouds for each target in the first frame. Filter out LiDAR sequences with a length of less than 50 frames and LiDAR sequences with less than 50 point clouds for the target in the first frame to obtain the final 4D target segmentation dataset. Specifically, the 4D target segmentation dataset contains 200 LiDAR sequences, totaling 18370 frames. Among them, 125 sequences contain the car category, 39 sequences contain the human category, 18 sequences contain the truck category, and 18 sequences contain the bicycle category. The detailed distribution is as follows:

[0065] Table 1: Data distribution of the 4D object segmentation dataset, representing the total number of LiDAR frames / sequences for each category.

[0066]

[0067] Step S2: Design a 4D target segmentation network based on a point cloud spatiotemporal memory network and combined with a nonlocal / local attention mechanism. For example... Figure 2 The diagram shown is a block diagram of the attention mechanism designed in this invention. The specific steps are as follows:

[0068] S21. Design an input conversion module to convert two different types of input (i.e., point cloud template P) into a single input. R and its target mask M R And point cloud search area P Q Mapping to a unified feature space yields template features E. R and search feature E Q Specifically, the transformation module is divided into a template transformation module and a search transformation module. Both have the same structure, consisting of a 3-layer fully connected network with [64, 128] neurons in the hidden layers, followed by a ReLU layer and a BN layer. The only difference between these two modules is their input: the search transformation module uses the point cloud search region P of the LiDAR frame as input. Q As input, output search feature E Q The template conversion module uses the point cloud template P of the LiDAR frame. R and its target mask M R For input, output template feature E R ;

[0069] S22. Design a twin encoder using PointNet++ as the backbone network, and process the template features E respectively. R and search feature E Q Downsampling yields hierarchical features F R and F Q Specifically, the twin encoder uses the first three stages of PointNet++ as a shared encoder. Each stage performs local aggregation and 2x downsampling on the input, and the number of feature channels in the output are 128, 128 and 256, respectively.

[0070] S23. Design non-local attention and local attention mechanism modules to address the hierarchical features F in S22. R and F Q Calculate the matching feature O; the non-local attention mechanism module specifically includes: such as Figure 1 As shown in (b), first F R / F Q Concatenate with Cartesian coordinates, then use two parallel convolutions to map the features to key-value feature pairs (k). Q ,v Q ),(k R ,v RThen, feature matching is performed in a non-local manner to model the relationship between the search point and the template point, which can also be understood as retrieving target information from the template point; in the matching process, the search key feature k is first calculated. Q and template key feature k R The similarity between all point features is used as the weight; then, the weight is multiplied as the affinity matrix by the template value feature v. R And compare the results with the search value feature v Q The concatenation, resulting in the final matching result O, can be mathematically expressed as:

[0071]

[0072]

[0073]

[0074] in, and Hierarchical features F Q and F R Spatial Cartesian coordinates; Φ * A 1×1 convolution (Φ k The number of output channels is 64, Φ v The number of output channels is 128), [·,·] is the concatenation operation, p and q represent the point index of the search area and the point index of the template, respectively, '·' is the dot product, and 'σ' is the softmax function;

[0075] The local attention mechanism module specifically includes: Figure 1 As shown in (c), two local self-attention blocks (LSAs) are used to apply F to F. Q and F R Feature enhancement is performed, and feature matching is performed using a local cross-attention block (LCA) with a design similar to LSA; for example... Figure 2 As shown, LSA can be specifically represented as:

[0076] q * =Φ q (F * ),k * =Φ k (F * ),v * =Φ v (F * )

[0077]

[0078] O=α(y)+F *

[0079] Where * represents Q (point cloud search area) or R (point cloud template), Φ * For a 1×1 convolution, p refers to q * The point index, N(p) is the set of local regions in the neighborhood of point p, '·' is the dot product, 'σ' is the softmax function, and δ = θ(p) i -p j ) represents the positional encoding, p i ,p j α, γ, θ are 3D Cartesian coordinates, and α, γ, θ are multilayer perceptrons (MLPs) with two linear layers and one ReLU layer, and the number of channels in the hidden layer is 256.

[0080] like Figure 2 As shown, LCA can be specifically represented as:

[0081] q Q =Φ q (F Q ),k R =Φ k (F R ),v R =Φ v (F R )

[0082]

[0083] O=α(y)+F Q

[0084] Among them, Ω R This is the template point set.

[0085] This invention discovers that the spatiotemporal attention module used in the baseline method is inherently similar to the nonlocal matching mechanism, lacking locality. 3D point clouds exhibit local density characteristics, reflecting the surface structure of objects. Furthermore, overcoming attention dispersion and undersegmentation problems is challenging in the case of global attention. Therefore, to leverage the temporal and locally dense structure of LiDAR video, this invention replaces the nonlocal attention mechanism module in the baseline method with a local attention mechanism module (local self-attention and local cross-attention), which is more effective in learning the connections between surface aggregation points. With the help of the local attention mechanism, the improved model can better learn the target representation in the point cloud, thereby achieving more accurate segmentation prediction.

[0086] S24. Based on the segmentation head of PointNet++, design a multi-stage decoder. Initially, take the matching feature O as input, then take the output of the previous stage and features from the same level from the Siamese encoder as input, and gradually upsample to obtain the final output M. Q Specifically, the multi-stage decoder consists of three stages, with each stage performing a 2x upsampling.

[0087] Step S3: Train the 4D target segmentation network model by reducing the network loss function and optimizing the network parameters until the network converges, thus obtaining a 4D target segmentation method based on a point cloud spatiotemporal memory network. Figure 1 The diagram shows the algorithm framework of a 4D target segmentation method based on a point cloud spatiotemporal memory network according to the present invention. The specific steps are as follows:

[0088] S31. Utilize the server to execute the training point cloud template and point cloud search area generation unit to generate the point cloud template. and point cloud search area Specifically, LIDAR video is extracted from the 4D target segmentation dataset described in step S1, and then two LIDAR frames are randomly extracted from the LIDAR video: {X t-1 ,X t}, where t is the current time; according to the LIDAR frame {X t-1 ,X t} and its segmentation labels, generating axis-aligned boundaries {B t-1 B t}; A multivariate Gaussian distribution with parameters (1, 1, 5°) is used to generate parameters in the X, Y axis directions and rotation direction for the bounding box B. t-1 Perform data augmentation, then extract the X values ​​from the augmented bounding box. t-1 The 3D point cloud in the image is normalized by translation and rotation operations and sampled (1024 points) to obtain the point cloud template P. R Then, using a multivariate uniform distribution with parameters (-0.3, 0.3, 5°), parameters in the X, Y axis directions and rotation direction are generated for the bounding box B. t Data augmentation is performed, followed by translation (2m offset in the X direction) and scaling (doubling the size) of the augmented bounding box to increase the search area. Then, the X-axis values ​​within the transformed bounding box are taken. t The 3D point cloud in the image is normalized and sampled (512 points) after translation and rotation operations to obtain the point cloud search area P. Q ;

[0089] S32. Utilize the server to execute the input conversion module described in step S2, converting two different types of input (i.e., point cloud template P) into two different types of input. R and its target mask M RAnd point cloud search area P Q Mapping to a unified feature space yields template features. and search features

[0090] S33. Execute the twin encoder described in step S2 using the server, and input the feature E. R and E Q Downsampling is performed to obtain hierarchical features. and

[0091] S34. Execute the attention mechanism module described in step S2 using the server, based on the hierarchical feature F. R and F Q To obtain matching features O contains specific target information within the search area;

[0092] S35. Execute the multi-stage decoder described in step S2 using the server, inputting matching features O, and outputting the final segmentation result for the search region.

[0093] S36. Network training is performed using a server in an end-to-end manner; the segmentation loss function L is the cross-entropy loss function, and its mathematical expression is:

[0094]

[0095] Where Y represents the true value of the target mask; M represents the predicted result of the target mask; and Ω represents the set of all points in the target mask.

[0096] S37. Optimize the objective function using the server to obtain locally optimal network parameters; specifically, take the loss function L in step S36 as the objective function, use the Adam optimizer to iteratively update the network parameters, reduce the objective loss function until it converges to a local optimum, and the training ends, obtaining the trained network weights for 4D target segmentation based on point cloud spatiotemporal memory network.

[0097] Step S4: Segment the given target in the new LiDAR video sequence using the 4D target segmentation method based on the point cloud spatiotemporal memory network. The specific steps are as follows:

[0098] S41. For a given LiDAR video, using the first frame and a specific target mask, call the point cloud template generation unit described in S31 to initialize the point cloud template. Segmentation begins from the second frame of the video sequence;

[0099] S42. For the current LiDAR frame, leveraging temporal continuity, generate an axis-aligned bounding box using the target mask from the previous frame. Then, offset (2m offset in the X direction) and scale (double the size) to create the search area for the current LiDAR frame. Take the point cloud within the search area and perform translation-free rotation normalization to obtain the point cloud search area P of the current LiDAR frame. Q And a search area indicator mask (using 0 and 1 to indicate the point cloud belonging to the search area in the LIDAR frame); unlike S314, the generation of the point cloud search area in this step does not involve sampling;

[0100] S43, transfer the point cloud template P R and its target mask M R And point cloud search area P Q The input is fed into the input conversion module described in step S2 to obtain template features. and search feature E Q ;

[0101] S44, Feature E R and E Q The input is downsampled to the twin encoder described in step S2 to obtain hierarchical features. and F Q

[0102] S45, Hierarchical feature F R and F Q The input is fed into the attention mechanism module described in step S2 to obtain matching feature O;

[0103] S46. Input the matching feature O into the multi-stage decoder described in step S2, and output the segmentation result M of the search region. Q Finally, based on the search area indicator mask generated in S42, the segmentation result of the search area is extended into a global LIDAR frame to obtain the final segmentation result M;

[0104] S47. Using the first frame and its annotations, and the current frame and its segmentation results, generate a new point cloud template, and repeat S42-S46 until the segmentation of the entire LiDAR video is complete. Figure 3 The image shown is a segmentation result diagram of a 4D target segmentation method based on a point cloud memory network according to the present invention.

[0105] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A 4D target segmentation method based on point cloud spatiotemporal memory network, characterized in that, Includes the following steps: S1. Construct a 4D object segmentation task dataset; S2. A 4D target segmentation network is constructed based on a point cloud spatiotemporal memory network and combined with a nonlocal / local attention mechanism. S3. Train the 4D target segmentation network model, optimize the network parameters by reducing the network loss function until the network converges, and obtain the 4D target segmentation method based on point cloud spatiotemporal memory network. S4. Use the 4D target segmentation method based on point cloud spatiotemporal memory network to segment a given target in a new LIDAR video sequence; Step S2 specifically includes the following steps: S21. Take two different types of inputs, namely point cloud template P. R and its target mask M R And point cloud search area P Q The template features E are obtained by mapping the input to a unified feature space through the input transformation module. R and search feature E Q ; S22. Using a twin encoder constructed with PointNet++ as the backbone network, the template features E are processed respectively. R and search feature E Q Downsampling yields hierarchical features F R and F Q ; S23. Utilize non-local attention mechanism modules and local attention mechanism modules to process the hierarchical feature F. R and F Q Calculate the matching feature O; S24. Based on the PointNet++ segmentation head, a multi-stage decoder is constructed. Initially, the matching feature O is used as input, followed by the output of the previous stage and features from the same level of the Siamese encoder as input. Upsampling is performed step by step to obtain the final output M. Q .

2. The 4D target segmentation method based on point cloud spatiotemporal memory network according to claim 1, characterized in that, Step S1 specifically includes the following steps: S11. Count the target IDs that belong to moving targets in the KITTI dataset, and use the target IDs to extract the LIDAR data of each category to obtain a coarse dataset; S12. Calculate the length of each LIDAR sequence and the number of point clouds of each target in the first frame. Filter out LIDAR sequences with a length of less than 50 frames and LIDAR sequences with a target point cloud number of less than 50 in the first frame to obtain the final 4D target segmentation dataset.

3. The 4D target segmentation method based on point cloud spatiotemporal memory network according to claim 1, characterized in that, The input conversion module is specifically divided into a template conversion module and a search conversion module. Both the template conversion module and the search conversion module consist of a 3-layer fully connected network with the number of neurons in the hidden layers being [64, 128], and each hidden layer is followed by a ReLU layer and a BN layer. The difference between the template conversion module and the search conversion module is their input. The search conversion module uses the point cloud search area P of the LiDAR frame as its input. Q As input, output search feature E Q The template conversion module uses the point cloud template P of the LiDAR frame. R and its target mask M R For input, output template feature E R .

4. The 4D target segmentation method based on point cloud spatiotemporal memory network according to claim 1, characterized in that, The twin encoder specifically uses the first three stages of PointNet++ as a shared encoder. Each stage performs local aggregation and 2x downsampling on the input, and the number of feature channels of the output are 128, 128 and 256, respectively.

5. The 4D target segmentation method based on point cloud spatiotemporal memory network according to claim 1, characterized in that, The nonlocal attention mechanism module specifically involves: first, F... R / F Q Concatenate with Cartesian coordinates, then use two parallel convolutions to map the features to key-value feature pairs (k). Q ,v Q ),(k R ,v R Then, feature matching is performed in a non-local manner to model the relationship between the search point and the template point. During the matching process, the search key feature k is first calculated. Q and template key feature k R The similarity between all point features is used as the weight; then, the weight is multiplied as the affinity matrix by the template value feature v. R And compare the results with the search value feature v Q The concatenation is used as the final matching feature O.

6. The 4D target segmentation method based on point cloud spatiotemporal memory network according to claim 1, characterized in that, The local attention mechanism module specifically uses two local self-attention blocks to respectively focus on F. Q and F R Feature enhancement is performed, and feature matching is performed with a local cross-attention block that has a similar design to the local self-attention block.

7. The 4D target segmentation method based on point cloud spatiotemporal memory network according to claim 1, characterized in that, Step S3 specifically includes the following steps: S31. Utilize the server to execute the training point cloud template generation unit and the point cloud search area generation unit to generate point cloud templates. and point cloud search area S32. Utilize the server to execute the input conversion module to convert the two different types of input, namely point cloud template P... R and its target mask M R And point cloud search area P Q Mapping to a unified feature space yields template features. and search features S33. Execute the twin encoder using the server, and input the feature E R and E Q Downsampling is performed to obtain hierarchical features. and S34. Execute the non-local attention mechanism module and the local attention mechanism module using the server, based on the hierarchical feature F. R and E Q Obtain matching features O contains specific target information within the search area; S35. Execute the multi-stage decoder using the server, inputting matching features O, and outputting the final segmentation result for the search region. S36. Network training is performed using a server in an end-to-end manner; the segmentation loss function L is the cross-entropy loss function, and its mathematical expression is: Where Y represents the true value of the target mask; M represents the predicted result of the target mask; and Ω represents the set of all points in the target mask. S37. Utilize the server to optimize the objective function and obtain locally optimal network parameters.

8. A 4D target segmentation method based on a point cloud spatiotemporal memory network according to claim 7, characterized in that, Step S31 specifically includes the following steps: S311. Extract LiDAR video from the 4D object segmentation task dataset, and then randomly extract 2 LiDAR frames from the LiDAR video: {X t-1 ,X t }, where t is the current time; S312, According to LIDAR: {X t-1 ,X t } and its target mask, generate axis-aligned bounding box {B t-1 B t }; S313, The usage parameters are (1, 1, 5) o The multivariate Gaussian distribution of the bounding box B is generated in the X, Y axis directions and rotation direction parameters. t-1 Perform data augmentation, then extract the X values ​​from the augmented bounding box. t-1 The 3D point cloud in the image is normalized and sampled after translation and rotation operations to obtain the point cloud template P. R ; S314, the operating parameters are (-0.3, 0.3, 5). o The multivariate uniform distribution of parameters in the X, Y axis directions and rotation direction is generated for the bounding box B. t Data augmentation is performed, followed by translation and scaling of the augmented bounding box to increase the search area. Finally, the X-axis values ​​within the transformed bounding box are extracted. t The 3D point cloud in the image is normalized and sampled after translation and rotation operations to obtain the point cloud search area P. Q .

9. A 4D target segmentation method based on a point cloud spatiotemporal memory network according to claim 7, characterized in that, Step S4 specifically includes the following steps: S41. For a new LiDAR video sequence, using the first frame and a specific target mask, call the point cloud template generation unit to initialize the point cloud template. Segmentation begins from the second frame of the video sequence; S42. For the current LiDAR frame, leveraging temporal continuity, use the target mask from the previous frame to generate an axis-aligned bounding box, then offset and scale it to create the search area for the current LiDAR frame. Extract the point cloud within the search area and perform translation-reduction, rotation-reduction, and normalization to obtain the point cloud search area P of the current LiDAR frame. Q and search area indicator mask; S43, transfer the point cloud template P R and its target mask M R And point cloud search area P Q The input is fed into the input conversion module to obtain template features. and search feature E Q ; S44, Feature E R and E Q The input is downsampled in the twin encoder to obtain hierarchical features. and F Q ; S45, Hierarchical feature F R and F Q The input is fed into the attention mechanism module to obtain the matching feature O; S46. Input the matching feature O into the multi-stage decoder and output the segmentation result M of the search region. Q Finally, based on the search area indicator mask, the segmentation result of the search area is expanded into a global LIDAR frame to obtain the final segmentation result M; S47. Use the first frame and its target mask, and the current frame and its segmentation results to generate a new point cloud template, and repeat S42-S46 until the segmentation of the entire LiDAR video is completed.

Images