A continuous-time canonical moving point cloud video understanding method

Abstract

The application discloses a continuous time canonical motion point cloud video understanding method, which takes a static point cloud as a canonical state, constructs a pseudo dynamic point cloud through global rigid transformation and local residual displacement, generates a pseudo state at any continuous time, trains a point cloud encoder by using state prediction and velocity consistency loss, combines mask geometry modeling and multi-modal semantic alignment, and enables a pre-training model to obtain continuous motion priori. In the fine-tuning stage, representative frames are selected from three different time stages of the point cloud video to construct a shared shape anchor point, motion-aware features are obtained by subtracting the anchor point from frame-level features, and finally, the understanding result is output by inputting the task head. The application can learn continuous and smooth motion representation by using a static point cloud, suppresses the interference of static shape bias on motion features, improves the motion sensitivity and time sequence consistency of the point cloud video model under limited frame input, and can be transferred to action recognition, action segmentation, point cloud video completion and denoising tasks.

Description

Technical Field

[0001] This invention belongs to the field of point cloud video understanding technology, specifically relating to a method for understanding continuous-time standardized motion point cloud videos. Background Technology

[0002] Point cloud videos consist of three-dimensional point cloud frames arranged in chronological order. They can simultaneously represent the spatial geometry of an object or scene and its motion state changing over time, making them suitable for action recognition, action segmentation, scene flow prediction, point cloud video denoising, point cloud video completion, and four-dimensional semantic segmentation. Point cloud video understanding requires processing both the geometric relationships within a single frame and the motion relationships between consecutive frames.

[0003] Existing supervised point cloud video understanding methods directly train spatiotemporal networks using labeled dynamic point cloud sequences, which are highly dependent on the scale of dynamic data and the quality of annotations. While unsupervised or self-supervised point cloud video learning methods reduce the dependence on labels, they still require the acquisition of a large number of dynamic point cloud sequences, and the costs of acquiring, synchronizing, storing, and preprocessing dynamic point clouds are higher than those of static point clouds.

[0004] Existing static 3D point cloud pre-training methods mainly acquire geometric priors through mask reconstruction, contrastive learning, or generative modeling, and then add a time adaptation module to the point cloud video task. Since the motion evolution process is not explicitly established in the static pre-training stage, the transferred model tends to rely on static shape and appearance information, making it difficult to obtain stable motion features under conditions of few frames or subtle motion changes.

[0005] Random rotation, translation, or local perturbation can generate different static augmentation samples, but discrete augmentation samples lack a clear temporal order, cannot directly represent intermediate states in the motion trajectory, and cannot constrain the rate of change of adjacent states. When predicting only a single augmentation result, the model may learn transformation invariance but cannot learn the law of continuous evolution of point cloud states over time.

[0006] Furthermore, frame-level features obtained from point cloud video fine-tuning typically contain both static shape information shared across frames and time-varying motion information. When shared shape information dominates the feature components, fine-grained motion changes are easily masked by the static structure, thus affecting action segmentation, scene flow prediction, and other motion-sensitive tasks.

[0007] Therefore, there is a need for a point cloud video understanding method that constructs pseudo-dynamic trajectories with global and local motion from static point clouds, learns arbitrary time states and their evolution speeds, and separates shared shape information and dynamic change information from frame-level features during the point cloud video fine-tuning stage. Summary of the Invention

[0008] This invention proposes a continuous-time, canonical motion point cloud video understanding method to address the problems of static 3D pre-training lacking continuous-time motion priors, discrete geometric augmentation failing to constrain state evolution speed, and point cloud video frame-level features being interfered with by shared static shape information. Specifically, it involves a method that utilizes static 3D point clouds to construct a pseudo-dynamic process containing global rigid motion and local residual motion, learns dynamic priors through continuous-time state prediction and velocity consistency, and separates static shape information from inter-frame motion information using shared shape anchors during the point cloud video transfer stage.

[0009] The distinguishing features of this invention include: defining the static point cloud as a normal state, constructing overall rigid motion through global rotation and translation, and distributing the local residual displacement sampled during the pseudo-dynamic process construction to each point according to the spatial proximity relationship between the point and the local anchoring region, thereby obtaining a pseudo-dynamic point cloud that simultaneously contains global motion and local non-rigid changes; generating pseudo intermediate states at any time based on the continuous rotation path from unit rotation to target rotation, and training the point cloud encoder using state prediction loss and velocity consistency loss of adjacent time positions; in the point cloud video fine-tuning stage, constructing shared shape anchors from representative frames of three different time stages, and subtracting the shared shape anchors from the frame-level features of each frame to obtain motion perception features that highlight the dynamic changes between frames.

[0010] The technical problem addressed by the above distinctions is: to enable static point cloud pre-trained models to acquire global motion, local motion, and continuous-time state evolution capabilities without relying on large-scale dynamic point cloud pre-training data, and to reduce the interference of shared static shape information on motion features during point cloud video migration.

[0011] The technical effects achieved by the above-mentioned differences are as follows: by using static point clouds to establish pseudo-dynamic supervision with a clear temporal sequence, the point cloud encoder learns continuous and smooth motion representations; at the same time, by using shape motion decomposition to highlight the dynamic changes of each frame relative to the shared shape anchor point, the sensitivity of the point cloud video model to finite frame input, local motion and fine-grained temporal changes is improved.

[0012] The technical solution adopted in this invention is a method for understanding continuous-time standardized motion point cloud videos, which includes the following steps: Step S1: Perform global rigid transformation and local residual displacement construction on the obtained static points to obtain pseudo-dynamic point cloud; Step S2: Based on the pseudo-dynamic point cloud, perform motion prediction and continuous-time state learning, and train the pre-trained network using mask geometric modeling and multimodal semantics to obtain the pre-trained point cloud encoder. Step S3: Using the pre-trained point cloud encoder, extract the frame-level features corresponding to each time frame in the acquired point cloud video. Step S4: Construct shared shape anchor points based on the frame-level features and representative frames at different time stages in the point cloud video; Step S5: Based on each time point, generate motion-aware features according to the frame-level features and the shared shape anchor points; output point cloud video understanding results based on the motion-aware features.

[0013] Preferably, step S1, which involves performing a global rigid transformation and constructing local residual displacements on the acquired static points to obtain a pseudo-dynamic point cloud, includes the following steps: Step S1.1: The acquired static point cloud is centered, scaled, and sampled to obtain a processed static point cloud; the processed static point cloud is used as the normalized state. Step S1.2: Using farthest point sampling, select multiple local point cloud block centers in the normalized state, and use nearest neighbor query to obtain the aggregated neighborhood points of each local point cloud block center in the normalized state, thereby obtaining multiple center local point cloud blocks corresponding to each center; Step S1.3: Generate a rotation matrix based on the randomly sampled axis-angle rotation vector, and perform a global rigid transformation on each of the central local point cloud blocks based on the rotation matrix and the randomly sampled global translation vector to obtain the point cloud after global transformation; Step S1.4: Select multiple local anchoring regions from the standard state, assign corresponding local residual displacements to each point according to the spatial proximity relationship between each point in the standard state and each local anchoring region, and superimpose the assigned local residual displacements onto the point cloud after global transformation to obtain a pseudo-dynamic point cloud. The rotation matrix satisfies the following formula:

[0014] In the above formula, Represents the rotation matrix. Represents matrix exponentiation. Represents the identity matrix. Represents the axis-angle rotation vector antisymmetric matrix, Represents the sine function. The L2 norm of the axis-angle rotation vector is represented. This represents the cosine function.

[0015] Preferably, step S2, which involves performing motion prediction and continuous-time state learning based on the pseudo-dynamic point cloud, and training the pre-trained network using mask geometric modeling and multimodal semantics to obtain a pre-trained point cloud encoder, includes the following steps: Step S2.1: Input the pseudo-dynamic point cloud into the point cloud encoder to obtain coded features; input the coded features into the global motion prediction head to obtain rotation parameter estimates and translation vector estimates; and calculate the global motion prediction loss based on the differences between the rotation parameter estimates and translation vector estimates and the randomly sampled axis angle rotation vector and the sampled global translation vector. The global motion prediction loss satisfies the following formula:

[0016] In the above formula, This represents the global motion prediction loss. This represents the estimated value of the rotation parameters. Indicates rotation parameters, The square of the L2 norm, This represents the learnable weights used to balance rotation and translation errors. This represents the estimated translation vector. Represents the global translation vector. express and The distance between them; Step S2.2: Input the coded markers of the centers of each local point cloud block into the local flow prediction head to obtain the predicted residual displacement of each local point cloud block center, and calculate the local motion prediction loss based on the difference between the predicted residual displacement and the local residual displacement of each local point cloud block center. The local motion prediction loss satisfies the following formula:

[0017] In the above formula, This represents the local motion prediction loss. Represents a set of local dynamic point cloud blocks. This represents the predicted residual displacement at the center of each local point cloud block. This represents the local residual displacement at the center of each local point cloud block. This represents the difference between the predicted residual displacement and the local residual displacement; Step S2.3: Based on the randomly sampled axis-angle rotation vector, global translation vector, and local residual displacement corresponding to each point, generate real pseudo-states corresponding to any continuous time interval between zero and one; input the normalized state and its corresponding time into the state prediction network to obtain the predicted pseudo-state; calculate the state prediction loss based on the difference between the predicted pseudo-state and the real pseudo-state. Step S2.4: Generate real pseudo-states and predicted pseudo-states at adjacent time positions respectively; obtain the real state velocity based on the changes in adjacent real pseudo-states; obtain the predicted state velocity based on the changes in adjacent predicted pseudo-states; calculate the velocity consistency loss based on the difference between the predicted state velocity and the real state velocity, according to the normed state. Step S2.5: Select some local point cloud blocks from each local point cloud block as mask regions, reconstruct the mask regions based on the unmasked local point cloud blocks to obtain reconstructed local point cloud blocks; calculate the mask geometric modeling loss based on the reconstructed local point cloud blocks and the local point cloud blocks in the mask regions. Step S2.6: Extract image features and text features corresponding to the static point cloud using the frozen image encoder and text encoder; calculate the alignment loss of multimodal semantics based on image-related tags, text-related tags, the image features, and the text features. Step S2.7: Update the network parameters of the pre-trained network based on the global motion prediction loss, the local motion prediction loss, the state prediction loss, the velocity consistency loss, the mask geometry modeling loss, and the multimodal semantic alignment loss to obtain the pre-trained point cloud encoder.

[0018] Preferably, step S2.3, which involves generating the true / false state corresponding to any continuous time interval within the time range of zero to one based on the randomly sampled axis-angle rotation vector, global translation vector, and local residual displacement corresponding to each point, includes the following steps: Step S2.3.1: Based on the randomly sampled axis-angle rotation vector, global translation vector, and local residual displacement corresponding to each point, sample the time within the time interval from zero to one, and generate the continuous rotation matrix corresponding to the sampling time based on the continuous rotation path from the unit rotation matrix to the rotation matrix. Step S2.3.2: Generate the true pseudo-state corresponding to the sampling time based on the continuous rotation matrix, the normalized state, the global translation vector, and the local residual displacement; The true / false state satisfies the following formula:

[0019] In the above formula, Indicates time The corresponding real and pseudo-state points, Indicates time The corresponding continuous rotation matrix, and satisfying , Indicates the rotation parameters of the sampling. Represents the set of points in the normal state. Indicates time, Represents the global translation vector. Indicates local residual displacement; The predicted pseudo-state satisfies the following formula:

[0020] In the above formula, Indicates time The corresponding predicted pseudo-state, The parameter is A state prediction network.

[0021] Preferably, step S2.4, which involves generating true pseudo-states and predicted pseudo-states at adjacent time positions, obtaining the true state velocity based on the changes in adjacent true pseudo-states, and obtaining the predicted state velocity based on the changes in adjacent predicted pseudo-states, includes the following steps: Step S2.4.1: Based on the method of generating the real pseudo-state and the predicted pseudo-state, obtain the real pseudo-state and the predicted pseudo-state corresponding to two adjacent time positions respectively; Step S2.4.2: Based on the changes and time intervals between the real and pseudo states corresponding to two adjacent time positions, obtain the real state velocity, and based on the changes and time intervals between the predicted pseudo states corresponding to two adjacent time positions, obtain the predicted state velocity. The speed consistency loss satisfies the following formula:

[0022] In the above formula, This represents the speed consistency loss, where N represents the number of points in the normalized state. Indicates the first One point, Indicates time Predicted state speed, Indicates time The actual speed at that point This represents the difference between the predicted state velocity and the actual state velocity.

[0023] Preferably, the multimodal semantic alignment loss in step S2.6 satisfies the following formula:

[0024] In the above formula, Indicates alignment loss. Indicates image-related tags, Representing image features, Indicates the distance between text-related tags. Indicates text-related tags, Represents text features.

[0025] Preferably, step S2.7, which updates the network parameters of the pre-trained network based on the global motion prediction loss, the local motion prediction loss, the state prediction loss, the velocity consistency loss, the mask geometry modeling loss, and the multimodal semantic alignment loss to obtain the pre-trained point cloud encoder, includes the following steps: Step S2.7.1: Based on the global motion prediction loss, the local motion prediction loss, the state prediction loss, the velocity consistency loss, the mask geometry modeling loss, and the multimodal semantic alignment loss, update the network parameters of the pre-trained network with the pre-trained overall function as the objective. Step S2.7.2: After completing the pre-training, retain the point cloud encoder and its pre-training parameters in the pre-trained network to obtain the pre-trained point cloud encoder; The pre-trained overall function satisfies the following formula:

[0026] In the above formula, This indicates the overall goal of pre-training. The weights represent the weights of the global motion prediction loss. This represents the global motion prediction loss. The weights representing the local motion prediction loss, This represents the local motion prediction loss. The weights represent the state prediction loss. This represents the state prediction loss. The weights representing the speed consistency loss This indicates the loss of speed consistency. The weights represent the mask geometry modeling loss. This represents the mask geometry modeling loss. The weights represent the multimodal semantic alignment loss. This represents the multimodal semantic alignment loss.

[0027] Preferably, step S3, which involves using the pre-trained point cloud encoder to extract frame-level features corresponding to each time frame in the acquired point cloud video, includes the following steps: Step S3.1: Using the pre-trained point cloud encoder, divide each time frame in the point cloud video into multiple local point cloud groups; Step S3.2: Encode each local point cloud group to obtain the local markers corresponding to each time frame; Step S3.3: Based on each time frame, aggregate all local tags of each time frame to obtain frame-level features; The local marker corresponding to each time frame satisfies the following formula:

[0028] In the above formula, Indicates the first Frame local markers, This indicates a local point cloud group encoding operation. Indicates the first Frame local point cloud group, Indicates the time frame index.

[0029] Preferably, step S4, which involves constructing shared shape anchors based on each time period, according to the frame-level features and representative frames from different time stages in the point cloud video, includes the following steps: Step S4.1: Select a representative frame from each of the three different time stages of the point cloud video; Step S4.2: Normalize the three representative frames respectively, and input the normalized three representative frames into the shape encoder to obtain three global shape features; Step S4.3: Normalize the three global shape features respectively, and calculate the arithmetic mean of the three normalized global shape features to obtain the average shape feature; Step S4.4: Project the average shape feature onto the feature space where the frame-level feature is located to obtain the shared shape anchor point; The three global shape features satisfy the following formula:

[0030] In the above formula, Indicates the first A global shape feature representing a frame. Indicates shape encoder, This indicates a normalization operation. Indicates time phase The corresponding representative frame, Indicates the first A time phase; The shared shape anchor points satisfy the following formula:

[0031] In the above formula, Indicates a shared shape anchor point. This represents the mapping that projects the average shape features onto the feature space containing the frame-level features. This indicates a feature normalization operation.

[0032] Preferably, step S5, which involves generating motion-aware features based on the frame-level features and the shared shape anchors at each time point, and outputting point cloud video understanding results based on the motion-aware features, includes the following steps: Step S5.1: Based on each time, subtract the shared shape anchor point from the frame-level features to obtain the motion-aware features corresponding to the frame; Step S5.2: Perform temporal aggregation on the motion sensing features corresponding to each time frame to obtain the aggregated motion features; Step S5.3: Input the aggregated motion features into the task head and output the point cloud video understanding results; The point cloud video understanding result satisfies the following formula:

[0033] In the above formula, This represents the point cloud video understanding results. The parameter is Task head, This indicates a time aggregation operation. Indicates the first Motion-sensing features of frames Indicates the time frame index. This indicates the number of frames in the point cloud video.

[0034] The beneficial effects of this invention are as follows: (1) The present invention constructs a pseudo-dynamic point cloud by combining global rigid motion and local residual motion, so that the static point cloud pre-training can simultaneously obtain supervision information on the overall posture change and the local dynamic region change.

[0035] (2) The present invention generates pseudo intermediate states at any time and does not limit dynamic pre-training to the initial state and the final state, thereby establishing a continuous state evolution process with a clear time sequence.

[0036] (3) The present invention constrains the predicted velocity at adjacent time positions by velocity consistency loss, so that the predicted state changes smoothly along the preset pseudo-dynamic trajectory, thereby improving the temporal continuity of the pre-trained representation.

[0037] (4) This invention retains mask geometry modeling while performing continuous motion learning and uses image and text features for semantic alignment, so that the pre-trained encoder takes into account local geometry, motion evolution and high-level semantic information.

[0038] (5) The present invention constructs shared shape anchor points from multiple time stages of point cloud video and subtracts them from the features of each frame, thereby reducing the interference of cross-frame shared static shape information on motion changes.

[0039] (6) The pre-training process of the present invention uses static point cloud as the basic data, reducing the dependence on large-scale point cloud video pre-training data, and the resulting encoder can be transferred to sequence-level, frame-level, point-by-point prediction and point cloud sequence reconstruction tasks. Attached Figure Description

[0040] Figure 1 This is the overall flowchart of the continuous-time normed motion pre-training and shape motion decomposition of the present invention; Figure 2 This is a pre-training framework diagram for global and local pseudo-motion construction, continuous-time state prediction, mask geometric modeling, and multimodal semantic alignment of the present invention. Figure 3 This is a transfer application framework diagram of the point cloud video frame-level feature extraction and shape motion decomposition of the present invention; Figure 4 This is a schematic diagram showing the results of point cloud video completion and point cloud video denoising using the present invention; Detailed Implementation

[0041] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The following embodiments are used to illustrate the technical solutions of the present invention and are not intended to add technical features not described in the original paper.

[0042] like Figure 1 As shown, the present invention provides a method for understanding continuous-time standardized motion point cloud videos, comprising the following steps: Step S1: Perform global rigid transformation and local residual displacement construction on the obtained static points to obtain pseudo-dynamic point cloud; Step S2: Based on the pseudo-dynamic point cloud, perform motion prediction and continuous-time state learning, and train the pre-trained network using mask geometric modeling and multimodal semantics to obtain the pre-trained point cloud encoder. Step S3: Using the pre-trained point cloud encoder, extract the frame-level features corresponding to each time frame in the acquired point cloud video. Step S4: Construct shared shape anchor points based on the frame-level features and representative frames at different time stages in the point cloud video; Step S5: Based on each time point, generate motion-aware features according to the frame-level features and the shared shape anchor points; output point cloud video understanding results based on the motion-aware features.

[0043] Further, step S1 involves performing a global rigid transformation and constructing local residual displacements on the acquired static points to obtain a pseudo-dynamic point cloud, including the following steps: Step S1.1: The acquired static point cloud is centered, scaled, and sampled to obtain a processed static point cloud; the processed static point cloud is used as the normalized state. Step S1.2: Using farthest point sampling, select multiple local point cloud block centers in the normalized state, and use nearest neighbor query to obtain the aggregated neighborhood points of each local point cloud block center in the normalized state, thereby obtaining multiple center local point cloud blocks corresponding to each center; Step S1.3: Generate a rotation matrix based on the randomly sampled axis-angle rotation vectors. Mathematically, this will transform the static point cloud... As a normal state, the sampling axis angle rotation vector Calculate the rotation matrix according to Rodrigues' formula:

[0044] In the above formula, Represents the rotation matrix. Represents matrix exponentiation. Represents the axis-angle rotation vector antisymmetric matrix, Represents the identity matrix. Represents the sine function. The L2 norm of the axis-angle rotation vector. This represents the cosine function.

[0045] Based on the rotation matrix and the randomly sampled global translation vector, a global rigid transformation is performed on each of the central local point cloud blocks to obtain the globally transformed point cloud:

[0046] In the above formula, This represents the point cloud after global transformation. This represents a central local point cloud block. This represents the global translation vector.

[0047] Step S1.4: Select multiple local anchoring regions from the normalized state. Based on the spatial proximity between each point in the normalized state and each of the local anchoring regions, assign corresponding local residual displacements to each point. Then, superimpose the assigned local residual displacements onto the point cloud after the global transformation to obtain a pseudo-dynamic point cloud.

[0048] In the above formula, Represents pseudo-dynamic points, This represents the local residual displacement.

[0049] Furthermore, such as Figure 2 As shown, step S2, which involves performing motion prediction and continuous-time state learning based on the pseudo-dynamic point cloud, and training the pre-trained network using mask geometric modeling and multimodal semantics to obtain a pre-trained point cloud encoder, includes the following steps: Step S2.1: Input the pseudo-dynamic point cloud into the point cloud encoder to obtain coded features; input the coded features into the global motion prediction head to obtain rotation parameter estimates and translation vector estimates; and calculate the global motion prediction loss based on the differences between the rotation parameter estimates and translation vector estimates and the randomly sampled axis angle rotation vector and the sampled global translation vector. The global motion prediction loss satisfies the following formula:

[0050] In the above formula, This represents the global motion prediction loss. This represents the estimated value of the rotation parameters. Indicates rotation parameters, Represents the square of the L2 norm. This represents the learnable weights used to balance rotation and translation errors. This represents the estimated value of the global translation vector. Represents the global translation vector. express and The distance between them; Step S2.2: Input the coded markers of the centers of each local point cloud block into the local flow prediction head to obtain the predicted residual displacement of each local point cloud block center, and calculate the local motion prediction loss based on the difference between the predicted residual displacement and the local residual displacement of each local point cloud block center. The local motion prediction loss satisfies the following formula:

[0051] In the above formula, This represents the local motion prediction loss. Represents a local set of point cloud blocks. This represents the predicted residual displacement at the center of each local point cloud block. This represents the local residual displacement at the center of each local point cloud block. This represents the difference between the predicted residual displacement and the local residual displacement; Step S2.3: Based on the randomly sampled axis-angle rotation vector, global translation vector, and local residual displacements corresponding to each point, generate real and pseudo-states corresponding to any continuous time interval between zero and one; input the normalized state and the corresponding time into the state prediction network ( A state prediction network is used to obtain predicted pseudo-states; specifically, this includes: Step S2.3.1: Based on the randomly sampled axis-angle rotation vector, global translation vector, and local residual displacement corresponding to each point, sample the time within the time interval from zero to one, and generate the continuous rotation matrix corresponding to the sampling time based on the continuous rotation path from the unit rotation matrix to the rotation matrix. Step S2.3.2: Generate the true pseudo-state corresponding to the sampling time based on the continuous rotation matrix, the normalized state, the global translation vector, and the local residual displacement; The true / false state satisfies the following formula:

[0052] In the above formula, Indicates time The corresponding real and pseudo-state points, Indicates time The corresponding continuous rotation matrix, and satisfying , Indicates the rotation parameters of the sampling. Represents the set of points in the normal state. Indicates time, Represents the global translation vector. Indicates local residual displacement; The predicted pseudo-state satisfies the following formula:

[0053] In the above formula, Indicates time Predicted pseudo-states express State prediction network, Indicates the normal state. Indicates time.

[0054] The state prediction loss is calculated based on the difference between the predicted pseudo-state and the true pseudo-state. The state prediction loss satisfies the following formula:

[0055] In the above formula, This represents the state prediction loss. Indicates time The corresponding predicted pseudo-state point, Indicates time The corresponding real and pseudo-state points, This represents the difference between the predicted pseudo-state point and the true pseudo-state point; Step S2.4: Generate true pseudo-states and predicted pseudo-states at adjacent time positions respectively. Obtain the true state velocity based on the changes in adjacent true pseudo-states, and obtain the predicted state velocity based on the changes in adjacent predicted pseudo-states; specifically including the following: Step S2.4.1: Based on the method of generating the real pseudo-state and the predicted pseudo-state, obtain the real pseudo-state and the predicted pseudo-state corresponding to two adjacent time positions respectively; Step S2.4.2: Based on the changes and time intervals between the real and pseudo states corresponding to two adjacent time positions, obtain the real state velocity, and based on the changes and time intervals between the predicted pseudo states corresponding to two adjacent time positions, obtain the predicted state velocity. Example: in time and The true pseudo-state and the predicted pseudo-state are obtained at the respective locations, and the speed of the true state is... and predicted state velocity They are represented as follows:

[0056] In the above formula, Indicates time The actual speed at that point Indicates time The corresponding real and pseudo states, Indicates time The corresponding real and pseudo states, Indicates the time increment between adjacent time positions. Indicates time Predicted state velocity at the location, Indicates time The corresponding predicted pseudo-state, Indicates time The corresponding predicted pseudo-state.

[0057] Based on the standard state, calculate the speed consistency loss according to the difference between the predicted state speed and the actual state speed; The speed consistency loss satisfies the following formula:

[0058] In the above formula, This indicates the loss of speed consistency. This represents the number of points in the normalized state. Indicates the first One point, Indicates time Predicted state speed, Indicates time The actual speed at that point This represents the difference between the predicted state velocity and the actual state velocity.

[0059] Step S2.5: Select some local point cloud blocks from each local point cloud block as mask regions, reconstruct the mask regions based on the unmasked local point cloud blocks to obtain reconstructed local point cloud blocks; calculate the mask geometric modeling loss based on the reconstructed local point cloud blocks and the local point cloud blocks in the mask regions. Select a portion of the local point cloud blocks from each local point cloud block, for example: A local point cloud patch Local point cloud block set As a masked region, the point cloud encoder only receives the unmasked local point cloud blocks (visible), and the point cloud decoder reconstructs the masked region based on the visible context and mask markers. Let... This represents a local point cloud patch within the masked region. If we represent the reconstruction of a local point cloud patch, then the mask geometry modeling loss is expressed as:

[0060] In the above formula, This represents the mask geometry modeling loss. This represents the distance between local point cloud patches in the reconstructed mask region using the L2 norm point distance and other local point cloud patches in the mask region. Step S2.6: Extract image features and text features corresponding to the static point cloud using the frozen image encoder and text encoder; calculate the alignment loss of multimodal semantics based on image-related tags, text-related tags, the image features, and the text features. Specifically, this involves extracting image features corresponding to the static point cloud using a frozen image encoder and text encoder. and text features Image-related tags Text-related tags After mapping to the corresponding semantic space, the alignment loss is calculated as follows:

[0061] In the above formula, Indicates alignment loss. Indicates image-related tags, Representing image features, Indicates the distance between text-related tags. Indicates text-related tags, Represents text features.

[0062] Step S2.7: Update the network parameters of the pre-trained network based on the global motion prediction loss, the local motion prediction loss, the state prediction loss, the velocity consistency loss, the mask geometry modeling loss, and the multimodal semantic alignment loss to obtain the pre-trained point cloud encoder. Specifically, this includes: Step S2.7.1: Based on the global motion prediction loss, the local motion prediction loss, the state prediction loss, the velocity consistency loss, the mask geometry modeling loss, and the multimodal semantic alignment loss, update the network parameters of the pre-trained network with the pre-trained overall function as the objective. Step S2.7.2: After completing the pre-training, retain the point cloud encoder and its pre-training parameters in the pre-trained network to obtain the pre-trained point cloud encoder; The pre-trained overall function satisfies the following formula:

[0063] In the above formula, This indicates the overall goal set for pre-training. The weights represent the weights of the global motion prediction loss. This represents the global motion prediction loss. The weights represent the state prediction loss. This represents the local motion prediction loss. The weights represent the state prediction loss. This represents the state prediction loss. The weights representing the speed consistency loss This indicates the loss of speed consistency. The weights represent the mask geometry modeling loss. This represents the mask geometry modeling loss. The weights represent the multimodal semantic alignment loss. This represents the multimodal semantic alignment loss.

[0064] Furthermore, such as Figure 3 As shown, step S3, which involves using the pre-trained point cloud encoder to extract frame-level features corresponding to each time frame in the acquired point cloud video, includes the following steps: Step S3.1: Using the pre-trained point cloud encoder, divide each time frame in the point cloud video into multiple local point cloud groups; Step S3.2: Encode each local point cloud group to obtain the local label corresponding to each time frame; for example, let each local point cloud group be... ,in Indicates the number of frames. Each frame is divided into... The local point cloud group, for the first Local point cloud group of time frame Encoding is performed to obtain the local marker corresponding to each time frame, satisfying the following formula:

[0065] In the above formula, Indicates the first Frame local markers, This indicates a local point cloud group encoding operation. Indicates the first Frame local point cloud group, Indicates the time frame index.

[0066] Step S3.3: Based on each time frame, aggregate all local tags of each time frame to obtain frame-level features; Furthermore, step S4, which involves constructing shared shape anchor points based on each time period, according to the frame-level features and representative frames from different time stages in the point cloud video, includes the following steps: Step S4.1: Select a representative frame from each of the three different time stages of the point cloud video. The time stages are denoted as follows: , and ; Step S4.2: Normalize the three representative frames respectively, and input the normalized three representative frames into the shape encoder. This yields three global shape features:

[0067] In the above formula, Indicates the first A global shape feature, Indicates shape encoder, This indicates a normalization operation. Indicates time phase The corresponding representative frame, Indicates the first Each time phase.

[0068] Step S4.3: Normalize the three global shape features respectively, and calculate the arithmetic mean of the three normalized global shape features to obtain the average shape feature; Step S4.4: Project the average shape feature onto the feature space where the frame-level feature is located to obtain the shared shape anchor point; The shared shape anchor points satisfy the following formula:

[0069] In the above formula, Indicates a shared shape anchor point. This represents the mapping that projects the average shape features onto the feature space containing the frame-level features. This indicates a feature normalization operation.

[0070] Further, step S5, which involves generating motion-aware features based on each time period, according to the frame-level features and the shared shape anchors, and outputting point cloud video understanding results based on the motion-aware features, includes the following steps: Step S5.1: Based on each time, subtract the shared shape anchor point from the frame-level features to obtain the motion-aware features corresponding to the frame; The motion sensing features satisfy the following formula:

[0071] In the above formula, Indicates the first Motion-sensing features of frames Indicates the first Frame-level features Indicates a shared shape anchor point. Indicates the time frame index; Step S5.2: Perform temporal aggregation on the motion sensing features corresponding to each time frame to obtain the aggregated motion features; Step S5.3: Input the aggregated motion features into the task head and output the point cloud video understanding results; The point cloud video understanding result satisfies the following formula:

[0072] In the above formula, This represents the point cloud video understanding results. The parameter is Task head, This indicates a time aggregation operation. Indicates the first Motion-sensing features of frames Indicates the time frame index. The number of frames in the point cloud video is indicated. The task header is either a frame-level action segmentation header or a point cloud video completion and reconstruction header. The point cloud video understanding result is the action label of each time frame or the completed point cloud video.

[0073] Figure 3 The upper part is the pre-trained point cloud encoder that divides and encodes local point cloud groups for each frame to obtain frame-level features; the lower part is the selection of representative frames from three time stages, which are then processed by a shape encoder, feature normalization and averaging, and projection mapping to obtain shared shape anchors. The frame-level features of each frame are then subtracted from the anchors to obtain motion-aware features, and finally the results are output through time aggregation and task header.

[0074] To illustrate the method of the present invention, four specific examples are provided below: Specific use case 1: The final point cloud video understanding output was defined as an action recognition task, specifically implemented on human action dataset 1 (MSR-Action3D) and human action dataset 2 (UTD-MHAD) using 4, 8, 16, and 24 frames of input, respectively. Comparison methods included Point Cloud Mask Autoencoder (Point-MAE) and Point Cloud Generative Pre-trained Transformer - Miniature (PointGPT-S). The results are shown in Table 1.

[0075] Table 1. 3D motion recognition accuracy at different frame rates

[0076] Table 1 lists the action recognition accuracy of the three methods under different frame inputs. As shown in Table 1, the method of the present invention can transfer the pre-trained knowledge of static point clouds to the action recognition task under different frame input settings, and maintains a high recognition accuracy, especially with limited frame input.

[0077] Specific use case 2: Point cloud video completion and noise reduction.

[0078] like Figure 4 As shown, this embodiment constructs a point cloud video completion task on human action recognition dataset 1 (MSR-Action3D) and human action recognition dataset 2 (UTD-MHAD), employing random missing data and spherical region missing data respectively, and setting missing proportions of 25%, 50%, and 75%. Evaluation metrics are based on... The chamfer distance is calculated as the point distance, where the mean represents the average error across all frames, and the median represents the error in intermediate frames. The comparison method is a point cloud probabilistic quantization autoencoder (Point-PQAE). Table 2 lists the mean results of Point-PQAE and the method of this invention at 25% and 75% missing percentages.

[0079] Table 2: Point Cloud Video Completion Results

[0080] This use case also performs point cloud video denoising on MSR-Noise and UTD-Noise, injecting Gaussian perturbations at three intensities: 1%, 2%, and 3% of the point cloud radius, and evaluating using 4-frame and 8-frame sequences. Table 3 lists Point-PQAE and the method of this invention. distance.

[0081] Table 3. Noise Removal Results for Point Cloud Videos

[0082] As shown in Tables 2 and 3, the continuous motion prior learned in this invention can be used for structure restoration and noise suppression of point cloud videos, while maintaining the consistency of dynamic states across frames.

[0083] Specific use case 3: Point cloud video motion segmentation.

[0084] This embodiment transfers the point cloud video understanding task to the large-scale action segmentation dataset (HOI4D) to complete the segmentation task, predicting action labels for each time frame in the point cloud video. The evaluation metric is frame-level accuracy. Clip editing distance Fragment level with intersection-to-union ratio threshold of 50% Score These metrics are used to measure frame-by-frame classification accuracy, temporal sequence consistency of action segments, and segment boundary quality, respectively. The comparison methods are categorized by training type into supervised methods (P4Transformer, PPTr projection pursuit threshold regression, MAMBA4D state-space model), self-supervised methods (P4Trans+C2P point-to-patch, PPTr+C2P point cloud processing network based on Transformer architecture, X4D self-supervised 4D point cloud video understanding model, CrossVideo self-supervised cross-modal contrastive learning method), and static pre-trained adaptation methods (PointATA). The results are shown in Table 4.

[0085] Table 4. HOI4D Point Cloud Video Action Segmentation Results

[0086] As shown in Table 4, the method of the present invention is applied to HOI4D. , and The corresponding figures are 86.7%, 91.9%, and 86.3%, respectively, all higher than the comparison methods in the table. This indicates that continuous-time normalized motion pre-training and shape motion decomposition can enhance the fine-grained temporal understanding of point cloud videos and improve the accuracy of frame-level action label prediction and action segment boundary recognition.

[0087] Specific use case 4: Validation of continuous time state prediction and shape motion decomposition effects.

[0088] This use case employs comparative experiments to verify the effectiveness of continuous-time state prediction and velocity consistency constraints. Under the same pre-training and fine-tuning settings, the results are compared sequentially with no continuous-time state prediction, intermediate state prediction, continuous-time state prediction, and the combined constraint of continuous-time state prediction and velocity consistency. The results are shown in Table 5.

[0089] Table 5. Comparison Results of Continuous-Time State Prediction and Velocity Consistency

[0090] As shown in Table 5, intermediate state prediction is improved compared to continuous-time state prediction. Continuous-time state prediction further improves the recognition accuracy. On this basis, adding velocity consistency constraints achieves the highest results, indicating that arbitrary time state modeling and adjacent state velocity constraints can jointly enhance the continuity of dynamic representation.

[0091] In the shape motion decomposition comparison, under the same point tube embedding and spatiotemporal feature extraction settings, the three feature designs of directly using the original frame-level features, using adjacent frame difference, and using shared shape anchor points to perform shape motion decomposition are compared. The HOI4D frame-level accuracy results are shown in Table 6.

[0092] Table 6 Comparison Results of Shape Motion Decomposition Feature Design

[0093] As shown in Table 6, the difference between adjacent frames can improve the model's sensitivity to dynamic changes. Furthermore, the higher accuracy obtained by constructing shared shape anchors based on representative frames at three different time stages and subtracting the shared shape anchors from the features of each frame indicates that this decomposition method can more effectively suppress the sharing of static shape information across frames.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit its protection scope. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading the present invention, they can still make various changes, modifications or equivalent substitutions to the specific implementation of the application, but these changes, modifications or equivalent substitutions are all within the protection scope of the claims pending approval.

Claims

1. A method for understanding continuous-time standardized motion point cloud videos, characterized in that, Includes the following steps: Step S1: Perform global rigid transformation and local residual displacement construction on the obtained static points to obtain pseudo-dynamic point cloud; Step S2: Based on the pseudo-dynamic point cloud, perform motion prediction and continuous-time state learning, and train the pre-trained network using mask geometric modeling and multimodal semantics to obtain the pre-trained point cloud encoder. Step S3: Using the pre-trained point cloud encoder, extract the frame-level features corresponding to each time frame in the acquired point cloud video. Step S4: Construct shared shape anchor points based on the frame-level features and representative frames at different time stages in the point cloud video; Step S5: Based on each time point, generate motion-aware features according to the frame-level features and the shared shape anchor points; output point cloud video understanding results based on the motion-aware features.

2. The method according to claim 1, characterized in that, Step S1, which describes performing a global rigid transformation and constructing local residual displacements on the acquired static points to obtain a pseudo-dynamic point cloud, includes the following steps: Step S1.1: The acquired static point cloud is centered, scaled, and sampled to obtain a processed static point cloud; the processed static point cloud is used as the normalized state. Step S1.2: Using farthest point sampling, select multiple local point cloud block centers in the normalized state, and use nearest neighbor query to obtain the aggregated neighborhood points of each local point cloud block center in the normalized state, thereby obtaining multiple center local point cloud blocks corresponding to each center; Step S1.3: Generate a rotation matrix based on the randomly sampled axis-angle rotation vector, and perform a global rigid transformation on each of the central local point cloud blocks based on the rotation matrix and the randomly sampled global translation vector to obtain the point cloud after global transformation; Step S1.4: Select multiple local anchoring regions from the standard state, assign corresponding local residual displacements to each point according to the spatial proximity relationship between each point in the standard state and each local anchoring region, and superimpose the assigned local residual displacements onto the point cloud after global transformation to obtain a pseudo-dynamic point cloud. The rotation matrix satisfies the following formula: ； In the above formula, Represents the rotation matrix. Represents matrix exponentiation. Represents the identity matrix. Represents the axis-angle rotation vector antisymmetric matrix, Represents the sine function. The L2 norm of the axis-angle rotation vector is represented. This represents the cosine function.

3. The method according to claim 2, characterized in that, Step S2, which describes performing motion prediction and continuous-time state learning based on the pseudo-dynamic point cloud, and training the pre-trained network using mask geometric modeling and multimodal semantics to obtain a pre-trained point cloud encoder, includes the following steps: Step S2.1: Input the pseudo-dynamic point cloud into the point cloud encoder to obtain coded features; input the coded features into the global motion prediction head to obtain rotation parameter estimates and translation vector estimates; and calculate the global motion prediction loss based on the differences between the rotation parameter estimates and translation vector estimates and the randomly sampled axis angle rotation vector and the sampled global translation vector. The global motion prediction loss satisfies the following formula: ； In the above formula, This represents the global motion prediction loss. This represents the estimated value of the rotation parameters. Indicates rotation parameters, Represents the square of the L2 norm. This represents the learnable weights used to balance rotation and translation errors. This represents the estimated translation vector. Represents the global translation vector. express and The distance between them; Step S2.2: Input the coded markers of the centers of each local point cloud block into the local flow prediction head to obtain the predicted residual displacement of each local point cloud block center, and calculate the local motion prediction loss based on the difference between the predicted residual displacement and the local residual displacement of each local point cloud block center. The local motion prediction loss satisfies the following formula: ； In the above formula, This represents the local motion prediction loss. Represents a set of local dynamic point cloud blocks. This represents the predicted residual displacement at the center of each local point cloud block. This represents the local residual displacement at the center of each local point cloud block. This represents the difference between the predicted residual displacement and the local residual displacement; Step S2.3: Based on the randomly sampled axis-angle rotation vector, global translation vector, and local residual displacement corresponding to each point, generate real pseudo-states corresponding to any continuous time interval between zero and one; input the normalized state and its corresponding time into the state prediction network to obtain the predicted pseudo-state; calculate the state prediction loss based on the difference between the predicted pseudo-state and the real pseudo-state. Step S2.4: Generate real pseudo-states and predicted pseudo-states at adjacent time positions respectively; obtain the real state velocity based on the changes in adjacent real pseudo-states; obtain the predicted state velocity based on the changes in adjacent predicted pseudo-states; calculate the velocity consistency loss based on the difference between the predicted state velocity and the real state velocity, according to the normed state. Step S2.5: Select some local point cloud blocks from each local point cloud block as mask regions, reconstruct the mask regions based on the unmasked local point cloud blocks to obtain reconstructed local point cloud blocks; calculate the mask geometric modeling loss based on the reconstructed local point cloud blocks and the local point cloud blocks in the mask regions. Step S2.6: Extract image features and text features corresponding to the static point cloud using the frozen image encoder and text encoder; calculate the alignment loss of multimodal semantics based on image-related tags, text-related tags, the image features, and the text features. Step S2.7: Update the network parameters of the pre-trained network based on the global motion prediction loss, the local motion prediction loss, the state prediction loss, the velocity consistency loss, the mask geometry modeling loss, and the multimodal semantic alignment loss to obtain the pre-trained point cloud encoder.

4. The method according to claim 3, characterized in that, Step S2.3, which involves generating the true pseudo-state for any consecutive time interval between zero and one based on the randomly sampled axis-angle rotation vector, global translation vector, and local residual displacements corresponding to each point, includes the following steps: Step S2.3.1: Based on the randomly sampled axis-angle rotation vector, global translation vector, and local residual displacement corresponding to each point, sample the time within the time interval from zero to one, and generate the continuous rotation matrix corresponding to the sampling time based on the continuous rotation path from the unit rotation matrix to the rotation matrix. Step S2.3.2: Generate the true pseudo-state corresponding to the sampling time based on the continuous rotation matrix, the normalized state, the global translation vector, and the local residual displacement; The true / false state satisfies the following formula: ； In the above formula, Indicates time The corresponding real and pseudo-state points, Indicates time The corresponding continuous rotation matrix, and satisfying , Indicates the rotation parameters of the sampling. Represents the set of points in the normal state. Indicates time, Represents the global translation vector. Indicates local residual displacement; The predicted pseudo-state satisfies the following formula: ； In the above formula, Indicates time The corresponding predicted pseudo-state, The parameter is A state prediction network.

5. The method according to claim 3, characterized in that, Step S2.4, which involves generating true pseudo-states and predicted pseudo-states at adjacent time positions, obtaining the true state velocity based on the changes in adjacent true pseudo-states, and obtaining the predicted state velocity based on the changes in adjacent predicted pseudo-states, includes the following steps: Step S2.4.1: Based on the method of generating the real pseudo-state and the predicted pseudo-state, obtain the real pseudo-state and the predicted pseudo-state corresponding to two adjacent time positions respectively; Step S2.4.2: Based on the changes and time intervals between the real and pseudo states corresponding to two adjacent time positions, obtain the real state velocity, and based on the changes and time intervals between the predicted pseudo states corresponding to two adjacent time positions, obtain the predicted state velocity. The speed consistency loss satisfies the following formula: ； In the above formula, This represents the speed consistency loss, where N represents the number of points in the normalized state. Indicates the first One point, Indicates time Predicted state speed, Indicates time The actual speed at that point This represents the difference between the predicted state velocity and the actual state velocity.

6. The method according to claim 3, characterized in that, The multimodal semantic alignment loss described in step S2.6 satisfies the following formula: ; In the above formula, Indicates alignment loss. Indicates image-related tags, Representing image features, Indicates the distance between text-related tags. Indicates text-related tags, Represents text features.

7. The method according to claim 3, characterized in that, Step S2.7, which updates the network parameters of the pre-trained network based on the global motion prediction loss, the local motion prediction loss, the state prediction loss, the velocity consistency loss, the mask geometry modeling loss, and the multimodal semantic alignment loss to obtain the pre-trained point cloud encoder, includes the following steps: Step S2.7.1: Based on the global motion prediction loss, the local motion prediction loss, the state prediction loss, the velocity consistency loss, the mask geometry modeling loss, and the multimodal semantic alignment loss, update the network parameters of the pre-trained network with the pre-trained overall function as the objective. Step S2.7.2: After completing the pre-training, retain the point cloud encoder and its pre-training parameters in the pre-trained network to obtain the pre-trained point cloud encoder; The pre-trained overall function satisfies the following formula: ； In the above formula, This indicates the overall goal of pre-training. The weights represent the weights of the global motion prediction loss. This represents the global motion prediction loss. The weights representing the local motion prediction loss, This represents the local motion prediction loss. The weights represent the state prediction loss. This represents the state prediction loss. The weights representing the speed consistency loss This indicates the loss of speed consistency. The weights represent the mask geometry modeling loss. This represents the mask geometry modeling loss. The weights represent the multimodal semantic alignment loss. This represents the multimodal semantic alignment loss.

8. The method according to claim 1, characterized in that, Step S3, which involves using the pre-trained point cloud encoder to extract frame-level features corresponding to each time frame in the acquired point cloud video, includes the following steps: Step S3.1: Using the pre-trained point cloud encoder, divide each time frame in the point cloud video into multiple local point cloud groups; Step S3.2: Encode each local point cloud group to obtain the local markers corresponding to each time frame; Step S3.3: Based on each time frame, aggregate all local tags of each time frame to obtain frame-level features; The local marker corresponding to each time frame satisfies the following formula: ； In the above formula, Indicates the first Frame local markers, This indicates a local point cloud group encoding operation. Indicates the first Frame local point cloud group, Indicates the time frame index.

9. The method according to claim 2, characterized in that, Step S4, which involves constructing shared shape anchors based on each time period, according to the frame-level features and representative frames from different time stages in the point cloud video, includes the following steps: Step S4.1: Select a representative frame from each of the three different time stages of the point cloud video; Step S4.2: Normalize the three representative frames respectively, and input the normalized three representative frames into the shape encoder to obtain three global shape features; Step S4.3: Normalize the three global shape features respectively, and calculate the arithmetic mean of the three normalized global shape features to obtain the average shape feature; Step S4.4: Project the average shape feature onto the feature space where the frame-level feature is located to obtain the shared shape anchor point; The three global shape features satisfy the following formula: ； In the above formula, Indicates the first A global shape feature representing a frame. Indicates shape encoder, This indicates a normalization operation. Indicates time phase The corresponding representative frame, Indicates the first A time phase; The shared shape anchor points satisfy the following formula: ； In the above formula, Indicates a shared shape anchor point. This represents the mapping that projects the average shape features onto the feature space containing the frame-level features. This indicates a feature normalization operation.

10. The method according to claim 1, characterized in that, Step S5, which involves generating motion-aware features based on each time period, according to the frame-level features and the shared shape anchors, and outputting point cloud video understanding results based on the motion-aware features, includes the following steps: Step S5.1: Based on each time, subtract the shared shape anchor point from the frame-level features to obtain the motion-aware features corresponding to the frame; Step S5.2: Perform temporal aggregation on the motion sensing features corresponding to each time frame to obtain the aggregated motion features; Step S5.3: Input the aggregated motion features into the task head and output the point cloud video understanding results; The point cloud video understanding result satisfies the following formula: ； In the above formula, This represents the point cloud video understanding results. The parameter is Task head, This indicates a time aggregation operation. Indicates the first Motion-sensing features of frames Indicates the time frame index. This indicates the number of frames in the point cloud video.

Images