Method and device for identifying point cloud state

Abstract

The invention discloses a point cloud state identification method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the following steps: determining a full-amount optical flow result based on obtained point cloud data and image data at a first moment and a second moment; wherein the first moment is a moment before the second moment; determining a self-vehicle movement optical flow component according to the point cloud data; and performing difference calculation on the full-amount optical flow result and the vehicle movement optical flow component to obtain a target movement optical flow component, and identifying a point cloud state at a second moment based on the target movement optical flow component. According to the embodiment of the invention, the self-vehicle motion and the target motion can be decoupled, the real optical flow information of the target motion is calculated, and the point cloud state is judged based on the real optical flow information, so that the accuracy of point cloud state recognition is improved, and the user experience is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a point cloud state recognition method and device. BACKGROUND

[0002] In the automatic driving scene, point cloud state recognition is an important task, which aims to determine which of the point clouds collected by the laser radar sensor is in a motion state and which is in a static state. At present, the recognition and determination of the dynamic and static state of the point cloud is mainly realized through an image optical flow algorithm. However, due to the double influence of the ego motion and the target motion, the motion information calculated by the conventional image optical flow algorithm is the superposition state of the ego motion and the target motion, and the real dynamic and static state of the target cannot be directly determined, which leads to poor accuracy of point cloud state recognition and reduces the user experience. SUMMARY

[0003] Therefore, the embodiments of the present application provide a point cloud state recognition method and device, which can decouple the ego motion and the target motion, calculate the real optical flow information of the target motion, and determine the point cloud state based on the real optical flow information, thereby improving the accuracy of point cloud state recognition and improving the user experience.

[0004] To achieve the above object, according to an aspect of the embodiments of the present application, a point cloud state recognition method is provided, comprising:

[0005] determining a full optical flow result based on the acquired point cloud data and image data at the first time and the second time; wherein the first time is a time before the second time;

[0006] determining an ego motion optical flow component according to the point cloud data;

[0007] determining a target motion optical flow component by difference calculation of the full optical flow result and the ego motion optical flow component, and recognizing the point cloud state at the second time based on the target motion optical flow component.

[0008] Optionally, the point cloud data comprises three-dimensional points, an inter-frame coordinate transformation matrix and a camera parameter matrix; determining the ego motion optical flow component according to the point cloud data comprises:

[0009] determining the projection position of the three-dimensional points at the first time and the projection position of the three-dimensional points at the second time based on the inter-frame coordinate transformation matrix and the camera parameter matrix;

[0010] calculating the difference between the projection position of the three-dimensional points at the second time and the projection position of the three-dimensional points at the first time to obtain the ego motion optical flow component.

[0011] Optionally, the camera parameter matrix comprises a camera extrinsic parameter matrix and a camera intrinsic parameter matrix; based on the inter-frame coordinate transformation matrix and the camera parameter matrix, the projection position of the three-dimensional point at the first time and the projection position of the three-dimensional point at the second time are determined, comprising:

[0012] based on the camera extrinsic parameter matrix and the coordinate system corresponding to the first time, the three-dimensional point is coordinate-converted to obtain the three-dimensional coordinates of the three-dimensional point at the first time, and based on the three-dimensional coordinates at the first time, the projection position of the three-dimensional point at the first time is determined by using the camera intrinsic parameter matrix;

[0013] based on the inter-frame coordinate transformation matrix, the three-dimensional point is transformed from the coordinate system at the first time to the coordinate system at the second time;

[0014] based on the camera extrinsic parameter matrix and the coordinate system corresponding to the second time, the three-dimensional point is coordinate-converted to obtain the three-dimensional coordinates of the three-dimensional point at the second time, and based on the three-dimensional coordinates at the second time, the projection position of the three-dimensional point at the second time is determined by using the camera intrinsic parameter matrix.

[0015] Optionally, based on the obtained point cloud data and image data at the first time and the second time, the full optical flow result is determined, comprising:

[0016] projecting the point cloud at the first time and the second time onto the corresponding image respectively to obtain a point cloud enhanced image;

[0017] performing optical flow calculation according to the point cloud enhanced image to obtain the full optical flow result.

[0018] Optionally, based on the target motion optical flow component, the state of the point cloud at the second time is identified, comprising:

[0019] back-projecting the target motion optical flow component to the point cloud corresponding point cloud space to obtain the three-dimensional velocity modulus value of the point cloud;

[0020] in response to the three-dimensional velocity modulus value of the point cloud being greater than a preset velocity threshold, determining that the state of the point cloud at the second time is dynamic;

[0021] in response to the three-dimensional velocity modulus value of the point cloud being less than or equal to the preset velocity threshold, determining that the state of the point cloud at the second time is static.

[0022] Optionally, after identifying the state of the point cloud at the second time based on the target motion optical flow component, further comprising:

[0023] generating a point cloud state label, and performing a classification task corresponding to the point cloud based on the point cloud state label.

[0024] According to the second aspect of the embodiment of the application, a point cloud state identification device is provided, comprising:

[0025] The acquisition module is configured to determine a full optical flow result based on the acquired point cloud data and image data at the first time and the second time, wherein the first time is a time before the second time.

[0026] The determination module is configured to determine a self-motion optical flow component according to the point cloud data.

[0027] The recognition module is configured to perform difference calculation on the full optical flow result and the self-motion optical flow component to obtain a target motion optical flow component, and recognize the point cloud state at the second time based on the target motion optical flow component.

[0028] According to a third aspect of the embodiments of the present application, an electronic device is provided, comprising:

[0029] one or more processors;

[0030] a memory configured to store one or more programs,

[0031] when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any of the above embodiments.

[0032] According to a fourth aspect of the embodiments of the present application, a computer readable medium is provided, which stores a computer program, and the program is executed by a processor to implement the method of any of the above embodiments.

[0033] According to a fifth aspect of the embodiments of the present application, a computer program product is provided, which comprises a computer program, and the computer program is executed by a processor to implement the method of any of the above embodiments.

[0034] One of the above embodiments has the following advantages or beneficial effects: by determining a full optical flow result based on the acquired point cloud data and image data at the first time and the second time, wherein the first time is a time before the second time; determining a self-motion optical flow component according to the point cloud data; performing difference calculation on the full optical flow result and the self-motion optical flow component to obtain a target motion optical flow component, and recognizing the point cloud state at the second time based on the target motion optical flow component; thus, the self-motion and the target motion can be decoupled, the real optical flow information of the target motion is calculated, and the point cloud state is determined, thereby improving the accuracy of point cloud state recognition and enhancing the user experience.

[0035] The further effects of the above non-conventional optional mode will be described in the following combined with the specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings are used to better understand the present application, and do not constitute an improper limitation on the present application. Among them:

[0037] Figure 1is a schematic diagram of the main flow of a point cloud state recognition method according to an embodiment of the present application;

[0038] Figure 2 is a schematic diagram of the main flow of a point cloud state recognition method according to an optional embodiment of the present application;

[0039] Figure 3 is a schematic diagram of the main modules of a point cloud state recognition device according to an embodiment of the present application;

[0040] Figure 4 is an exemplary system architecture diagram to which embodiments of the present application can be applied;

[0041] Figure 5 is a structural schematic diagram of a computer system of a terminal device or a server suitable for implementing embodiments of the present application. DETAILED DESCRIPTION

[0042] Exemplary embodiments of the present application are described below with reference to the accompanying drawings, which include various details of the embodiments of the present application to assist in understanding, which should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Also, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0043] It should be noted that the acquisition, storage and application of personal information and the like involved in embodiments of the present application comply with relevant laws and regulations and do not violate public order and good customs.

[0044] In the autonomous driving scenario, point cloud state recognition is an important task, which aims to determine which of the point clouds collected by the laser radar sensor are in motion and which are in a static state. At present, the recognition and determination of point cloud dynamic and static states are mainly achieved through image optical flow algorithm. However, due to the double influence of the ego motion and the target motion, the motion information calculated by the conventional image optical flow algorithm is the superposition state of the ego motion and the target motion, which cannot directly determine the true dynamic and static state of the target, resulting in poor accuracy of point cloud state recognition and reducing user experience.

[0045] Therefore, according to an aspect of an embodiment of the present application, a point cloud state recognition method is provided.

[0046] Figure 1 is a schematic diagram of the main flow of a point cloud state recognition method according to an embodiment of the present application. As shown in Figure 1 The point cloud state recognition method according to an embodiment of the present application includes the following steps S101 to S103.

[0047] In step S101, based on the obtained point cloud data and image data at the first time and the second time, a full-quantity optical flow result is determined; wherein the first time is a time before the second time.

[0048] The first time and the second time are two time nodes that are adjacent or spaced on a time axis, wherein the first time is a time before the second time. Specifically, the first time and the second time can be two adjacent continuous time nodes on the time axis, for example, the collection time t-1 and t of the adjacent two frames of point cloud and image data. In this way, the disparity between the point cloud and the image is small, the scene changes continuously and smoothly, the matching error and the calculation complexity can be effectively reduced, and it is suitable for vehicle perception scenes with high real-time requirements, such as continuous frame motion estimation of an automatic driving system. In another way, the first time and the second time can be two collection nodes that are spaced on the time axis, for example, t-1 and t+Δt after being spaced by several frames. At this time, the pose difference between the point cloud and the image data is more significant, which can capture more range of environmental changes and dynamic target motion information, enhance the sensitivity of the optical flow result to long-term motion trends, and improve the robustness and environmental understanding ability of the system in a large range of scenes or low frame rate perception conditions.

[0049] The first time and the second time can be the time of point cloud collection or the time of image collection, and a unified time reference is used and alignment processing is performed to ensure spatio-temporal consistency. Specifically, the point cloud and the image share the same timestamp, and in an implementation in which the point cloud timestamp is used as a reference, the image data is aligned to the point cloud timestamp. The alignment method can use hardware triggering to synchronize the laser radar and the camera for synchronous collection, or use interpolation or nearest neighbor matching based on the respective timestamps to align the collected image to the point cloud time, or use time interpolation based on the vehicle body pose to transform the image or the point cloud in the time dimension to eliminate the time deviation. For example, the first time can be a time point before the current time, denoted as t-1; the second time can be a current time point after the first time, denoted as t. In this embodiment, the point cloud data and the image data at the first time are collected at the time t-1, and the point cloud data and the image data at the second time are collected at the time t. The point cloud data is used to represent the spatial position and structural features of objects in the environment at the time, and the image data provides two-dimensional visual information of the scene.

[0050] The determination of the full optical flow result according to the point cloud data and the image data can be achieved by an image optical flow algorithm, such as Lucas-Kanade optical flow algorithm. Alternatively, a neural network-based optical flow model can also be used, which learns the spatial feature correspondence between images through a neural network and outputs a dense optical flow map to obtain the full optical flow result. The full optical flow result is a displacement vector of each pixel on the image plane calculated between the first time and the second time, including background changes caused by the motion of the ego vehicle and changes caused by the target itself in the scene. Specifically, the point cloud data at the first time and the second time is obtained, and the two frames of point cloud are converted to the camera coordinate system through the extrinsic matrix; the above three-dimensional point cloud is projected to the image plane at the corresponding time by using the intrinsic matrix of the camera to generate a point cloud enhanced image; the point cloud enhanced image is input into the optical flow calculation component, and the Lucas-Kanade optical flow algorithm is used to locally fit and calculate the pixel gray scale changes of adjacent frames to estimate the motion displacement vector of each pixel, thereby obtaining the full optical flow result. In another implementation, the point cloud at the first time and the second time is also projected into a point cloud enhanced image, and the point cloud enhanced image pair is normalized; the normalized two frames of point cloud enhanced images are input into the optical flow model, and the optical flow model predicts the motion displacement vector of each pixel in the point cloud enhanced image in the time dimension through feature extraction, correlation matching and multi-scale aggregation, and outputs the full optical flow result.

[0051] In step S102, the ego motion optical flow component is determined according to the point cloud data.

[0052] The ego motion optical flow component refers to the translation effect of the pixel points in the visual image caused by the movement or attitude change of the vehicle between the two times, which is specifically manifested as a two-dimensional displacement vector formed by the change of the projection position of a three-dimensional point in the point cloud on the image plane over time.

[0053] In this embodiment, the determination of the vehicle motion optical flow component based on point cloud data can be achieved using a three-dimensional geometric transformation calculation method. This method utilizes the LiDAR point cloud, inter-frame coordinate transformation matrix, and camera parameter matrix to calculate the position difference of a 3D point projected onto the image plane at two different time points. Specifically, the 3D point at the first time point is transformed to the camera coordinate system using the camera's extrinsic parameter matrix, and then projected using the camera's intrinsic parameter matrix to obtain the position of the 3D point in the image at the first time point. The inter-frame coordinate transformation matrix is ​​then used to transform the 3D point from the first time point to the second time point coordinate system, and the same projection process is used to obtain its position in the image at the second time point. Subtracting the two values ​​gives the image plane displacement caused by the vehicle motion, forming the vehicle motion optical flow component. Alternatively, an image space inverse projection method can be used. First, pixels are selected on the image plane, and combined with the depth map or sparse depth information provided by the LiDAR, these pixels are inversely projected into 3D points at the first time point. These 3D points are then transformed to the second time point coordinate system using the inter-frame coordinate transformation matrix, and then projected back onto the image plane to obtain the corresponding image position at the second time point. The difference in projection position between two frames is used to represent the optical flow component of the vehicle's motion. Both methods described above use spatiotemporal transformation to track changes in the point cloud projection position to remove the influence of vehicle motion, providing an accurate reference for extracting the true optical flow of the target.

[0054] Step S103: Calculate the difference between the full optical flow result and the vehicle motion optical flow component to obtain the target motion optical flow component, and identify the point cloud state at the second moment based on the target motion optical flow component.

[0055] The target motion optical flow component refers to the motion changes of pixels on the image plane caused by moving objects in the environment (such as pedestrians, cars, bicycles, etc.) after excluding the vehicle's own motion. In this embodiment, the difference between the full optical flow result and the vehicle motion optical flow component is first calculated to remove background optical flow interference caused by the vehicle's own motion. Specifically, based on dense optical flow, vector subtraction can be performed directly on the image plane. That is, at the pixel position of the LiDAR point cloud projection, the dense full optical flow result calculated by the image optical flow algorithm is extracted, and a vector difference operation is performed with the vehicle motion optical flow component calculated through geometric transformation at that pixel position, retaining the difference as the target motion optical flow component. Alternatively, based on sparse point cloud, for each pixel with LiDAR point projection, its full motion vector and vehicle motion vector in the image optical flow are calculated separately, and point-by-point difference calculation is performed at the point cloud level, finally mapping to a motion vector on the image to form the target motion optical flow component.

[0056] Point cloud state refers to the judgment result of whether the physical object to which each point in the 3D point cloud belongs is in motion at the current moment. It can be divided into two categories: static points and dynamic points. In this embodiment, the current moment can be understood as the second moment mentioned above. The process of identifying the point cloud state is to associate the target motion optical flow component with the point cloud point by point and determine whether each 3D point is in motion. Specifically, a threshold rule-based judgment method can be used, setting a modulus threshold or direction change threshold for the motion optical flow. If the target motion optical flow component corresponding to a certain 3D point exceeds the modulus threshold or direction change threshold, it is judged as a dynamic point; otherwise, it is judged as a static point. This method is simple to calculate and has strong real-time performance. Alternatively, a machine learning model or deep neural network method can be used, taking the target motion optical flow component as one of the input features, and combining it with multimodal information such as point cloud intensity, depth change, and time series to train a point-level classification model. The point-level classification model can then be used to achieve point cloud motion and static identification.

[0057] This invention determines the full optical flow result based on point cloud data and image data acquired at the first and second time points; determines the vehicle motion optical flow component based on the point cloud data; calculates the difference between the full optical flow result and the vehicle motion optical flow component to obtain the target motion optical flow component; and identifies the point cloud state at the second time point based on the target motion optical flow component. This decouples the vehicle motion from the target motion, calculates the true optical flow information of the target motion, and uses this information to determine the point cloud state, improving the accuracy of point cloud state identification and enhancing the user experience.

[0058] Figure 2 This is a schematic diagram of the main flow of a point cloud state recognition method according to an optional embodiment of the present invention. The point cloud data includes 3D points, inter-frame coordinate transformation matrices, and camera parameter matrices. For example... Figure 2 As shown, the optical flow component of the vehicle motion is determined based on the point cloud data, including the following steps S201 to S202.

[0059] Step S201: Based on the inter-frame coordinate transformation matrix and the camera parameter matrix, determine the projection position of the 3D point at the first time moment and the projection position at the second time moment.

[0060] Step S202: Calculate the difference between the projection position of the three-dimensional point at the second time and the projection position at the first time to obtain the optical flow component of the vehicle motion.

[0061] In this embodiment, a mapping relationship between three-dimensional space and the image plane is established by acquiring point cloud data, including 3D points, inter-frame coordinate transformation matrices, and camera parameter matrices. Specifically, the 3D points are transformed from the LiDAR coordinate system to the camera coordinate system using the camera extrinsic parameter matrix, and then projected onto the image plane at the first moment using the camera intrinsic parameter matrix, thereby obtaining the projected position of the 3D points at the first moment. Combining the vehicle attitude change represented by the inter-frame coordinate transformation matrix, the position of the 3D points at the second moment is transformed and projected, thereby obtaining its projected position in the image at the second moment. By calculating the difference between the coordinates of the projected positions at the two moments, the displacement result of the 3D points on the image plane caused by the vehicle motion can be obtained, i.e., the vehicle motion optical flow component. Alternatively, a self-supervised estimation model can be constructed by fusing the images from the first and second moments with the LiDAR data and a LiDAR learning network. Guided by existing high-precision maps or pseudo-labels, the distribution trend of the vehicle motion optical flow can be indirectly learned to obtain the vehicle motion optical flow component. By obtaining the optical flow component of the vehicle motion using the above method, the background motion information caused by the vehicle's movement can be effectively removed, providing a more accurate motion reference for tasks such as dynamic object detection, point cloud dynamic and static discrimination, semantic segmentation, and path planning, thereby enhancing the overall system's perception stability and decision-making accuracy in complex traffic environments.

[0062] Optionally, the camera parameter matrix includes a camera extrinsic parameter matrix and a camera intrinsic parameter matrix. Based on the inter-frame coordinate transformation matrix and the camera parameter matrix, determining the projection position of the 3D point at the first time step and the projection position at the second time step includes: transforming the coordinates of the 3D point based on the camera extrinsic parameter matrix and the coordinate system corresponding to the first time step to obtain the 3D coordinates of the 3D point at the first time step; determining the projection position of the 3D point at the first time step using the camera intrinsic parameter matrix based on the 3D coordinates of the first time step; transforming the 3D point from the coordinate system of the first time step to the coordinate system of the second time step based on the inter-frame coordinate transformation matrix; transforming the coordinates of the 3D point based on the camera extrinsic parameter matrix and the coordinate system corresponding to the second time step to obtain the 3D coordinates of the 3D point at the second time step; determining the projection position of the 3D point at the second time step using the camera intrinsic parameter matrix based on the 3D coordinates of the second time step.

[0063] The camera extrinsic matrix is ​​used to realize the coordinate transformation between the LiDAR coordinate system and the camera coordinate system; the camera intrinsic matrix is ​​used to realize the projection from the camera coordinate system to the image plane. Specifically, based on the camera extrinsic matrix, the 3D points in the point cloud are transformed from the LiDAR coordinate system to the camera coordinate system at the first moment, obtaining the camera 3D coordinates of the 3D points at the first moment. The camera intrinsic matrix is ​​then used to project these 3D coordinates onto the image plane, thus obtaining the image plane projection position of the 3D points at the first moment. The inter-frame coordinate transformation matrix is ​​used to transform the coordinates of the 3D points in the time dimension, mapping them from the coordinate system at the first moment to the coordinate system at the second moment, thus obtaining their spatial position at the second moment. Similarly, based on the camera extrinsic matrix and the coordinate system corresponding to the second moment, the coordinates of the 3D points are transformed to obtain their camera 3D coordinates at the second moment, and then projected onto the image plane using the camera intrinsic matrix, thus obtaining the image projection position at the second moment. By calculating the difference between the projection positions at the second moment and the first moment, the displacement of the 3D points on the image plane can be obtained, and thus the optical flow component of the vehicle motion can be obtained. In addition to the geometric mapping derivation method mentioned above, an end-to-end learning method can also be introduced. By inputting point cloud data and image frame pairs, the sparse optical flow features caused by the vehicle's motion are predicted with the help of an optical flow network structure. Then, attention mechanism is combined to focus on learning the regions in the image that overlap with the point cloud.

[0064] The optical flow components of vehicle motion derived in this embodiment based on the camera parameter matrix and the inter-frame coordinate transformation matrix have physical interpretability under geometric constraints. They can accurately eliminate background changes caused by vehicle motion and provide high-quality, sparse, and reliable optical flow references for downstream modules such as dynamic obstacle recognition, perception fusion, and path planning.

[0065] Optionally, based on the point cloud data and image data acquired at the first and second time points, the full optical flow result is determined, including: projecting the point clouds at the first and second time points onto the corresponding images to obtain point cloud enhanced images; and performing optical flow calculations based on the point cloud enhanced images to obtain the full optical flow result.

[0066] Based on the point cloud data and image data acquired at the first and second time points, an enhanced point cloud image incorporating spatial geometric information is constructed. Specifically, the point cloud data corresponding to the first and second time points are projected onto the image plane in three-dimensional space based on the camera extrinsic and intrinsic parameter matrices, respectively. This marks the spatial structure information obtained by LiDAR scanning on the image, forming a point cloud enhanced image with depth perception capabilities. This enhanced point cloud image not only contains the texture information of the original image but also integrates spatial features such as depth or density provided by the point cloud, making the image more sensitive to dynamic changes. After obtaining the enhanced point cloud image, an image optical flow algorithm or optical flow model can be used to perform registration calculations on the enhanced point cloud images at the two time points. Since spatial depth information has been introduced into the image, it is beneficial to enhance the model's perception capabilities of moving boundaries, occluded areas, and highly dynamic targets, thereby improving the accuracy and robustness of optical flow calculations and ultimately obtaining the full optical flow result. In addition to the methods mentioned above, point cloud data can be first converted into depth maps or sparse voxel maps, and then trained end-to-end with image information through a multimodal fusion network. This method can fuse information from different modalities within a unified neural network framework.

[0067] The point cloud enhancement image strategy adopted in this embodiment effectively supplements the shortcomings of traditional images in areas with changes in illumination and single texture. The obtained full optical flow results have stronger dynamic responsiveness and edge accuracy, providing a more reliable input data foundation for subsequent perception tasks such as vehicle motion decoupling, target detection, and dynamic mapping.

[0068] Optionally, identifying the point cloud state at the second moment based on the target motion optical flow component includes: back-projecting the target motion optical flow component onto the point cloud space corresponding to the point cloud to obtain the three-dimensional velocity magnitude of the point cloud; determining that the point cloud state at the second moment is dynamic in response to the three-dimensional velocity magnitude of the point cloud being greater than a preset velocity threshold; and determining that the point cloud state at the second moment is static in response to the three-dimensional velocity magnitude of the point cloud being less than or equal to the preset velocity threshold.

[0069] The target motion optical flow component obtained through interpolation is back-projected into the 3D point cloud space. This involves using the inverse process of the camera parameter matrix to reconstruct the target motion optical flow vector on the image plane into the spatial motion vector corresponding to the 3D point cloud, thereby obtaining the velocity information of each 3D point in 3D space. Specifically, by combining the camera intrinsic and extrinsic parameter matrices, the velocity direction and magnitude of each point cloud are derived using a projection model, thus achieving motion estimation reconstruction from image space to point cloud space. A reasonable velocity threshold is set as the criterion for distinguishing between static and dynamic point clouds. When the velocity magnitude of a point cloud in 3D space is greater than the preset velocity threshold, it indicates that the point cloud has undergone a positional change over time, and is therefore classified as a dynamic point; conversely, when the velocity magnitude is less than or equal to the velocity threshold, the point cloud is classified as a static point. In this way, moving target objects, such as pedestrians, vehicles, and animals, can be identified in dense point clouds and distinguished from static structures such as the ground and buildings. Besides the velocity threshold method, discrimination methods based on cluster analysis or temporal consistency modeling can also be used. For example, the velocity vector field can be clustered to analyze whether the point cloud within a certain region exhibits a consistent motion trend, thus determining whether it is a dynamic target; or multi-frame point cloud historical information can be used in conjunction with a state transition probability model for more stable dynamic determination. This embodiment achieves accurate identification of the point cloud state through the fusion analysis of the target motion optical flow component and point cloud data.

[0070] Optionally, after identifying the point cloud state at the second moment based on the target motion optical flow component, the method further includes: generating point cloud state labels and performing the corresponding classification task based on the point cloud state labels.

[0071] In this embodiment, based on the completion of dynamic or static state discrimination for each point cloud, this state information is mapped into structured point cloud state labels. For example, dynamic or static points are used as category markers and appended to the attribute information of the point cloud data to form a point cloud dataset with motion attribute annotations. A classification task is then performed using the point cloud data with these state labels. The classification task can perform fine-grained distinctions based on different target types, structural attributes, or functional regions. For example, in an autonomous driving environment, dynamic points such as moving vehicles, cyclists, or pedestrians can be identified and classified based on dynamic state labels, and static structures such as roads, buildings, and road signs can be further classified based on static state labels. The above process can be performed using feature engineering methods or combined with deep learning models, such as point cloud classification networks, using input features fused with state labels for end-to-end training and inference. Besides directly generating labels and participating in classification, techniques such as weakly supervised learning and pseudo-label generation can be combined to extend some state labels to unlabeled data, further improving classification accuracy and generalization ability.

[0072] By adding point cloud state labels to point cloud state recognition and guiding the classification task, not only can the accuracy and efficiency of point cloud classification be improved, but the entire perception system can also have a stronger scene understanding ability and behavior prediction ability, especially in complex traffic environments and multi-object scenarios, it can demonstrate a higher level of intelligent perception.

[0073] According to a preferred embodiment of the present invention, the vehicle motion is first decoupled from the target motion, and the optical flow component representing the target motion is separated from the full optical flow result. Then, this target motion optical flow component is back-projected into three-dimensional space as an important basis for judging the dynamic and static status of the point cloud. The two processes are highly unified, which can conveniently and quickly obtain high-quality point cloud dynamic and static classification results.

[0074] Specifically, two consecutive point cloud frames at times t-1 and t are projected onto their corresponding images to obtain an enhanced point cloud image. The full optical flow result F(u,v) is then obtained through the optical flow module. t-1， Where u and v are pixel coordinates (horizontal and vertical coordinates) on the image plane. The optical flow module can specifically employ the Lucas-Canade algorithm, the Licas-Kanade algorithm, or a neural network-based optical flow model. This embodiment does not limit the specific algorithm used.

[0075] As mentioned earlier, the total optical flow result F(u,v) t-1 It includes the effects of both the vehicle's motion and the target's motion. For example, when the vehicle and the target are moving forward at the same speed, the target's position in the image remains essentially unchanged, and the corresponding total optical flow result will be approximately 0, indicating that the target is not moving. However, in reality, a total optical flow result of 0 only indicates that the target is not moving relative to the vehicle (i.e., the relative velocity is 0), and does not reflect the target's true motion state.

[0076] Therefore, it is necessary to first calculate the optical flow component of the vehicle's motion. This calculation is based on the point cloud, the inter-frame coordinate transformation matrix, and the camera parameter matrix. The specific calculation formula is F. ego (u,v) t-1 = KT L->C T t-1->t p i,t-1 – KT L->C p i,t-1 Among them, p i,t-1 Let i be a 3D point in the lidar point cloud, and let T be the i-th 3D point. L->C This is the camera extrinsic matrix, where L represents Lidar, C represents Camera, and K is the camera intrinsic matrix, KT L->C p i,t-1 This represents the position of the 3D point projected onto the camera plane in the coordinate system at the current time (t-1), where T is the position of the point.t-1->t Let KT be the coordinate transformation matrix from time t-1 to time t. L-> C T t-1->t p i,t-1 This represents the position of the 3D point projected onto the camera plane in the coordinate system at the next time step (t). Subtracting the two gives the displacement of the 3D point in the image plane caused by the vehicle's motion, i.e., the optical flow component F of the vehicle's motion. ego (u,v) t-1 Subtracting the vehicle's motion optical flow component from the total optical flow result yields the decoded target motion optical flow component, i.e., F. obj (u,v) t-1 =F(u,v) t-1 –F ego (u,v) t-1 It is worth noting that F(u,v) t-1 It is calculated using an image optical flow algorithm, with optical flow results for each pixel, resulting in a dense output; while F ego (u,v) t-1 Calculations can only be performed on areas scanned by lidar point clouds, therefore the results are sparse; the subtraction of the two only occurs at F... ego (u,v) t-1 Calculations are performed where values ​​are available. Subsequently, the 3D points in the original point cloud can be queried in relation to F. obj (u,v) t-1 The optical flow results at the corresponding locations are used to determine the dynamics of the point cloud, resulting in the final dynamics assessment result. This point cloud dynamics result can serve as crucial information for downstream autonomous driving modules such as fusion tracking and control decision-making.

[0077] According to a second aspect of the present invention, a point cloud state recognition device is provided.

[0078] Figure 3 This is a schematic diagram of the main modules of a point cloud state recognition device according to an embodiment of the present invention. Figure 3 As shown, the point cloud state recognition device 300 includes:

[0079] The acquisition module 301 is used to determine the full optical flow result based on the point cloud data and image data acquired at the first and second time points, wherein the first time point is the time point before the second time point;

[0080] The determination module 302 is used to determine the optical flow components of the vehicle motion based on the point cloud data;

[0081] The identification module 303 is used to calculate the difference between the full optical flow result and the vehicle motion optical flow component to obtain the target motion optical flow component, and to identify the point cloud state at the second moment based on the target motion optical flow component.

[0082] Optionally, the point cloud data includes 3D points, inter-frame coordinate transformation matrices, and camera parameter matrices; the determination module 302 is also used for:

[0083] Based on the inter-frame coordinate transformation matrix and the camera parameter matrix, the projection positions of the 3D points at the first time moment and the projection positions at the second time moment are determined.

[0084] The difference between the projected position of the three-dimensional point at the second moment and the projected position at the first moment is calculated to obtain the optical flow component of the vehicle motion.

[0085] Optionally, the camera parameter matrix includes a camera extrinsic parameter matrix and a camera intrinsic parameter matrix; the determination module 302 is also used for:

[0086] Based on the camera extrinsic matrix and the coordinate system corresponding to the first moment, the coordinates of the three-dimensional point are transformed to obtain the three-dimensional coordinates of the three-dimensional point at the first moment. Based on the three-dimensional coordinates at the first moment, the projection position of the three-dimensional point at the first moment is determined using the camera intrinsic matrix.

[0087] Based on the inter-frame coordinate transformation matrix, the coordinate system of the three-dimensional point is transformed from the coordinate system of the first time step to the coordinate system of the second time step;

[0088] Based on the camera extrinsic matrix and the coordinate system corresponding to the second time step, the coordinates of the 3D point are transformed to obtain the 3D coordinates of the 3D point at the second time step. Based on the 3D coordinates at the second time step, the projection position of the 3D point at the second time step is determined using the camera intrinsic matrix.

[0089] Optionally, the acquisition module 301 is also used for:

[0090] The point clouds at the first and second time points are projected onto the corresponding images to obtain point cloud augmented images.

[0091] Optical flow is calculated based on the point cloud augmented image to obtain the full optical flow result.

[0092] Optionally, the identification module 303 is also used for:

[0093] The optical flow component of the target motion is back-projected onto the point cloud space corresponding to the point cloud to obtain the three-dimensional velocity modulus of the point cloud;

[0094] If the three-dimensional velocity modulus of the point cloud is greater than a preset velocity threshold, the point cloud state at the second moment is determined to be dynamic.

[0095] If the three-dimensional velocity modulus of the point cloud is less than or equal to a preset velocity threshold, the point cloud state at the second moment is determined to be static.

[0096] Optionally, the identification device 300 further includes a generation module, which is used for:

[0097] Generate point cloud state labels, and perform the corresponding classification task based on the point cloud state labels.

[0098] It should be noted that the specific implementation of the point cloud state recognition device of the present invention has been described in detail in the point cloud state recognition method above, so the content will not be repeated here.

[0099] According to a third aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the first aspect of the present invention.

[0100] According to a fourth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method provided in the first aspect of the present invention.

[0101] According to a fifth aspect of the present invention, a computer program product is provided, including a computer program that, when executed by a processor, implements the method provided in the first aspect of the present invention.

[0102] Figure 4 An exemplary system architecture 400 is shown, in which the point cloud state identification method or point cloud state identification device of the present invention can be applied.

[0103] like Figure 4 As shown, system architecture 400 may include terminal devices 401, 402, and 403, a network 404, and a server 405. Network 404 serves as the medium for providing communication links between terminal devices 401, 402, and 403 and server 405. Network 404 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0104] Users can use terminal devices 401, 402, and 403 to interact with server 405 via network 404 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 401, 402, and 403, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0105] Terminal devices 401, 402, and 403 can be various electronic devices with displays that support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0106] Server 405 can be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 401, 402, and 403 (for example only). The backend management server can analyze and process data such as received point cloud status recognition requests, and feed back the processing results (e.g., point cloud status - for example only) to the terminal devices.

[0107] It should be noted that the point cloud state recognition method provided in this embodiment of the invention is generally run by server 405, and correspondingly, the point cloud state recognition device is generally set in server 405.

[0108] It should be understood that Figure 4 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0109] The following is for reference. Figure 5 It shows a schematic diagram of the structure of a computer system 500 suitable for implementing a terminal device of the present invention. Figure 5 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0110] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 502 or programs loaded from storage section 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the system 500. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0111] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. Drive 510 is also connected to I / O interface 505 as needed. Removable media 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 510 as needed so that computer programs read from them can be installed into storage section 508 as needed.

[0112] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is run by the central processing unit (CPU) 501, it performs the functions defined above in the system of this invention.

[0113] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0114] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more operable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually operate substantially in parallel, and they may sometimes operate in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0115] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including an acquisition module, a determination module, and an identification module. The names of these modules do not necessarily limit the module itself; for example, the determination module can also be described as "a module for determining the optical flow components of vehicle motion based on point cloud data."

[0116] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to include: determining a full optical flow result based on point cloud data and image data acquired at a first time and a second time; wherein the first time is a time prior to the second time; determining a vehicle motion optical flow component based on the point cloud data; calculating the difference between the full optical flow result and the vehicle motion optical flow component to obtain a target motion optical flow component; and identifying the point cloud state at the second time based on the target motion optical flow component.

[0117] The computer program product provided in this embodiment of the invention includes a computer program that, when executed by a processor, implements the point cloud state recognition method in this embodiment of the invention.

[0118] According to the technical solution of the present invention, the following advantages or beneficial effects are achieved: A full optical flow result is determined based on point cloud data and image data acquired at a first and second time point; wherein the first time point is a time prior to the second time point; the vehicle motion optical flow component is determined based on the point cloud data; the difference between the full optical flow result and the vehicle motion optical flow component is calculated to obtain the target motion optical flow component; and the point cloud state at the second time point is identified based on the target motion optical flow component. This decouples the vehicle motion from the target motion, calculates the true optical flow information of the target motion, and uses this information to determine the point cloud state, thereby improving the accuracy of point cloud state identification and enhancing the user experience.

[0119] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

[0120] It should be noted that the acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

Claims

1. A method for identifying the state of a point cloud, characterized in that, include: Based on the point cloud data and image data acquired at the first and second time points, the full optical flow result is determined; wherein, the first time point is the time point before the second time point; The optical flow components of the vehicle motion are determined based on the point cloud data. The difference between the full optical flow result and the vehicle motion optical flow component is calculated to obtain the target motion optical flow component, and the point cloud state at the second moment is identified based on the target motion optical flow component.

2. The method according to claim 1, characterized in that, The point cloud data includes 3D points, inter-frame coordinate transformation matrices, and camera parameter matrices; the vehicle motion optical flow components are determined based on the point cloud data, including: Based on the inter-frame coordinate transformation matrix and camera parameter matrix, the projection position of the 3D point at the first moment and the projection position at the second moment are determined. The difference between the projected position of the three-dimensional point at the second moment and the projected position at the first moment is calculated to obtain the optical flow component of the vehicle motion.

3. The method according to claim 2, characterized in that, The camera parameter matrix includes a camera extrinsic parameter matrix and a camera intrinsic parameter matrix; Based on the inter-frame coordinate transformation matrix and camera parameter matrix, the projection positions of the 3D points at the first time moment and the projection positions at the second time moment are determined, including: Based on the camera extrinsic matrix and the coordinate system corresponding to the first time moment, the coordinates of the three-dimensional point are transformed to obtain the three-dimensional coordinates of the three-dimensional point at the first time moment. Based on the three-dimensional coordinates at the first time moment, the projection position of the three-dimensional point at the first time moment is determined using the camera intrinsic matrix. Based on the inter-frame coordinate transformation matrix, the three-dimensional point is transformed from the coordinate system at the first time moment to the coordinate system at the second time moment; Based on the camera extrinsic matrix and the coordinate system corresponding to the second time step, the coordinates of the three-dimensional point are transformed to obtain the three-dimensional coordinates of the three-dimensional point at the second time step. Based on the three-dimensional coordinates at the second time step, the projection position of the three-dimensional point at the second time step is determined using the camera intrinsic matrix.

4. The method according to claim 1, characterized in that, Based on the point cloud data and image data acquired at the first and second time points, the full optical flow results are determined, including: The point clouds at the first and second time points are projected onto the corresponding images to obtain point cloud enhanced images; Optical flow calculations are performed on the enhanced point cloud image to obtain the full optical flow result.

5. The method according to claim 1, characterized in that, Identifying the point cloud state at the second moment based on the target motion optical flow component includes: The target motion optical flow component is back-projected onto the point cloud space corresponding to the point cloud to obtain the three-dimensional velocity modulus of the point cloud. In response to the point cloud's three-dimensional velocity modulus being greater than a preset velocity threshold, the point cloud state at the second moment is determined to be dynamic; In response to the point cloud's three-dimensional velocity modulus being less than or equal to a preset velocity threshold, the point cloud state at the second moment is determined to be static.

6. The method according to claim 1, characterized in that, After identifying the point cloud state at the second moment based on the target motion optical flow component, the method further includes: Generate point cloud state labels, and perform the corresponding classification task based on the point cloud state labels.

7. A point cloud state recognition device, characterized in that, include: The acquisition module is used to determine the full optical flow result based on the point cloud data and image data acquired at the first and second time points; wherein, the first time point is the time point before the second time point; The determination module is used to determine the optical flow components of the vehicle motion based on the point cloud data; The identification module is used to calculate the difference between the full optical flow result and the vehicle motion optical flow component to obtain the target motion optical flow component, and to identify the point cloud state at the second moment based on the target motion optical flow component.

8. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images