Method and system for self-supervised learning of body motion for autonomous driving

Abstract

The present invention provides a method, apparatus, storage medium, and electronic device for self-supervised learning, wherein the method comprises: organizing a column of real points in a vertical direction into a body column, wherein the body column is provided with body column motion parameters, and each real point in the body column has motion parameters identical to the body column motion parameters; determining a predicted point for the next frame; determining a first loss term based on the minimum distance between the predicted point of the next frame and the real point of the next frame, and generating a loss function including the first loss term; and performing self-supervised learning processing according to the loss function. The method, apparatus, storage medium, and electronic device for self-supervised learning provided by the embodiments of the present invention determine body column motion parameters that can represent the motion of real points using body columns as units, thereby enhancing the association between point clouds. Self-supervised learning can also be achieved when there is no precise correspondence between predicted points and real points, and training can be performed using a large number of unlabeled point clouds.

Description

Technical Field

[0001] The present invention relates to the technical field of self-supervised learning, and in particular to a method, device, storage medium and electronic device for self-supervised learning. Background Art

[0002] Understanding and predicting the motion of various behavioral objects (such as vehicles and pedestrians) is crucial for autonomous vehicles to operate safely in dynamic environments. Autonomous vehicles are typically equipped with multiple sensors, and the most commonly used sensor is LiDAR. Therefore, how to estimate motion based on point clouds collected by LiDAR is one of the fundamental research problems in autonomous driving. However, this is challenging in the following aspects: (1) there are many behavioral object categories, and each category exhibits specific motion behaviors; (2) point clouds are sparse and lack precise correspondence between scans.

[0003] Current motion estimation methods typically require a large amount of annotated training data from autonomous driving scenarios. However, manually labeling point clouds is notoriously difficult, error-prone, and time-consuming. While self-supervised learning for language and vision has gained increasing attention in recent years, self-supervision for point clouds still lags behind.

[0004] Another active research direction is to estimate scene flow from point clouds to understand dense 3D motion fields. However, current methods typically take hundreds of milliseconds to process a point cloud, even if the point cloud is significantly downsampled. In addition, this method is effective for synthetic data with precise correspondences (e.g., FlyingThings3D) or densely processed data (e.g., KITTI scene flow), but for raw point clouds scanned by lidar, there is usually no correspondence between points in different frames, making it difficult to directly estimate scene flow from lidar. Summary of the Invention

[0005] In order to solve the problem that point clouds are difficult to implement self-supervised learning, the purpose of the embodiments of the present invention is to provide a method, device, storage medium and electronic device for self-supervised learning.

[0006] In a first aspect, an embodiment of the present invention provides a method for self-supervised learning, comprising:

[0007] Acquire an unlabeled data set, wherein the data set includes multiple frames of point clouds, and each frame of the point cloud includes multiple real points;

[0008] Organizing a row of the real points in a vertical direction into a body column, wherein the body column is provided with body column motion parameters, and each of the real points in the body column has motion parameters identical to the body column motion parameters;

[0009] Moving each of the real points of the current frame to the next frame according to the corresponding body column motion parameters, and determining a prediction point of the next frame;

[0010] Determining a first loss term according to a minimum distance between the predicted point of the next frame and the true point of the next frame, and generating a loss function including the first loss term;

[0011] A self-supervised learning process is performed according to the loss function to determine body column motion parameters of the body column.

[0012] In a possible implementation, determining the first loss term according to the minimum distance between the predicted point of the next frame and the real point of the next frame includes:

[0013] The first loss item is determined according to the first minimum distance and / or the second minimum distance, wherein the first minimum distance is the minimum value of the distance between the predicted point of the next frame and the multiple real points of the next frame, and the second minimum distance is the minimum value of the distance between the real point of the next frame and the multiple predicted points of the next frame.

[0014] In a possible implementation manner, determining the first loss item according to the first minimum distance and / or the second minimum distance includes:

[0015] The sum of the first minimum distances corresponding to the plurality of predicted points in the next frame and the sum of the second minimum distances corresponding to the plurality of real points in the next frame are added as a first loss term;

[0016]

[0017] in, represents the first loss term, represents the i-th predicted point in the t-th frame, represents a set of multiple prediction points of the t-th frame, represents the jth real point in the tth frame, P t Represents a set of multiple real points in the t-th frame.

[0018] In a possible implementation, the data set further includes multiple frames of image data matching the point cloud;

[0019] The method further comprises:

[0020] Projecting the real point of the target frame onto the image plane where the image data is located, determining a first pixel point, and determining an ego optical flow of the first pixel point based on a change in the posture of the ego vehicle; the target frame is the current frame or the next frame, and the first pixel point is a pixel point in the image data of the target frame that matches the real point of the target frame;

[0021] determining an overall optical flow of the first pixel according to the image data, and determining an object optical flow of the first pixel according to the overall optical flow and the self optical flow;

[0022] Projecting the body column motion parameters corresponding to the real point of the target frame onto the image plane to determine the corresponding projected optical flow;

[0023] A second loss term is determined according to a difference between the object optical flow and the projection optical flow, and the second loss term is added to the loss function.

[0024] In a possible implementation manner, determining the second loss term according to the difference between the object optical flow and the projection optical flow includes:

[0025] Determine the second loss term:

[0026]

[0027] in, represents the second loss term, (u i ,v i ) t Represents the t-th frame and the i-th real point The corresponding first pixel point, Indicates the first pixel point (u i ,v i ) t The projected optical flow, F obj (u i ,v i ) t Indicates the first pixel point (u i ,v i ) t The object optical flow.

[0028] In a possible implementation, the data set further includes multiple frames of image data matching the point cloud;

[0029] The method further comprises:

[0030] Projecting the real point of the target frame onto the image plane where the image data is located, determining a first pixel point, and determining an ego optical flow of the first pixel point based on a change in the posture of the ego vehicle; the target frame is the current frame or the next frame, and the first pixel point is a pixel point in the image data of the target frame that matches the real point of the target frame;

[0031] determining an overall optical flow of the first pixel according to the image data, and determining an object optical flow of the first pixel according to the overall optical flow and the self optical flow; and

[0032] Determining a probability that the first pixel corresponds to a dynamic object based on the magnitude of the object optical flow, and then determining a probability that the real point in the target frame corresponding to the first pixel corresponds to the dynamic object; the probability is positively correlated with the magnitude of the object optical flow;

[0033] Determining the first loss term according to the minimum distance between the predicted point of the next frame and the real point of the next frame includes:

[0034] The probability that the real point corresponds to the dynamic object is used as a weighting coefficient, the minimum distance between the predicted point of the next frame and the real point of the next frame is weighted, and the first loss term is determined according to the minimum distance after weighted processing.

[0035] In a possible implementation, determining the probability that the first pixel corresponds to a dynamic object according to the magnitude of the object optical flow includes:

[0036] Determine the probability that the first pixel corresponds to a dynamic object:

[0037]

[0038] Among them, (u i ,v i ) t represents the first pixel point corresponding to the i-th real point in the t-th frame, F obj (u i ,v i ) t Indicates the first pixel point (u i ,v i ) t The object optical flow, Indicates the first pixel point (u i ,v i ) t The probability of corresponding dynamic objects, α is the smoothing factor, and τ is the tolerance.

[0039] In one possible implementation, projecting the real point of the target frame onto the image plane where the image data is located, determining a first pixel point, and determining the ego optical flow of the first pixel point based on a change in the posture of the ego vehicle includes:

[0040] Determine the relative position T between the laser radar for collecting the point cloud and the camera device for collecting the image data L→C , and determine the intrinsic parameter K of the camera device;

[0041] Projecting the real point of the target frame onto the image plane where the image data is located according to the relative posture and the intrinsic parameters to determine a first pixel point;

[0042] Determine the pose change T of the ego vehicle in the target frame t→t+1 , and determine the self-optical flow of the first pixel according to:

[0043]

[0044] in, represents the i-th real point in the t-th frame, (u i ,v i ) t represents the first pixel point corresponding to the i-th real point in the t-th frame, F ego (u i ,v i ) t Indicates the first pixel point (u i ,v i ) t Self-light flow.

[0045] In a possible implementation manner, the component of the body column motion parameter in the vertical direction is zero.

[0046] In a possible implementation, the method further includes:

[0047] generating a body column motion field, wherein the body column motion field includes a plurality of body column motion parameters of the body columns;

[0048] A third loss term is determined according to the components and gradients of the body column motion field in multiple directions, and the third loss term is added to the loss function.

[0049] In a possible implementation, determining the third loss term according to the components and gradients of the body column motion field in multiple directions includes:

[0050] Determine the components and gradients of the body column motion field in the x-direction and the y-direction in the horizontal plane, and determine the third loss term:

[0051]

[0052] in, represents the component of the body column motion field in the x direction at the t-th frame, represents the component of the body column motion field in the y direction at the t-th frame, represents the gradient in the x direction, Represents the gradient in the x direction.

[0053] In a possible implementation, generating a loss function including the first loss term includes:

[0054] When there are other loss terms except the first loss term, a balance coefficient is set for the first loss term and the other loss terms, and a loss function is generated.

[0055] In a second aspect, an embodiment of the present invention further provides a self-supervised learning apparatus, comprising:

[0056] An acquisition module is used to acquire an unlabeled data set, wherein the data set includes multiple frames of point clouds, and each frame of the point cloud includes multiple real points;

[0057] a body column module, configured to organize a row of the real points in a vertical direction into a body column, wherein the body column is provided with body column motion parameters, and each of the real points in the body column has motion parameters identical to the body column motion parameters;

[0058] A prediction module, configured to move each of the real points of the current frame to the next frame according to the corresponding body column motion parameters, and determine a prediction point for the next frame;

[0059] a loss determination module, configured to determine a first loss term based on a minimum distance between the predicted point of the next frame and the true point of the next frame, and generate a loss function including the first loss term;

[0060] A processing module is used to perform self-supervised learning processing according to the loss function to determine the body column motion parameters of the body column.

[0061] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in any one of the above-described self-supervised learning methods.

[0062] In a fourth aspect, an embodiment of the present invention provides an electronic device comprising a bus, a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the transceiver, the memory, and the processor are connected via the bus, and when the computer program is executed by the processor, the steps in the self-supervised learning method described above are implemented.

[0063] In the solution provided by the first aspect of the embodiments of the present invention, each real point within a body column has identical body column motion parameters. Predicted points are derived based on the body column motion parameters, and a loss function is established using the minimum distance between the predicted points and the real points to achieve self-supervised learning. This method organizes the real points into corresponding body columns and determines body column motion parameters that represent the motion of the real points on a per-body-column basis, thereby enhancing the association between point clouds. Self-supervised learning is also possible even when there is no precise correspondence between the predicted and real points, and training can utilize large numbers of unlabeled point clouds. Furthermore, this method does not require distinguishing between object categories and can estimate category-independent motion based on point clouds.

[0064] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0066] Figure 1 A flowchart of a self-supervised learning method provided by an embodiment of the present invention is shown;

[0067] Figure 2 A schematic diagram of dividing body columns in the self-supervised learning method provided by an embodiment of the present invention is shown;

[0068] Figure 3 A flow chart of determining regularization loss provided by an embodiment of the present invention is shown;

[0069] Figure 4 A flowchart of determining the probability that a real point corresponds to a dynamic object provided by an embodiment of the present invention is shown;

[0070] Figure 5 A flow chart for determining smoothness loss provided by an embodiment of the present invention is shown;

[0071] Figure 6 A comparison table of different models provided by the embodiments of the present invention is shown;

[0072] Figure 7 A schematic diagram of the structure of a self-supervised learning device provided by an embodiment of the present invention is shown;

[0073] Figure 8 A schematic structural diagram of an electronic device for executing a self-supervised learning method provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0074] The embodiments of the present invention are described below with reference to the accompanying drawings.

[0075] An embodiment of the present invention provides a self-supervised learning method for implementing self-supervised learning of point clouds in an autonomous driving scenario. Figure 1 FIG. 1 is a flow chart showing a method for self-supervised learning provided by an embodiment of the present invention. Figure 1 As shown, the method includes steps 101-105.

[0076] Step 101: Acquire an unlabeled dataset, where the dataset includes multiple frames of point clouds, and each frame of point cloud includes multiple real points.

[0077] In an embodiment of the present invention, a data set is a set generated based on pre-collected data. The point cloud in the data set is collected by a laser radar. The laser radar scans once every period of time (such as 50ms), thereby forming a frame of point cloud. Each frame of point cloud corresponds to a timestamp. Each frame of point cloud contains multiple points (points). Since the points in the point cloud are actually collected, they are called "real points". For example, the point cloud can be obtained by a sensor such as a laser radar on an ego vehicle, wherein the ego vehicle can be an ego vehicle (ego-vehicle). Optionally, the point clouds of multiple frames in the data set are continuous, that is, the point clouds of multiple frames correspond to a time period, and there is no phenomenon of missing a frame of point cloud.

[0078] Furthermore, this method primarily utilizes unlabeled point clouds for self-supervised learning, so there is no manual labeling of the actual points in the point cloud. For example, the point cloud can be a raw, unprocessed point cloud after acquisition.

[0079] Step 102: Organize a column of real points in the vertical direction into a body column, where the body column is provided with body column motion parameters, and each real point in the body column has the same motion parameters as the body column motion parameters.

[0080] Due to the sparsity of point clouds, in autonomous driving scenarios, the vehicle itself may be moving, and objects such as pedestrians and other vehicles scanned by the lidar may also be moving. This leads to a lack of precise correspondence between the two point cloud frames, making it difficult to determine which real-world point in one frame corresponds to which real-world point in the other frame. In this embodiment of the present invention, the motion of the real-world points is described based on a pillar, in the hope of enhancing the correlation between the two point cloud frames.

[0081] In an embodiment of the present invention, objects in an autonomous driving scene mainly move in the horizontal direction, with a very small amplitude of vertical movement or even no movement; moreover, the movement of objects within the same body column has consistency, that is, objects within the same body column can basically be regarded as moving synchronously, and self-supervised learning of point clouds is achieved based on this consistency.

[0082] Specifically, the three-dimensional space is divided into a plurality of body columns arranged in the vertical direction; for example, Figure 2 As shown, the horizontal plane of three-dimensional space is divided into a×b grids. The space corresponding to each grid in the vertical direction (i.e., the z-axis) is a pillar. For each frame of the point cloud, a column of real points in the vertical direction lies within the same pillar, so the real points in that column can be organized into corresponding pillars. Furthermore, because the objects within the pillars are consistent, the real points within the pillars are also consistent, and each real point within the pillars has the same motion. In this embodiment of the present invention, pillar motion is used to represent the motion of each point in the point cloud. Furthermore, the motion of the pillar is represented based on pillar motion parameters, and each real point within the pillar moves according to these pillar motion parameters. Specifically, these pillar motion parameters can be two-dimensional motion vectors, for example, including velocity and direction. Optionally, the vertical motion of the pillar can be ignored, meaning that the vertical component of the pillar motion parameters is zero, thereby simplifying the calculation process and improving processing efficiency.

[0083] Step 103: Move each real point of the current frame to the next frame according to the corresponding body column motion parameter, and determine the prediction point of the next frame.

[0084] In an embodiment of the present invention, body motion parameters represent the motion of each real point within a body column; wherein, each frame's timestamp corresponds to a body motion parameter. A real point in the current frame is located within the corresponding body column, and the body column motion parameters corresponding to the real point can be determined, allowing the real point to move to the next frame according to the body column motion parameters. In an embodiment of the present invention, "moving the real point to the next frame" means that the real point's time dimension moves from the timestamp of the current frame to the timestamp of the next frame, and the real point moves according to the body column motion parameters, such as moving a certain distance in a certain direction or not moving at all. Thus, the corresponding position of the real point in the next frame can be determined. The real point that moves to the next frame is called the predicted point of the next frame.

[0085] For example, the current frame is the t-th frame, and a real point p in the t-th frame corresponds to the body column motion parameter M of the t-th frame. Then, the real point p moves to the next frame according to the body column motion parameter M, that is, moves to the t+1-th frame. In this way, the position of the real point p in the t-th frame when it moves to the t+1-th frame can be determined, and then the predicted point of the t+1-th frame can be determined.

[0086] Step 104: Determine a first loss term based on the minimum distance between the predicted point of the next frame and the true point of the next frame, and generate a loss function including the first loss term.

[0087] In this embodiment of the present invention, the dataset includes at least a point cloud for the current frame and a point cloud for the next frame. Based on step 103 above, each real point in the current frame can be converted into a predicted point for the next frame. This means that a point cloud containing multiple real points is converted into a point cloud containing multiple predicted points, and this point cloud containing multiple predicted points is referred to as the predicted point cloud for the next frame. The dataset also includes the point cloud for the next frame acquired from actual capture, i.e., the real point cloud for the next frame. If the body column motion parameters are correct, the predicted point cloud for the next frame should be similar or even identical to the real point cloud for the next frame. As described above, although there is a lack of precise correspondence between the point clouds of the two frames, making it difficult to determine which real point in the next frame a predicted point in the next frame corresponds to, in this embodiment of the present invention, all real points in the body column have the same motion parameters. There is a strong correspondence between the body column corresponding to the predicted point in the next frame and the real point in the next frame. This means that there is structural consistency between the predicted point cloud for the next frame and the real point cloud for the next frame. Therefore, self-supervised learning based on this structural consistency can learn the body column motion, i.e., the body column motion parameters.

[0088] In an embodiment of the present invention, there are multiple predicted points in the next frame and multiple real points. A loss term, i.e., a first loss term, is determined based on the minimum distance between the predicted points of the next frame and the real points of the next frame. This loss term is then used to generate the loss function required for self-supervised learning.

[0089] Optionally, the step of “determining the first loss term based on the minimum distance between the predicted point of the next frame and the true point of the next frame” includes:

[0090] Step A1: Determine a first loss item based on a first minimum distance and / or a second minimum distance, where the first minimum distance is the minimum value of the distance between the predicted point of the next frame and multiple real points of the next frame, and the second minimum distance is the minimum value of the distance between the real point of the next frame and multiple predicted points of the next frame.

[0091] In an embodiment of the present invention, in the next frame, there are a pair of predicted points and multiple real points. For a predicted point, the minimum distance between the predicted point and the multiple real points, i.e., the first minimum distance, can be determined. For a real point, the minimum distance between the real point and the multiple predicted points, i.e., the second minimum distance, can be determined. This embodiment of the present invention determines a first loss term based on the first minimum distance or the second minimum distance; alternatively, the first loss term is determined based on the first minimum distance and the second minimum distance. The first loss term can represent a loss in structural consistency.

[0092] Optionally, to ensure a strong correspondence between the predicted point and the actual point, the embodiment of the present invention determines the first loss term based on the first minimum distance and the second minimum distance. The process specifically includes:

[0093] The sum of the first minimum distances corresponding to multiple predicted points of the next frame and the sum of the second minimum distances corresponding to multiple real points of the next frame are taken as the first loss term:

[0094]

[0095] in, represents the first loss term, represents the i-th prediction point of the t-th frame, represents a set of multiple prediction points of the t-th frame, represents the jth real point in the tth frame, P t Represents a set of multiple real points in the t-th frame.

[0096] In the embodiment of the present invention, the first loss term can be calculated for each frame according to the above formula (1). Taking the next frame as the t-th frame as an example, a plurality of real points form a set P t , multiple prediction points form a set Among them, the above two sets can contain all points of the t-th frame, such as P t Contains all the real points of the t-th frame; or, the above two sets can also contain only some points of the t-th frame, such as P t Inclusion and prediction points All the real points in the corresponding body column. For the i-th predicted point The minimum distance between it and multiple real points is Correspondingly, for the jth real point The minimum distance between it and multiple prediction points is Then, the first loss term is calculated according to formula (1), and the loss function including the first loss term is determined.

[0097] Step 105: Perform self-supervised learning processing according to the loss function to determine the body column motion parameters of the body column.

[0098] In an embodiment of the present invention, after determining the loss function, training processing can be performed according to existing self-supervised learning methods, and ultimately the motion parameters of the pillars, i.e., the pillar motion parameters, can be determined. The pillar motion parameters of multiple pillars can form a pillar motion field. Based on this pillar motion field, the motion of each pillar can be predicted, that is, the motion of the objects within each pillar can be determined. Furthermore, even if there are objects of different categories, such as vehicles, pedestrians, and buildings, in an embodiment of the present invention, these different categories of objects can all be mapped to corresponding pillars. Therefore, the embodiment of the present invention focuses on the motion of the pillars and can predict the motion of the objects without distinguishing the categories of the objects, thereby achieving point cloud-based estimation of class-agnostic motion.

[0099] An embodiment of the present invention provides a self-supervised learning method in which each real point within a body column has the same body column motion parameters. Predicted points are derived based on these body column motion parameters, and a loss function is established using the minimum distance between the predicted and real points to achieve self-supervised learning. This method organizes the real points into corresponding body columns and determines body column motion parameters that represent the motion of the real points on a per-body-column basis, thereby enhancing the association between point clouds. Self-supervised learning is also possible even when there is no precise correspondence between the predicted and real points, and training can utilize large numbers of unlabeled point clouds. Furthermore, this method does not require distinguishing between object categories and can estimate category-independent motion based on point clouds.

[0100] Although body pillars can enhance the structural consistency between point clouds, this structural consistency still relies on the existence of a correspondence between two consecutive point clouds (for example, the point cloud of the t-th frame and the point cloud of the t+1-th frame). Point clouds are sparse, especially point clouds in the distance, where the points are extremely sparse, resulting in the fact that the real points in the current frame cannot accurately correspond to the real points of the next frame, that is, the predicted points of the next frame cannot accurately correspond to the real points of the next frame. The loss term determined based on the minimum distance may be ambiguous. Therefore, the above-mentioned structural consistency matching based on body pillars will still inevitably introduce noise. For example, the body pillar motion parameters of static objects (such as buildings, etc.) should be zero, but due to the presence of these noises, it may be believed that a static object has slight motion. The embodiment of the present invention complements the structural consistency by image data matched with the point cloud to mitigate the ambiguity caused by the sparsity of the point cloud.

[0101] The data set in the embodiment of the present invention also includes multiple frames of image data that match the point cloud. For example, each frame of the point cloud corresponds to corresponding image data. Specifically, the image data can be collected by a paired camera device due to a laser radar, and the camera device can specifically be a camera. For example, a self-vehicle is equipped with multiple laser radars and camera devices. Every time stamp, for example, every 20ms, the laser radar and camera device collect data once. The laser radar can collect the point cloud of each time stamp, that is, the point cloud of each frame; the camera device can collect the image data of each time stamp, that is, the image data of each frame.

[0102] Although there are solutions for estimating scene flow based on images, as described in the background art, directly estimating scene flow is difficult and inaccurate. Therefore, the embodiments of the present invention utilize optical flow to perform cross-sensor self-supervised motion regularization, thereby regularizing motion learning across sensors to alleviate the problem of lack of precise correspondence between scans. In the embodiments of the present invention, in addition to generating the first loss term, the current frame or the next frame is also used as the target frame, and the second loss term corresponding to the target frame is calculated, and the second loss term is used as regularization; see Figure 3 As shown, the regularization process includes:

[0103] Step 301: Project the real point of the target frame onto the image plane where the image data is located, determine the first pixel point, and determine the self-optical flow of the first pixel point based on the posture change of the self-vehicle; the target frame is the current frame or the next frame, and the first pixel point is the pixel point in the image data of the target frame that matches the real point of the target frame.

[0104] Step 302: Determine the global optical flow of the first pixel according to the image data, and determine the object optical flow of the first pixel according to the global optical flow and the self optical flow.

[0105] In the embodiment of the present invention, the image data is two-dimensional data, which corresponds to a certain plane, namely, the image plane; by projecting the real point of the target frame onto the image data of the target frame, the pixel point in the image data that matches the real point can be determined, namely, the first pixel point. For example, the target frame is the t-th frame, in which the i-th real point and the pixel points in the image data (u i ,v i ) t If the pixel (u i ,v i ) t That is, a first pixel point. Those skilled in the art will understand that step 301 can be performed for each frame. When determining the first loss term based on the predicted points of the next frame and the actual points of the next frame, the current frame or the next frame can be used as the target frame. The first pixel point is a type of pixel point, that is, a pixel point that can correspond to a real point; due to the sparsity of the point cloud, some pixels in the image data may not correspond to a real point, and such pixels are not first pixels.

[0106] In the embodiment of the present invention, the laser radar for collecting point clouds and the camera device for collecting image data are fixed in position, that is, based on the relative position T between the two L→C The correspondence between the real point and the first pixel can be determined by , and then the real point can be mapped to the corresponding first pixel. For example, the i-th real point of the t-th frame is It is three-dimensional, that is If the relative position between the laser radar and the camera device is T L→C , the inherent parameters (such as internal parameters, external parameters, etc.) of the camera device are K, then the corresponding first pixel point (u i ,v i ) t for:

[0107]

[0108] Although the optical flow estimation is relatively accurate, the optical flow determined based on the image data (i.e., the overall optical flow) includes the optical flow caused by the movement of the ego vehicle itself (i.e., the self-optical flow) and the optical flow caused by the movement of the object (i.e., the object optical flow). Therefore, the overall optical flow cannot directly represent the movement of the object, and the movement of the ego vehicle itself needs to be decomposed from the overall optical flow. In an embodiment of the present invention, when the ego vehicle collects point cloud and image data, its pose will change. This pose change can represent the movement of the ego vehicle itself, so the self-optical flow of the first pixel can be determined based on the pose change. For example, the target frame is the t-th frame, and the pose change of the ego vehicle from the t-th frame to the t+1-th frame is taken as the pose change T of the target frame. t→t+1 , then the real point The corresponding first pixel point is located at the t frame (u i ,v i ) t , at the t+1th frame, the change caused by the ego vehicle causes the first pixel to move to Therefore, the self-optical flow F of the first pixel is ego (u i ,v i ) t for:

[0109]

[0110] Furthermore, the overall optical flow of the first pixel can be determined based on the image data of two adjacent frames. For example, the overall optical flow of the first pixel of the t-th frame can be determined based on the image data of the t-th frame and the t+1-th frame. i ,v i ) t The overall optical flow is F(u i ,v i ) t , whose object optical flow is F obj (u i ,v i ) t ,but:

[0111] F(u i ,v i ) t =F ego (u i ,v i ) t +F obj (u i ,v i ) t #(4)

[0112] Based on the above formulas (3) and (4), the object optical flow F of the first pixel can be determined obj (u i ,v i ) t The object optical flow is the portion obtained by decomposing the self optical flow from the overall optical flow. Since the self optical flow can only be determined at the first pixel corresponding to the real point, the object optical flow can only be determined by compensating for the first pixel. The object optical flow cannot be determined for other pixels in the image data.

[0113] Step 303: Project the body column motion parameters corresponding to the real points of the target frame onto the image plane to determine the corresponding projected optical flow.

[0114] In the embodiment of the present invention, as shown in the above formula (2), there is a corresponding relationship between the real point and the first pixel point. Accordingly, the body column motion parameter of the real point can also be projected onto the image plane based on the formula (2), so that the body column motion parameter in the three-dimensional world can be projected onto the two-dimensional image plane. The projected body column motion parameter is the motion of the body column in the image plane, which is referred to as projected optical flow in the embodiment of the present invention.

[0115] Step 304: Determine a second loss term according to the difference between the object optical flow and the projection optical flow, and add the second loss term to the loss function.

[0116] In an embodiment of the present invention, the projected optical flow is the optical flow determined by projecting the body column motion parameters onto the image plane. It is an optical flow predicted based on the body column motion parameters. The object optical flow is the optical flow determined based on the image data and the changes in the position of the ego vehicle. This object optical flow is real. Therefore, based on the difference between the object optical flow and the projected optical flow, an association can be established between the body column motion and the optical flow. This difference can also represent the difference between the body column motion parameters and the actual motion situation. Based on this difference, a second loss term is determined. This second loss term represents the loss across the two sensors, the lidar and the camera. Self-supervised learning based on a loss function that includes this second loss term can make the predicted projected optical flow closer to the actual object optical flow.

[0117] Optionally, the second loss term determined in step 304 may be:

[0118]

[0119] in, represents the second loss term, (u i ,v i ) t Represents the t-th frame and the i-th real point The corresponding first pixel point, Indicates the first pixel (u i ,v i ) t The projected optical flow, F obj (u i ,v i ) t Indicates the first pixel (u i ,v i ) t The object optical flow.

[0120] In this embodiment of the present invention, the second loss term, a cross-sensory loss, is used as an auxiliary regularization to supplement the structural consistency loss (i.e., the first loss term) and mitigate the lack of correspondence between point clouds due to their sparsity. Furthermore, optical flow-guided regularization can be viewed as motion knowledge extracted from the camera to the lidar during training, enhancing the effectiveness of self-supervised learning.

[0121] Optionally, since the ego vehicle is generally moving during the actual scanning process of the LiDAR, the real points contained in the body column of the static object in the point cloud of two adjacent frames are not the same, resulting in the introduction of noise into the static object when the nearest neighbor matching is enforced based on the above-mentioned first loss term. The embodiment of the present invention reduces the impact of noise by introducing the probability that each real point is in motion, thereby enhancing the first loss term. In the embodiment of the present invention, the impact of the above-mentioned noise is reduced based on the image data of multiple frames that match the point cloud; see Figure 4 As shown, the method further includes the following steps 401-403, and the process of determining the first loss term in the above step 104 further includes the following step B1.

[0122] Step 401: Project the real point of the target frame onto the image plane where the image data is located, determine the first pixel point, and determine the self-optical flow of the first pixel point based on the posture change of the self-vehicle; the target frame is the current frame or the next frame, and the first pixel point is the pixel point in the image data of the target frame that matches the real point of the target frame.

[0123] Step 402: Determine the global optical flow of the first pixel according to the image data, and determine the object optical flow of the first pixel according to the global optical flow and the self optical flow.

[0124] In this embodiment of the present invention, steps 401-402 are the process of determining the object's optical flow, which is the same as steps 301-302 and is not further described here. Furthermore, if a second loss term needs to be added to the loss function, either steps 301-302 or steps 401-402 can be performed, meaning that the process of determining the object's optical flow only needs to be performed once.

[0125] Step 403: Determine the probability that the first pixel corresponds to a dynamic object based on the size of the object optical flow, and then determine the probability that the real point of the target frame corresponding to the first pixel corresponds to the dynamic object; this probability is positively correlated with the size of the object optical flow.

[0126] In an embodiment of the present invention, the object optical flow can represent the movement of an object. If the object is static, the magnitude of the object optical flow should be zero; if the object is dynamic, the greater the degree of displacement of the object, the greater the magnitude of the object optical flow. Therefore, the larger the magnitude of the object optical flow, the greater the probability that the corresponding object is dynamic, that is, there is a positive correlation between the probability and the magnitude of the object optical flow. Among them, since there is a corresponding relationship between the first pixel point and the real point, the "probability that the first pixel point corresponds to a dynamic object" is the same as the corresponding "probability that the real point of the target frame corresponds to a dynamic object", and both are positively correlated with the magnitude of the object optical flow.

[0127] In the embodiment of the present invention, the norm of the object optical flow can be used as the size of the object optical flow, that is, the object optical flow F obj (u i ,v i ) t The size of ||F obj (u i ,v i ) t ||. Optionally, since errors are inevitable in the process of calculating the optical flow of an object, an embodiment of the present invention pre-sets a tolerance (tolerance) τ, which can be fixed. The probability of the object optical flow being smaller than the tolerance τ is set to 0, that is, the object is considered to be static at this time. Specifically, the max(||F obj (u i ,v i ) t ||-τ,0) to calculate the corresponding probability. For example, the probability that the first pixel corresponds to a dynamic object is:

[0128]

[0129] Among them, (u i ,v i ) t represents the first pixel corresponding to the i-th real point in the t-th frame, Fobj (u i ,v i ) t Indicates the first pixel (u i ,v i ) t The object optical flow, Indicates the first pixel (u i ,v i ) t The probability of the corresponding dynamic object, α is the smoothing factor, and τ is the tolerance, which is generally fixed.

[0130] After determining the probability that each real point corresponds to a dynamic object, the first loss term can be adjusted. In the embodiment of the present invention, the process of determining the first loss term in step 104 further includes the following step B1.

[0131] Step B1: Using the probability that the real point corresponds to the dynamic object as the weighting coefficient, weight the minimum distance between the predicted point of the next frame and the real point of the next frame, and determine the first loss term based on the weighted minimum distance.

[0132] In this embodiment of the present invention, the probability that a ground truth point corresponds to a dynamic object is used as a weighting factor to weight the minimum distance. This gives ground truth points corresponding to dynamic objects a higher weight during training. Probabilistic motion masking is used to reduce the weight of ground truth points in static body pillars, thereby reducing the impact of noise introduced by static body pillars. Furthermore, in autonomous driving scenarios, the point cloud often contains more static objects than dynamic ones. For example, there are fewer points corresponding to dynamic vehicles and pedestrians, while there are more points corresponding to static buildings. This means that static objects or static body pillars dominate. Therefore, this authentication strategy also helps balance the contributions of static and dynamic body pillars when calculating the structural consistency loss (i.e., the first loss term).

[0133] For example, if the probability that the jth real point in the tth frame corresponds to a dynamic object is Then the above formula (1) can be weighted and the first loss term can be determined:

[0134]

[0135] Based on the above embodiment, the method further includes applying local smoothness loss to body column motion learning, see Figure 5 As shown, the process includes:

[0136] Step 501: Generate a body column motion field, which includes body column motion parameters of multiple body columns.

[0137] Step 502: Determine a third loss term based on the components and gradients of the body column motion field in multiple directions, and add the third loss term to the loss function.

[0138] In this embodiment of the present invention, multiple pillars exist in each frame. The pillar motion parameters of these pillars can represent the overall motion, i.e., the pillar motion field. A smoothness loss, i.e., the third loss term, is determined based on the components and gradients of this pillar motion field in multiple directions. This enables the model to predict similar motions of pillars belonging to the same subject.

[0139] Optionally, in step 502, “determining a third loss term based on the components and gradients of the body column motion field in multiple directions” includes:

[0140] Determine the components and gradients of the body column motion field in the x and y directions in the horizontal plane, and determine the third loss term:

[0141]

[0142] in, represents the component of the body column motion field in the x direction of the t-th frame, represents the component of the body column motion field in the tth frame in the y direction, represents the gradient in the x direction, Represents the gradient in the x direction.

[0143] Alternatively, if the loss function contains multiple loss terms, that is, other loss terms (such as a second loss term and a third loss term) in addition to the first loss term, a weighted approach is used to determine the loss function. That is, when there are other loss terms in addition to the first loss term, a balance coefficient is set for the first loss term and the other loss terms, and a loss function is generated. For example, if a loss function is determined based on the first loss term, the second loss term, and the third loss term, the loss function can be:

[0144]

[0145] Among them, λ1, λ2, and λ3 are the balance coefficients of the corresponding loss terms.

[0146] The following describes the process of the method in detail through an embodiment. It should be noted that i and j in the following formulas or parameters in the embodiments of the present invention represent an index, and may have different meanings in different formulas or parameters, that is, represent different indexes.

[0147] In the embodiment of the present invention, the self-vehicle is equipped with a laser radar and a multi-camera device, and captures the corresponding point cloud and image data at each time stamp t. The time stamp t corresponds to the tth frame, and the point cloud P of the tth frame t Expressed as The image data of frame t I t Expressed as Among them, N1 represents the point cloud P t The number of real points contained, point clouds of different frames may have different N1; N2 represents the image data I t In addition, since the vehicle can be provided with multiple cameras in different directions, that is, the image data of the t-th frame can include multiple images, this embodiment only takes one of the images as an example for description.

[0148] In the embodiment of the present invention, the point cloud P t Can be discretized into multiple non-overlapping columns (When dividing the body columns, some body columns do not contain real points. The embodiment of the present invention only focuses on non-empty body columns containing real points.) Each body column has a corresponding body column motion parameter. Multiple body column motion parameters Can form a body column sports field For example,

[0149] For the point cloud of frame t, the body column motion parameters Under the action of It can be moved to the position corresponding to the t+1th frame to form the body column predicted in the t+1th frame It can be expressed as: Correspondingly, the i-th ground truth point in the t-th frame It can also be moved to the corresponding position to form the prediction point of the t+1 frame Right now All forecast points It can also form a predicted point cloud and In addition, the point cloud of frame t+1 has real points Therefore, based on this prediction point And be real The structural consistency loss, i.e., the first loss term, can be determined. The first loss term can represent the predicted point cloud Compared with the real point cloud P t+1 The above formula (1) represents the calculation of the first loss term of the t-th frame. Similarly, the first loss term of the t+1-th frame can also be calculated.

[0150] In the embodiment of the present invention, since the laser radar and the camera device are matched, and the image data collected by the imaging device has more dense information, the real point of the point cloud of the tth frame is Can correspond to the t-th frame image data I t The corresponding pixel point in the , that is, the first pixel point (u i ,v i ) t And, according to the image data of two frames I t and I t+1 To estimate the corresponding overall optical flow F(u,v) t , and then determine the overall optical flow F(u i ,v i ) t Based on the above formula (3), the self-optical flow F is determined ego (u i ,v i ) t After that, the object optical flow F can be determined obj (u i ,v i ) t .

[0151] Furthermore, the body column motion parameter of the first pixel point can be Project onto the image plane and determine the projected optical flow The regularization loss for cross-sensor motion, i.e., the second loss term, is determined based on formula (5).

[0152] In addition, when determining the object optical flow F obj (u i ,v i ) t After that, calculate the probability that the real point corresponds to the dynamic object Therefore, the introduction of probabilistic motion masking into the first loss term can optimize the structural consistency loss. Can be divided into the x-direction component and the y-direction component Then, the smoothness loss, i.e., the third loss term, is determined based on formula (7). Finally, the loss function is determined by weighting, as shown in formula (8). After the loss function is determined, self-supervised learning can be performed.

[0153] The self-supervised learning framework provided by the embodiment of the present invention does not rely on a specific backbone network and has a wide applicability. In addition, the embodiment of the present invention compares models using different combinations, divides non-empty body columns into three speed groups, namely static, slow (≤5m / s) and fast (>5m / s), and estimates all non-empty body columns, all foreground object body columns and all moving object body columns. The mean error and median error of different models can be found in Figure 6 The table shown, Figure 6 The numbers in bold are the minimum values.

[0154] exist Figure 6 In the case of structural consistency loss, only The trained base model (a) performs poorly for the static group, which is consistent with the previous description that the static body column introduces noise. As the only supervision, model (b) achieves good results for the static group, but performs much worse for the fast group, as it can reliably recover the ego-motion of static points from optical flow. This is not surprising, as regularizing motion only in the image plane of the 2D camera is ambiguous, and multiple ground-truth points in the 3D point cloud may have different body-column motion parameters, but project to the same pixel in the 2D image plane.

[0155] The model (c) that combines the structural consistency loss and the motion regularization loss performs well in the fast group, but is still not the best choice for the static and slow groups, which is mainly due to the inconsistency between the two losses in the static and slow motion regions. By integrating probabilistic motion masking into model (c), model (e) achieves improvements over the static and slow groups. This is because by suppressing static pillars, the model is less confused by the noisy motion caused by the moving ego vehicle and can therefore better focus on learning the real object motion. Model (d) only uses probabilistic motion masking to enhance the structural consistency loss. Compared with model (a), the improvement of the static and slow groups is obvious, but model (d) is still inferior to model (e), which further verifies the effectiveness of cross-sensor motion regularization in providing complementary motion supervision.

[0156] The comparison results of the model trained based on the embodiment of the present invention and other models can be seen in Table 1 below.

[0157] Table 1

[0158]

[0159] Table 1 is divided into two parts, the upper part of which shows the self-supervised model provided by the embodiment of the present invention compared with the models FlowNet3D and HPLFlowNet of the application scenario flow.

[0160] The self-supervised model provided in this embodiment is largely superior to the supervised pre-trained model. In fact, the self-supervised model is better than or close to certain methods that have been fully supervised trained on the benchmark dataset. For example, for the fast group, our model performs better than FlowNet3D, HPLFlowNet and PointRCNN. All these comparisons clearly demonstrate the advantages of the self-supervised learning method proposed in the embodiment of the present invention and the importance of self-supervised training on the target domain. Moreover, compared with other existing models, the model provided in this embodiment has higher computational efficiency (about 0.02s), thereby enabling real-time processing of large-scale point clouds.

[0161] The above describes in detail the method for self-supervised learning provided by an embodiment of the present invention. This method can also be implemented by a corresponding device. The following describes in detail the device for self-supervised learning provided by an embodiment of the present invention.

[0162] Figure 7 FIG. 1 shows a schematic diagram of the structure of a self-supervised learning device provided by an embodiment of the present invention. Figure 7 As shown, the self-supervised learning device includes:

[0163] An acquisition module 71 is configured to acquire an unlabeled data set, wherein the data set includes multiple frames of point clouds, and each frame of the point cloud includes multiple real points;

[0164] a body column module 72 for organizing a row of the real points in a vertical direction into a body column, wherein the body column is provided with body column motion parameters, and each of the real points in the body column has motion parameters identical to the body column motion parameters;

[0165] A prediction module 73 is configured to move each of the real points of the current frame to the next frame according to the corresponding body column motion parameters, and determine a prediction point for the next frame;

[0166] a loss determination module 74, configured to determine a first loss term based on a minimum distance between the predicted point of the next frame and the true point of the next frame, and to generate a loss function including the first loss term;

[0167] The processing module 75 is configured to perform self-supervised learning processing according to the loss function to determine the body column motion parameters of the body column.

[0168] Based on the above embodiment, the loss determination module 74 includes a first loss item determination unit; the first loss item determination unit is configured to:

[0169] The first loss item is determined according to the first minimum distance and / or the second minimum distance, wherein the first minimum distance is the minimum value of the distance between the predicted point of the next frame and the multiple real points of the next frame, and the second minimum distance is the minimum value of the distance between the real point of the next frame and the multiple predicted points of the next frame.

[0170] Based on the above embodiment, the first loss term determining unit determines the first loss term according to the first minimum distance and / or the second minimum distance, including:

[0171] The sum of the first minimum distances corresponding to the plurality of predicted points in the next frame and the sum of the second minimum distances corresponding to the plurality of real points in the next frame are added as a first loss term;

[0172]

[0173] in, represents the first loss term, represents the i-th predicted point in the t-th frame, represents a set of multiple prediction points of the t-th frame, represents the jth real point in the tth frame, P t Represents a set of multiple real points in the t-th frame.

[0174] Based on the above embodiment, the data set further includes multiple frames of image data matching the point cloud;

[0175] The loss determination module 74 includes a second loss item determination unit; the second loss item determination unit is configured to:

[0176] Projecting the real point of the target frame onto the image plane where the image data is located, determining a first pixel point, and determining an ego optical flow of the first pixel point based on a change in the posture of the ego vehicle; the target frame is the current frame or the next frame, and the first pixel point is a pixel point in the image data of the target frame that matches the real point of the target frame;

[0177] determining an overall optical flow of the first pixel according to the image data, and determining an object optical flow of the first pixel according to the overall optical flow and the self optical flow;

[0178] Projecting the body column motion parameters corresponding to the real point of the target frame onto the image plane to determine the corresponding projected optical flow;

[0179] A second loss term is determined according to a difference between the object optical flow and the projection optical flow, and the second loss term is added to the loss function.

[0180] Based on the above embodiment, the second loss term determining unit determines the second loss term according to the difference between the object optical flow and the projection optical flow, including:

[0181] Determine the second loss term:

[0182]

[0183] in, represents the second loss term, (u i ,v i ) t Represents the t-th frame and the i-th real point The corresponding first pixel point, Indicates the first pixel point (u i ,v i ) t The projected optical flow, F obj (u i ,v i ) t Indicates the first pixel point (u i ,v i ) t The object optical flow.

[0184] Based on the above embodiment, the data set further includes multiple frames of image data matching the point cloud;

[0185] The loss determination module 74 includes a dynamic probability determination unit; the dynamic probability determination unit is used to:

[0186] Projecting the real point of the target frame onto the image plane where the image data is located, determining a first pixel point, and determining an ego optical flow of the first pixel point based on a change in the posture of the ego vehicle; the target frame is the current frame or the next frame, and the first pixel point is a pixel point in the image data of the target frame that matches the real point of the target frame;

[0187] determining an overall optical flow of the first pixel according to the image data, and determining an object optical flow of the first pixel according to the overall optical flow and the self optical flow; and

[0188] Determining a probability that the first pixel corresponds to a dynamic object based on the magnitude of the object optical flow, and then determining a probability that the real point in the target frame corresponding to the first pixel corresponds to the dynamic object; the probability is positively correlated with the magnitude of the object optical flow;

[0189] The loss determination module 74 determines the first loss term according to the minimum distance between the predicted point of the next frame and the real point of the next frame, including:

[0190] The probability that the real point corresponds to the dynamic object is used as a weighting coefficient, the minimum distance between the predicted point of the next frame and the real point of the next frame is weighted, and the first loss term is determined according to the minimum distance after weighted processing.

[0191] Based on the above embodiment, the dynamic probability determination unit determines the probability that the first pixel corresponds to a dynamic object according to the magnitude of the object optical flow, including:

[0192] Determine the probability that the first pixel corresponds to a dynamic object:

[0193]

[0194] Among them, (u i ,v i ) t represents the first pixel point corresponding to the i-th real point in the t-th frame, F obj (u i ,v i ) t Indicates the first pixel point (u i ,v i ) t The object optical flow, Indicates the first pixel point (u i ,v i ) t The probability of corresponding dynamic objects, α is the smoothing factor, and τ is the tolerance.

[0195] Based on the above embodiment, the second loss term determination unit or the dynamic probability determination unit projects the real point of the target frame onto the image plane where the image data is located, determines a first pixel point, and determines the ego optical flow of the first pixel point based on the posture change of the ego vehicle, including:

[0196] Determine the relative position T between the laser radar for collecting the point cloud and the camera device for collecting the image data L→C , and determine the intrinsic parameter K of the camera device;

[0197] Projecting the real point of the target frame onto the image plane where the image data is located according to the relative posture and the intrinsic parameters to determine a first pixel point;

[0198] Determine the pose change T of the ego vehicle in the target frame t→t+1 , and determine the self-optical flow of the first pixel according to:

[0199]

[0200] in, represents the i-th real point in the t-th frame, (u i ,v i ) t represents the first pixel point corresponding to the i-th real point in the t-th frame, F ego (u i ,v i ) t Indicates the first pixel point (u i ,v i ) t Self-light flow.

[0201] Based on the above embodiment, the component of the body column motion parameter in the vertical direction is zero.

[0202] Based on the above embodiment, the loss determination module 74 includes a third loss item determination unit; the third loss item determination unit is configured to:

[0203] generating a body column motion field, wherein the body column motion field includes a plurality of body column motion parameters of the body columns;

[0204] A third loss term is determined according to the components and gradients of the body column motion field in multiple directions, and the third loss term is added to the loss function.

[0205] Based on the above embodiment, the third loss term determining unit determines the third loss term according to the components and gradients of the body column motion field in multiple directions, including:

[0206] Determine the components and gradients of the body column motion field in the x-direction and the y-direction in the horizontal plane, and determine the third loss term:

[0207]

[0208] in, represents the component of the body column motion field in the x direction at the t-th frame, represents the component of the body column motion field in the y direction at the t-th frame, represents the gradient in the x direction, Represents the gradient in the x direction.

[0209] Based on the above embodiment, the loss determination module 74 generates a loss function including the first loss term, including:

[0210] When there are other loss terms except the first loss term, a balance coefficient is set for the first loss term and the other loss terms, and a loss function is generated.

[0211] In addition, an embodiment of the present invention also provides an electronic device, including a bus, a transceiver, a memory, a processor, and a computer program stored in the memory and runnable on the processor. The transceiver, the memory, and the processor are respectively connected via a bus. When the computer program is executed by the processor, each process of the above-mentioned self-supervised learning method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0212] For details, see Figure 8 As shown, an embodiment of the present invention further provides an electronic device, which includes a bus 1110 , a processor 1120 , a transceiver 1130 , a bus interface 1140 , a memory 1150 and a user interface 1160 .

[0213] In an embodiment of the present invention, the electronic device further includes: a computer program stored in the memory 1150 and executable on the processor 1120, and when the computer program is executed by the processor 1120, each process of the above-mentioned self-supervised learning method embodiment is implemented.

[0214] The transceiver 1130 is configured to receive and send data under the control of the processor 1120 .

[0215] In an embodiment of the present invention, a bus architecture (represented by bus 1110) may include any number of interconnected buses and bridges, and bus 1110 connects various circuits including one or more processors represented by processor 1120 and a memory represented by memory 1150.

[0216] Bus 1110 represents one or more of any of several types of bus structures, including a memory bus and memory controller, a peripheral bus, an Accelerated Graphical Port (AGP), a processor, or a local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA), and a Peripheral Component Interconnect (PCI) bus.

[0217] The processor 1120 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above-mentioned processor includes: a general-purpose processor, a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), a programmable logic array (PLA), a microcontroller unit (MCU) or other programmable logic devices, discrete gates, transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. For example, the processor can be a single-core processor or a multi-core processor, and the processor can be integrated into a single chip or located on multiple different chips.

[0218] The processor 1120 can be a microprocessor or any conventional processor. The method steps disclosed in conjunction with the embodiments of the present invention can be directly executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a readable storage medium known in the art, such as a random access memory (RAM), a flash memory (Flash Memory), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), or a register. The readable storage medium is located in a memory, and the processor reads the information in the memory and performs the steps of the above method in conjunction with its hardware.

[0219] The bus 1110 may also connect various other circuits, such as peripheral devices, voltage regulators, or power management circuits. The bus interface 1140 provides an interface between the bus 1110 and the transceiver 1130. These are all well known in the art and are therefore not further described in this embodiment of the present invention.

[0220] The transceiver 1130 can be a single component or multiple components, such as multiple receivers and transmitters, providing a means for communicating with various other devices over a transmission medium. For example, the transceiver 1130 receives external data from other devices and transmits data processed by the processor 1120 to other devices. Depending on the nature of the computer system, a user interface 1160 may also be provided, such as a touch screen, physical keyboard, display, mouse, speaker, microphone, trackball, joystick, or stylus.

[0221] It should be understood that in an embodiment of the present invention, the memory 1150 may further include a memory remotely located relative to the processor 1120, and these remotely located memories may be connected to a server via a network. One or more parts of the aforementioned network may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a wireless wide area network (WWAN), a metropolitan area network (MAN), the Internet, a public switched telephone network (PSTN), a plain old telephone service network (POTS), a cellular telephone network, a wireless network, a wireless fidelity (Wi-Fi) network, or a combination of two or more of the aforementioned networks. For example, the cellular telephone network and the wireless network can be a Global System for Mobile Communications (GSM) system, a Code Division Multiple Access (CDMA) system, a Worldwide Interoperability for Microwave Access (WiMAX) system, a General Packet Radio Service (GPRS) system, a Wideband Code Division Multiple Access (WCDMA) system, a Long Term Evolution (LTE) system, an LTE Frequency Division Duplex (FDD) system, an LTE Time Division Duplex (TDD) system, an Advanced Long Term Evolution (LTE-A) system, a Universal Mobile Telecommunications (UMTS) system, an Enhanced Mobile Broadband (eMBB) system, a Massive Machine Type of Communication (mMTC) system, an Ultra Reliable Low Latency Communications (uRLLC) system, and the like.

[0222] It should be understood that the memory 1150 in the embodiment of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Non-volatile memories include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory.

[0223] Volatile memory includes random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 1150 of the electronic device described in the embodiments of the present invention includes, but is not limited to, the above and any other suitable types of memory.

[0224] In the embodiment of the present invention, the memory 1150 stores the following elements of the operating system 1151 and the application 1152: executable modules, data structures, or subsets thereof, or extended sets thereof.

[0225] Specifically, the operating system 1151 includes various system programs, such as a framework layer, a core library layer, and a driver layer, which are used to implement various basic services and process hardware-based tasks. The application 1152 includes various application programs, such as a media player and a browser, which are used to implement various application services. The program that implements the method of the embodiment of the present invention may be included in the application 1152. The application 1152 includes applets, objects, components, logic, data structures, and other computer system executable instructions that perform specific tasks or implement specific abstract data types.

[0226] In addition, an embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned self-supervised learning method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0227] Computer-readable storage media include: permanent and non-permanent, removable and non-removable media, which are tangible devices that can retain and store instructions for use by instruction execution devices. Computer-readable storage media include: electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, and any suitable combination of the above. Computer-readable storage media include: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette storage, magnetic disk storage or other magnetic storage devices, memory sticks, mechanical encoding devices (such as punched cards or raised structures with grooves in which instructions are recorded), or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined in the embodiments of the present invention, computer-readable storage media does not include temporary signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (such as light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0228] In the several embodiments provided in this application, it should be understood that the disclosed devices, electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be an electrical, mechanical or other form of connection.

[0229] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (including: a personal computer, a server, a data center or other network device) to perform all or part of the steps of the method described in each embodiment of the present invention. The above-mentioned storage medium includes the various media that can store program codes as listed above.

[0230] In describing the embodiments of the present invention, those skilled in the art should understand that the embodiments of the present invention can be implemented as methods, apparatuses, electronic devices, and computer-readable storage media. Therefore, the embodiments of the present invention can be specifically implemented in the following forms: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software. In addition, in some embodiments, the embodiments of the present invention can also be implemented in the form of a computer program product in one or more computer-readable storage media, wherein the computer-readable storage medium contains computer program code.

[0231] The embodiments of the present invention describe the provided methods, devices, and electronic devices through flowcharts and / or block diagrams.

[0232] It should be understood that each block in the flowchart and / or block diagram, as well as combinations of blocks in the flowchart and / or block diagram, can be implemented by computer-readable program instructions. These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine. These computer-readable program instructions are executed by the computer or other programmable data processing device to produce a device that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0233] These computer-readable program instructions may also be stored in a computer-readable storage medium that enables a computer or other programmable data processing device to operate in a specific manner. In this way, the instructions stored in the computer-readable storage medium produce an instruction device product that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0234] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby enabling the instructions executed on the computer or other programmable data processing apparatus to provide a process that implements the functions / operations specified by the blocks in the flowchart and / or block diagram.

[0235] The above description is merely a specific implementation of the embodiments of the present invention, but the scope of protection of the embodiments of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the embodiments of the present invention should be included in the scope of protection of the embodiments of the present invention. Therefore, the scope of protection of the embodiments of the present invention should be based on the scope of protection of the claims.

Claims

1. A method of self-supervised learning, characterized in that include: Acquire an unlabeled data set, wherein the data set includes multiple frames of point clouds, and each frame of the point cloud includes multiple real points; Organizing a row of the real points in a vertical direction into a body column, wherein the body column is provided with body column motion parameters, and each of the real points in the body column has motion parameters identical to the body column motion parameters; Moving each of the real points of the current frame to the next frame according to the corresponding body column motion parameters, and determining a prediction point of the next frame; Determine a first loss term by adding a sum of first minimum distances corresponding to the plurality of predicted points in the next frame to a sum of second minimum distances corresponding to the plurality of real points in the next frame, and generate a loss function including the first loss term, where the first minimum distance is the minimum value of the distances between the predicted point in the next frame and the plurality of real points in the next frame, and the second minimum distance is the minimum value of the distances between the real point in the next frame and the plurality of predicted points in the next frame; A self-supervised learning process is performed according to the loss function to determine body column motion parameters of the body column.

2. The method according to claim 1, characterized in that The relationship between the first loss term is as follows: in, represents the first loss term, represents the i-th predicted point in the t-th frame, represents a set of multiple prediction points of the t-th frame, represents the jth real point in the tth frame, p t Represents a set of multiple real points in the t-th frame.

3. The method according to claim 1, characterized in that The data set also includes multiple frames of image data that match the point cloud; The method further comprises: Projecting the real point of the target frame onto the image plane where the image data is located, determining a first pixel point, and determining an ego optical flow of the first pixel point based on a change in the posture of the ego vehicle; the target frame is the current frame or the next frame, and the first pixel point is a pixel point in the image data of the target frame that matches the real point of the target frame; determining an overall optical flow of the first pixel according to the image data, and determining an object optical flow of the first pixel according to the overall optical flow and the self optical flow; Projecting the body column motion parameters corresponding to the real point of the target frame onto the image plane to determine the corresponding projected optical flow; A second loss term is determined according to a difference between the object optical flow and the projection optical flow, and the second loss term is added to the loss function.

4. The method according to claim 3, characterized in that Determining the second loss term according to the difference between the object optical flow and the projection optical flow includes: Determine the second loss term: in, represents the second loss term, (u i ,v i ) t Represents the t-th frame and the i-th real point The corresponding first pixel, Indicates the first pixel (u i ,v i ) t The projected optical flow, F obj (u i ,v i ) t Indicates the first pixel (u i ,v i ) t The object optical flow.

5. The method according to claim 1, wherein The data set also includes multiple frames of image data that match the point cloud; The method further comprises: Projecting the real point of the target frame onto the image plane where the image data is located, determining a first pixel point, and determining an ego optical flow of the first pixel point based on a change in the posture of the ego vehicle; the target frame is the current frame or the next frame, and the first pixel point is a pixel point in the image data of the target frame that matches the real point of the target frame; determining an overall optical flow of the first pixel based on the image data, and determining an object optical flow of the first pixel based on the overall optical flow and the self optical flow; and determining a probability that the first pixel corresponds to a dynamic object based on the magnitude of the object optical flow, and further determining a probability that the real point of the target frame corresponding to the first pixel corresponds to the dynamic object; the probability is positively correlated with the magnitude of the object optical flow; Determining a first loss term according to a minimum distance between the predicted point of the next frame and the true point of the next frame includes: The probability that the real point corresponds to the dynamic object is used as a weighting coefficient, the minimum distance between the predicted point of the next frame and the real point of the next frame is weighted, and the first loss term is determined according to the minimum distance after weighted processing.

6. The method according to claim 5, characterized in that The determining, based on the magnitude of the object optical flow, the probability that the first pixel corresponds to the dynamic object includes: Determine the probability that the first pixel corresponds to a dynamic object: Among them, (u i ,v i ) t represents the first pixel point corresponding to the i-th real point in the t-th frame, F obj (u i ,v i ) t Indicates the first pixel point (u i ,v i ) t The object optical flow, Indicates the first pixel point (u i ,v i ) t The probability of corresponding dynamic objects, α is the smoothing factor, and τ is the tolerance.

7. The method according to claim 3 or 5, characterized in that The projecting the real point of the target frame onto the image plane where the image data is located, determining a first pixel point, and determining the self-optical flow of the first pixel point according to the change in the posture of the self-vehicle, includes: Determine the relative position T between the laser radar for collecting the point cloud and the camera device for collecting the image data L→C , and determine the intrinsic parameter K of the camera device; Projecting the real point of the target frame onto the image plane where the image data is located according to the relative posture and the intrinsic parameters to determine a first pixel point; Determine the pose change T of the ego vehicle in the target frame t→t+1 , and determine the self-optical flow of the first pixel according to: in, represents the i-th real point in the t-th frame, (u i ,v i ) t represents the first pixel corresponding to the i-th real point in the t-th frame, F ego (u i ,v i ) t Indicates the first pixel point (u i ,v i ) t Self-light flow.

8. The method according to claim 1, characterized in that The component of the body column motion parameter in the vertical direction is zero.

9. The method according to claim 1 or 3, characterized in that Also includes: generating a body column motion field, wherein the body column motion field includes a plurality of body column motion parameters of the body columns; A third loss term is determined according to the components and gradients of the body column motion field in multiple directions, and the third loss term is added to the loss function.

10. The method according to claim 9, characterized in that Determining the third loss term according to the components and gradients of the body column motion field in multiple directions includes: Determine the components and gradients of the body column motion field in the x-direction and the y-direction in the horizontal plane, and determine the third loss term: in, represents the component of the body column motion field in the x direction at the t-th frame, represents the component of the body column motion field in the y direction at the t-th frame, represents the gradient in the x direction, Represents the gradient in the y direction.

11. The method according to claim 1, wherein Generating a loss function including the first loss term includes: When there are other loss terms except the first loss term, a balance coefficient is set for the first loss term and the other loss terms, and a loss function is generated.

12. A self-supervised learning device, characterized in that include: An acquisition module is used to acquire an unlabeled data set, wherein the data set includes multiple frames of point clouds, and each frame of the point cloud includes multiple real points; a body column module, configured to organize a row of the real points in a vertical direction into a body column, wherein the body column is provided with body column motion parameters, and each of the real points in the body column has motion parameters identical to the body column motion parameters; A prediction module, configured to move each of the real points of the current frame to the next frame according to the corresponding body column motion parameters, and determine a prediction point for the next frame; a loss determination module, configured to determine a first loss term based on a sum of first minimum distances corresponding to the plurality of predicted points in the next frame and a sum of second minimum distances corresponding to the plurality of real points in the next frame, and generate a loss function including the first loss term, wherein the first minimum distance is a minimum value of distances between the predicted point in the next frame and the plurality of real points in the next frame, and the second minimum distance is a minimum value of distances between the real point in the next frame and the plurality of predicted points in the next frame; A processing module is used to perform self-supervised learning processing according to the loss function to determine the body column motion parameters of the body column.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the self-supervised learning method according to any one of claims 1 to 11 are implemented.

14. An electronic device comprising a bus, a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the transceiver, the memory, and the processor are connected via the bus, wherein: When the computer program is executed by the processor, the steps of the self-supervised learning method according to any one of claims 1 to 11 are implemented.

Images