Verification method and model training method and device for registered point cloud in high-definition map

Abstract

This disclosure provides a method for verifying registered point clouds and a training method and apparatus for optical flow value estimation models, relating to the field of artificial intelligence, specifically to intelligent transportation, computer vision, virtual reality, and augmented reality, and applicable to high-precision map production and 3D scene generation. The specific implementation of the point cloud verification method is as follows: based on the transformation relationship between the two registered point clouds, the two point clouds are transformed to the same coordinate space to obtain the transformed two point clouds; multiple optical flow values ​​are determined for the transformed two point clouds; these multiple optical flow values ​​include the optical flow value between the two graticles in each of multiple raster pairs; the two graticles in each raster pair are corresponding graticles in two raster grids; the two raster grids correspond to the transformed two point clouds respectively; and the registration accuracy of the two registered point clouds is determined based on the multiple optical flow values.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, specifically to technologies such as intelligent transportation, computer vision, virtual reality, and augmented reality, and can be applied to high-precision map production and 3D scene generation. Background Technology

[0002] With the development of computer and electronic technologies, computer vision technology has been widely applied in various scenarios. For example, it can be used for high-precision map creation and 3D scene generation. Point cloud registration is one of the key technologies in high-precision map creation and 3D scene generation. High-precision point cloud registration is essential for generating realistic high-precision maps and 3D scenes. Summary of the Invention

[0003] This disclosure aims to provide a method, apparatus, device, and medium for verifying the accuracy of point cloud registration and training a model for optical flow estimation.

[0004] According to one aspect of this disclosure, a method for verifying registered point clouds is provided, comprising: transforming the two registered point clouds to the same coordinate space according to the transformation relationship between the two frames of point clouds, to obtain two transformed point clouds; determining multiple optical flow values ​​for the two transformed point clouds; the multiple optical flow values ​​include optical flow values ​​between two graticles in each of multiple graticle pairs; the two graticles in each graticle pair are corresponding two graticles in two graticles; the two graticles correspond to the two transformed point clouds respectively; and determining the registration accuracy of the two registered point clouds according to the multiple optical flow values.

[0005] According to another aspect of this disclosure, a training method for an optical flow estimation model is provided, comprising: transforming two registered point clouds to the same coordinate space according to the transformation relationship between the two registered point clouds, obtaining two transformed point clouds; randomly transforming the first point cloud in the two transformed point clouds to obtain a third point cloud; determining the optical flow value for the third point cloud and the first point cloud according to the positional relationship between the third point cloud and the first point cloud, as the true optical flow value; processing the third point cloud and the second point cloud in the two transformed point clouds using the optical flow estimation model to obtain multiple optical flow prediction values; the multiple optical flow prediction values ​​include the optical flow prediction value between the two graticles in each of the multiple graticle pairs; the two graticles in each graticle pair are two corresponding graticles in two graticles; the two graticles include a first graticle corresponding to the third point cloud and a second graticle corresponding to the second point cloud; and training the optical flow estimation model according to the difference between the multiple optical flow prediction values ​​and the true optical flow value.

[0006] According to another aspect of this disclosure, a verification device for registered point clouds is provided, comprising: a point cloud transformation module, configured to transform the two registered point clouds to the same coordinate space according to the transformation relationship between the two registered point clouds, to obtain the transformed two point clouds; an optical flow value estimation module, configured to determine multiple optical flow values ​​for the transformed two point clouds; the multiple optical flow values ​​include the optical flow value between two graticles in each of multiple graticle pairs; the two graticles in each graticle pair are corresponding two graticles in two graticles; the two graticles correspond to the two transformed point clouds respectively; and a registration verification module, configured to determine the registration accuracy of the two registered point clouds according to the multiple optical flow values.

[0007] According to another aspect of this disclosure, a training apparatus for an optical flow value estimation model is provided, comprising: a point cloud transformation module, configured to transform two registered point clouds to the same coordinate space according to the transformation relationship between the two registered point clouds, to obtain two transformed point clouds; a random transformation module, configured to randomly transform the first point cloud in the two transformed point clouds to obtain a third point cloud; an optical flow value determination module, configured to determine the optical flow value for the third point cloud and the first point cloud according to the positional relationship between the third point cloud and the first point cloud, as the true value of the optical flow value; and an optical flow value estimation module, configured to: An optical flow estimation model is used to process the point cloud of the third frame and the second frame of the transformed two-frame point cloud to obtain multiple optical flow prediction values. The multiple optical flow prediction values ​​include the optical flow prediction values ​​between the two grids in each grid pair. The two grids in each grid pair are two corresponding grids in two grid networks. The two grid networks include a first grid network corresponding to the third frame point cloud and a second grid network corresponding to the second frame point cloud. A model training module is also used to train the optical flow estimation model based on the differences between the multiple optical flow prediction values ​​and the true optical flow values.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the verification method for registration point clouds and / or the training method for optical flow value estimation models provided in this disclosure.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the verification method for registration point clouds and / or the training method for optical flow value estimation models provided in this disclosure.

[0010] According to another aspect of this disclosure, a computer program product is provided, including a computer program / instructions stored on at least one of a readable storage medium and an electronic device, wherein the computer program / instructions, when executed by a processor, implement the verification method for registration point clouds and / or the training method for optical flow value estimation models provided in this disclosure.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0013] Figure 1 This is a schematic diagram illustrating the application scenario of the verification method for registration point clouds and the training method and apparatus for optical flow value estimation models according to embodiments of this disclosure.

[0014] Figure 2 This is a flowchart illustrating the verification method for registered point clouds according to an embodiment of the present disclosure;

[0015] Figure 3 This is a schematic diagram illustrating the principle of obtaining multiple optical flow values ​​according to embodiments of the present disclosure;

[0016] Figure 4 This is a schematic diagram illustrating the principle of determining registration accuracy according to embodiments of this disclosure;

[0017] Figure 5 This is a schematic diagram of the structure of an optical flow value estimation model according to an embodiment of the present disclosure;

[0018] Figure 6 This is a flowchart of a training method for an optical flow value estimation network according to an embodiment of the present disclosure;

[0019] Figure 7 This is a schematic diagram of the structure of a verification device for registration point clouds according to an embodiment of the present disclosure;

[0020] Figure 8 This is a structural block diagram of a training apparatus for an optical flow value estimation network according to an embodiment of the present disclosure;

[0021] Figure 9 This is a block diagram of an electronic device used to implement the verification method for registration point clouds and / or the training method for optical flow value estimation networks according to embodiments of the present disclosure. Detailed Implementation

[0022] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0023] The following is an explanation of the technical terms used in this disclosure:

[0024] Optical flow is the instantaneous velocity of pixels moving on the imaging plane of a moving object in space. It is a method that uses the changes of pixels in the time domain in an image sequence and the correlation between adjacent frames to find the correspondence between the previous frame and the current frame, thereby calculating the motion information of objects between adjacent frames.

[0025] A point cloud frame is defined in this disclosure as the point cloud data accumulated within 100ms (corresponding to a typical value of 10Hz).

[0026] Point cloud gridding can be divided into 2D rasterization and 3D voxelization. 2D rasterization primarily processes point cloud data acquired by airborne LiDAR, and the resulting data is observed from a top-down perspective. The data obtained from 2D rasterization resembles regular image data, and 2D rasterization is achieved by projecting the point cloud onto a grid on the XOY plane.

[0027] Point cloud registration aims to compare or fuse images of the same object acquired under different conditions. Specifically, point cloud registration refers to taking two frames of point clouds as input and outputting a transformation T such that the point cloud obtained after transforming one frame of point cloud by transformation T has the highest possible degree of overlap with the point cloud of the other frame, so that points corresponding to the same spatial location in the two frames of point clouds correspond one-to-one.

[0028] The accuracy of point cloud registration affects the realism of the generated high-precision maps and 3D scenes. For example, in autonomous driving scenarios, the horizontal registration accuracy of point clouds also affects the positioning accuracy of autonomous vehicles. To improve point cloud registration accuracy and continuously optimize the algorithm, the accuracy of point cloud registration can be verified. For example, the accuracy of point cloud registration can be verified based on local ghosting, local variance changes, etc. However, these methods only focus on local features, and the verification effect is not ideal when double-layer planes or high noise occur locally.

[0029] To address this problem, this disclosure provides a method for verifying registered point clouds and a method, apparatus, device, and medium for training an optical flow estimation model. The following will first combine... Figure 1The application scenarios of the methods and apparatus provided in this disclosure are described.

[0030] Figure 1 This is a schematic diagram illustrating the application scenario of the registration point cloud verification method and the optical flow value estimation model training method and apparatus according to embodiments of this disclosure.

[0031] like Figure 1 As shown, the application scenario 100 of this embodiment may include an electronic device 110, which may be a variety of electronic devices with processing functions, including but not limited to smartphones, tablets, laptops, desktop computers, servers, and autonomous driving systems, etc.

[0032] The electronic device 110 can, for example, process the point cloud 120 registered by the autonomous driving system on the autonomous vehicle to determine the registration accuracy 130 of the registered point cloud, thereby determining whether to use the registered point cloud for the localization and path planning of the autonomous vehicle.

[0033] In one embodiment, the matching degree of the two registered point clouds can be determined, for example, based on the matching relationship of features in the top-view direction, i.e., the matching relationship of the bird's-eye view of the two registered point clouds. Alternatively, the two point clouds can first be transformed into point clouds in the same coordinate space according to their transformation relationship, then the two transformed point clouds in the same coordinate space can be projected onto a horizontal grid, and the matching degree of the two registered point clouds can be determined based on the optical flow values ​​of corresponding grid cells in the grid. This avoids the problems inherent in verification methods that only consider local features.

[0034] In one embodiment, an optical flow estimation model 140 can be used to process two frames of point clouds transformed to the same spatial coordinates to predict optical flow values. This optical flow estimation model 140 can be a pre-trained model.

[0035] In one embodiment, the application scenario 100 may further include a server 150. The electronic device 110 can communicate with the server 150 via a network. The server 150 can be a background management server that supports the operation of client applications installed on the electronic device 110. For example, an optical flow estimation model 140 can be trained by the server 150 and then sent to the electronic device 110. The electronic device 110 can use this optical flow estimation model to verify the registration of two point clouds. In this way, the point cloud registration algorithm used by the autonomous driving system of the autonomous vehicle can be optimized based on the verification results.

[0036] It should be noted that the registration point cloud verification method provided in this disclosure can be executed by electronic device 110. Correspondingly, the registration point cloud verification device provided in this disclosure can be installed in electronic device 110. The optical flow value estimation model training method provided in this disclosure can be executed by server 150. Correspondingly, the optical flow value estimation model training device provided in this disclosure can be installed in server 150.

[0037] It should be understood that Figure 1 The number and type of electronic devices 110 and servers 150 shown are merely illustrative. Depending on implementation requirements, any number and type of electronic devices 110 and servers 150 may be used.

[0038] The following will combine Figures 2-5 The verification method for the registration point cloud provided in this disclosure is described in detail.

[0039] Figure 2 This is a flowchart illustrating the verification method for registration point clouds according to embodiments of the present disclosure.

[0040] like Figure 2 As shown, the registration point cloud verification method 200 of this embodiment may include operations S210 to S230.

[0041] In operation S210, based on the transformation relationship between the two registered point clouds, the two point clouds are transformed to the same coordinate space to obtain the transformed two point clouds.

[0042] For example, the two point clouds to be registered include point cloud P s (source) and point cloud P t (target), the transformation relationship between the two point clouds is T = (R, t), and this embodiment can use the following formula (1) to transform the point cloud P. s (source) is transformed to obtain the point cloud P s (source) Convert to point cloud P t The coordinate space where (target) is located. Here, R represents the point cloud P. t (target) relative to point cloud P s The rotation matrix of (source) is given, where t is the rotation matrix of the point cloud P. t (target) relative to point cloud P s The displacement matrix of (source). Then the transformed point cloud P s '(source) and point cloud P t (target) refers to the transformed point cloud from the two frames.

[0043] RP s (source)+t=P s'(source) formula (1)

[0044] In operation S220, multiple optical flow values ​​are determined for the two transformed point clouds.

[0045] In this embodiment, the two transformed point cloud frames P can be first... s '(source) and P t The target is projected onto a horizontal plane and rasterized in two dimensions to obtain a grid. The number of grid cells in the grid can be set according to actual needs, and the size of a single grid cell (i.e., the resolution of the two-dimensional grid) can also be set according to actual needs. For example, the size of a single grid cell can be 20cm*20cm, and the grid can include 512 rows and 512 columns of cells.

[0046] For example, given two transformed point clouds, if the first point cloud in the first frame is registered with the second point cloud in the second frame, the optical flow value between the first grid cell containing the projection point of the first point cloud projected onto the horizontal plane and the second grid cell containing the projection point of the second point cloud projected onto the horizontal plane can be used as an optical flow value for the two transformed point clouds. Correspondingly, the first grid cell and the second grid cell are two corresponding grid cells, forming a grid pair. Thus, for the first point cloud, projecting onto the horizontal plane and rasterizing it in two dimensions yields a corresponding grid; for the second point cloud, projecting onto the horizontal plane and rasterizing it in two dimensions yields a corresponding grid. The registered first and second point clouds can form a point cloud pair. If there are multiple point cloud pairs between the first and second point clouds, multiple optical flow values ​​can be obtained for the two transformed point clouds.

[0047] It is understandable that if point cloud P s (source) and point cloud P t If the registration accuracy of (target) is high, then the point cloud P s (source) The first raster obtained by projecting onto a horizontal plane and then rasterizing it in two dimensions, and the point cloud P t The second raster obtained by rasterizing the target point onto the horizontal plane will overlap. However, if the registration is poor, the positions of the two raster points will differ. In the two registered point clouds, the raster position of the projected point of one point cloud in the first raster should be the same as the raster position of the projected point of the other point cloud in the second raster.

[0048] The optical flow value between the first and second grids can be determined, for example, by their positional relationship. For instance, if the position of the first grid on the horizontal plane differs from the position of the second grid on the same horizontal plane by 2 grids in the row direction and 3 grids in the column direction, then the optical flow value can be represented by (2, 3). Alternatively, the optical flow value can be...

[0049] In operation S230, the registration accuracy of the two point clouds to be registered is determined based on multiple optical flow values.

[0050] According to embodiments of this disclosure, if the registration of the two point clouds is accurate, the optical flow value determined by operation S220 should be very small or even close to 0. Conversely, if the registration of the two point clouds is inaccurate, the optical flow value determined by operation S220 should be relatively large. Therefore, in operation S230, a threshold value can be set for the optical flow value, for example. If the average of multiple optical flow values ​​is greater than the threshold, it can be determined that the registration of the two point clouds is inaccurate. If the average of multiple optical flow values ​​is less than or equal to the threshold, it can be determined that the registration of the two point clouds is accurate. The threshold value set for the optical flow value can be set according to actual needs; for example, it can be set to a value greater than 0 and close to 0, such as 2 or 3. The threshold value can be an integer or a non-integer value, and this disclosure does not limit this.

[0051] The technical solution of this disclosure verifies the registration of two point clouds based on grid-by-grid optical flow values. This avoids focusing on only some features and prevents overfitting. Therefore, it improves the reliability of registration accuracy verification and provides accurate information for optimizing point cloud registration algorithms.

[0052] In one embodiment, optical flow values ​​can be determined using deep learning algorithms to improve the sensitivity of the determined optical flow values ​​to noise. By comprehensively considering global features through deep learning, dependence on local features can be further avoided, thus improving the accuracy of the determined optical flow values.

[0053] For example, this embodiment can use an optical flow estimation model to process the two transformed point clouds to obtain multiple optical flow values. For instance, the two transformed point clouds can be input into the optical flow estimation model, which then outputs multiple optical flow values. The optical flow estimation model can include at least one of the following: an optical flow network (FlowNet) or a recurrent all-pairs field transforms for optical flow (RAFT). For example, the optical flow estimation model can also include a network for transforming the two transformed point clouds into a two-dimensional raster image, providing a two-dimensional image for the optical flow estimation network (FlowNet or RAFT). The following will combine... Figure 3 The principle of obtaining multiple optical flow values ​​using deep learning algorithms is described in detail.

[0054] Figure 3 This is a schematic diagram illustrating the principle of obtaining multiple optical flow values ​​according to an embodiment of the present disclosure.

[0055] like Figure 3 As shown, in this embodiment 300, the optical flow estimation model 310 may include a feature extraction network 311 and an optical flow estimation network 312. The feature extraction network 311 can be a feature extraction network for extracting bird's-eye view features of the point cloud, converting the two transformed point cloud frames 301 and 302 into two-dimensional feature maps, respectively, to obtain two first feature maps 303 and 304. That is, the feature extraction network 311 extracts the feature maps of the two transformed point cloud frames at their respective top-view angles. The two obtained first feature maps 303 and 304 can be used as inputs to the optical flow estimation network 312, which outputs multiple optical flow values ​​305.

[0056] The feature extraction network 311 can be composed of, for example, the Pillar Feature Net and the backbone network (e.g., a 2D CNN) in the PointPillars 3D point cloud object detection algorithm. Alternatively, the feature extraction network 311 can be a portion of a PointNet point cloud neural network, which may be a network preceding the Multilayer Perceptron (MLP) in the classification network of the PointNet point cloud neural network. Alternatively, the feature extraction network 311 can be any network capable of converting point cloud data into two-dimensional features; this disclosure does not limit this.

[0057] The optical flow estimation network 312 can be either FlowNet or RAFT, and this disclosure does not limit it.

[0058] This embodiment obtains global features of the point cloud by setting a feature extraction network in the optical flow estimation model to extract feature maps of the point cloud from a top-down angle. This improves the comprehensiveness of the data representation information input to the optical flow estimation network and reduces the computational load of the optical flow estimation network to some extent compared to directly inputting a 2D image. Furthermore, by using the Pillar FeatureNet from the PointPillars 3D point cloud object detection algorithm as the feature extraction network, the expressive power of the two obtained first feature maps can be further improved. This is because the features extracted by the Pillar FeatureNet are the fused features from multiple different scales.

[0059] Figure 4 This is a schematic diagram illustrating the principle of determining registration accuracy according to an embodiment of this disclosure.

[0060] According to embodiments of this disclosure, the optical flow value estimated using the optical flow value estimation model includes the optical flow values ​​of all grid pairs consisting of two grids at the same position in the two grids corresponding to the transformed two frame point clouds. For example, if the set grid includes a 512-row, 512-column grid, the predicted optical flow value can be represented by a 512×512 optical flow value matrix, where each element represents the optical flow value of a grid pair. Therefore, when estimating the optical flow value using the optical flow value estimation model, the obtained optical flow value may include, in addition to the first optical flow value between the two grids corresponding to the two registered point clouds in the processed two frame point clouds, a second optical flow value between the grid in one grid and the corresponding grid in the other grid for the unregistered point cloud. The presence of these second optical flow values ​​may affect the accuracy of the registration verification. An unregistered point cloud in one frame can be understood as a point whose spatial location was not captured when acquiring another frame point cloud.

[0061] For example, such as Figure 4 As shown, in this embodiment 400, an overlap estimation network 413 can be set in the optical flow value estimation model 410 to estimate whether two grids with corresponding positions in the two grids corresponding to the two transformed point clouds have an overlap relationship. An overlap relationship indicates that both grids include the projected points of the point cloud. For example, for the first and second grids in two overlapping grids, in the two transformed point clouds, the first frame point cloud includes the point cloud projected onto the first grid, and the second frame point cloud includes the point cloud projected onto the second grid.

[0062] After obtaining the overlap relationships of all raster pairs, target raster pairs with overlapping graticules can be selected from all raster pairs. The two graticules in this target raster pair are the graticules corresponding to the two point clouds determined by the registration results. This embodiment can determine the registration accuracy solely based on the optical flow values ​​of this target raster pair. Thus, by eliminating the optical flow values ​​of raster pairs corresponding to point clouds that were not registered according to the registration results, the accuracy of the determined registration is improved.

[0063] For example, such as Figure 4As shown, in this embodiment 400, when determining the registration accuracy, the transformed point clouds 401 and 402 can be input into the feature extraction network 411 to obtain two first feature maps 403 and 404. Subsequently, in this embodiment, the two first feature maps 403 and 404 can be input into the optical flow value estimation network 412, which outputs multiple optical flow values ​​405. Simultaneously, based on the two first feature maps 403 and 404, the overlap estimation network 413 can estimate the overlap relationship between the two grids in each grid pair. For example, the two first feature maps 403 and 404 can be input into the overlap estimation network 413, which outputs an overlap probability matrix 406. The size of the overlap probability matrix 406 can be equal to the size of the optical flow value matrix described above.

[0064] In one embodiment, the two first feature maps 403 and 404 can be concatenated along the channel dimension, and the concatenated image can be used as the input to the overlap estimation network 413. The overlap estimation network 413 can be composed of convolutional layers and activation layers, for example. The activation layers can be based on activation functions such as sigmoid. The number and size of the convolutional kernels in the convolutional layers can be set according to actual needs, and this disclosure does not limit them.

[0065] After obtaining the overlap probability matrix 406, this embodiment 400 can determine the target element in the overlap probability matrix 406 whose value is greater than or equal to the probability threshold, and take the raster pair corresponding to the target element as the target raster pair, that is, the probability value that the two raster pairs included in the target raster pair have an overlap relationship is greater than or equal to the probability threshold. In this embodiment, the optical flow value represented by the element at the position corresponding to the target element in the optical flow value matrix representing multiple optical flow values ​​405 can be taken as the optical flow value 407 of the target raster pair. This embodiment can compare the mean of the optical flow values ​​of the target raster pair with the threshold. If the mean is greater than the threshold, it is determined that the registration of the two frame point clouds is inaccurate. The probability threshold can be any value such as 0.6 or 0.8, and this disclosure does not limit it.

[0066] Figure 5 This is a schematic diagram of the optical flow value estimation model according to an embodiment of the present disclosure.

[0067] In one embodiment, when determining the overlap relationship, the feature map of the input overlap estimation network can be adjusted based on the optical flow value predicted by the optical flow value estimation network. In this way, the information represented by the features of the input overlap estimation network can eliminate the influence of optical flow on the overlap relationship, thereby improving the accuracy of the estimated overlap relationship.

[0068] For example, multiple predicted optical flow values ​​can be used to adjust one of the two first feature maps. Then, the adjusted feature map is concatenated with the other first feature map along the channel dimension and input into the overlap estimation network, which outputs the overlap probability matrix.

[0069] In one embodiment, the optical flow estimation network may include at least a feature extraction subnetwork and an optical flow estimation subnetwork. The feature extraction subnetwork is used to further extract features from the two first feature maps obtained by the feature extraction network, so as to extract the features of each pixel in the two first feature maps and provide more accurate information for the estimation of optical flow values. Specifically, the feature extraction subnetwork can be used to process the two first feature maps separately to obtain two second feature maps. After obtaining the two second feature maps, the optical flow estimation subnetwork can be used to process the two second feature maps to obtain an optical flow matrix including multiple optical flow values.

[0070] In overlap estimation, two second feature maps extracted by the feature extraction sub-network can be used. This provides pixel-by-pixel features to the overlap estimation network, improving its estimation accuracy. Specifically, during overlap estimation, one of the two second feature maps can be transformed using the optical flow matrix obtained from the optical flow estimation network, resulting in a transformed feature map. Then, based on this transformed feature map and the other feature map from the two second feature maps, the overlap estimation network estimates the overlap relationship between the two grids in each grid pair. Specifically, the transformed feature map and the other feature map can be concatenated along the channel dimension, and the concatenated feature map is input into the overlap estimation network, which outputs an overlap probability matrix.

[0071] In one embodiment, the optical flow estimation network in the optical flow estimation model can employ the RAFT framework. For example... Figure 5 As shown, the optical flow estimation model 500 in this embodiment includes a feature extraction network 510, an optical flow estimation network 520, and an overlap estimation network 530. The optical flow estimation network 520 may include a feature extraction sub-network 521, a semantic feature extraction sub-network 522, a similarity calculation sub-network 523, and an update iteration sub-network 524.

[0072] After the transformed point clouds 501 and 502 are processed by the feature extraction network 510, two first feature maps 503 and 504 are obtained. Subsequently, in this embodiment, the two first feature maps 503 and 504 can be input into the feature extraction sub-network 521 respectively, so that the feature extraction sub-network 521 extracts the features of the two first feature maps pixel by pixel, thereby obtaining two second feature maps corresponding to the two first feature maps respectively. At the same time, one of the two first feature maps can be input into the semantic feature extraction sub-network 522, so that the semantic feature extraction sub-network 522 extracts the semantic features of the one feature map.

[0073] Two second feature maps can be used as inputs to a similarity calculation subnetwork 523, which calculates the dot product between corresponding feature vectors in the two second feature maps, thereby constructing a 4D correlation space.

[0074] Subsequently, in this embodiment, the 4D relevant spatial features output by the similarity calculation subnetwork 523 and the semantic features extracted by the semantic feature extraction subnetwork 522 can be input into the update iteration subnetwork 524. This update iteration subnetwork 524 uses GRU and convolution to iteratively calculate the residual optical flow, and then adds it to the optical flow output from the previous iteration for compensation. The number of iterations can be set according to actual needs, and this disclosure does not limit this. In this embodiment, the optical flow obtained from the last iteration can be used as the optical flow value matrix 505.

[0075] After obtaining two second feature maps, this embodiment can concatenate the two second feature maps in the channel direction and input them into the convolutional layer CNN 531 in the overlap estimation network 530. The features output by the convolutional layer CNN 531 are input into the activation layer 532, and the activation layer 532 outputs the overlap probability matrix 506.

[0076] To facilitate the implementation of operations for determining multiple optical flow values, this disclosure also provides a training method for an optical flow value estimation model, which will be discussed below in conjunction with... Figure 6 The method is described in detail.

[0077] Figure 6 This is a flowchart of a training method for an optical flow value estimation network according to an embodiment of the present disclosure.

[0078] like Figure 6 As shown, the training method 600 of the optical flow value estimation network in this embodiment may include operations S610 to S650.

[0079] In operation S610, based on the transformation relationship between the two registered point clouds, the two point clouds are transformed to the same coordinate space, resulting in two transformed point clouds. The implementation principle of operation S610 is similar to that of operation S210 described above, and will not be repeated here. The difference is that the two registered point clouds in operation S610 are accurately registered point clouds, which are used as samples for training the optical flow value estimation model.

[0080] In operation S620, the first frame of the transformed point cloud is randomly transformed to obtain the third frame of the point cloud.

[0081] According to embodiments of this disclosure, the random transformation may include at least one of rotational transformation and translational transformation. The purpose of this random transformation is to add motion to the first frame point cloud relative to the second frame point cloud among the two transformed frames, thereby generating optical flow. The principle of this random transformation is similar to that embodied in formula (1) described above; for example, the random transformation can be represented by T = (R, t). This facilitates the optical flow value estimation network in predicting the predicted optical flow value.

[0082] In operation S630, the optical flow value for the third frame point cloud and the first frame point cloud is determined based on the positional relationship between them, and is used as the true value of the optical flow value.

[0083] According to embodiments of this disclosure, based on the aforementioned random perturbation, the actual optical flow value of the third frame point cloud relative to the first frame point cloud can be calculated as the true optical flow value. This actual optical flow value can be calculated, for example, using the Lucas-Kanade algorithm.

[0084] In one embodiment, when determining the true value of the optical flow value, for example, the third grid corresponding to the first frame point cloud and the first grid corresponding to the third frame point cloud can be determined first based on the first frame point cloud and the third frame point cloud, respectively. That is, the first frame point cloud and the third frame point cloud are projected and rasterized in two dimensions to obtain the third grid and the first grid respectively. For example, if the coordinates of the projection point of point cloud p3 in the third frame point cloud in the grid width direction on the horizontal plane are set to x3 and the coordinates of the projection point in the grid width direction on the horizontal plane are set to y3, then the value of the grid in the first grid corresponding to the third frame point cloud can be determined to be 1, which can be expressed as the following formula (2).

[0085] M1[int(y3 / dy), int(x3 / dx)]=1 Formula (2)

[0086] Where M1[i,j] represents the value of the grid cell in the i-th row and j-th column of the first grid, and int() represents rounding up. The grid cell in the first grid corresponding to the point cloud of the third frame that has no projection point is 0.

[0087] Subsequently, a grid pair is formed by combining one grid cell from the first grid with a grid cell at the same position in the third grid. This results in multiple grid pairs. In this embodiment, the actual optical flow value for each grid pair can be determined based on the positional difference between the two grid cells, thus obtaining multiple actual optical flow values.

[0088] In one embodiment, when determining the true value of optical flow, the optical flow value can be determined only for point clouds in the third frame point cloud that indicate the same spatial point as the point clouds in the second frame point cloud. This is because calculating the optical flow value is meaningless for point clouds indicating different spatial points. Therefore, when determining the true value of optical flow, for example, a second grid corresponding to the second frame point cloud can also be determined based on the second frame point cloud. The principle of determining the second grid is similar to the principle of determining the first grid. After obtaining the second grid and the third grid, two grids with an overlapping relationship can be identified to form a target grid pair. The two grids with an overlapping relationship are: the first grid has the projection point of the point cloud in the first frame point cloud, and the second grid has the projection point of the point cloud in the second frame point cloud. That is, the first frame point cloud includes the first point cloud projected onto the first grid, and the second frame point cloud includes the second point cloud projected onto the second grid. It can be understood that the position of the first grid in the third grid corresponds to the position of the second grid in the second grid. For example, if the third grid is set as M3 and the second grid is set as M2, the overlap relationship between the two grids can be determined by the following formula (3).

[0089] Overlap = M3 & M2 Formula (3)

[0090] That is, if the values ​​of two grid cells at the same position are both 1, it indicates that the two grid cells have an overlapping relationship; otherwise, it is determined that the two grid cells do not have an overlapping relationship.

[0091] After obtaining the overlap relationship, the true optical flow value between the third and second grids can be determined based on the positional relationship between the third grid and the first grid at the position corresponding to the first grid in the first grid corresponding to the third frame point cloud. This is because, in two accurately registered point clouds, the first and second grids coincide, and therefore the optical flow value between the third and second grids is the same as the optical flow value between the third and first grids.

[0092] Once the true optical flow value is obtained, it can be used as a supervision signal for training the optical flow estimation model.

[0093] In operation S640, the optical flow value estimation model is used to process the point cloud of the third frame and the second frame of the transformed two-frame point cloud to obtain multiple optical flow value predictions.

[0094] The principle behind operation S640 obtaining multiple optical flow prediction values ​​is similar to that in the registration point cloud verification method described above, where an optical flow estimation model is used to obtain multiple optical flow values, and will not be repeated here. It can be understood that these multiple optical flow prediction values ​​include the optical flow prediction values ​​between the two graticles in each of the multiple graticle pairs. The multiple graticle pairs can be composed of a graticle from the first graticle network described above and a graticle at the corresponding position in the second graticle network. For example, if the graticle network includes 512 rows and 512 columns of graticles, then the number of optical flow prediction values ​​obtained is 512*512.

[0095] When operating the S650, the optical flow estimation model is trained based on the differences between multiple predicted optical flow values ​​and the true optical flow values.

[0096] According to embodiments of this disclosure, the number of true optical flow values ​​can be equal to the number of predicted optical flow values. This embodiment uses the sum of the absolute values ​​of the differences between multiple true optical flow values ​​and their corresponding multiple predicted optical flow values ​​as the loss value of the optical flow estimation model. The optical flow estimation model is trained with the goal of minimizing this loss value. For example, if the predicted optical flow value is flow... pred If the true value of optical flow is flow, then the loss of the model can be expressed as the following formula (4).

[0097] loss = sum(abs(f1ow) pred -flow)) formula (4)

[0098] According to embodiments of this disclosure, when the true optical flow value is determined solely based on overlapping gratings, the predicted value corresponding to the true optical flow value can be selected from the predicted optical flow values. The model's loss value is then determined based on the absolute value of the difference between the true optical flow value and the corresponding predicted value. Alternatively, a fixed loss value can be added to the predicted optical flow values ​​that do not correspond to the true optical flow value. The sum of this fixed loss value and the loss value determined by the absolute value of the difference is used as the model's total loss.

[0099] In one embodiment, as described above for Figure 3 As described, the optical flow estimation model can include a feature map extraction network and an optical flow estimation network. When multiple optical flow values ​​are obtained, the feature extraction network can first extract feature maps of the third and second frame point clouds from the top-view angle, resulting in two first feature maps. Then, the optical flow estimation network processes these two first feature maps to obtain multiple optical flow values.

[0100] In one embodiment, as described above for Figure 4As described, the optical flow estimation model also includes an overlap estimation network. In this embodiment, the overlap estimation network can also be used to estimate the predicted overlap relationship between the two grids in each grid pair based on the two first feature maps. For the fourth and fifth grids among the two grids with an overlap relationship, the third frame point cloud includes the point cloud projected onto the fourth grid, and the second frame point cloud includes the point cloud projected onto the fifth grid. It should be noted that when using the overlap estimation network to estimate the overlap relationship, for example, the first feature map of the third frame point cloud can be processed inversely to the random perturbation, and the processed feature map can be concatenated with the first feature map of the second frame point cloud in the channel direction before being input into the overlap estimation network.

[0101] In this embodiment, while training the optical flow estimation network, an overlap estimation network can also be trained, for example. After identifying two overlapping grids in the third and second grids using the method described above, and obtaining the target grid pair, the ground truth value of the overlap relationship between the two grids included in the target grid can be determined, which indicates an overlap relationship, with an overlap probability of 1. During training, this ground truth value can be used as a supervisory signal to train the overlap estimation network. For example, the optical flow estimation model can be trained as a whole based on the predicted overlap relationship of the target grid. The predicted overlap relationship can be represented, for example, by the predicted overlap probability. This embodiment can determine the loss value of the optical flow estimation model by using the difference between the predicted overlap probability corresponding to the target grid pair and 1.

[0102] In one embodiment, a binary cross-entropy loss function can be used to determine the loss value for predicting overlap relationships. In this embodiment, the weighted sum of the loss value for predicting overlap relationships and the loss value for predicting optical flow values ​​can be used as the total loss value of the optical flow value estimation model. The weighting coefficients can be set according to actual needs, and this disclosure does not limit them.

[0103] In one embodiment, when using an overlap estimation network to estimate the predicted overlap relationship between the two grids in each grid pair, for example, the optical flow value matrix output by an optical flow value estimation network can be used to transform one of the two first feature maps. The transformed feature map is then concatenated with the other first feature map along the channel dimension and input into the overlap estimation network. This allows the predicted overlap relationship to be influenced by the accuracy of the optical flow value estimation network. Therefore, during model training, the ground truth of the overlap relationship can also be used as a supervisory signal for training the optical flow value estimation network, which helps improve the accuracy of the trained optical flow value estimation model.

[0104] In one embodiment, as described above, the optical flow estimation network may include a first feature extraction subnetwork and an optical flow estimation subnetwork. When processing two first feature maps to obtain multiple optical flow values, the first feature extraction subnetwork can be used to process the two first feature maps separately to obtain two second feature maps. Subsequently, the optical flow estimation network is used to process the two second feature maps to obtain an optical flow matrix, which includes the multiple optical flow values. Correspondingly, when predicting overlap relationships, the overlap estimation network can process the two second feature maps. For example, based on the optical flow matrix, one of the two second feature maps can be transformed to obtain a transformed feature map. Subsequently, based on the transformed feature map and the other feature map from the two second feature maps, the overlap estimation network is used to estimate the overlap relationship between the two grids included in each grid pair. Specifically, the transformed feature map and the other feature map from the two second feature maps can be concatenated along the channel dimension and input into the overlap estimation network, which then outputs an overlap probability matrix.

[0105] Based on the registration point cloud verification method provided in this disclosure, this disclosure also provides a registration point cloud verification device. The following will be combined with... Figure 7 The device is described in detail.

[0106] Figure 7 This is a structural block diagram of a registration point cloud verification device according to an embodiment of the present disclosure.

[0107] like Figure 7 As shown, the registration point cloud verification device 700 of this embodiment may include a point cloud transformation module 710, an optical flow value estimation module 720, and a registration verification module 730.

[0108] The point cloud transformation module 710 is used to transform the two registered point clouds to the same coordinate space according to the transformation relationship between them, thereby obtaining the transformed two point clouds. In one embodiment, the point cloud transformation module 710 can be used to perform the operation S210 described above, which will not be repeated here.

[0109] The optical flow estimation module 720 is used to determine multiple optical flow values ​​for the two transformed point clouds. These multiple optical flow values ​​include the optical flow value between the two graticles in each of multiple raster pairs; each raster pair consists of two corresponding graticles in two raster grids; and each raster grid corresponds to one of the two transformed point clouds. In one embodiment, the optical flow estimation module 720 can be used to perform the operation S220 described above, which will not be repeated here.

[0110] The registration verification module 730 is used to determine the registration accuracy of two registered point clouds based on multiple optical flow values. In one embodiment, the registration verification module 730 can be used to perform the operation S230 described above, which will not be repeated here.

[0111] According to an embodiment of this disclosure, the optical flow value estimation module 720 is specifically used to process the transformed two frames of point clouds using an optical flow value estimation model to obtain multiple optical flow values.

[0112] According to embodiments of this disclosure, the optical flow estimation model includes a feature extraction network and an optical flow estimation network. The aforementioned optical flow estimation module 720 may include a feature extraction submodule and an optical flow estimation submodule. The feature extraction submodule is used to extract feature maps of the transformed point clouds at their respective top-view angles using the feature extraction network, resulting in two first feature maps. The optical flow estimation submodule is used to process the two first feature maps using the optical flow estimation network to obtain multiple optical flow values.

[0113] According to embodiments of this disclosure, the optical flow estimation model further includes an overlap estimation network. The registration point cloud verification device 700 may further include an overlap relationship estimation module, used to estimate the overlap relationship between the two grates in each grid pair based on two first feature maps using the overlap estimation network; for the first and second grates in two overlapping grates, the first frame point cloud in the transformed two-frame point cloud includes the point cloud projected onto the first grates, and the second frame point cloud in the transformed two-frame point cloud includes the point cloud projected onto the second grates. The registration verification module 730 may include an overlapping grates determination submodule and a registration verification submodule. The overlapping grates determination submodule is used to determine a target grates pair among multiple grid pairs based on the overlap relationship, wherein the two grates in the target grates pair have an overlap relationship. The registration verification submodule is used to determine the registration accuracy of the two-frame point clouds based on the optical flow value for the target grates pair.

[0114] According to embodiments of this disclosure, the optical flow estimation network includes a feature extraction subnetwork and an optical flow estimation subnetwork. The optical flow estimation submodule may include a feature extraction unit and an optical flow estimation unit. The feature extraction unit processes two first feature maps using the feature extraction subnetwork to obtain two second feature maps. The optical flow estimation unit processes the two second feature maps using the optical flow estimation subnetwork to obtain an optical flow value matrix. The aforementioned overlap estimation module may include a feature transformation submodule and a relationship estimation submodule. The feature transformation submodule transforms one of the two second feature maps according to the optical flow value matrix to obtain a transformed feature map. The relationship estimation submodule estimates the overlap relationship between the two grids in each grid pair using the overlap estimation network based on the transformed feature map and the other feature map in the two second feature maps.

[0115] Based on the training method of the optical flow estimation model provided in this disclosure, this disclosure also provides a training device for the optical flow estimation model, which will be described below in conjunction with... Figure 8 The device is described in detail.

[0116] Figure 8 This is a structural block diagram of a training apparatus for an optical flow value estimation model according to an embodiment of the present disclosure.

[0117] like Figure 8 As shown, the training device 800 for the optical flow value estimation model in this embodiment may include a point cloud transformation module 810, a random transformation module 820, an optical flow value determination module 830, an optical flow value estimation module 840, and a model training module 850.

[0118] The point cloud transformation module 810 is used to transform the two registered point clouds to the same coordinate space according to the transformation relationship between them, thereby obtaining the transformed two point clouds. In one embodiment, the point cloud transformation module 810 can be used to perform the operation S610 described above, which will not be repeated here.

[0119] The random transformation module 820 is used to randomly transform the first frame of the transformed point cloud to obtain the third frame of the point cloud. In one embodiment, the random transformation module 820 can be used to perform the operation S620 described above, which will not be repeated here.

[0120] The optical flow value determination module 830 is used to determine the optical flow value for the third frame point cloud and the first frame point cloud based on the positional relationship between the third frame point cloud and the first frame point cloud, and use this value as the true optical flow value. In one embodiment, the optical flow value determination module 830 can be used to perform the operation S630 described above, which will not be repeated here.

[0121] The optical flow estimation module 840 is used to process the third frame point cloud and the second frame point cloud in the transformed two-frame point cloud using an optical flow estimation model to obtain multiple optical flow prediction values. These multiple optical flow prediction values ​​include the optical flow prediction values ​​between the two graticles in each of the multiple graticle pairs; the two graticles in each graticle pair are corresponding graticles in two grids; the two grids include a first grid corresponding to the third frame point cloud and a second grid corresponding to the second frame point cloud. In one embodiment, the optical flow estimation module 840 can be used to perform the operation S640 described above, which will not be repeated here.

[0122] The model training module 850 is used to train an optical flow estimation model based on the differences between multiple predicted optical flow values ​​and true optical flow values. In one embodiment, the model training module 850 can be used to perform the operation S650 described above, which will not be repeated here.

[0123] According to embodiments of this disclosure, the optical flow value determination module 830 may include a grid determination submodule, an overlapping grid determination submodule, and an optical flow value determination submodule. The grid determination submodule is used to determine a third grid, a second grid, and a first grid corresponding to the first frame point cloud, respectively, based on the first frame point cloud, the second frame point cloud, and the third frame point cloud. The overlapping grid determination submodule is used to determine two grids with an overlapping relationship in the third grid and the second grid to obtain a target grid pair. The two grids with an overlapping relationship include a first grid belonging to the third grid and a second grid belonging to the second grid; the first frame point cloud includes a first point cloud projected onto the first grid, and the second frame point cloud includes a second point cloud projected onto the second grid. The optical flow value determination submodule is used to determine the true optical flow value between the third grid and the second grid based on the positional relationship between the first grid and the third grid corresponding to the first grid in the first grid.

[0124] According to embodiments of this disclosure, the optical flow estimation model includes a feature map extraction network and an optical flow estimation network. The aforementioned optical flow estimation module 840 may include a feature extraction submodule and an optical flow estimation submodule. The feature extraction submodule is used to extract feature maps of the third frame point cloud and the second frame point cloud at a top-view angle using the feature extraction network, obtaining two first feature maps. The optical flow estimation submodule is used to process the two first feature maps using the optical flow estimation network to obtain multiple optical flow values.

[0125] According to embodiments of this disclosure, the optical flow estimation model further includes an overlap estimation network. The training apparatus 800 for the optical flow estimation model may further include an overlap relationship estimation module, a grid determination module, and an overlapping grid determination module. The overlap relationship estimation module is used to estimate the predicted overlap relationship between the two grids in each grid pair based on two first feature maps using the overlap estimation network. Specifically, for the fourth and fifth grids among the two grids with an overlap relationship, the third frame point cloud includes a point cloud projected onto the fourth grid, and the second frame point cloud includes a point cloud projected onto the fifth grid. The grid determination module is used to determine the third grid corresponding to the first frame point cloud and the second grid corresponding to the second frame point cloud. The overlapping grid determination module is used to determine the two grids with an overlap relationship in the third grid and the second grid, obtaining the target grid pair. The model training module 850 is further used, for example, to train the optical flow estimation model based on the target grid pair and the predicted overlap relationship.

[0126] According to embodiments of this disclosure, the flow estimation network includes a first feature extraction subnetwork and an optical flow estimation subnetwork. The optical flow estimation submodule may include a feature extraction unit and an optical flow estimation unit. The feature extraction unit processes two first feature maps using the first feature extraction subnetwork to obtain two second feature maps. The optical flow estimation unit processes the two second feature maps using the optical flow estimation network to obtain an optical flow matrix. The aforementioned overlap estimation module may include a feature transformation submodule and a relationship estimation submodule. The feature transformation submodule transforms one of the two second feature maps according to the optical flow matrix to obtain a transformed feature map. The relationship estimation submodule estimates the overlap relationship between the two graticles in each grid pair using the overlap estimation network based on the transformed feature map and the other feature map in the two second feature maps.

[0127] It should be noted that the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information in this disclosed technical solution comply with relevant laws and regulations, necessary confidentiality measures have been taken, and it does not violate public order and good morals. In this disclosed technical solution, user authorization or consent has been obtained before acquiring or collecting user personal information.

[0128] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0129] Figure 9 A schematic block diagram of an example electronic device 900 is shown, which can be used to implement the verification method for registration point clouds and / or the training method for optical flow value estimation models of embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0130] like Figure 9As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0131] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0132] The computing unit 901 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the registration point cloud verification method and / or the optical flow value estimation model training method. For example, in some embodiments, the registration point cloud verification method and / or the optical flow value estimation model training method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by computing unit 901, one or more steps of the registration point cloud verification method and / or optical flow value estimation model training method described above can be performed. Alternatively, in other embodiments, the computing unit 901 may be configured by any other suitable means (e.g., by means of firmware) to perform a verification method for the registration point cloud and / or a training method for the optical flow value estimation model.

[0133] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0134] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0135] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0136] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0137] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0138] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0139] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0140] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for verifying registered point clouds, comprising: Based on the transformation relationship between the two registered point clouds, the two registered point clouds are transformed into the same coordinate space to obtain the two transformed point clouds. An optical flow estimation model is used to process the two transformed point clouds to obtain multiple optical flow values. These multiple optical flow values ​​include the optical flow values ​​between the two grids in each grid pair. The two grids in each grid pair are corresponding grids in two grid networks. The two grid networks correspond to the two transformed point clouds respectively. as well as Determining the registration accuracy of the two-frame point clouds based on the plurality of optical flow values ​​includes: if the mean of at least some of the plurality of optical flow values ​​is less than or equal to a threshold, then the registration accuracy of the two-frame point clouds is determined. If the mean value is greater than the threshold, then the registration of the two point clouds is determined to be inaccurate.

2. The method according to claim 1, wherein, The optical flow estimation model includes a feature extraction network and an optical flow estimation network; the process of using the optical flow estimation model to process the transformed two frames of point clouds to obtain the plurality of optical flow values ​​includes: The feature extraction network is used to extract feature maps of the two transformed point clouds at the top-view angle, resulting in two first feature maps; and The optical flow estimation network is used to process the two first feature maps to obtain the plurality of optical flow values.

3. The method according to claim 2, wherein, The optical flow estimation model further includes an overlap estimation network; the method further includes: Based on the two first feature maps, the overlap estimation network is used to estimate the overlap relationship between the two grids included in each grid pair; for the first grid and the second grid in the two grids with an overlap relationship, the first frame point cloud in the transformed two-frame point cloud includes the point cloud projected onto the first grid, and the second frame point cloud in the transformed two-frame point cloud includes the point cloud projected onto the second grid. The step of determining the registration accuracy of the two point clouds based on the plurality of optical flow values ​​includes: Based on the overlap relationship, a target grid pair is determined among the plurality of grid pairs, wherein the two grids in the target grid pair have an overlap relationship; and The registration accuracy of the two point clouds being registered is determined based on the optical flow values ​​for the target raster pair.

4. The method according to claim 3, wherein, The optical flow estimation network includes a feature extraction subnetwork and an optical flow estimation subnetwork; The process of using the optical flow value estimation network to process the two first feature maps to obtain the plurality of optical flow values ​​includes: The feature extraction sub-network is used to process the two first feature maps respectively to obtain two second feature maps; and The optical flow estimation subnetwork is used to process the two second feature maps to obtain the optical flow matrix. Specifically, estimating the overlap relationship between the two grids in each of the plurality of grid pairs using the overlap estimation network based on the two first feature maps includes: Based on the optical flow value matrix, one of the two second feature maps is transformed to obtain the transformed feature map; and Based on the transformed feature map and another feature map in the two second feature maps, the overlap estimation network is used to estimate the overlap relationship between the two grids included in each grid pair.

5. A training method for an optical flow value estimation model, comprising: Based on the transformation relationship between the two registered point clouds, the two registered point clouds are transformed into the same coordinate space to obtain the two transformed point clouds. The first frame of the transformed point cloud is randomly transformed to obtain the third frame of the point cloud; Based on the positional relationship between the third frame point cloud and the first frame point cloud, the optical flow value for the third frame point cloud and the first frame point cloud is determined as the true value of the optical flow value. An optical flow estimation model is used to process the third frame point cloud and the second frame point cloud in the transformed two-frame point cloud to obtain multiple optical flow prediction values. These multiple optical flow prediction values ​​include the optical flow prediction values ​​between the two graticles in each of multiple raster pairs. Each raster pair consists of two corresponding graticles in two raster grids. The two raster grids include a first raster grid corresponding to the third frame point cloud and a second raster grid corresponding to the second frame point cloud. as well as The optical flow estimation model is trained based on the differences between the multiple predicted optical flow values ​​and the true optical flow values. The trained optical flow estimation model is used to process two frames of point clouds to obtain multiple optical flow values, so as to determine that the registration of the two frames of point clouds is accurate when the mean of at least some of the multiple optical flow values ​​is less than or equal to a threshold, and to determine that the registration of the two frames of point clouds is inaccurate when the mean is greater than the threshold.

6. The method according to claim 5, wherein, The step of determining the optical flow value for the third frame point cloud and the first frame point cloud based on the positional relationship between the third frame point cloud and the first frame point cloud, as the true value of the optical flow value, includes: Based on the first frame point cloud, the second frame point cloud, and the third frame point cloud, respectively, determine the third grid, the second grid, and the first grid corresponding to the first frame point cloud; Two overlapping grids in the third grid and the second grid are identified to obtain a target grid pair; the two overlapping grids include a first grid belonging to the third grid and a second grid belonging to the second grid; the first frame point cloud includes a first point cloud projected onto the first grid, and the second frame point cloud includes a second point cloud projected onto the second grid; and The true value of the optical flow between the third grid and the second grid is determined based on the positional relationship between the first grid and the third grid corresponding to the first grid.

7. The method according to claim 5, wherein, The optical flow estimation model includes a feature extraction network and an optical flow estimation network; the optical flow estimation model is used to process the third frame point cloud and the second frame point cloud of the transformed two frames to obtain multiple optical flow prediction values, including: The feature extraction network is used to extract feature maps of the third frame point cloud and the second frame point cloud at the top-view angle, respectively, to obtain two first feature maps; and The optical flow estimation network is used to process the two first feature maps to obtain the multiple optical flow prediction values.

8. The method according to claim 7, wherein, The optical flow estimation model further includes an overlap estimation network; the method further includes: Based on the two first feature maps, the overlap estimation network is used to estimate the predicted overlap relationship between the two grids included in each grid pair; for the fourth and fifth grids of the two grids with an overlap relationship, the third frame point cloud includes the point cloud projected onto the fourth grid, and the second frame point cloud includes the point cloud projected onto the fifth grid; Determine the third grid corresponding to the point cloud of the first frame and the second grid corresponding to the point cloud of the second frame; To obtain a target grid pair, two overlapping grid cells in the third grid and the second grid are identified; and The optical flow estimation model is trained based on the overlap relationship between the target grid pair and the prediction.

9. The method according to claim 8, wherein, The optical flow estimation network includes a first feature extraction subnetwork and an optical flow estimation subnetwork; The process of using the optical flow value estimation network to process the two first feature maps to obtain the plurality of optical flow values ​​includes: The first feature extraction sub-network is used to process the two first feature maps respectively to obtain two second feature maps; and The optical flow estimation network is used to process the two second feature maps to obtain the optical flow matrix. Specifically, estimating the predicted overlap relationship between the two grids in each grid pair using the overlap estimation network based on the two first feature maps includes: Based on the optical flow value matrix, one of the two second feature maps is transformed to obtain the transformed feature map; and Based on the transformed feature map and another feature map in the two second feature maps, the overlap estimation network is used to estimate the overlap relationship between the two grids included in each grid pair.

10. A verification device for registered point clouds, comprising: The point cloud transformation module is used to transform the two registered point clouds into the same coordinate space according to the transformation relationship between the two frames of point clouds, so as to obtain the transformed two frames of point clouds. An optical flow estimation module is used to process the two transformed point clouds using an optical flow estimation model to obtain multiple optical flow values. The multiple optical flow values ​​include the optical flow values ​​between the two grids in each grid pair. The two grids in each grid pair are corresponding grids in two grid networks. The two grid networks correspond to the two transformed point clouds respectively. as well as The registration verification module is used to determine the registration accuracy of the two-frame point cloud based on the plurality of optical flow values, including: if the mean of at least some of the plurality of optical flow values ​​is less than or equal to a threshold, then the registration accuracy of the two-frame point cloud is determined. If the mean value is greater than the threshold, then the registration of the two point clouds is determined to be inaccurate.

11. The apparatus according to claim 10, wherein, The optical flow estimation model includes a feature extraction network and an optical flow estimation network; the optical flow estimation module includes: The feature extraction submodule is used to extract feature maps of the two transformed point clouds from the top-view angle using the feature extraction network, resulting in two first feature maps; and The optical flow estimation submodule is used to process the two first feature maps using the optical flow value estimation network to obtain the plurality of optical flow values.

12. The apparatus according to claim 11, wherein, The optical flow estimation model further includes an overlap estimation network; the device further includes: The overlap estimation module is used to estimate the overlap relationship between the two grids included in each grid pair based on the two first feature maps and using the overlap estimation network. For the first grid and the second grid in two grids with an overlap relationship, the first frame point cloud in the transformed two-frame point cloud includes the point cloud projected onto the first grid, and the second frame point cloud in the transformed two-frame point cloud includes the point cloud projected onto the second grid. The registration verification module includes: An overlapping grid determination submodule is used to determine a target grid pair among the plurality of grid pairs based on the overlapping relationship, wherein the target grid pair comprises two grids that have an overlapping relationship; and The registration verification submodule is used to determine the registration accuracy of the two frame point clouds based on the optical flow values ​​for the target raster pair.

13. The apparatus according to claim 12, wherein, The optical flow estimation network includes a feature extraction subnetwork and an optical flow estimation subnetwork; The optical flow estimation submodule includes: The feature extraction unit is configured to process the two first feature maps using the feature extraction sub-network to obtain two second feature maps; and The optical flow estimation unit is used to process the two second feature maps using the optical flow value estimation subnetwork to obtain an optical flow value matrix. The overlap relationship estimation module includes: The feature transformation submodule is used to transform one of the two second feature maps according to the optical flow value matrix to obtain the transformed feature map; and The relationship estimation submodule is used to estimate the overlap relationship between the two grids included in each grid pair using the overlap estimation network, based on the transformed feature map and another feature map in the two second feature maps.

14. A training device for an optical flow value estimation model, comprising: The point cloud transformation module is used to transform the two registered point clouds into the same coordinate space according to the transformation relationship between the two frames of point clouds, so as to obtain the transformed two frames of point clouds. The random transformation module is used to randomly transform the first frame of the point cloud in the two transformed point clouds to obtain the third frame of the point cloud. The optical flow value determination module is used to determine the optical flow value between the third frame point cloud and the first frame point cloud based on the positional relationship between the third frame point cloud and the first frame point cloud, and use it as the true value of the optical flow value. An optical flow estimation module is used to process the third frame point cloud and the second frame point cloud in the transformed two-frame point cloud using an optical flow estimation model to obtain multiple optical flow prediction values. These multiple optical flow prediction values ​​include the optical flow prediction values ​​between the two graticles in each of multiple grid pairs. Each grid pair consists of two corresponding graticles in two grid networks. The two grid networks include a first grid network corresponding to the third frame point cloud and a second grid network corresponding to the second frame point cloud. as well as The model training module is used to train the optical flow value estimation model based on the differences between the multiple predicted optical flow values ​​and the true optical flow values. The trained optical flow estimation model is used to process two frames of point clouds to obtain multiple optical flow values, so as to determine that the registration of the two frames of point clouds is accurate when the mean of at least some of the multiple optical flow values ​​is less than or equal to a threshold, and to determine that the registration of the two frames of point clouds is inaccurate when the mean is greater than the threshold.

15. The apparatus according to claim 14, wherein, The optical flow value determination module includes: The grid determination submodule is used to determine the third grid, the second grid, and the first grid corresponding to the first frame point cloud based on the first frame point cloud, the second frame point cloud, and the third frame point cloud, respectively. An overlapping grid determination submodule is used to determine two grids that overlap in the third grid and the second grid to obtain a target grid pair; the two overlapping grids include a first grid belonging to the third grid and a second grid belonging to the second grid; the first frame point cloud includes a first point cloud projected onto the first grid, and the second frame point cloud includes a second point cloud projected onto the second grid; and The optical flow value determination submodule is used to determine the true value of the optical flow value between the third grid and the second grid based on the positional relationship between the first grid and the third grid in the first grid corresponding to the first grid.

16. The apparatus according to claim 14, wherein, The optical flow estimation model includes a feature extraction network and an optical flow estimation network; the optical flow estimation module includes: The feature extraction submodule is used to extract feature maps of the third frame point cloud and the second frame point cloud from the top-view angle using the feature extraction network, resulting in two first feature maps; and The optical flow estimation submodule is used to process the two first feature maps using the optical flow estimation network to obtain the multiple optical flow prediction values.

17. The apparatus according to claim 16, wherein, The optical flow estimation model further includes an overlap estimation network; the device further includes: An overlap relationship estimation module is used to estimate the predicted overlap relationship between the two grids included in each grid pair using the overlap estimation network based on the two first feature maps; for the fourth grid and the fifth grid in the two grids with an overlap relationship, the third frame point cloud includes the point cloud projected onto the fourth grid, and the second frame point cloud includes the point cloud projected onto the fifth grid; A grid determination module is used to determine a third grid corresponding to the first frame point cloud and a second grid corresponding to the second frame point cloud; and The overlapping grid determination module is used to identify two grids that overlap in the third grid and the second grid to obtain a target grid pair. The model training module is further configured to: train the optical flow value estimation model based on the overlap relationship between the target grid pair and the prediction.

18. The apparatus according to claim 17, wherein, The optical flow estimation network includes a first feature extraction subnetwork and an optical flow estimation subnetwork; the optical flow estimation submodule includes: The feature extraction unit is configured to process the two first feature maps using the first feature extraction sub-network to obtain two second feature maps; and The optical flow estimation unit is used to process the two second feature maps using the optical flow estimation network to obtain an optical flow matrix. The overlap relationship estimation module includes: The feature transformation submodule is used to transform one of the two second feature maps according to the optical flow value matrix to obtain the transformed feature map; and The relationship estimation submodule is used to estimate the overlap relationship between the two grids included in each grid pair using the overlap estimation network, based on the transformed feature map and another feature map in the two second feature maps.

19. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 9.

21. A computer program product comprising a computer program / instructions stored on at least one of a readable storage medium and an electronic device, wherein the computer program / instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

Images