Point cloud registration method, device, equipment, medium and autonomous driving vehicle

Abstract

The present disclosure provides a point cloud registration method, apparatus, device, medium, and autonomous driving vehicle, and relates to the field of artificial intelligence, and more specifically, to autonomous driving technology, computer vision technology, and point cloud registration technology. The method includes: performing a matching operation on a first point cloud input and a second point cloud input based on an estimated pose to obtain at least one key point pair, wherein each key point pair includes a first key point and a second key point; performing a registration operation on the at least one key point pair, including: for each key point pair: performing a coordinate transformation on the first key point based on at least the estimated pose to obtain a third key point; and determining an estimated color parameter of the third key point based on a true color parameter of the second key point and a relationship between the second key point and the third key point; and determining first target update information for the estimated pose based on the true color parameter of the first key point and the estimated color parameter of the third key point to update the estimated pose.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, specifically to autonomous driving technology, computer vision technology and point cloud registration technology, and in particular to a point cloud registration method, a point cloud registration device, an electronic device, a computer-readable storage medium and a computer program product, and an autonomous driving vehicle. Background Art

[0002] Artificial intelligence (AI) is the study of how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily encompass computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graphs.

[0003] Point cloud registration technology transforms one set of point cloud data in order to match it with another. Point cloud registration technology is crucial in a variety of scenarios, including autonomous driving. Highly accurate point cloud registration can determine the precise position of the vehicle, thus ensuring autonomous driving.

[0004] The approaches described in this section are not necessarily approaches that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any approach described in this section is prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise indicated, the issues raised in this section should not be considered as having been recognized in any prior art. Summary of the Invention

[0005] The present disclosure provides a point cloud registration method, a point cloud registration device, an electronic device, a computer-readable storage medium, a computer program product, and an autonomous driving vehicle.

[0006] According to one aspect of the present disclosure, a point cloud registration method is provided. The method includes: obtaining a first point cloud, a second point cloud, and an estimated pose of a reference system of the second point cloud relative to a reference system of the first point cloud; determining a first point cloud input based on the first point cloud and determining a second point cloud input based on the second point cloud; performing a matching operation on the first point cloud input and the second point cloud input based on the estimated pose to obtain at least one key point pair, wherein each key point pair in the at least one key point pair includes a first key point belonging to the first point cloud input and a second key point belonging to the second point cloud input; and performing a registration operation on the at least one key point pair, wherein the registration operation includes: for each key point pair in the at least one key point pair: performing a coordinate transformation on the first key point based on at least the estimated pose to obtain a third key point in the reference system of the second point cloud; and determining an estimated color parameter of the third key point based on a true color parameter of the second key point and a relationship between the second key point and the third key point; determining first target update information for the estimated pose based on the true color parameter of the first key point in the at least one key point pair and the estimated color parameter of the third key point corresponding to the first key point; and updating the estimated pose based on at least the first target update information.

[0007] According to another aspect of the present disclosure, a point cloud registration device is provided. The device includes: an acquisition unit configured to acquire a first point cloud, a second point cloud, and an estimated pose of the reference system of the second point cloud relative to the reference system of the first point cloud; a determination unit configured to determine a first point cloud input based on the first point cloud and a second point cloud input based on the second point cloud; a matching unit configured to perform a matching operation on the first point cloud input and the second point cloud input based on the estimated pose to obtain at least one key point pair, wherein each key point pair in the at least one key point pair includes a first key point belonging to the first point cloud input and a second key point belonging to the second point cloud input; and a registration unit configured to perform a registration operation on the at least one key point pair, wherein the registration unit includes : A color estimation subunit is configured to: for each key point pair in at least one key point pair: perform coordinate transformation on the first key point at least based on the estimated pose to obtain a third key point in the reference system of the second point cloud; and determine the estimated color parameters of the third key point based on the true color parameters of the second key point and the relationship between the second key point and the third key point; a determination subunit is configured to determine first target update information for the estimated pose based on the true color parameters of the first key point in at least one key point pair and the estimated color parameters of the third key point corresponding to the first key point; and an update subunit is configured to update the estimated pose based on at least the first target update information.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above method.

[0010] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0011] According to another aspect of the present disclosure, an autonomous driving vehicle is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned method.

[0012] According to one or more embodiments of the present disclosure, by using an objective function including a point cloud reflectance value, a constraint on the point cloud reflectance value (i.e., the color of the point) is introduced, thereby improving the estimation result of the pose. Specifically, by performing a coordinate transformation on the first key point based on the estimated pose, a third key point located in the same reference system as the second key point is obtained. By using the true color parameters of the corresponding second key point and the positional relationship between the second key point and the third key point, an accurate estimate of the color of the position of the third key point can be obtained, thereby updating the estimated pose according to the estimated color parameters of the third key point and the true color parameters of the first key point to obtain an accurate pose estimation result.

[0013] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings illustrate exemplary embodiments and constitute a part of the specification. Together with the description of the specification, they serve to explain exemplary implementation of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals designate similar, but not necessarily identical, elements.

[0015] Figure 1 A schematic diagram illustrating an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0016] Figure 2 A flow chart of a point cloud registration method according to an exemplary embodiment of the present disclosure is shown;

[0017] Figure 3 A flowchart of a registration operation according to an exemplary embodiment of the present disclosure is shown;

[0018] Figure 4 A flow chart of a point cloud registration method according to an exemplary embodiment of the present disclosure is shown;

[0019] Figure 5 A structural block diagram of a point cloud registration device according to an exemplary embodiment of the present disclosure is shown; and

[0020] Figure 6 FIG2 shows a block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure. DETAILED DESCRIPTION

[0021] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] In this disclosure, unless otherwise specified, the use of terms such as "first" and "second" to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements. Such terms are only used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of the element, while in some cases, based on the context of the description, they may also refer to different instances.

[0023] The terms used in the descriptions of the various examples described in this disclosure are for the purpose of describing specific examples only and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element may be one or more. In addition, the term "and / or" used in this disclosure encompasses any one and all possible combinations of the listed items.

[0024] In the related art, the existing point cloud registration method does not constrain the point cloud reflectance value, so it is impossible to obtain an accurate estimation of the estimated pose when other constraints are poor.

[0025] In order to solve the above problems, the present disclosure introduces constraints on the point cloud reflection value (i.e., the color of the point) by using an objective function including the point cloud reflection value, so as to obtain an improved estimation result of the pose. Specifically, the coordinates of the first key point are transformed based on the estimated pose to obtain a third key point located in the same reference system as the second key point. By using the real color parameters of the corresponding second key point and the positional relationship between the second key point and the third key point, an accurate estimate of the color of the position of the third key point can be obtained, thereby updating the estimated pose according to the estimated color parameters of the third key point and the real color parameters of the first key point to obtain an accurate pose estimation result.

[0026] Figure 1 FIG2 is a schematic diagram of an exemplary system 100 in which the various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes a motor vehicle 110 , a server 120 , and one or more communication networks 130 coupling the motor vehicle 110 to the server 120 .

[0027] In an embodiment of the present disclosure, the motor vehicle 110 may include a computing device according to an embodiment of the present disclosure and / or be configured to perform a method according to an embodiment of the present disclosure.

[0028] The server 120 may run one or more services or software applications that enable the method of point cloud registration. In some embodiments, the server 120 may also provide other services or software applications that may include non-virtual environments and virtual environments. Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. The user of the motor vehicle 110 may, in turn, utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0029] Server 120 may include one or more general-purpose computers, specialized server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0030] The computing units in the server 120 may run one or more operating systems including any of the operating systems described above as well as any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, and the like.

[0031] In some embodiments, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from motor vehicle 110. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of motor vehicle 110.

[0032] The network 130 may be any type of network known to those skilled in the art that can support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a satellite communication network, a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (including, for example, Bluetooth, WiFi), and / or any combination of these and other networks.

[0033] The system 100 may also include one or more databases 150. In some embodiments, these databases can be used to store data and other information. For example, one or more of the databases 150 can be used to store information such as audio files and video files. The data repository 150 can reside in a variety of locations. For example, the data repository used by the server 120 can be local to the server 120, or can be remote from the server 120 and can communicate with the server 120 via a network-based or dedicated connection. The data repository 150 can be of different types. In some embodiments, the data repository used by the server 120 can be a database, such as a relational database. One or more of these databases can store, update, and retrieve data to and from the database in response to commands.

[0034] In some embodiments, one or more of the databases 150 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0035] Motor vehicle 110 may include sensors 111 for sensing its surroundings. Sensors 111 may include one or more of the following: visual cameras, infrared cameras, ultrasonic sensors, millimeter-wave radar, and laser radar (LiDAR). Different sensors offer different detection accuracy and range. Cameras may be mounted on the front, rear, or other locations of the vehicle. Visual cameras can capture real-time information about the vehicle's interior and exterior and present it to the driver and / or passengers. Furthermore, by analyzing the images captured by the visual cameras, information such as traffic light indications, intersection conditions, and the operating status of other vehicles can be obtained. Infrared cameras can detect objects in night vision conditions. Ultrasonic sensors can be mounted on all sides of the vehicle, utilizing the strong directionality of ultrasonic waves to measure the distance of external objects from the vehicle. Millimeter-wave radars can be mounted on the front, rear, or other locations of the vehicle, utilizing the properties of electromagnetic waves to measure the distance of external objects from the vehicle. LiDARs can be mounted on the front, rear, or other locations of the vehicle, detecting object edges and shapes for object recognition and tracking. Due to the Doppler effect, radar devices can also measure changes in the speed of the vehicle and moving objects.

[0036] The motor vehicle 110 may also include a communication device 112. The communication device 112 may include a satellite positioning module that can receive satellite positioning signals (e.g., Beidou, GPS, GLONASS, and GALILEO) from satellites 141 and generate coordinates based on these signals. The communication device 112 may also include a module for communicating with a mobile communication base station 142. The mobile communication network may implement any suitable communication technology, such as GSM / GPRS, CDMA, LTE, and other current or evolving wireless communication technologies (e.g., 5G technology). The communication device 112 may also have a vehicle-to-everything (V2X) module that is configured to implement vehicle-to-vehicle (V2V) communication with other vehicles 143 and vehicle-to-infrastructure (V2I) communication with infrastructure 144, for example. In addition, the communication device 112 may also include a module configured to communicate with a user terminal 145 (including but not limited to a smartphone, tablet computer, or wearable device such as a watch) via a wireless local area network or Bluetooth using the IEEE 802.11 standard, for example. Using the communication device 112, the motor vehicle 110 may also access the server 120 via the network 130.

[0037] The motor vehicle 110 may also include a control device 113. The control device 113 may include a processor that communicates with various types of computer-readable storage devices or media, such as a central processing unit (CPU) or a graphics processing unit (GPU), or other dedicated processors. The control device 113 may include an autonomous driving system for automatically controlling various actuators in the vehicle. The autonomous driving system is configured to control the powertrain, steering system, and braking system of the motor vehicle 110 (not shown) via multiple actuators in response to input from multiple sensors 111 or other input devices to control acceleration, steering, and braking, respectively, without human intervention or limited human intervention. Some processing functions of the control device 113 may be implemented through cloud computing. For example, some processing may be performed using an on-board processor, while other processing may be performed using computing resources in the cloud. The control device 113 may be configured to execute the method according to the present disclosure. In addition, the control device 113 may be implemented as an example of a computing device on the motor vehicle side (client) according to the present disclosure.

[0038] Figure 1 The system 100 may be configured and operated in various ways to enable application of the various methods and apparatuses described in accordance with the present disclosure.

[0039] According to one aspect of the present disclosure, a point cloud registration method is provided. Figure 2 As shown, the method includes: step S201, obtaining a first point cloud, a second point cloud, and an estimated pose of a reference system of the second point cloud relative to a reference system of the first point cloud; step S202, determining a first point cloud input based on the first point cloud and determining a second point cloud input based on the second point cloud; step S203, performing a matching operation on the first point cloud input and the second point cloud input based on the estimated pose to obtain at least one key point pair, wherein each key point pair in the at least one key point pair includes a first key point belonging to the first point cloud input and a second key point belonging to the second point cloud input; and step S204, performing a registration operation on the at least one key point pair, wherein the registration operation The operation includes: step S205, for each key point pair in at least one key point pair, performing coordinate transformation on the first key point based on at least one estimated pose to obtain a third key point in the reference system of the second point cloud, and determining the estimated color parameter of the third key point based on the true color parameter of the second key point and the relationship between the second key point and the third key point; step S206, determining first target update information for the estimated pose based on the true color parameter of the first key point in at least one key point pair and the estimated color parameter of the third key point corresponding to the first key point; and step S207, updating the estimated pose based on at least one first target update information.

[0040] Thus, by using an objective function including the point cloud reflectance value, a constraint on the point cloud reflectance value (i.e., the color of the point) is introduced, thereby improving the pose estimation result. Specifically, by performing a coordinate transformation on the first key point based on the estimated pose, a third key point is obtained that is located in the same reference system as the second key point. By using the true color parameters of the corresponding second key point and the positional relationship between the second key point and the third key point, an accurate estimate of the color of the position of the third key point can be obtained, thereby updating the estimated pose according to the estimated color parameters of the third key point and the true color parameters of the first key point to obtain an accurate pose estimation result.

[0041] According to some embodiments, the first point cloud may be generated based on data collected by onboard sensors, and the second point cloud may be generated based on high-precision map data. It is understood that the first point cloud and the second point cloud may also be point cloud data obtained by other methods, which are not limited here.

[0042] According to some embodiments, step S202 of determining a first point cloud input based on the first point cloud and determining a second point cloud input based on the second point cloud may include downsampling the first point cloud and the second point cloud, respectively, to obtain the first point cloud input and the second point cloud input. The significance and effect of such downsampling will be described below.

[0043] According to some embodiments, the estimated position and posture may be, for example, an estimated value of the vehicle's current position and posture. The estimated position and posture may be obtained, for example, via a satellite navigation system, calculated based on accelerometer data, or obtained by other means, without limitation herein.

[0044] According to some embodiments, the estimated pose T k May include a rotational component ω k and the translation component t k The update of the estimated pose can be performed iteratively, so k can be used to represent the current iteration round. In an exemplary embodiment, the rotation component ω k It further includes three subcomponents α k , β k , γ k , translation component t k It further includes three subcomponents a k 、b k 、c k The estimated pose can be expressed as the following matrix:

[0045]

[0046] According to some embodiments, matching a first point cloud input with a second point cloud input may include: performing a coordinate transformation on the first point cloud input based on an estimated pose; and matching the transformed first point cloud input with the second point cloud input. In some embodiments, performing a coordinate transformation on the first point cloud input based on the estimated pose may, for example, perform a coordinate transformation on all points in the first point cloud based on the estimated pose. Thus, by first performing a pose transformation on the first point cloud before matching it with the second point cloud, matching accuracy and point matching rate can be improved.

[0047] After obtaining the matching results of the first point cloud and the second point cloud, registration may be performed on at least one key point pair that is matched.

[0048] According to some embodiments, for each key point pair in the at least one key point pair, a coordinate transformation is performed on the corresponding first key point based on at least the estimated pose to obtain a third key point in the reference frame of the second point cloud input.

[0049] According to some embodiments, the true color parameter includes a true color value of the second key point and a color gradient at the second key point.

[0050] According to some embodiments, a virtual orthographic camera may be introduced for the second keypoint, which is configured to be along the normal n pObserve the second key point p. The image plane of this virtual camera is the tangent plane at p, that is, the reference plane at the second key point. The points on the reference plane can be represented as a continuous color function C p (u), where u is the vector emitted from the second keypoint p along the tangent plane: u*n p = 0. Function C p (u) can be approximated as its first-order approximation:

[0051]

[0052] Among them, C(p) is the true color value of the second key point p, d p It is C p The gradient of (u) is the color gradient at the second key point. After obtaining the reference surface at the second key point, the corresponding first key point can be transformed based on the estimated pose and further projected onto the reference surface to obtain the third key point.

[0053] It can be understood that the registration operation in step S204 may include, for example, steps S205 to S207.

[0054] According to some embodiments, Figure 3 As shown, the registration operation may further include: step S301, determining a color gradient at the second key point based on a true color value of the second key point, the true color values of each of a plurality of fourth key points in the second point cloud input, and a positional relationship between the second key point and the plurality of fourth key points, wherein the plurality of fourth key points are all located within a first preset range centered on the second key point. Figure 3 The operations of step S302, step S303 and step S306 in Figure 2 The operations of step S205 to step S207 are similar and are not limited here.

[0055] Therefore, by determining the color gradient at the second key point based on the true color values of the second key point's local neighbors and the positional relationship between the second key point and these local neighbors, an accurate color gradient can be calculated, thereby obtaining an accurate estimation of the color of the third key point.

[0056] According to some embodiments, determining the color gradient at the second key point may include: for each of a plurality of fourth key points, based on the true color value of the fourth key point, the true color value of the second key point, and the estimated color gradient, as well as the positional relationship between the fourth key point and the second key point, determining target update sub-information corresponding to the fourth key point; determining second target update information for the estimated color gradient based on the target update sub-information corresponding to each of the plurality of fourth key points; and determining the color gradient at the second key point based on the second target update information. Thus, by using the method of determining the target update information, the accuracy of the obtained color gradient can be further improved, thereby further improving the accuracy of the estimated color of the third key point.

[0057] In one exemplary embodiment, the color gradient is obtained by applying least squares to C p′ |p′∈N p To estimate, N p Is the local neighbor of p, that is, multiple fourth key points. Let f(s) be the function of projecting a three-dimensional point s to the reference surface at the second key point p:

[0058] f(s)=sn p (sp) T n p

[0059] Used to calculate the color gradient d p The second target update information is:

[0060]

[0061] The second target update sub-information can be, for example, the term for each local neighbor p′ in the above formula. The estimated color gradient is the initial value of the color gradient used when calculating the color gradient based on the second target update information. It is understood that those skilled in the art can select the corresponding initial value, i.e., the estimated color gradient, to solve the color gradient, and this is not limited here.

[0062] According to some embodiments, the color gradient at the second key point may be coplanar with the reference plane, i.e. n p = 0. It is understandable that the color gradient at the second key point can be calculated during the registration operation, or during the matching operation, or in the preprocessing stage by calculating the color gradient for all points included in the second point cloud to obtain the color gradient of the second key point in each key point pair, which is not limited here.

[0063] According to some embodiments, determining first target update information regarding the estimated pose may include: for each keypoint pair, determining first target update sub-information corresponding to the keypoint pair based on the difference between the true color value of the first keypoint in the keypoint pair and the estimated color value of the third keypoint corresponding to the first keypoint; and determining first target update information regarding the estimated pose based on the first target update sub-information corresponding to each of at least one keypoint pair. Thus, by determining the first target update sub-information based on the difference between the true color value of the first keypoint and the estimated color value of the third keypoint, constraints on the point cloud reflectance are implemented to obtain a more accurate pose estimation result.

[0064] In some embodiments, the first target update information may be, for example, the target function E constrained by the point cloud reflectance value. C (T), as shown in the following equation. For at least one keypoint pair K = {(p,q)}, the first keypoint q is transformed s(·) based on the estimated pose T and then projected to the third keypoint q′ on the reference plane of the second keypoint p:

[0065] s(q,T)=T·q

[0066] q′=f(s(q,T))

[0067]

[0068] Among them, C(q) is the true color value of the first key point, C p (q′) is the estimated color value of the third keypoint.

[0069] According to some embodiments, updating the estimated pose based on at least the first target update information may include: determining a first residual and a first Jacobian matrix based on the first target update information and the estimated pose; determining pose update information based on the first residual and the first Jacobian matrix; and updating the estimated pose based on the pose update information. Thus, by performing an optimization solution using the residual and the Jacobian matrix, rapid convergence of the estimated pose is achieved.

[0070] According to some embodiments, the first residual r C For example, it can be obtained based on the residual corresponding to each key point pair (p, q) in the first target update information, as follows:

[0071]

[0072] First Jacobian matrix For example, the first residual r C For the estimated pose T k The Gauss-Newton method can be used to solve and optimize the following equation based on the residual and Jacobian matrix:

[0073]

[0074] The pose update information ξ can be converted to the estimated pose T. The pose update information ξ includes a one-dimensional vector estimated pose T k All components of . In an exemplary embodiment, ξ=(α',β',γ',a',b',c'). The pose update information ξ can also be mapped to SE(3). k The update can be expressed as:

[0075]

[0076] On the basis of the point cloud reflectance value constraint, other constraints can be introduced to further improve the accuracy of the registration results.

[0077] According to some embodiments, Figure 3 As shown, the registration operation may further include: step S305, determining third target update information for the estimated pose based on at least one of the true color value and coordinate value of each of the first key point and the second key point in the at least one key point pair and the estimated pose. Step S306, updating the estimated pose based on at least the first target update information may include: updating the estimated pose based on the first target update information and the third target update information. Thus, by introducing other constraints and updating the target information corresponding to different constraints, the accuracy of the registration result can be further improved.

[0078] According to some embodiments, Figure 3 As shown, the registration operation may further include: step S304, for each key point pair, determining a first covariance matrix at the first key point based on the respective coordinate values of multiple fifth key points in the first point cloud input, and determining a second covariance matrix at the second key point based on the respective coordinate values of multiple sixth key points in the second point cloud input, wherein the multiple fifth key points are all located in a second preset range centered on the first key point, and the multiple sixth key points are all located in a third preset range centered on the second key point. The multiple fifth key points and the multiple sixth key points may be local neighbors of the first key point and the second key point.

[0079] According to some embodiments, determining third target update information for the estimated pose may include determining the third target update information based on a positional relationship between the second keypoint and a third keypoint corresponding to the first keypoint, the first covariance matrix, the second covariance matrix, and the estimated pose. Thus, by using the respective covariance matrices of the first keypoint and the second keypoint, geometric constraints can be implemented, thereby supplementing color constraints to improve point cloud registration.

[0080] In some embodiments, the third target update information may be, for example, the target function E of the geometric constraint. G (T), as shown below. Assume that Cov p , Cov q are the covariance matrices calculated based on the coordinates of the second key point p and the points near the first key point q, respectively, then:

[0081]

[0082] in, is the positional relationship between the second key point and the third key point.

[0083] According to some embodiments, updating the estimated pose based on the first target update information and the third target update information includes: determining the weights of the first target update information and the third target update information; determining a first residual and a first Jacobian matrix based on the first target update information and the estimated pose; determining a second residual and a second Jacobian matrix based on the third target update information and the estimated pose; concatenating the first residual and the second residual based on the weights of the first target update information and the third target update information to obtain a fused residual; concatenating the first Jacobian matrix and the second Jacobian matrix based on the weights of the first target update information and the third target update information to obtain a fused Jacobian matrix; determining pose update information based on at least the fused residual and the fused Jacobian matrix; and updating the estimated pose based on the pose update information. Thus, multiple target update information corresponding to different constraints are combined and optimized to obtain pose estimation results under these constraints.

[0084] In some embodiments, σ∈[0,1] and (1-σ) are used as weights to balance color constraints and geometric constraints, then:

[0085] E(T)=(1-σ)E C (T)+σ·E G (T)

[0086] The second residual r G For example, it can be obtained based on the residual corresponding to each key point pair (p, q) in the second target update information, as follows:

[0087]

[0088] Second Jacobian matrix For example, the second residual r G For the estimated pose T kThe first residual and the second residual can be concatenated based on the weights of the first target update information and the third target update information to obtain the fused residual:

[0089]

[0090] The first Jacobian matrix and the second Jacobian matrix can be concatenated based on the weights of the first target update information and the third target update information to obtain a fused Jacobian matrix:

[0091]

[0092] It is understandable that other constraints and corresponding target update information may be used, or more constraints and corresponding target update information may be introduced to update the estimated pose, which is not limited here.

[0093] According to some embodiments, Figure 4 As shown, the point cloud registration method may also include: step S405, after updating the estimated pose, performing a matching operation on the first point cloud input and the second point cloud input based on the updated estimated pose to obtain at least one updated key point pair; and step S406, in response to determining that the first preset condition is not met, performing a registration operation on the at least one updated key point pair. Figure 4 The operations of steps S401 to S404 in Figure 2 The operations of step S201 to step S204 in are similar and will not be described in detail here. Thus, by setting the preset conditions, it is possible to iteratively update the estimated pose to further improve the accuracy of the registration result.

[0094] According to some embodiments, the first preset condition may include at least one of the following: the difference between the average value of the distance between the first key point and the second key point included in each key point pair of the at least one updated key point pair and the average value of the distance between the first key point and the second key point included in each key point pair of the at least one updated key point pair is less than a first preset threshold; the difference between a first ratio of the number of the at least one updated key point pair to the number of points included in the first point cloud input and a second ratio of the number of the at least one updated key point pair to the number of points included in the first point cloud input is less than a second preset threshold; and the difference between a third ratio of the number of the at least one updated key point pair to the number of points included in the second point cloud input and a fourth ratio of the number of the at least one updated key point pair to the number of points included in the second point cloud input is less than a third preset threshold. Thus, the iteration can be terminated when the change in the distance between the key point pairs is small or the change in the ratio of the number of key point pairs to the number of points included in the point cloud is small.

[0095] It is understandable that those skilled in the art can set the first preset threshold, the second preset threshold and the third preset threshold by themselves, and can also set corresponding convergence conditions according to their needs, such as the maximum number of iterations, etc., which are not limited here.

[0096] According to some embodiments, obtaining the first point cloud and the second point cloud may include: downsampling the source point cloud and the target point cloud respectively to obtain the first point cloud and the second point cloud. Figure 4 As shown, the point cloud registration method may further include: step S407: after updating the estimated pose, in response to determining that the first preset condition is satisfied, performing a registration operation on the source point cloud and the target point cloud based on the updated estimated pose. Thus, by using the estimated pose obtained based on the low-resolution first point cloud and the second point cloud as the initial value for high-resolution registration, the problem of inaccurate initial values encountered when registering high-resolution point clouds can be solved, thereby improving the accuracy of the point cloud registration results and shortening the time required for convergence.

[0097] According to some embodiments, the input point cloud can be downsampled using a voxel grid from large to small to construct a point cloud pyramid. Registration is then performed starting from the lowest resolution level in the pyramid. The pose obtained after convergence of the registration at the lowest level is used as the initial pose for the next lowest level. Registration is then performed layer by layer until the estimated pose at the highest resolution is obtained, which serves as the final pose. This approach makes the objective function smoother at each level, and the final estimated pose more accurate.

[0098] According to another aspect of the present disclosure, a point cloud registration device is disclosed. Figure 5As shown, the point cloud registration device 500 includes: an acquisition unit 510, configured to acquire a first point cloud, a second point cloud, and an estimated pose of the reference system of the second point cloud relative to the reference system of the first point cloud; a determination unit 520, configured to determine a first point cloud input based on the first point cloud and a second point cloud input based on the second point cloud; a matching unit 530, configured to perform a matching operation on the first point cloud input and the second point cloud input based on the estimated pose to obtain at least one key point pair, wherein each key point pair in the at least one key point pair includes a first key point belonging to the first point cloud input and a second key point belonging to the second point cloud input; and a registration unit 540, configured to perform a registration operation on the at least one key point pair, wherein the matching The quasi-unit includes: a color estimation subunit 541, which is configured to: for each key point pair in at least one key point pair: perform coordinate transformation on the first key point at least based on the estimated pose to obtain a third key point in the reference system of the second point cloud; and determine the estimated color parameters of the third key point based on the true color parameters of the second key point and the relationship between the second key point and the third key point; a determination subunit 542, which is configured to determine first target update information for the estimated pose based on the true color parameters of the first key point in at least one key point pair and the estimated color parameters of the third key point corresponding to the first key point; and an update subunit 543, which is configured to update the estimated pose based on at least the first target update information.

[0099] It can be understood that the operations of the units 510-540 and the subunits 541-543 in the point cloud registration device 500 are respectively Figure 2 The operations of step S201 to step S207 are similar and will not be described in detail here.

[0100] Thus, by using an objective function including the point cloud reflectance value, a constraint on the point cloud reflectance value (i.e., the color of the point) is introduced, thereby improving the pose estimation result. Specifically, by performing a coordinate transformation on the first key point based on the estimated pose, a third key point is obtained that is located in the same reference system as the second key point. By using the true color parameters of the corresponding second key point and the positional relationship between the second key point and the third key point, an accurate estimate of the color of the position of the third key point can be obtained, thereby updating the estimated pose according to the estimated color parameters of the third key point and the true color parameters of the first key point to obtain an accurate pose estimation result.

[0101] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0102] According to an embodiment of the present disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0103] refer to Figure 6 , a block diagram of an electronic device 600 that can serve as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0104] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the device 600 can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0105] Multiple components within device 600 are connected to I / O interface 605, including an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. Input unit 606 can be any type of device capable of inputting information into device 600. Input unit 606 can receive input numeric or character information and generate key signal input related to user settings and / or function control of the electronic device. It may include, but is not limited to, a mouse, keyboard, touch screen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 607 can be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. Storage unit 608 may include, but is not limited to, a magnetic disk or an optical disk. Communication unit 609 allows device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks. It may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver and / or chipset, such as a Bluetooth™ device, a 1302.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0106] The computing unit 601 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning network algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the point cloud registration method. For example, in some embodiments, the point cloud registration method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the point cloud registration method described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the point cloud registration method by any other suitable means (e.g., via firmware).

[0107] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0108] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0109] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0110] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0111] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0112] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0113] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0114] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but is only limited by the claims after authorization and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. In addition, the steps may be performed in an order different from that described in this disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that as technology evolves, many of the elements described herein may be replaced by equivalent elements that appear after this disclosure.

Claims

1. A point cloud registration method, comprising: Obtaining a first point cloud, a second point cloud, and an estimated pose of a reference frame of the second point cloud relative to a reference frame of the first point cloud; determining a first point cloud input based on the first point cloud and determining a second point cloud input based on the second point cloud; Based on the estimated pose, performing a matching operation on the first point cloud input and the second point cloud input to obtain at least one keypoint pair, wherein each keypoint pair in the at least one keypoint pair includes a first keypoint belonging to the first point cloud input and a second keypoint belonging to the second point cloud input; and Performing a registration operation on the at least one keypoint pair, wherein the registration operation comprises: For each key point pair in the at least one key point pair: performing a coordinate transformation on the first key point based at least on the estimated pose to obtain a third key point in the reference frame of the second point cloud; and Determining an estimated color parameter of the third key point based on a true color parameter of the second key point and a relationship between the second key point and the third key point, wherein the true color parameter includes a true color value of the second key point and a color gradient at the second key point; determining first target update information for the estimated pose based on a true color parameter of a first key point in the at least one key point pair and an estimated color parameter of a third key point corresponding to the first key point, the first target update information being an objective function constrained by a point cloud reflectance value; and The estimated pose is updated based on at least the first target update information.

2. The method according to claim 1, wherein The registration operation further includes: Determine a color gradient at the second key point based on the true color value of the second key point, the true color values of each of multiple fourth key points in the second point cloud input, and the positional relationship between the second key point and the multiple fourth key points, wherein the multiple fourth key points are all located within a first preset range centered on the second key point.

3. The method according to claim 2, wherein: Determining the color gradient at the second key point includes: For each of the plurality of fourth key points, determining target update sub-information corresponding to the fourth key point based on a true color value of the fourth key point, the true color value and the estimated color gradient of the second key point, and a positional relationship between the fourth key point and the second key point; determining second target update information for the estimated color gradient based on the target update sub-information corresponding to each of the plurality of fourth key points; and Based on the second target update information, a color gradient at the second key point is determined.

4. The method according to claim 1, wherein The registration operation further includes: Determine the reference surface at the second key point, wherein the color gradient at the second key point is coplanar with the reference surface, and The step of performing coordinate transformation on the first key point based at least on the estimated pose includes: The corresponding first key point is subjected to coordinate transformation based on the estimated posture and is further projected onto the reference surface to obtain the third key point.

5. The method of claim 1, wherein The true color parameter includes a true color value of the second key point, the estimated color parameter includes an estimated color value, and determining first target update information about the estimated pose includes: For each keypoint pair, determining first target update sub-information corresponding to the keypoint pair based on a difference between a true color value of a first keypoint in the keypoint pair and an estimated color value of a third keypoint corresponding to the first keypoint, the first target update sub-information being an objective function constrained by the point cloud reflectance value; and Based on the first target update sub-information corresponding to each of the at least one key point pair, first target update information about the estimated pose is determined.

6. The method of claim 1, wherein: The first target update information includes an item for each key point pair in the at least one key point pair, and updating the estimated pose based at least on the first target update information includes: Based on the first target update information and the estimated pose, a first residual and a first Jacobian matrix are determined, wherein the first residual is obtained by the following formula: Where (p, q) is each key point pair in the at least one key point pair, T k is the estimated pose, The first Jacobian matrix is obtained by differentiating each component of the estimated pose by the first residual. Determining pose update information using a Gauss-Newton method based on the first residual and the first Jacobian matrix; and Based on the pose update information, the estimated pose is updated.

7. The method of claim 3, wherein: The true color parameter includes a true color value of the second key point, the estimated color parameter includes an estimated color value, and the registration operation further includes: For each key point pair: determining a first covariance matrix at the first key point based on respective coordinate values of a plurality of fifth key points in the first point cloud input, wherein the plurality of fifth key points are all located within a second preset range centered on the first key point; and Determining a second covariance matrix at the second key point based on respective coordinate values of a plurality of sixth key points in the second point cloud input, wherein the plurality of sixth key points are all located within a third preset range centered on the second key point; Determining third target update information for the estimated pose based on a positional relationship between the second key point and a third key point corresponding to the first key point, the first covariance matrix, the second covariance matrix, and the estimated pose, Wherein, updating the estimated pose based at least on the first target update information includes: The estimated pose is updated based on the first target update information and the third target update information.

8. The method of claim 7, wherein: The first target update information includes an item for each key point pair in the at least one key point pair, the second target update information includes an item for each key point pair in the at least one key point pair, and updating the estimated pose based on the first target update information and the third target update information includes: Determining respective weights of the first target update information and the third target update information; Based on the first target update information and the estimated pose, a first residual and a first Jacobian matrix are determined, wherein the first residual is obtained by the following formula: Where (p, q) is each key point pair in the at least one key point pair, T k is the estimated pose, The first Jacobian matrix is obtained by differentiating each component of the estimated pose by the first residual; Based on the third target update information and the estimated pose, a second residual and a second Jacobian matrix are determined, wherein the second residual is obtained by the following formula: Where (p, q) is each key point pair in the at least one key point pair, T k is the estimated pose, The residual corresponding to the item for each key point pair (p, q) in the second target update information is obtained by differentiating each component in the estimated pose by the second residual; concatenating the first residual and the second residual based on respective weights of the first target update information and the third target update information to obtain a fused residual; concatenating the first Jacobian matrix and the second Jacobian matrix based on respective weights of the first target update information and the third target update information to obtain a fused Jacobian matrix; Determining pose update information based at least on the fused residual and the fused Jacobian matrix; and Based on the pose update information, the estimated pose is updated.

9. The method of claim 1, further comprising: After updating the estimated pose, performing a matching operation on the first point cloud input and the second point cloud input based on the updated estimated pose to obtain at least one updated keypoint pair; as well as In response to determining that the first preset condition is not satisfied, performing the registration operation on the updated at least one keypoint pair.

10. The method of claim 9, wherein: The first preset condition includes at least one of the following: a difference between an average value of distances between a first key point and a second key point included in each of the at least one key point pair after the update and an average value of distances between the first key point and the second key point included in each of the at least one key point pair before the update being less than a first preset threshold; a difference between a first ratio of the number of the at least one key point pair after the update to the number of points included in the first point cloud input and a second ratio of the number of the at least one key point pair before the update to the number of points included in the first point cloud input is less than a second preset threshold; and A difference between a third ratio of the number of at least one key point pair after the update to the number of points included in the second point cloud input and a fourth ratio of the number of at least one key point pair before the update to the number of points included in the second point cloud input is less than a third preset threshold.

11. The method of claim 9, wherein: Determining a first point cloud input based on the first point cloud and determining a second point cloud input based on the second point cloud includes: Downsampling the first point cloud and the second point cloud respectively to obtain the first point cloud input and the second point cloud input, The method further comprises: After the estimated pose is updated, in response to determining that the first preset condition is satisfied, the registration operation is performed on the first point cloud and the second point cloud based on the updated estimated pose.

12. The method of claim 1, wherein The matching operation includes: Performing coordinate transformation on the first point cloud input based on the estimated pose; and Matching the first point cloud input and the second point cloud input after coordinate transformation.

13. The method according to any one of claims 1 to 12, wherein The estimated pose includes a rotation component and a translation component.

14. The method according to any one of claims 1 to 12, wherein The first point cloud is generated based on data collected by vehicle-mounted sensors, and the second point cloud is generated based on high-precision map data.

15. A point cloud registration device, comprising: an acquisition unit configured to acquire a first point cloud, a second point cloud, and an estimated pose of a reference frame of the second point cloud relative to a reference frame of the first point cloud; a determining unit configured to determine a first point cloud input based on the first point cloud and to determine a second point cloud input based on the second point cloud; a matching unit configured to perform a matching operation on the first point cloud input and the second point cloud input based on the estimated pose to obtain at least one keypoint pair, wherein each keypoint pair in the at least one keypoint pair includes a first keypoint belonging to the first point cloud input and a second keypoint belonging to the second point cloud input; and A registration unit is configured to perform a registration operation on the at least one key point pair, wherein the registration unit comprises: The color estimation subunit is configured to, for each of the at least one key point pair: perform a coordinate transformation on a first key point based on at least the estimated pose to obtain a third key point in a reference system of the second point cloud; and determine an estimated color parameter of the third key point based on a true color parameter of the second key point and a relationship between the second key point and the third key point, wherein the true color parameter includes a true color value of the second key point and a color gradient at the second key point; a determining subunit configured to determine first target update information for the estimated pose based on a true color parameter of a first key point in the at least one key point pair and an estimated color parameter of a third key point corresponding to the first key point, wherein the first target update information is an objective function constrained by a point cloud reflectance value; and An updating subunit is configured to update the estimated pose based at least on the first target update information.

16. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14.

17. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-14.

18. A computer program product comprising a computer program, wherein When the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

19. An autonomous driving vehicle comprising: The electronic device according to claim 16.

Images