Point cloud fusion method and device, electronic equipment and medium

Abstract

The present disclosure provides a point cloud fusion method and device, electronic equipment and medium, relates to the technical field of autonomous driving, and particularly relates to the fields of high-precision maps and cloud computing. The specific implementation scheme is: determining a to-be-registered point cloud image from historical point cloud images according to a target collection position of a target point cloud image; determining an auxiliary road element associated with a target road element from the to-be-registered point cloud image according to a target element position of the target road element included in the target point cloud image; and registering the target point cloud image and the to-be-registered point cloud image according to the target element position and an auxiliary element position of the auxiliary road element. The present disclosure achieves the effect of reducing the amount of data required for point cloud image registration, shortens the time required for point cloud image registration, and thus improves the efficiency of point cloud image registration.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of automatic driving, in particular to the technical field of high-definition map and cloud computing, and specifically to a point cloud fusion method and device, an electronic device and a medium. BACKGROUND

[0002] A high-definition map, also known as a high-precision map, is used by an autonomous vehicle. The high-definition map has accurate vehicle position information and rich road element data information, which can help the vehicle to predict complex road information such as slope, curvature, heading, etc., and better avoid potential risks. Currently, the map collected by a laser radar usually exists in the form of a point cloud image. The point cloud image is a form of representation of a three-dimensional object or a three-dimensional scene, and is composed of a set of irregularly distributed discrete points expressing the spatial structure and surface properties of a three-dimensional object or a three-dimensional scene.

[0003] Before point cloud fusion is performed on the point cloud image, each point cloud image needs to be registered. Currently, the registration is usually performed based on the geometric information of the full point cloud. SUMMARY

[0004] The present disclosure provides a method, device, electronic device and medium for registering point cloud images.

[0005] According to an aspect of the present disclosure, a point cloud fusion method is provided, comprising:

[0006] determining a point cloud image to be registered from historical point cloud images according to a target collection position of a target point cloud image;

[0007] determining an auxiliary road element associated with a target road element from the point cloud image to be registered according to a target element position of the target road element included in the target point cloud image;

[0008] registering the target point cloud image and the point cloud image to be registered according to the target element position and an auxiliary element position of the auxiliary road element.

[0009] According to another aspect of the present disclosure, a point cloud fusion device is provided, comprising:

[0010] a point cloud image determining module configured to determine a point cloud image to be registered from historical point cloud images according to a target collection position of a target point cloud image;

[0011] a road element determining module configured to determine an auxiliary road element associated with a target road element from the point cloud image to be registered according to a target element position of the target road element included in the target point cloud image;

[0012] a point cloud image registration module configured to register the target point cloud image and the point cloud image to be registered according to the target element position and the auxiliary element position of the auxiliary road element.

[0013] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0014] at least one processor; and

[0015] a memory connected to the at least one processor in communication; wherein,

[0016] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of the present disclosure.

[0017] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to make the computer perform the method of any one of the present disclosure.

[0018] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program for performing the method of any one of the present disclosure when executed by a processor.

[0019] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings are used to better understand the present scheme, and do not limit the present disclosure. Among them:

[0021] Figure 1 is a flowchart of some point cloud fusion methods disclosed according to embodiments of the present disclosure;

[0022] Figure 2 is a flowchart of another point cloud fusion method disclosed according to embodiments of the present disclosure;

[0023] Figure 3 is a structural schematic diagram of some point cloud fusion devices disclosed according to embodiments of the present disclosure;

[0024] Figure 4 is a block diagram of an electronic device for implementing the point cloud fusion method disclosed according to embodiments of the present disclosure. DETAILED DESCRIPTION

[0025] Exemplary embodiments of the present disclosure are described herein below with reference to the accompanying drawings, in which various details are set forth to facilitate an understanding of the present disclosure. It should be appreciated that various embodiments of the present disclosure can be practiced with variation of details not set forth herein, without departing from the scope and spirit of the present disclosure. Also, it should be noted that well-known functions or constructions are not described in detail for brevity and clarity.

[0026] The applicant found in the development process that, before the point cloud image is fused, the pose offset matrix between the point cloud images is usually determined based on the geometric information of the full-quantity point cloud data contained in each point cloud image, and then each point cloud image is registered based on the pose offset matrix, and finally the point cloud fusion is performed based on the registered point cloud images.

[0027] However, the existing point cloud image registration method directly uses the geometric information of the full-quantity point cloud data, which undoubtedly causes the problem of too large data quantity required for registration when the number of point cloud images to be registered is large, resulting in long time required for point cloud image registration, and thus low efficiency of point cloud image registration.

[0028] Figure 1 is a flowchart of some point cloud fusion methods disclosed according to an embodiment of the present disclosure. The embodiment can be applicable to the case of registering each point cloud image. The embodiment method can be executed by a point cloud fusion device disclosed by an embodiment of the present disclosure, which can be implemented by software and / or hardware, and can be integrated on any electronic device with computing capability.

[0029] As shown in Figure 1 The point cloud fusion method disclosed by the embodiment can include:

[0030] S101, determining a point cloud image to be registered from a historical point cloud image according to a target acquisition position of a target point cloud image.

[0031] The point cloud image represents a point cloud data set obtained by scanning a target object or a target scene by a measuring instrument, such as a three-dimensional laser scanner or a photographic scanner. The point cloud image in the embodiment refers to a point cloud data set obtained by scanning a traffic scene such as a street, a highway or an expressway. The target point cloud image refers to a frame of point cloud image that needs to be registered in the historical point cloud image that has been collected. The embodiment does not limit the selection method of the target point cloud image, which can be randomly selected from the historical point cloud image, or selected from the historical point cloud image according to the actual business demand by a relevant technical personnel, etc. The target acquisition position represents the 3D position coordinates of the measuring instrument in the world coordinate system when the target point cloud image is collected.

[0032] In an embodiment, during the point cloud image acquisition, the correspondence between the acquisition positions and the point cloud images is recorded in real time. According to the recorded correspondence between the acquisition positions and the point cloud images, a target acquisition position corresponding to a target point cloud image is determined. The target acquisition position is matched with the acquisition positions of all historical point cloud images, and the acquisition positions within a preset distance threshold from the target acquisition position are taken as auxiliary acquisition positions. Then, according to the recorded correspondence between the acquisition positions and the point cloud images, the point cloud images corresponding to the auxiliary acquisition positions are taken as to-be-registered point cloud images, wherein the number of to-be-registered point cloud images can be one frame or at least two frames. Further, the target point cloud image and the to-be-registered point cloud images are taken as a to-be-registered image sequence for subsequent registration of the target point cloud image and the to-be-registered point cloud images.

[0033] By determining the to-be-registered point cloud images from the historical point cloud images according to the target acquisition position of the target point cloud image, a data basis is laid for subsequent registration of the target point cloud image and the to-be-registered point cloud images.

[0034] S102, determining an auxiliary road element associated with the target road element from the to-be-registered point cloud image according to a target element position of a target road element included in the target point cloud image.

[0035] The target road element refers to a road-related entity element contained in the target point cloud image, and the auxiliary road element refers to a road-related entity element contained in the to-be-registered point cloud image. In this embodiment, the element types of the target road element and the auxiliary road element include but are not limited to lane lines, road arrows, road signs, and street lamp poles, etc. The target element position refers to the image 3D coordinate position of the target road element in the target point cloud image. In this embodiment, the center point 3D coordinate position of the road element in the point cloud image is optionally taken as the element position.

[0036] In an embodiment, the target point cloud image and the to-be-registered point cloud image are respectively subjected to semantic segmentation and element extraction, to determine the target road element included in the target point cloud image and the auxiliary road element included in the to-be-registered point cloud image. According to the target element position of the target road element, a search position is determined as the coordinate position same as the target element position in the to-be-registered point cloud image, and a road element search is performed in the vicinity of the search position according to a preset distance threshold, to determine whether there is a road element with the same element type as the target road element and within the distance threshold from the element position, and if so, the road element is taken as the auxiliary road element associated with the target road element in the to-be-registered point cloud image.

[0037] For example, assuming that the element type of the target road element A is a lane line, the target element position of the target road element A in the target point cloud image is (A, B, C), the position (A, B, C) in the point cloud image to be registered is taken as the search position. Assuming that the distance threshold is 10 m, a spherical search region is formed with (A, B, C) as the center and 10 m as the radius, and a search is performed in the spherical search region. Assuming that there is a road element A' in the spherical search region, the element type of which is a lane line and the element position of which is in the spherical search region, the road element A' is taken as the auxiliary road element associated with the target road element A in the point cloud image to be registered.

[0038] By determining the auxiliary road element associated with the target road element from the target point cloud image according to the target element position of the target road element included in the target point cloud image, the effect of determining the auxiliary road element associated with the target road element in the point cloud image to be registered is achieved, which lays a data foundation for subsequent registration of the target point cloud image and the point cloud image to be registered based on the target road element and the auxiliary road element.

[0039] S103, registering the target point cloud image and the point cloud image to be registered according to the target element position and the auxiliary element position of the auxiliary road element.

[0040] In an implementation, since the collection positions of the target point cloud image and the point cloud image to be registered are close, and the element positions of the target road element and the auxiliary road element in the images are also close, the target road element and the auxiliary road element can be considered as the same road element. For example, the same lane line, or the same road sign, etc.

[0041] Further, since the target road element and the auxiliary road element are the same road element, the pose matrices of the two in the world coordinate system should be the same. Then, the target collection position and the target collection pose of the target point cloud image, and the auxiliary collection position and the auxiliary collection pose of the point cloud image to be registered are determined, wherein the target collection pose represents the pose of the measuring instrument in the world coordinate system when the target point cloud image is collected, and the auxiliary collection position and the auxiliary collection pose represent the 3D position coordinates and the pose of the measuring instrument in the world coordinate system when the point cloud image to be registered is collected.

[0042] According to the target collection position, the target collection attitude and the target element position, a target pose matrix of the target road element in a world coordinate system is determined. According to the auxiliary collection position, the auxiliary collection attitude and the auxiliary element position, an auxiliary pose matrix of the auxiliary road element in the world coordinate system is determined. Then, the target pose matrix and the auxiliary pose matrix are subtracted, and the target pose matrix and the auxiliary pose matrix are adjusted according to an optimization algorithm, so as to determine a first pose matrix of the target pose matrix after adjustment and a second pose matrix of the auxiliary pose matrix after adjustment when the difference value is the smallest. Finally, the target point cloud image is registered according to the first pose matrix, and the point cloud image to be registered is registered according to the second pose matrix.

[0043] The present disclosure determines the point cloud image to be registered from the historical point cloud image according to the target collection position of the target point cloud image, and determines the auxiliary road element associated with the target road element from the point cloud image to be registered according to the target element position of the target road element included in the target point cloud image. Then, the target point cloud image and the point cloud image to be registered are registered according to the target element position and the auxiliary element position of the auxiliary road element, so that the registration of the point cloud images can be realized only according to the position information of the associated road elements between the point cloud images, without using the geometric information of the full-quantity point cloud data for registration. The effect of reducing the data amount required for point cloud image registration is realized, and the time required for point cloud image registration is shortened, thereby improving the efficiency of point cloud image registration.

[0044] Figure 2 The flowchart of another point cloud fusion method disclosed according to the embodiments of the present disclosure is based on the further optimization and expansion of the above technical solutions, and can be combined with the above various optional embodiments.

[0045] As shown in Figure 2 , the point cloud fusion method disclosed in the present embodiment can include:

[0046] S201, searching in a multi-dimensional space tree according to a target collection position of a target point cloud image and a distance threshold to determine an auxiliary collection position.

[0047] The multi-dimensional space tree is constructed according to the collection positions of each historical point cloud image, and the multi-dimensional space tree is KD-Tree, which is a data structure for partitioning K-dimensional data space. In the present embodiment, since the collection position is a 3D position, the multi-dimensional space tree is a three-dimensional space tree.

[0048] In an embodiment, the target collection position is taken as a search point, and a neighborhood search is performed in the three-dimensional space tree according to a preset distance threshold, so as to determine all collection positions within the distance threshold as the auxiliary collection position. The distance threshold can optionally include 30m.

[0049] S202, take the historical point cloud image corresponding to the auxiliary acquisition position as the point cloud image to be registered.

[0050] In an embodiment, the historical point cloud image corresponding to the auxiliary acquisition position is taken as the point cloud image to be registered according to the correspondence between the acquisition position and the historical point cloud image. Since the acquisition positions of the point cloud image to be registered and the target point cloud image are close, the point cloud data contained in the two are also close.

[0051] S203, determine the reference target road element from the target road elements included in the target point cloud image, and determine the reference auxiliary road element associated with the reference target road element from the point cloud image to be registered according to the reference target element position of the reference target road element.

[0052] The reference target road element is any target road element selected from the target road elements. The reference target element position represents the 3D position coordinates of the reference target road element in the target point cloud image. The random sample consensus algorithm is the Random Sample Consensus (RANSAC) algorithm.

[0053] In an embodiment, according to actual business experience, any target road element of the element type of the street lamp pole is selected as the reference target road element in this embodiment. The reference target element position of the reference target road element and the element positions of the road elements in the point cloud image to be registered are collectively used to form a data set. The random sample consensus algorithm is used to find the element position that best matches the reference target element position in the data set, and the road element corresponding to the element position is taken as the reference auxiliary road element.

[0054] S204, determine the element position relationship between the other target road elements and the reference target road element in the target point cloud image, and determine the other auxiliary road elements associated with the other target road elements except the reference auxiliary road element from the point cloud image to be registered according to the element position relationship.

[0055] The other target road elements are target road elements other than the reference target road element in the target road elements.

[0056] In an embodiment, the element position relationship between the other target road elements and the reference target road element is determined according to the other target element positions of the other target road elements and the reference target element position of the reference target road element. And according to the element position relationship and the reference auxiliary element position of the reference auxiliary road element, the other auxiliary road elements associated with the other target road elements except the reference auxiliary road element are determined from the point cloud image to be registered.

[0057] Optionally, S204 comprises steps A and B as follows:

[0058] A. determining a search position from the to-be-registered point cloud image according to the element position relationship and the reference auxiliary element position of the reference auxiliary road element.

[0059] For example, assuming that the other target element position of any other target road element is (A1, B1, C1) and the reference target element position is (A2, B2, C2), (A1-A2, B1-B2, C1-C2) is taken as the element position relationship between the other target road element and the reference target road element. Assuming that the reference auxiliary element position of the reference auxiliary road element is (A3, B3, C3), (A3+A1-A2, B3+B1-B2, C3+C1-C2) is taken as the search position.

[0060] B. performing neighborhood search at the search position according to the element type of the other target road element and the distance threshold value to determine other auxiliary road elements associated with the other target road element.

[0061] In an embodiment, road elements whose element positions are within the distance threshold value of the search position and whose element types are the same as the element type of the other target road element are taken as the other auxiliary road elements associated with the other target road element.

[0062] For example, assuming that the element type of any other target road element is a lane line and the search position is (X, Y, Z) and the distance threshold value is (X1, Y1, Z1), road elements whose element positions are within (X, Y, Z)~(X+X1, Y+Y1, Z+Z1) and whose element types are lane lines are taken as the other auxiliary road elements associated with the other target road element.

[0063] By determining a search position from the to-be-registered point cloud image according to the element position relationship and the reference auxiliary element position of the reference auxiliary road element and performing neighborhood search at the search position according to the element type of the other target road element and the distance threshold value to determine other auxiliary road elements associated with the other target road element, it is realized that only the determined element position relationship can be used to determine the other auxiliary road elements associated with the other target road element, and it is not necessary to execute a random sample consensus algorithm to determine the associated other auxiliary road elements for each other target road element, thereby reducing the algorithm processing process and improving the efficiency.

[0064] S205. determining a target collection pose of the target point cloud image and an auxiliary collection position and an auxiliary collection pose of the to-be-registered point cloud image.

[0065] In an implementation, a target acquisition pose corresponding to the target point cloud image and an auxiliary acquisition position and an auxiliary acquisition pose corresponding to the point cloud image to be registered are determined according to the correspondence between the acquired historical point cloud image and the acquisition pose.

[0066] In S206, the target point cloud image and the point cloud image to be registered are registered according to the target acquisition position, the target acquisition pose, the auxiliary acquisition position, the auxiliary acquisition pose, the target element position, and the auxiliary element position.

[0067] The target element position includes a reference target element position of a reference target road element and other target element positions of other target road elements. The auxiliary element position includes a reference auxiliary element position of a reference auxiliary road element and other auxiliary element positions of other auxiliary road elements.

[0068] The disclosure searches in the multi-dimensional space tree according to the target acquisition position of the target point cloud image and the distance threshold, determines the auxiliary acquisition position, takes the historical point cloud image corresponding to the auxiliary acquisition position as the point cloud image to be registered, ensures that the acquisition positions of the point cloud image to be registered and the target point cloud image are close, and further ensures the accuracy of subsequent road element association. The disclosure determines the reference target road element from the target road elements included in the target point cloud image, determines the reference auxiliary road element associated with the reference target road element from the point cloud image to be registered according to the reference target element position of the reference target road element, determines the element position relationship between the other target road elements and the reference target road element in the target point cloud image, and determines the other auxiliary road elements associated with the other target road elements except the reference auxiliary road element from the point cloud image to be registered according to the element position relationship, which realizes that the other auxiliary road elements associated with the other target road elements can be determined only by relying on the element position relationship, without executing the random sample consensus algorithm to determine the associated other auxiliary road elements for each other target road element, reduces the algorithm processing process, and improves the efficiency. The disclosure determines the target acquisition pose of the target point cloud image and the auxiliary acquisition position and the auxiliary acquisition pose of the point cloud image to be registered, registers the target point cloud image and the point cloud image to be registered according to the target acquisition position, the target acquisition pose, the auxiliary acquisition position, the auxiliary acquisition pose, the target element position, and the auxiliary element position, does not need to use the geometric information of the full-quantity point cloud data for registration, realizes the effect of reducing the data quantity required for point cloud image registration, shortens the time required for point cloud image registration, and thus improves the efficiency of point cloud image registration.

[0069] Optionally, S206 includes the following steps A1, A2, and A3:

[0070] A1, determining a target pose matrix of the target road element in a world coordinate system according to the target collection position, the target collection pose and the target element position.

[0071] In an implementation, the target element position is multiplied by the target collection pose in a matrix, and the multiplication result is added to the target collection position in a matrix, and the addition result is taken as the target pose matrix of the target road element in the world coordinate system.

[0072] A2, determining an auxiliary pose matrix of the auxiliary road element in the world coordinate system according to the auxiliary collection position, the auxiliary collection pose and the auxiliary element position.

[0073] In an implementation, the auxiliary element position is multiplied by the auxiliary collection pose in a matrix, and the multiplication result is added to the auxiliary collection position in a matrix, and the addition result is taken as the auxiliary pose matrix of the auxiliary road element in the world coordinate system.

[0074] A3, registering the target point cloud image and the to-be-registered point cloud image according to the target pose matrix and the auxiliary pose matrix.

[0075] In an implementation, since the collection positions of the target point cloud image and the to-be-registered point cloud image are close, and the element positions of the target road element and the auxiliary road element in the images are also close, the target road element and the auxiliary road element can be considered as the same road element, such as the same lane line or the same road sign. Further, since the target road element and the auxiliary road element are the same road element, the pose matrices of the two in the world coordinate system should be the same. Therefore, the target pose matrix and the auxiliary pose matrix are subtracted in a matrix, and an optimization algorithm is used to determine a first pose matrix corresponding to the target pose matrix and a second pose matrix corresponding to the auxiliary pose matrix when the subtraction result is the smallest. Finally, the target point cloud image and the to-be-registered point cloud image are registered according to the first pose matrix and the second pose matrix.

[0076] The target pose matrix of the target road element in the world coordinate system is determined according to the target acquisition position, the target acquisition attitude and the target element position, the auxiliary pose matrix of the auxiliary road element in the world coordinate system is determined according to the auxiliary acquisition position, the auxiliary acquisition attitude and the auxiliary element position, and then the target point cloud image and the point cloud image to be registered are registered according to the target pose matrix and the auxiliary pose matrix. Since the acquisition positions of the target point cloud image and the point cloud image to be registered are close, and the element positions of the target road element and the auxiliary road element in the images are also close, it can be considered that the target road element and the auxiliary road element are the same road element. Therefore, the effect of registering the target point cloud image and the point cloud image to be registered by taking the target pose matrix and the auxiliary pose matrix as anchor points is achieved. The registration of the point cloud image does not need to use the geometric information of the full amount of point cloud data, and the effect of reducing the amount of data required for point cloud image registration is achieved.

[0077] Optionally, step A3 includes the following steps A31, A32 and A33:

[0078] A31, constructing a constraint function according to the target pose matrix and the auxiliary pose matrix.

[0079] In an embodiment, the constraint function is constructed according to the matrix difference between the target pose matrix and the auxiliary pose matrix.

[0080] A32, adjusting the target pose matrix and the auxiliary pose matrix by using an optimization algorithm, and determining a first pose matrix after adjustment of the target pose matrix and a second pose matrix after adjustment of the auxiliary pose matrix when the function value of the constraint function is a target value.

[0081] In an embodiment, the target pose matrix and the auxiliary pose matrix are adjusted by using a nonlinear least square method, and a first pose matrix after adjustment of the target pose matrix and a second pose matrix after adjustment of the auxiliary pose matrix are determined when the difference between the two is the smallest.

[0082] A33, registering the target point cloud image and the point cloud image to be registered according to the first pose matrix and the second pose matrix.

[0083] In an embodiment, the offset matrix and the displacement matrix of the target point cloud image are determined according to the first pose, the offset matrix and the displacement matrix of the point cloud image to be registered are determined according to the second pose, and the target point cloud image and the point cloud image to be registered are registered according to the corresponding offset matrix and displacement matrix of the target point cloud image and the point cloud image to be registered.

[0084] By constructing a constraint function according to the target pose matrix and the auxiliary pose matrix, and adjusting the target pose matrix and the auxiliary pose matrix by using an optimization algorithm, and determining the first pose matrix after adjustment of the target pose matrix and the second pose matrix after adjustment of the auxiliary pose matrix when the function value of the constraint function is a target value, and then registering the target point cloud image and the point cloud image to be registered according to the first pose matrix and the second pose matrix, the effect of quickly determining the first pose matrix after adjustment of the target pose matrix and the second pose matrix after adjustment of the auxiliary pose matrix based on the constraint function and the optimization algorithm is realized, and the acquisition time of parameters required for point cloud image registration is shortened.

[0085] Optionally, step A33 is followed by steps A331, A332 and A333:

[0086] A331, determining a target rotation matrix and a target displacement matrix according to the first pose matrix and the target pose matrix.

[0087] In an embodiment, an equation is constructed from the first pose matrix and the target pose matrix, and the equation is solved to determine the target rotation matrix and the target displacement matrix.

[0088] For example, an equation Y = RX + T is constructed, where Y represents the first pose matrix, X represents the target pose matrix, R represents the target rotation matrix, and T represents the target displacement matrix, and then the equation is solved to determine R and T.

[0089] A332, determining an auxiliary rotation matrix and an auxiliary displacement matrix according to the second pose matrix and the auxiliary pose matrix.

[0090] In an embodiment, an equation is constructed from the second pose matrix and the auxiliary pose matrix, and the equation is solved to determine the auxiliary rotation matrix and the auxiliary displacement matrix.

[0091] For example, an equation Y1 = R1X1 + T1 is constructed, where Y1 represents the second pose matrix, X1 represents the auxiliary pose matrix, R1 represents the auxiliary rotation matrix, and T1 represents the auxiliary displacement matrix, and then the equation is solved to determine R1 and T1.

[0092] A333, registering the target point cloud image by using the target rotation matrix and the target displacement matrix, and registering the point cloud image to be registered by using the auxiliary rotation matrix and the auxiliary displacement matrix.

[0093] In an embodiment, the target rotation matrix and the target displacement matrix are used to perform pose conversion on all point cloud data in the target point cloud image, and correspondingly, the auxiliary rotation matrix and the auxiliary displacement matrix are used to perform pose conversion on all point cloud data in the point cloud image to be registered, so as to realize registration of the target point cloud image and the point cloud image to be registered.

[0094] By determining the target rotation matrix and the target displacement matrix according to the first pose matrix and the target pose matrix, and determining the auxiliary rotation matrix and the auxiliary displacement matrix according to the second pose matrix and the auxiliary pose matrix, and then using the target rotation matrix and the target displacement matrix to register the target point cloud image, and using the auxiliary rotation matrix and the auxiliary displacement matrix to register the point cloud image to be registered, the effect of jointly registering the target point cloud image and the point cloud image to be registered is realized, without using the geometric information of the full point cloud data for registration, and the effect of reducing the amount of data required for point cloud image registration is realized.

[0095] The step A31 is optimized in the embodiments of the present disclosure, and the step A31 comprises:

[0096] In the case that the element types of the target road element and the auxiliary road element include a road sign, the following e1 and e2 constraint functions are constructed:

[0097]

[0098]

[0099] wherein, The denotes a target pose matrix, and the R src denotes a target collection posture, and the T src denotes a target collection position, and the P src denotes a target element position, and the denotes an auxiliary pose matrix, and the R dst denotes an auxiliary collection posture, and the T dst denotes an auxiliary collection position, and the P dst denotes an auxiliary element position.

[0100] The denotes a normal vector of the target road element, that is, in the case that the element type of the target road element is a road sign, the normal vector of the road sign. The n dst denotes a normal vector of the auxiliary road element, that is, the normal vector of the road sign associated with the road sign in the point cloud image to be registered. w1 and w2 are fixed constants.

[0101] By constructing the e1 and e2 constraint functions in the case that the element types of the target road element and the auxiliary road element include a road sign, the effect of adaptively constructing the constraint function for the target road element and the auxiliary road element with the element type of the road sign is realized, and the accuracy of subsequently determining the first pose matrix and the second pose matrix is improved.

[0102] The embodiment of the present disclosure further optimizes the step A31, and the step A31 comprises:

[0103] In the case that the element types of the target road element and the auxiliary road element comprise lane lines and / or lamp posts, the following e3 and e4 constraint functions are constructed:

[0104]

[0105]

[0106] wherein, The represents a target pose matrix, the R src represents a target acquisition pose, the T src represents a target acquisition position, the P src represents a target element position, the represents an auxiliary pose matrix, the R dst represents an auxiliary acquisition pose, the T dst represents an auxiliary acquisition position, the P dst represents an auxiliary element position.

[0107] The v src represents the orientation of the target road element, that is, in the case that the element type of the target road element is a lane line, the orientation is the driving direction of the lane line; or in the case that the element type of the target road element is a lamp post, the orientation is the upward direction of the lamp post. The v dst represents the orientation of the auxiliary road element, that is, in the case that the lane line is associated with the driving direction of the lane line in the to-be-registered point cloud image; or in the case that the lamp post is associated with the upward direction of the lamp post in the to-be-registered point cloud image. w3 and w4 are fixed constants.

[0108] By constructing the e3 and e4 constraint functions in the case that the element types of the target road element and the auxiliary road element comprise lane lines and / or lamp posts, the effect of adaptively constructing the constraint functions for the target road element and the auxiliary road element with the element types of lane lines and / or lamp posts is achieved, and the accuracy of subsequently determining the first pose matrix and the second pose matrix is improved.

[0109] The embodiment of the present disclosure further optimizes the step A31, and the step A31 comprises:

[0110] In the case that the element types of the target road element and the auxiliary road element comprise road arrows, the following e5 constraint function is constructed:

[0111]

[0112] The road arrow represents a divided arrow in the road, such as a road left-turn arrow, a road straight arrow, or a road right-turn arrow, and the like.

[0113] The The target pose matrix is represented by T, the R src The target acquisition pose is represented by T, the R src The target acquisition position is represented by P, the R src The target element position is represented by P, the R The auxiliary pose matrix is represented by T, the R dst The auxiliary acquisition pose is represented by T, the R dst The auxiliary acquisition position is represented by P, the R dst The auxiliary element position is represented by P.

[0114] By constructing the e5 constraint function in the case where the element type of the target road element and the auxiliary road element includes a road arrow, the effect of adaptively constructing a constraint function for the target road element and the auxiliary road element with an element type of a road arrow is achieved, and the accuracy of subsequently determining the first pose matrix and the second pose matrix is improved.

[0115] It can be understood that if the element type of the target road element and the auxiliary road element includes a road sign, a lane line, a streetlight pole, and a road arrow, then the e1, e2, e3, e4, and e5 constraint functions are constructed together for constraint.

[0116] Figure 3 The structure diagram of some point cloud fusion devices disclosed according to the embodiments of the present disclosure, which can be applicable to the case of registering each point cloud image. The device of the present embodiment can be implemented by software and / or hardware, and can be integrated on any electronic device with computing capability.

[0117] As Figure 3 shown, the point cloud fusion device 30 disclosed in the present embodiment can include a point cloud image determination module 31, a road element determination module 32, and a point cloud image registration module 32, wherein:

[0118] The point cloud image determination module 31 is configured to determine a to-be-registered point cloud image from historical point cloud images according to a target acquisition position of a target point cloud image;

[0119] The road element determination module 32 is configured to determine an auxiliary road element associated with a target road element from the to-be-registered point cloud image according to a target element position of the target road element included in the target point cloud image;

[0120] The point cloud image registration module 33 is configured to register the target point cloud image and the to-be-registered point cloud image according to the target element position and an auxiliary element position of the auxiliary road element.

[0121] Optionally, the road element determining module 32 is specifically configured to:

[0122] determine a reference target road element from the target road elements, and determine reference auxiliary road elements associated with the reference target road element from the point cloud images to be registered according to a reference target element position of the reference target road element by using a random sample consensus algorithm;

[0123] determine an element position relationship between other target road elements in the target point cloud image and the reference target road element;

[0124] determine other auxiliary road elements associated with the other target road elements except the reference auxiliary road elements from the point cloud images to be registered according to the element position relationship.

[0125] Optionally, the road element determining module 32 is specifically further configured to:

[0126] determine a search position from the point cloud images to be registered according to the element position relationship and a reference auxiliary element position of the reference auxiliary road element;

[0127] perform neighborhood search at the search position according to an element type of the other target road elements and a distance threshold to determine the other auxiliary road elements associated with the other target road elements.

[0128] Optionally, the point cloud image registration module 33 is specifically configured to:

[0129] determine a target collection pose of the target point cloud image, and an auxiliary collection position and an auxiliary collection pose of the point cloud images to be registered;

[0130] register the target point cloud image and the point cloud images to be registered according to the target collection position, the target collection pose, the auxiliary collection position, the auxiliary collection pose, the target element position and the auxiliary element position.

[0131] Optionally, the point cloud image registration module 33 is specifically further configured to:

[0132] determine a target pose matrix of the target road elements in a world coordinate system according to the target collection position, the target collection pose and the target element position;

[0133] determine an auxiliary pose matrix of the auxiliary road elements in the world coordinate system according to the auxiliary collection position, the auxiliary collection pose and the auxiliary element position;

[0134] According to the target pose matrix and the auxiliary pose matrix, the target point cloud image and the point cloud image to be registered are registered.

[0135] Optionally, the point cloud image registration module 33 is specifically further configured to:

[0136] According to the target pose matrix and the auxiliary pose matrix, a constraint function is constructed.

[0137] An optimization algorithm is used to adjust the target pose matrix and the auxiliary pose matrix, and to determine a first pose matrix after adjustment of the target pose matrix and a second pose matrix after adjustment of the auxiliary pose matrix when a function value of the constraint function is a target value.

[0138] According to the first pose matrix and the second pose matrix, the target point cloud image and the point cloud image to be registered are registered.

[0139] Optionally, the point cloud image registration module 33 is specifically further configured to:

[0140] According to the first pose matrix and the target pose matrix, a target rotation matrix and a target displacement matrix are determined.

[0141] According to the second pose matrix and the auxiliary pose matrix, an auxiliary rotation matrix and an auxiliary displacement matrix are determined.

[0142] The target point cloud image is registered by using the target rotation matrix and the target displacement matrix, and the point cloud image to be registered is registered by using the auxiliary rotation matrix and the auxiliary displacement matrix.

[0143] Optionally, the point cloud image registration module 33 is specifically further configured to:

[0144] In a case where the element types of the target road element and the auxiliary road element include a road sign, the following e1 and e2 constraint functions are constructed:

[0145]

[0146]

[0147] wherein, The The target pose matrix is represented by R, the target rotation matrix is represented by R, the target displacement matrix is represented by T, the target acquisition pose is represented by P, the target acquisition position is represented by T, the target element position is represented by P, the auxiliary pose matrix is represented by R, the auxiliary rotation matrix is represented by R, the auxiliary displacement matrix is represented by T, the auxiliary acquisition pose is represented by P, and the auxiliary acquisition position is represented by T. src The target acquisition pose is represented by P, the target acquisition position is represented by T, the target element position is represented by P, the auxiliary pose matrix is represented by R, the auxiliary rotation matrix is represented by R, the auxiliary displacement matrix is represented by T, the auxiliary acquisition pose is represented by P, and the auxiliary acquisition position is represented by T. src The target acquisition pose is represented by P, the target acquisition position is represented by T, the target element position is represented by P, the auxiliary pose matrix is represented by R, the auxiliary rotation matrix is represented by R, the auxiliary displacement matrix is represented by T, the auxiliary acquisition pose is represented by P, and the auxiliary acquisition position is represented by T. src The target acquisition pose is represented by P, the target acquisition position is represented by T, the target element position is represented by P, the auxiliary pose matrix is represented by R, the auxiliary rotation matrix is represented by R, the auxiliary displacement matrix is represented by T, the auxiliary acquisition pose is represented by P, and the auxiliary acquisition position is represented by T. The target pose matrix is represented by R, the target rotation matrix is represented by R, the target displacement matrix is represented by T, the target acquisition pose is represented by P, the target acquisition position is represented by T, the target element position is represented by P, the auxiliary pose matrix is represented by R, the auxiliary rotation matrix is represented by R, the auxiliary displacement matrix is represented by T, the auxiliary acquisition pose is represented by P, and the auxiliary acquisition position is represented by T. dst The target acquisition pose is represented by P, the target acquisition position is represented by T, the target element position is represented by P, the auxiliary pose matrix is represented by R, the auxiliary rotation matrix is represented by R, the auxiliary displacement matrix is represented by T, the auxiliary acquisition pose is represented by P, and the auxiliary acquisition position is represented by T. dstrepresents an auxiliary element position, the v dst represents an auxiliary element position, the v represents a normal vector of the target road element, the n dst represents a normal vector of the auxiliary road element, w1 and w2 are fixed constants.

[0148] Optionally, the point cloud image registration module 33 is specifically further used for:

[0149] In the case that the element types of the target road element and the auxiliary road element include lane lines and / or lamp posts, the following e3 and e4 constraint functions are constructed:

[0150]

[0151]

[0152] wherein, the represents a target pose matrix, the R src represents a target collection pose, the T src represents a target collection position, the P src represents a target element position, the v represents an auxiliary pose matrix, the R dst represents an auxiliary collection pose, the T dst represents an auxiliary collection position, the P dst represents an auxiliary element position, the v src represents an orientation of the target road element, the v dst represents an orientation of the auxiliary road element, w3 and w4 are fixed constants.

[0153] Optionally, the point cloud image registration module 33 is specifically further used for:

[0154] In the case that the element types of the target road element and the auxiliary road element include road arrows, the following e5 constraint function is constructed:

[0155]

[0156] wherein, the represents a target pose matrix, the R src represents a target collection pose, the T src represents a target collection position, the P src represents a target element position, the v represents an auxiliary pose matrix, the R dst represents an auxiliary collection pose, the T dst represents an auxiliary collection position, the Pdst The auxiliary element position is indicated.

[0157] Optionally, the point cloud image determination module 31 is specifically configured to:

[0158] According to the target acquisition position and the distance threshold, searching in a multi-dimensional space tree to determine an auxiliary acquisition position, wherein the multi-dimensional space tree is constructed according to acquisition positions of each of the historical point cloud images;

[0159] The historical point cloud image corresponding to the auxiliary acquisition position is taken as the to-be-registered point cloud image.

[0160] The point cloud fusion device 30 disclosed in the embodiments of the present disclosure can execute the point cloud fusion method disclosed in the embodiments of the present disclosure, and has the corresponding function modules and beneficial effects of executing the method. The contents not described in detail in the embodiments can be referred to the description in the method embodiments of the present disclosure.

[0161] In the technical solutions of the present disclosure, the acquisition, storage and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0162] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0163] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0164] As shown in Figure 4 The device 400 includes a computing unit 401 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 402 or a computer program loaded into a random access memory (RAM) 403 from a storage unit 408. Various programs and data required for the operation of the device 400 can also be stored in the RAM 403. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0165] A number of components in device 400 are connected to I / O interface 405, including: an input unit 406, such as a keyboard, mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a magnetic disk, optical disk, etc.; and a communication unit 409, such as a network card, modem, wireless communication transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices over computer networks, such as the Internet, and / or various telecommunication networks.

[0166] Computing unit 401 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. Computing unit 401 performs various methods and processes described above, such as the point cloud fusion method. For example, in some embodiments, the point cloud fusion method can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed onto device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded onto RAM 403 and executed by computing unit 401, one or more steps of the point cloud fusion method described above can be performed. Alternatively, in other embodiments, computing unit 401 can be configured to perform the point cloud fusion method by any other appropriate means, such as by means of firmware.

[0167] Various implementations of the systems and techniques described above herein can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0168] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, or entirely on a remote machine or server.

[0169] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0170] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0171] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), blockchain network, and the Internet.

[0172] The computer system can include clients and servers. The clients and the servers are generally remote from each other and typically interact through a communication network. The relationship of client and server is one of communication and distribution, with the server receiving requests from the client and transmitting data to the client. The client and the server can be implemented by using a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of large management difficulty and weak business scalability in traditional physical hosts and VPS services.

[0173] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, in series, or in a different order, without departing from the desired results of the technical solutions of the present disclosure, and the present disclosure is not limited herein.

[0174] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A point cloud fusion method, comprising: determining a to-be-registered point cloud image from historical point cloud images according to a target collection position of a target point cloud image; determining auxiliary road elements associated with target road elements included in the target point cloud image from the to-be-registered point cloud image according to target element positions of the target road elements; registering the target point cloud image and the to-be-registered point cloud image according to the target element positions and auxiliary element positions of the auxiliary road elements; wherein the target road elements are road-related entity elements contained in the target point cloud image, and the auxiliary road elements are road-related entity elements contained in the to-be-registered point cloud image; the target element positions are used to determine a target pose matrix of the target road elements in a world coordinate system, and the auxiliary element positions are used to determine an auxiliary pose matrix of the auxiliary road elements in the world coordinate system; the target pose matrix and the auxiliary pose matrix are used to register the target point cloud image and the to-be-registered point cloud image.

2. The method of claim 1, wherein, The determining of the auxiliary road elements associated with the target road elements from the to-be-registered point cloud image according to the target element positions of the target road elements included in the target point cloud image comprises: determining a reference target road element from the target road elements, and determining a reference auxiliary road element associated with the reference target road element from the to-be-registered point cloud image using a random sample consensus algorithm according to a reference target element position of the reference target road element; determining element position relationships between other target road elements and the reference target road element in the target point cloud image; determining other auxiliary road elements associated with the other target road elements except the reference auxiliary road element from the to-be-registered point cloud image according to the element position relationships.

3. The method of claim 2, wherein, The determining of the other auxiliary road elements associated with the other target road elements except the reference auxiliary road element from the to-be-registered point cloud image according to the element position relationships comprises: determining a search position from the to-be-registered point cloud image according to the element position relationships and the reference auxiliary element position of the reference auxiliary road element; performing a neighborhood search at the search position according to an element type of the other target road elements and a distance threshold to determine the other auxiliary road elements associated with the other target road elements.

4. The method of any one of claims 1-3, wherein, The registering of the target point cloud image and the to-be-registered point cloud image according to the target element positions and the auxiliary element positions of the auxiliary road elements comprises: determining a target collection pose of the target point cloud image, and an auxiliary collection position and an auxiliary collection pose of the to-be-registered point cloud image; registering the target point cloud image and the to-be-registered point cloud image according to the target collection position, the target collection pose, the auxiliary collection position, the auxiliary collection pose, the target element positions and the auxiliary element positions.

5. The method of claim 4, wherein, The registration of the target point cloud image and the point cloud image to be registered according to the target acquisition position, the target acquisition pose, the auxiliary acquisition position, the auxiliary acquisition pose, the target element position and the auxiliary element position comprises: determining a target pose matrix of the target road element in a world coordinate system according to the target acquisition position, the target acquisition pose and the target element position; determining an auxiliary pose matrix of the auxiliary road element in the world coordinate system according to the auxiliary acquisition position, the auxiliary acquisition pose and the auxiliary element position; registering the target point cloud image and the point cloud image to be registered according to the target pose matrix and the auxiliary pose matrix.

6. The method of claim 5, wherein, The registration of the target point cloud image and the point cloud image to be registered according to the target pose matrix and the auxiliary pose matrix comprises: constructing a constraint function according to the target pose matrix and the auxiliary pose matrix; adjusting the target pose matrix and the auxiliary pose matrix by using an optimization algorithm, and determining a first pose matrix of the target pose matrix after adjustment and a second pose matrix of the auxiliary pose matrix after adjustment when a function value of the constraint function is a target value; registering the target point cloud image and the point cloud image to be registered according to the first pose matrix and the second pose matrix.

7. The method of claim 6, wherein, The registration of the target point cloud image and the point cloud image to be registered according to the first pose matrix and the second pose matrix comprises: determining a target rotation matrix and a target displacement matrix according to the first pose matrix and the target pose matrix; determining an auxiliary rotation matrix and an auxiliary displacement matrix according to the second pose matrix and the auxiliary pose matrix; registering the target point cloud image by using the target rotation matrix and the target displacement matrix, and registering the point cloud image to be registered by using the auxiliary rotation matrix and the auxiliary displacement matrix.

8. The method of any one of claims 6-7, wherein, The construction of the constraint function according to the target pose matrix and the auxiliary pose matrix comprises: in a case where the element types of the target road element and the auxiliary road element comprise a road sign, constructing the following e1 and e2 constraint functions: wherein, the denotes a target pose matrix, the R src denotes a target acquisition pose, the T src denotes a target acquisition position, the P src denotes a target element position, the denotes an auxiliary pose matrix, the R dst denotes an auxiliary acquisition pose, the T dst denotes an auxiliary acquisition position, the P dst denotes an auxiliary element position, the denotes a normal vector of the target road element, the n dst denotes a normal vector of the auxiliary road element, w1 and w2 are fixed constants.

9. The method of any one of claims 6-8, wherein, The construction of the constraint function according to the target pose matrix and the auxiliary pose matrix comprises: in a case where the element types of the target road element and the auxiliary road element comprise a lane line and / or a road lamp pole, constructing the following e3 and e4 constraint functions: wherein, the denotes a target pose matrix, the R src denotes a target acquisition pose, the T src denotes a target acquisition position, the P src denotes a target element position, the denotes an auxiliary pose matrix, the R dst denotes an auxiliary acquisition pose, the T dst denotes an auxiliary acquisition position, the P dst denotes an auxiliary element position, the v src denotes an orientation of the target road element, the v dst denotes an orientation of the auxiliary road element, w3 and w4 are fixed constants.

10. The method of any one of claims 6-9, wherein, The construction of the constraint function according to the target pose matrix and the auxiliary pose matrix comprises: in a case where the element types of the target road element and the auxiliary road element comprise a road arrow, constructing the following e5 constraint function: wherein, the denotes a target pose matrix, the R src denotes a target acquisition pose, the T src denotes a target acquisition position, the P src denotes a target element position, the denotes an auxiliary pose matrix, the R dst denotes an auxiliary acquisition pose, the T dst denotes an auxiliary acquisition position, the P dst denotes an auxiliary element position.

11. The method of any one of claims 1-10, wherein, The determination of the point cloud image to be registered from the historical point cloud images according to the target acquisition position of the target point cloud image comprises: searching in a multi-dimensional space tree according to the target acquisition position and a distance threshold to determine an auxiliary acquisition position; wherein the multi-dimensional space tree is constructed according to the acquisition positions of the historical point cloud images; The historical point cloud image corresponding to the auxiliary acquisition position is taken as the point cloud image to be registered.

12. A point cloud fusion device, comprising: a point cloud image determination module configured to determine a point cloud image to be registered from historical point cloud images according to a target acquisition position of a target point cloud image; a road element determination module configured to determine auxiliary road elements associated with target road elements included in the target point cloud image from the point cloud image to be registered according to target element positions of the target road elements; a point cloud image registration module configured to register the target point cloud image and the point cloud image to be registered according to target element positions and auxiliary element positions of the auxiliary road elements. The target road elements are entity elements related to roads included in the target point cloud image, and the auxiliary road elements are entity elements related to roads included in the point cloud image to be registered. The target element positions are used to determine target pose matrices of the target road elements in a world coordinate system, and the auxiliary element positions are used to determine auxiliary pose matrices of the auxiliary road elements in the world coordinate system. The target pose matrices and the auxiliary pose matrices are used to register the target point cloud image and the point cloud image to be registered.

13. The apparatus of claim 12, wherein, The road element determination module is specifically configured to: determine a reference target road element from the target road elements, and determine a reference auxiliary road element associated with the reference target road element from the point cloud image to be registered using a random sample consensus algorithm according to a reference target element position of the reference target road element; determine element position relationships between other target road elements and the reference target road element in the target point cloud image; determine other auxiliary road elements associated with the other target road elements except the reference auxiliary road element from the point cloud image to be registered according to the element position relationships.

14. The apparatus of claim 13, wherein, The road element determination module is specifically further configured to: determine a search position from the point cloud image to be registered according to the element position relationships and a reference auxiliary element position of the reference auxiliary road element; perform neighborhood search at the search position according to an element type of the other target road elements and a distance threshold to determine the other auxiliary road elements associated with the other target road elements.

15. The apparatus of any one of claims 12-14, wherein, The point cloud image registration module is specifically configured to: determine a target acquisition pose of the target point cloud image, and an auxiliary acquisition position and an auxiliary acquisition pose of the point cloud image to be registered; register the target point cloud image and the point cloud image to be registered according to the target acquisition position, the target acquisition pose, the auxiliary acquisition position, the auxiliary acquisition pose, the target element positions, and the auxiliary element positions.

16. The apparatus of claim 15, wherein, The point cloud image registration module is specifically further configured to: determine a target pose matrix of the target road elements in a world coordinate system according to the target acquisition position, the target acquisition pose, and the target element positions; determine an auxiliary pose matrix of the auxiliary road elements in the world coordinate system according to the auxiliary acquisition position, the auxiliary acquisition pose, and the auxiliary element positions; and According to the target pose matrix and the auxiliary pose matrix, the target point cloud image and the point cloud image to be registered are registered.

17. The apparatus of claim 16, wherein, The point cloud image registration module is specifically further configured to: According to the target pose matrix and the auxiliary pose matrix, a constraint function is constructed. An optimization algorithm is used to adjust the target pose matrix and the auxiliary pose matrix, and when the function value of the constraint function is a target value, a first pose matrix after adjustment of the target pose matrix and a second pose matrix after adjustment of the auxiliary pose matrix are determined. According to the first pose matrix and the second pose matrix, the target point cloud image and the point cloud image to be registered are registered.

18. The apparatus of claim 17, wherein, The point cloud image registration module is specifically further configured to: According to the first pose matrix and the target pose matrix, a target rotation matrix and a target displacement matrix are determined. According to the second pose matrix and the auxiliary pose matrix, an auxiliary rotation matrix and an auxiliary displacement matrix are determined. The target point cloud image is registered by using the target rotation matrix and the target displacement matrix, and the point cloud image to be registered is registered by using the auxiliary rotation matrix and the auxiliary displacement matrix.

19. The apparatus of any one of claims 17-18, wherein, The point cloud image registration module is specifically further configured to: In a case where the element types of the target road element and the auxiliary road element include a road sign, the following e1 and e2 constraint functions are constructed: wherein, the denotes a target pose matrix, the R src denotes a target acquisition pose, the T src denotes a target acquisition position, the P src denotes a target element position, the denotes an auxiliary pose matrix, the R dst denotes an auxiliary acquisition pose, the T dst denotes an auxiliary acquisition position, the P dst denotes an auxiliary element position, the denotes a normal vector of the target road element, the n dst denotes a normal vector of the auxiliary road element, w1 and w2 are fixed constants.

20. The apparatus of any one of claims 17-19, wherein, The point cloud image registration module is specifically further configured to: In a case where the element types of the target road element and the auxiliary road element include a lane line and / or a street lamp pole, the following e3 and e4 constraint functions are constructed: wherein, the denotes a target pose matrix, the R src denotes a target acquisition pose, the T src denotes a target acquisition position, the P src denotes a target element position, the denotes an auxiliary pose matrix, the R dst denotes an auxiliary acquisition pose, the T dst denotes an auxiliary acquisition position, the P dst denotes an auxiliary element position, the v src denotes an orientation of the target road element, the v dst denotes an orientation of the auxiliary road element, w3 and w4 are fixed constants.

21. The apparatus of any of claims 17-20, wherein, The point cloud image registration module is specifically further configured to: In a case where the element types of the target road element and the auxiliary road element include a road arrow, the following e5 constraint function is constructed: wherein, the denotes a target pose matrix, the R src denotes a target acquisition pose, the T src denotes a target acquisition position, the P src denotes a target element position, the denotes an auxiliary pose matrix, the R dst denotes an auxiliary acquisition pose, the T dst denotes an auxiliary acquisition position, the P dst denotes an auxiliary element position.

22. The apparatus of any one of claims 12-21, wherein, The point cloud image determination module is specifically configured to: According to the target collection position and a distance threshold, an auxiliary collection position is determined by searching in a multi-dimensional space tree, wherein the multi-dimensional space tree is constructed according to collection positions of the historical point cloud images. A historical point cloud image corresponding to the auxiliary collection position is taken as the point cloud image to be registered.

23. An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

24. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-11.

25. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-11.

Images