Road image and point cloud data matching parameter determination method, system and related equipment

Abstract

The present invention discloses a method, system and related equipment for determining matching parameters of road images and point cloud data. The method comprises: selecting a first pose parameter from optional pose parameters of a road image at a predetermined parameter interval; respectively obtaining a first matching evaluation value for characterizing the matching degree of a geographical element after the road image with the first pose parameter is matched with the corresponding point cloud data; reducing the parameter interval according to a preset rule, selecting a plurality of second pose parameters within a predetermined range centered on the first pose parameter corresponding to the maximum value in the first matching evaluation value with each reduced parameter interval, determining the maximum value of the second matching evaluation value after the road image with the second pose parameter is matched with the geographical element in the point cloud data and determining the matching parameters. The present invention efficiently determines the matching parameters of the road image and the point cloud data by means of coarse positioning at a predetermined parameter interval and fine positioning at a successively reduced parameter interval.

Description

Technical Field

[0001] The present invention relates to the field of map making, and in particular to a method, system and related equipment for determining matching parameters of road images and point cloud data. Background Art

[0002] With the development of science and technology, advanced driver assistance and autonomous driving have become hot topics in technology research because they can help drivers drive safely. At present, one of the technical implementation paths of advanced driver assistance and autonomous driving requires the reliance on "high-precision maps". Compared with ordinary maps, high-precision maps can express the real world in a more refined and accurate way. For example, ordinary maps do not express more detailed information about roads, while high-precision maps can express each lane of the road through lane lines.

[0003] At present, the production of high-precision maps is mainly made by laser point clouds (i.e. point cloud data) collected by laser equipment and pictures (road images) taken by camera equipment. Since laser point clouds and pictures have their own advantages and disadvantages, for example, the position accuracy of laser scanning points is high and is less affected by lighting conditions, but the cost is high, and the scanning reflectivity has low discrimination of road targets; pictures use instant large-area visible light imaging, which has high discrimination of road targets and low cost, but the position accuracy of pictures is low due to the influence of road image perspective, pixel discretization and camera distortion at a distance, and is greatly affected by lighting conditions. Therefore, in the process of high-precision map production and autonomous driving positioning, the solution of laser point cloud and picture fusion is used for positioning and mapping. However, in the actual mapping process, due to the inconsistency of the observation angle of laser point cloud and picture on road targets, the spectral range of detection targets, data interpretation method, etc., and the complex and changeable road environment, it is impossible to directly fuse laser point cloud with picture. Because the image has its own coordinate position in the world coordinates, that is, the six degrees of freedom of the image, it is necessary to determine the matching parameters of the road image translation and / or rotation when the image is fused with the point cloud data in order to accurately fuse the image and the point cloud data. Therefore, how to quickly and accurately determine the matching parameters of the image and the point cloud data is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0004] In view of the technical defects and technical drawbacks existing in the prior art, the embodiments of the present invention provide a method, system and related equipment for determining matching parameters of road images and point cloud data to overcome the above problems or at least partially solve the above problems.

[0005] As one aspect of an embodiment of the present disclosure, a method for determining matching parameters between a road image and point cloud data may include:

[0006] Selecting a first pose parameter from optional pose parameters of the road image at a predetermined parameter interval;

[0007] Obtaining first matching evaluation values ​​for characterizing the matching degree of geographical elements after the road image having the first posture parameter is matched with the corresponding point cloud data;

[0008] Reduce the parameter interval according to a preset rule, select a plurality of second posture parameters within a predetermined range centered on the first posture parameter corresponding to the maximum value in the first matching evaluation value with each reduced parameter interval, and determine the maximum value of the second matching evaluation value after the road image with the second posture parameter matches the geographic element in the point cloud data;

[0009] The second posture parameter corresponding to the maximum value of the second matching evaluation value is determined as the matching parameter.

[0010] As a second aspect of the embodiment of the present disclosure, a method for matching a road image with point cloud data may include:

[0011] Acquiring matching parameters of the road image according to the matching parameter determination method of the first aspect;

[0012] The road image is translated and / or rotated according to the matching parameters so that the road image matches the point cloud data.

[0013] As a third aspect of an embodiment of the present disclosure, a device for determining matching parameters between a road image and point cloud data may include:

[0014] A first pose parameter selection module is used to select a first pose parameter from optional pose parameters of the road image at a predetermined parameter interval;

[0015] A first matching evaluation value acquisition module, used to respectively acquire a first matching evaluation value for characterizing a matching degree of a geographic element after a road image having the first posture parameter is matched with corresponding point cloud data;

[0016] A second matching evaluation value acquisition module is configured to reduce the parameter interval according to a preset rule, select a plurality of second posture parameters within a predetermined range centered on the first posture parameter corresponding to the maximum value in the first matching evaluation value with each reduced parameter interval, and determine the maximum value of the second matching evaluation value after the road image having the second posture parameter matches the geographic element in the point cloud data;

[0017] A matching parameter determination module is used to determine the second posture parameter corresponding to the maximum value of the second matching evaluation value as the matching parameter.

[0018] As a fourth aspect of the embodiments of the present disclosure, a device for matching road images and point cloud data may include:

[0019] a matching parameter acquisition module, configured to acquire the matching parameters of the road image according to the matching parameter determination device according to the third aspect;

[0020] A matching module is used to translate and / or rotate the road image according to the matching parameters so that the road image matches the point cloud data.

[0021] As a fifth aspect of the embodiments of the present disclosure, it relates to a road image and point cloud data matching system, including: a server, an image acquisition device and a point cloud acquisition device;

[0022] The image acquisition device is used to acquire road images of the road;

[0023] The point cloud acquisition device is used to collect point cloud data of the road;

[0024] The server includes the matching device as described in the fourth aspect.

[0025] As the sixth aspect of an embodiment of the present disclosure, it relates to a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the method for determining road image and point cloud data matching parameters as described in the first aspect and the method for matching road image and point cloud data as described in the second aspect.

[0026] As the seventh aspect of the embodiment of the present disclosure, it relates to a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for determining parameters of matching road images and point cloud data as described in the first aspect and the method for matching road images and point cloud data as described in the second aspect are implemented.

[0027] The embodiments of the present disclosure achieve at least the following technical effects:

[0028] The disclosed embodiment selects the first pose parameter from the optional pose parameters of the road image at a predetermined parameter interval; obtains first matching evaluation values ​​for characterizing the matching degree of geographical elements after the road image with the first pose parameter is matched with the corresponding point cloud data; reduces the parameter interval according to a preset rule, selects multiple second pose parameters within a predetermined range centered on the first pose parameter corresponding to the maximum value in the first matching evaluation value for each reduced parameter interval, determines the maximum value of the second matching evaluation value after the road image with the second pose parameter is matched with the geographical elements in the point cloud data; determines the second pose parameter corresponding to the maximum value of the second matching evaluation value as the matching parameter. The disclosed embodiment efficiently determines the matching parameters of the road image and the point cloud data by means of the above-mentioned coarse positioning of the predetermined parameter interval and the fine positioning of the successively reduced parameter intervals.

[0029] Preferably, the lane line edge features of specific geographical elements are used as matching objects, which effectively solves the problem of inconsistent reflectivity between point cloud data and road images; by moving the position parameters of the spatial coordinate system, the lane line edges in the road image are matched with the lane line contours in the point cloud data, avoiding the problem of positioning errors caused by single-point matching errors.

[0030] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or be understood by implementing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures described in the written description, claims, and drawings.

[0031] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0033] Figure 1A A schematic diagram of a road image captured by an image acquisition device provided in an embodiment of the present disclosure;

[0034] Figure 1B A schematic diagram of point cloud data generated from point cloud data of a road collected by a point cloud collection device provided in an embodiment of the present disclosure;

[0035] Figure 2 A flowchart of a method for determining matching parameters of a road image and point cloud data provided by an embodiment of the present disclosure;

[0036] Figure 3 A flowchart for determining a correspondence between a road image and corresponding point cloud data provided by an embodiment of the present disclosure;

[0037] Figure 4 A schematic diagram of selecting a plurality of first pose parameters at a predetermined parameter interval provided in an embodiment of the present disclosure;

[0038] Figure 5 is a flowchart of the specific implementation process of step S12;

[0039] Fig. 6A The embodiments of the present disclosure provide Figure 1A Straight line induction diagram extracted from ;

[0040] Figure 6B The embodiments of the present disclosure provide Figure 1B The lane line contour map extracted from ;

[0041] Figure 7 A schematic diagram of selecting a plurality of second posture parameters at parameter intervals that are successively reduced exponentially provided in an embodiment of the present disclosure;

[0042] Figure 8 This is a flow chart of implementation method 1 of step S13;

[0043] Fig. 9 This is a flow chart of implementation method 2 of step S13;

[0044] Figures 10A-10D A schematic diagram of matching a road image and corresponding point cloud data provided by an embodiment of the present disclosure;

[0045] Fig.11 A flowchart of a specific method for determining matching parameters of road images and point cloud data provided by an embodiment of the present disclosure;

[0046] Fig.12 A schematic diagram of the structure of a device for determining parameters for matching road images and point cloud data provided by an embodiment of the present disclosure;

[0047] Fig.13 A schematic diagram of the effect of matching a road image with point cloud data provided by an embodiment of the present disclosure;

[0048] Fig.14 A schematic diagram of the structure of a road image and point cloud data matching system provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0050] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0051] Example 1

[0052] The present disclosure provides a method for determining matching parameters between a road image and point cloud data, wherein: Figure 1A As shown, the road image refers to the road image of the road collected by the image acquisition device; Figure 1B As shown, the point cloud data refers to the point cloud data of the road collected by the point cloud collection device; the matching parameters refer to the posture parameters of the road image in the point cloud data when matching the road image with the point cloud data.

[0053] like Figure 2 As shown, the method may include the following steps:

[0054] Step S11, selecting a first pose parameter from the optional pose parameters of the road image at a predetermined parameter interval.

[0055] In spatial geographic coordinates, the corresponding roads in the road image have spatial coordinates when performing spatial positioning, so the road image can be assigned a spatial position using six-degree-of-freedom spatial coordinate parameters. Among them, the optional posture parameters refer to all coordinate parameters in the six-degree-of-freedom space, which can be expressed as (x, y, z, α, β, γ). The above six degrees of freedom refer to the six degrees of freedom that an object has in space, namely the freedom of movement along the three rectangular coordinate axes of x, y, and z and the freedom of rotation around these three axes. The preset parameter interval refers to the same interval of translation in the directions of the three rectangular coordinate axes of x, y, and z or the same angle of rotation in the three axes of x, y, and z.

[0056] It should be noted that the parameter interval in this step is a parameter distance with a large range and a large step length, which can quickly assign the posture parameters of the road image with a large step length parameter distance in the spatial coordinates.

[0057] Step S12: Obtaining first matching evaluation values ​​for characterizing the matching degree of geographic elements after the road image with the first pose parameter is matched with the corresponding point cloud data.

[0058] The road image is positioned with the posture parameters assigned with a large step length parameter distance in the above step S11, and then the points representing the geographical elements in the road image and the point cloud data are matched. The most suitable first posture parameter for the road image is determined according to the size of the point matching degree, so that rough positioning can be performed quickly when matching the road image with the point cloud data.

[0059] Step S13, reducing the parameter interval according to the preset rules, selecting multiple second posture parameters within a predetermined range centered on the first posture parameter corresponding to the maximum value in the first matching evaluation value for each reduced parameter interval, and determining the maximum value of the second matching evaluation value after the road image with the second posture parameter is matched with the geographic elements in the point cloud data.

[0060] After completing the rough positioning, the step interval is shortened. It can be reduced gradually or all at once to a preset step length, so that multiple posture parameters can be determined to achieve the re-matching of the road image and the point cloud data, thereby obtaining a more accurate matching process.

[0061] Step S14: determine the second posture parameter corresponding to the maximum value of the second matching evaluation value as the matching parameter.

[0062] Through the matching result obtained in the above step S13, the posture parameters with the highest matching degree between the road image and the geographical elements representing the point cloud data are determined to represent the spatial degree of freedom coordinates of the road image.

[0063] The disclosed embodiment completes the coarse positioning of the road image by assigning the spatial coordinates of the road image with preset parameter intervals, and completes the fine positioning of the road image on the basis of the coarse positioning with each reduced parameter interval. It can quickly and efficiently complete the calculation of the spatial coordinate parameters of the road image, effectively save server resources, and improve positioning accuracy.

[0064] The above steps of the embodiment of the present disclosure are described in detail as follows:

[0065] In the above step S11, a first pose parameter is selected from the optional pose parameters of the road image at a predetermined parameter interval.

[0066] The embodiment of the present disclosure needs to determine the corresponding relationship between the road image and the point cloud data before determining the matching parameters between the road image and the point cloud data. For example, the road image and the point cloud data are both map data collected on the East Third Ring Road in Beijing. Figure 1A and Figure 1B As shown, it is necessary to determine the corresponding relationship between the above road image and the corresponding point cloud data, such as Figure 3 As shown, the following steps may be included:

[0067] Step S111, determining the initial position and posture of the road image according to the shooting time of the road image, the inertial navigation observation value of the camera shooting the road image, and the inertial navigation initial calibration parameter interpolation of the camera.

[0068] Among them, the inertial navigation observation value is obtained by interpolating the inertial navigation observation value in time, which can be calculated by methods such as high-speed random process, or linear interpolation of the translation component (representing the 3DOF in the directions of the three rectangular coordinate axes of x, y, and z) in the six degrees of freedom (6DOF), and spherical interpolation of the angle component (3DOF); the initial calibration parameters of the inertial navigation of the camera are obtained by Kalman filtering or manual measurement; the interpolation refers to interpolating a continuous function based on discrete data so that this continuous curve passes through all given discrete data points. Interpolation is an important method for approximating discrete functions. It can be used to estimate the approximate value of the function at other points through the value of the function at a finite number of points. The purpose of interpolation is to fill the gaps between pixels during image transformation.

[0069] The initial position and posture of the road image described in this step is obtained by interpolating the time of the road image after time synchronization, the inertial navigation observation value corresponding to the time of the road image shooting, and the initial calibration reference of the camera inertial navigation. When obtaining the position and posture of the road image, it is first necessary to synchronize the time of the road image shooting with the inertial navigation trajectory of the camera. The inertial navigation trajectory of the camera can be obtained through the trajectory data of the vehicle itself, or through other methods, which is not specifically limited in the embodiments of the present disclosure.

[0070] For example, the initial calibration value of the camera to inertial navigation pair is The observed value of the inertial navigation is Then the coordinates from the camera to the world coordinates are It can be calculated by the following formula:

[0071]

[0072] Step S112, performing spatial indexing in the point cloud data according to the initial posture, and determining the point cloud data corresponding to the road image.

[0073] The spatial index (also known as spatial data query) refers to arranging the summary information of spatial objects in a certain order according to the position and shape of the spatial objects or some spatial relationship between the spatial objects, such as the object's identifier, the circumscribed rectangle, and the pointer to the spatial object entity. The spatial index method in the disclosed embodiment is to perform fast query through RTree and KDTree. Of course, it can also be queried through BSPTree, R+Tree or CELLTree. The grid-type spatial search with a relatively simple structure is widely used in various GIS software and systems (such as ArcGIS). The disclosed embodiment does not specifically limit the above spatial index method.

[0074] In this step, since the point cloud data is a collection of a large amount of point data, and the range of the road image is limited, when determining the matching parameters between the road image and the point cloud data, it is necessary to determine the small point cloud map corresponding to the road image by means of spatial indexing. First, it is convenient for calculating the first matching evaluation value after the road image under various posture parameters is matched with the point cloud data (i.e., the small point cloud map) corresponding to the road image; second, it is to avoid using a large-scale map for matching. The farther the distance of the road image, the more serious the distortion of the road image, and the more difficult it is to successfully match the point cloud data. In the disclosed embodiment, the accuracy of the road image at a distance is reduced due to the influence of the distant perspective, pixel discretization, and camera distortion of the road image itself, so the distance of the spatial index of the point cloud data is controlled within the range of 25m. Of course, if the degree of pixel discretization or camera distortion of the road image changes, the distance of the above-mentioned spatial index can also be adjusted accordingly, and the disclosed embodiment does not make specific limitations on this.

[0075] For example, before executing step S11, the approximate positional relationship between the road image and the corresponding point cloud data has been obtained through steps S111 and S112, but the road image cannot be accurately matched to the point cloud data. Therefore, it is necessary to select multiple first pose parameters at a predetermined parameter interval, that is, in the spatial coordinates, refer to Figure 4 As shown, it is necessary to select multiple first pose parameters in a traversal manner and with a large step length at a preset parameter interval, such as a translation parameter interval of L and a rotation angle interval of ψ (not shown in the figure), such as Figure 4 As shown in , in one of the plane coordinate systems of the six-degree-of-freedom spatial coordinate system, a step size of L is selected for the translation parameter interval, and each node in the figure (such as the black point in the figure) represents the first pose parameter on the two axes. The method of selecting multiple first pose parameters at preset parameter intervals used in the embodiment of the present disclosure can be a grid search method. For example, the translation parameter interval L is 10 cm, and the rotation angle interval ψ is 0.1°. Therefore, the parameters of the road image at different positions and angles are given in the spatial coordinates. The above-mentioned grid search refers to giving initial parameters, a certain search step size and a search range, dividing the parameter space into a uniform grid, and conducting a rapid and large-scale search to find the optimal parameters and complete the coarse positioning of the road image.

[0076] In the above step S12, first matching evaluation values ​​for characterizing the matching degree of geographical elements are obtained after the road image having the first posture parameter is matched with the corresponding point cloud data.

[0077] The matching refers to projecting the road image under the first position parameter into the point cloud data; the first matching evaluation value refers to the degree of matching between the road image and the point cloud data.

[0078] Specifically, you can refer to Figure 5 As shown, the following steps may be specifically included:

[0079] Step S121, obtaining a first geographical element in the road image, and obtaining a second geographical element of the same type as the first geographical element in the point cloud data.

[0080] The geographic elements refer to elements that can represent the geographic features in the map in road images and point cloud data, such as lane lines, arrows, signs, zebra crossings, etc. at corresponding positions in the map.

[0081] Step S122: setting the posture of the road image using the first posture parameter.

[0082] The purpose of this step is to set the pose of the road image in the spatial coordinate system according to the multiple first pose parameters obtained in the above step S11. For example, the first pose parameters such as (x+L, y, z, α, β, γ), (xL, y, z, α, β, γ), (x+2L, y, z, α+ψ, β, γ) are assigned to the road image, so that the road image can match the point cloud data corresponding to the road image at different positions and angles in the spatial coordinate system.

[0083] Step S123: projecting the road image onto the point cloud data according to the above-mentioned posture.

[0084] Because when making a high-precision map, it is only necessary to translate and / or rotate the road image in the spatial coordinate system to match the road image with the corresponding point cloud data, so the purpose of this step is to project the road image set with the first pose parameter in step 220 into the point cloud data. Of course, if the point cloud data is projected into the road image, the purpose of matching the road image with the corresponding point cloud data can also be achieved, but when translating and / or rotating the road image, the first pose parameter needs to be inverted. Therefore, as long as the embodiment of the present disclosure can achieve the purpose of matching the road image with the corresponding point cloud data, the specific projection method is not specifically limited. The projection formula in the embodiment of the present disclosure is as follows: u, v are the coordinates of the point projected on the road image, Pi is the coordinates of the point on the geographic element of the point cloud data (such as the lane line outline below), and K is the camera intrinsic parameter matrix.

[0085]

[0086] Step S124: Obtain the sum of the products of the light intensities of the first geographical element and the second geographical element located in the same projection area as the first matching evaluation value.

[0087] The light intensity refers to the pixel value of each pixel in the road image and the laser reflection intensity of each point in the point cloud data. Since the road image is composed of different pixel points, for example, the pixel value is between 0 and 255, the larger the pixel value, the higher the brightness, and each point in the point cloud data generated by the laser point cloud also has a laser reflection intensity, the product of the pixel value of the point located in the same projection area and the laser reflection intensity can be used as the product of the light intensity, and the sum of all the light intensity products located in the same area projection is used as the matching evaluation value of the road image and the corresponding point cloud data. Because the road image and the corresponding point cloud data are matched under the first pose parameter in this step, it should be noted here that the obtained matching evaluation value is the first matching evaluation value.

[0088] Optionally, when the first geographic element and the second geographic element are lane lines, after acquiring the first geographic element in the road image, the method may also include: performing straight line detection on the road image, and gradient processing the light intensity of the detected lane lines so that the light intensity of the lane lines decreases from the center part to the edge part of the lane lines.

[0089] After obtaining a second geographical element of the same type as the first geographical element in the point cloud data, the method further includes: using a preset algorithm to extract the lane line profile from the point cloud data.

[0090] Because the road images collected by the image acquisition device and the point cloud data collected by the point cloud acquisition device are generally on the road, there are more linear elements distributed; at the same time, linear elements are more guaranteed in positioning accuracy. Therefore, the embodiments of the present disclosure can specifically use the lane line elements in the road image and the point cloud data as the objects of matching and positioning, which can representatively represent the matching of the road image and the point cloud data. It should be noted in the embodiments of the present disclosure that the lane line is a rectangular lane line in the actual geographical element, and cannot be understood as a straight line. The lane line in the embodiments of the present disclosure has a certain width in the direction perpendicular to the direction of the road.

[0091] Specifically, when executing this step, firstly, the lane lines in the road image are extracted. Since most of the lane lines are straight lines, the embodiments disclosed herein detect and extract the straight lines in the road image. The method for detecting the straight lines in the road image can be selected from the Canny edge detection algorithm and the Hough transform (Hough Transform), the straight line filter and other methods. You can also choose to extract the straight lines using the line segment detection algorithm (Line Segment Detector, referred to as lsd) and other straight line extraction algorithms. Before matching, you can use the road image lane line recognition results as a mask to remove unnecessary straight lines.

[0092] After extracting the straight lines from the road image, in order to match the lane lines in the point cloud data with the straight lines extracted from the road image, that is, to make the light intensity product more concentrated during the matching, the extracted straight lines can be weighted to generate a straight line induction map. For example, the inverse distance weighted calculation can be performed. The straight line feature induction map can be calculated by morphological dilation methods of different scales, inverse distance calculation, etc. In the induction map, the straight line pixel value is the largest, and the farther the distance, the smaller the value. Fig. 6A As shown, the extracted Figure 1AIn the straight line induction map, the straight line representing the frame supporting the image acquisition device on the acquisition vehicle is flooded. In order to ensure the validity of the feature, the contrast between the pixel value in the middle of the lane line and the pixel value at the edge of the lane line is large enough. This paper uses the following formula to calculate the inverse distance induction map. For example, the pixel value assigned to the pixel point at the center line position of the straight line is 255, which means that the center line position of the straight line is the brightest, and it decreases from the center part to the edge part, so that the pixel value of the pixel point at the edge has enough contrast with the center line. For example, E i,j is the edge value of the i-th row and j-th column in the image (the definition of x and y is the same), W i,j are the values ​​on the calculated induction diagram. α and γ are empirically taken as 1 / 3 and 0.98 respectively. Of course, other values ​​of α and γ can also achieve the result of generating a similar straight line induction diagram. The embodiments of the present disclosure do not specifically limit the above values.

[0093] W i,j =αE i，j +(1-α)E x,y γ max(|x-i|,|y-j|)

[0094] According to the above formula, the pixel values ​​of the pixels on the center line of the straight line (lane line) and the pixel values ​​of the pixels on the edge can be gradually reduced, so that the pixel values ​​of the pixels at the edge have sufficient contrast with the center line.

[0095] Optionally, when the first geographic element is a lane line, it is also necessary to extract the lane line contour in the point cloud data corresponding to the road image. Specifically, the embodiment of the present disclosure uses the lane line contour on the point cloud data as the positioning element (i.e., geographic element). You can first extract the ground in the point cloud data through methods such as region growing, and then use the reflectivity information to extract the lane lines on the ground in the point cloud data. When the reflectivity noise is less, you can use adaptive threshold segmentation such as Otsu (Otsu method or maximum inter-class variance method). When the noise is more, you can interpolate the ground into a picture and use the pyramid scene parsing network (PSPNet), fully convolutional neural network (UNet) and other neural network recognition algorithms to extract the lane lines to obtain the initial lane line segmentation result. Finally, use the edge tracking algorithm to extract the lane line contour in the point cloud data. Figure 6B As shown, it is an embodiment of the present disclosure Figure 1B Schematic diagram of the extracted lane line contour. There are more points collected on the center line of the lane line, and the points collected towards the edge of the lane line gradually decrease. In addition, the laser reflection intensity value of the points collected on the center line of the lane line is also larger, and the laser reflection intensity value of the points collected towards the edge of the lane line is also gradually reduced.

[0096] In the above step S13, the parameter interval is reduced according to the preset rules, and multiple second posture parameters are selected from a predetermined range centered on the first posture parameter corresponding to the maximum value in the first matching evaluation value for each reduced parameter interval to determine the maximum value of the second matching evaluation value after the road image with the second posture parameter is matched with the geographic elements in the point cloud data.

[0097] The preset range centered on the first posture parameter refers to a range centered on the first posture parameter and with the parameter interval in step S11 as a radius. For example, multiple second posture parameters of the parameter intervals that are successively reduced by predetermined conditions are selected within the range of the translation parameter interval L and the rotation angle interval ψ in step S11.

[0098] Specifically, this step may be implemented in two ways: one is to perform the step with a parameter interval lower limit smaller than the parameter interval in step S11, and the other is to perform the step with a gradually reduced parameter interval, as follows:

[0099] <Method 1> This method refers to Figure 7 As shown, the following steps may be included:

[0100] Step S1311: Reduce the parameter interval to a lower limit of the parameter interval according to a preset rule to obtain a plurality of reduced parameter intervals.

[0101] The lower limit of the parameter interval is a predetermined reduction value smaller than the parameter interval in step S11.

[0102] Step S1312: Select multiple second pose parameters from a predetermined range centered on the first pose parameter corresponding to the maximum value in the first matching evaluation value at each reduced parameter interval.

[0103] The purpose of this step is to finely locate the optimal first pose parameter in the rough matching result in step S12. For example, multiple second pose parameters can be selected with successively reduced parameter intervals smaller than a predetermined reduction value of the parameter interval, such as 0.1L and 0.1ψ.

[0104] Step S1313: respectively obtain second matching evaluation values ​​for characterizing the matching degree of geographical elements after the road image having the second posture parameter is matched with the corresponding point cloud data, and obtain the maximum value of the second matching evaluation values.

[0105] This method obtains the parameter interval through the lower limit of the parameter interval, and then assigns different second pose parameters to the road image, so that the step length is shortened when the road image is matched with the point cloud data, and the matching is more accurate.

[0106] <Method 2> This method refers to Figure 8 As shown, the following steps may be included:

[0107] Step S1321, reducing the parameter interval according to a preset rule to obtain a reduced parameter interval, and selecting a second posture parameter from a predetermined range centered on the first posture parameter corresponding to the maximum value in the first matching evaluation value with the reduced parameter interval.

[0108] For example, multiple second posture parameters are selected at the parameter intervals that are successively reduced exponentially. -n L and 2 -n Step S1322 selects multiple second posture parameters at successively reduced parameter intervals of ψ (where n is a natural number), and obtains a second matching evaluation value after the road image with the second posture parameter is matched with the geographic elements in the point cloud data.

[0109] Step S1323: If the second match evaluation value obtained is greater than the first match evaluation value or greater than the second match evaluation value obtained last time, return to continue executing the step of reducing the parameter interval according to the preset rule to obtain a reduced parameter interval, until the second match evaluation value obtained is no greater than the second match evaluation value obtained last time, and the maximum value of the second match evaluation value is obtained.

[0110] For example, in a specific embodiment, multiple second posture parameters are selected at parameter intervals that are successively reduced by exponentials. In the embodiment of the present disclosure, a binary tree search mode is used to perform an accurate search for the second posture parameters. Fig. 9 As shown:

[0111] The parameter interval L in the above step is used as the first interval (2 -0 *L); the first increment pose parameter and the first decrement pose parameter are obtained by adding and subtracting the first interval L from the first pose parameter, that is, Fig. 9 The second layer of black dots in the middle represents the first decremental posture parameter on the left and the first incremental posture parameter on the right; respectively obtain the first incremental matching evaluation value and the first decremental matching evaluation value after the road image of the first incremental posture parameter and the first decremental posture parameter are matched with the point cloud data; multiply the first interval to form a second interval (2 -1 *L).

[0112] The second incremental pose parameter and the second decrement pose parameter are obtained by adding and subtracting the second interval from the first incremental pose parameter or the first decrement pose parameter corresponding to the larger value of the determined first incremental matching evaluation value and the first decrement matching evaluation value, respectively. Fig. 9 If the first decrement matching evaluation value on the left side determined in the above is larger, then the first decrement posture parameter is added and subtracted from the second interval 2 respectively. -1*L obtains the second incremental pose parameter and the second decrement pose parameter. Then obtain the second incremental matching evaluation value and the second decrement matching evaluation value after the road image of the second incremental pose parameter and the second decrement pose parameter are matched with the point cloud data respectively; repeat the above steps to obtain the Nth incremental pose parameter and the Nth decrement pose parameter corresponding to the Nth interval as the second pose parameter, and the Nth incremental matching evaluation value and the Nth decrement matching evaluation value as the second matching evaluation value, wherein N is a natural number.

[0113] For example, when the Nth interval in step S13 satisfies a predetermined condition, the Nth incremental pose parameter or the Nth decrement pose parameter corresponding to the larger value of the Nth incremental matching evaluation value and the Nth decrement evaluation value is used as the matching parameter. Fig. 9 The parameter interval l after sequential reduction is less than or equal to 1cm, then N is 3 or 4 to meet the above conditions. When N is 3, that is, l = 2 -3 *L=0.125*L=1.25cm, when N is 4, that is, l=2 -4 *L=0.0625*L=0.625cm. The above can meet the accuracy requirements of high-precision maps, so the third incremental pose parameter or the third decremental pose parameter corresponding to the larger value of the third incremental matching evaluation value and the third decremental matching evaluation value when N is 3 can be selected as the second pose parameter.

[0114] For example, refer to Figures 10A-10D As shown in the figure, when the geographical element is a lane line, after the light intensity of the lane line in the road image is gradient processed, a schematic diagram of matching the road image with the corresponding point cloud data is shown. From the figure, it can be seen that the lane line in the road image has the largest (brightest) pixel value on the center line of the lane line, and the pixel value gradually decreases from the center line to the edge. The pixel value contrast between the pixel value in the middle of the lane line and the pixel value at the edge of the lane line is large enough; and the number of points collected on the center line of the lane line in the point cloud data is more than the points collected on the edge of the lane line, and the laser reflection intensity value of the points collected on the center line of the lane line is also greater than the laser reflection intensity value of the points collected on the edge of the lane line. Fig. 10A → Fig. 10B → Fig. 10C It can be seen that the matching degree is getting higher and higher, that is, the second matching evaluation value is getting larger and larger, so from Fig. 10A , Fig. 10B and Fig. 10C It can be seen from the three matching diagrams that Fig. 10C The best match. Fig. 10D It can be seen that when the rotational freedom of the three axes is different, there is also a comparison of the second matching evaluation value. The comparison method is the same as the above comparison method, which will not be repeated here.

[0115] In the above step S14, the second posture parameter corresponding to the maximum value of the second matching evaluation value is determined as the matching parameter.

[0116] In a specific embodiment, Fig.11 As shown, the method for determining the matching parameters of the road image (the photo in the figure) and the point cloud data can be: first, the road image and the inertial navigation trajectory are synchronized in time, and the initial posture of the road image is determined according to the shooting time of the road image, the inertial navigation observation value of the camera that shoots the road image, and the inertial navigation initial calibration parameter interpolation of the camera; then, spatial indexing is performed in the point cloud data according to the initial posture to find the point cloud data corresponding to the road image; straight line detection is performed on the geographical elements representing the lane lines in the road image respectively, and the detected straight lines are extracted and inverse distance weighted to generate a straight line induction map; the lane line contour of the geographical elements representing the lane line is extracted from the point cloud data corresponding to the road image; finally, the straight line induction map and the lane line contour are determined through a grid and binary tree mixed parameter search to determine the six-degree-of-freedom parameters of the road image in the spatial coordinate system.

[0117] The disclosed embodiment selects a plurality of first pose parameters at a predetermined parameter interval from all optional pose parameters of the road image, and then obtains the first matching evaluation value after the road image with each first pose parameter is matched with the point cloud data corresponding to the road image, and selects a plurality of second pose parameters from the optional pose parameters within the preset range of the first pose parameter corresponding to the maximum value among the obtained plurality of first matching evaluation values ​​at a parameter interval reduced in sequence according to a predetermined condition, and obtains the second matching evaluation value after the road image with each second pose parameter is matched with the point cloud data, and after fine positioning in a variable step length manner, the second pose parameter corresponding to the maximum value among the obtained plurality of second matching evaluation values ​​is determined as the matching parameter. The disclosed embodiment efficiently determines the matching parameters of the road image and the point cloud data by the above-mentioned predetermined parameter interval coarse positioning and sequentially reduced parameter interval fine positioning. Moreover, the lane line edge feature of a specific geographical element is used as a matching object, which effectively solves the problem of inconsistent reflectivity between the point cloud data and the road image; by moving the pose parameters of the spatial coordinate system, the lane line edge in the road image is matched with the lane line contour in the point cloud data, and the problem of positioning error caused by single point matching error is avoided.

[0118] Based on the same inventive concept, the embodiment of the present disclosure provides a device for determining matching parameters between a road image and point cloud data, such as Fig.12 As shown, it may include: a first pose parameter selection module 11, a first matching evaluation value acquisition module 12, a second matching evaluation value acquisition module 13 and a matching parameter determination module 14, and the corresponding Fig.11The working principle of the medium grid binary tree hybrid parameter search and road image 6DOF positioning parameter output module is as follows:

[0119] The first pose parameter selection module 11 selects the first pose parameter from the optional pose parameters of the road image at a predetermined parameter interval.

[0120] The first matching evaluation value acquisition module 12 respectively acquires the first matching evaluation value used to characterize the matching degree of geographical elements after the road image with the first posture parameter is matched with the corresponding point cloud data. Specifically, the first matching evaluation value acquisition module 12 acquires the first geographical element in the road image, acquires the second geographical element of the same type as the first geographical element in the point cloud data; sets the posture of the road image with the first posture parameter; projects the road image onto the point cloud data according to the posture; and the first matching evaluation value acquisition module 12 acquires the sum of the products of the light intensities of the first geographical element and the second geographical element located in the same projection area as the first matching evaluation value.

[0121] The second matching evaluation value acquisition module 13 reduces the parameter interval according to a preset rule, selects a plurality of second posture parameters from a predetermined range centered on the first posture parameter corresponding to the maximum value in the first matching evaluation value with each reduced parameter interval, and determines the maximum value of the second matching evaluation value after the road image with the second posture parameter matches the geographic element in the point cloud data. Specifically, the second matching evaluation value acquisition module 13 reduces the parameter interval to the lower limit of the parameter interval according to a preset rule to obtain a plurality of reduced parameter intervals; selects a plurality of second posture parameters from a predetermined range centered on the first posture parameter corresponding to the maximum value in the first matching evaluation value with each reduced parameter interval; obtains the second matching evaluation value for characterizing the degree of matching of the geographic element after the road image with the second posture parameter matches the corresponding point cloud data, and obtains the maximum value of the second matching evaluation value. Alternatively, the second matching evaluation value acquisition module 13 reduces the parameter interval according to a preset rule to obtain a reduced parameter interval, and selects the second posture parameter from a predetermined range centered on the first posture parameter corresponding to the maximum value in the first matching evaluation value with the reduced parameter interval; the second matching evaluation value acquisition module 13 obtains the second matching evaluation value after the road image with the second posture parameter is matched with the geographic element in the point cloud data; if the second matching evaluation value obtained by the second matching evaluation value acquisition module 13 is greater than the first matching evaluation value or greater than the second matching evaluation value obtained last time, then the module returns to continue executing the step of reducing the parameter interval according to the preset rule to obtain a reduced parameter interval, until the second matching evaluation value obtained is not greater than the second matching evaluation value obtained last time, and the maximum value of the second matching evaluation value is obtained.

[0122] The matching parameter determination module 14 determines the second posture parameter corresponding to the maximum value of the second matching evaluation value as the matching parameter.

[0123] The technical effects and other specific descriptions of this embodiment can be found in the relevant contents of the method for determining matching parameters of road images and point cloud data mentioned above, which will not be repeated here.

[0124] Example 2

[0125] The present disclosure provides a method for matching road images with point cloud data, which may include the following steps:

[0126] The matching parameters of the road image are obtained according to the method for determining matching parameters of the road image and the point cloud data in Example 1; and the road image is translated and / or rotated according to the matching parameters so that the road image matches the point cloud data.

[0127] Reference Fig.13 As shown, using the above matching parameters, the road image can be quickly and effectively translated and / or rotated to a spatial position that matches the point cloud data. The figure shows the effect after the road image and the corresponding point cloud data are successfully matched, where the multiple collection points circled by ellipses in the figure are point cloud data.

[0128] The technical effects and other specific descriptions of this embodiment can be found in the relevant contents of the method for determining matching parameters of road images and point cloud data mentioned above, which will not be repeated here.

[0129] The disclosed embodiment also provides a device for matching a road image with point cloud data, which may include: a matching parameter acquisition module and a matching module, and its working principle is as follows: the matching parameter acquisition module acquires the matching parameters of the road image according to the matching parameter determination device in Example 1; the matching module translates and / or rotates the road image according to the matching parameters so that the road image matches the point cloud data.

[0130] Based on the same inventive concept, the present disclosure also provides a road image and point cloud data matching system, referring to Fig.14 As shown, it may include: a server 1, an image acquisition device 2 and a point cloud acquisition device 3; the image acquisition device 2 is used to acquire road images of the road; the point cloud acquisition device 3 is used to acquire point cloud data of the road; the server 1 includes the matching device as described above.

[0131] The technical effects and other specific descriptions of the system of this embodiment can be found in the relevant content of the method for determining the matching parameters of the road image and point cloud data mentioned above, which will not be repeated here.

[0132] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0133] The present disclosure is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0134] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0135] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A method for determining matching parameters between a road image and point cloud data, comprising: Selecting a first pose parameter from optional pose parameters of the road image at a predetermined parameter interval; Obtaining first matching evaluation values ​​for characterizing the matching degree of geographical elements after the road image having the first posture parameter is matched with the corresponding point cloud data; The parameter interval is reduced according to a preset rule, and a plurality of second posture parameters are selected from a predetermined range centered on the first posture parameter corresponding to the maximum value in the first matching evaluation value with each reduced parameter interval, and the maximum value of the second matching evaluation value after the road image with the second posture parameter is matched with the geographic element in the point cloud data is determined; The second posture parameter corresponding to the maximum value of the second matching evaluation value is determined as the matching parameter.

2. According to the method of claim 1, the step of obtaining a first matching evaluation value for characterizing a matching degree of a geographic element after the road image having the first posture parameter is matched with the corresponding point cloud data, comprises: Acquire a first geographical element in the road image, and acquire a second geographical element of the same type as the first geographical element in the point cloud data; Setting the posture of the road image with the first posture parameter; Projecting the road image onto the point cloud data according to the posture; The sum of the products of the light intensities of the first geographical element and the second geographical element located in the same projection area is obtained as the first matching evaluation value.

3. The method according to claim 2, when the first geographical element and the second geographical element are lane lines, after acquiring the first geographical element in the road image, further comprising: Performing straight line detection on the road image, and performing gradient processing on the light intensity of the detected lane lines, so that the light intensity of the lane lines decreases from the center part to the edge part of the lane lines; After obtaining a second geographical element of the same type as the first geographical element in the point cloud data, the method further includes: The lane line contour is proposed by using a preset algorithm on the point cloud data.

4. The method according to claim 1, reducing the parameter interval according to a preset rule, selecting a plurality of second posture parameters within a predetermined range centered on the first posture parameter corresponding to the maximum value in the first matching evaluation value with the reduced parameter interval, and determining the maximum value of the second matching evaluation value after the road image with the second posture parameter matches the geographic element in the point cloud data, comprising: Reducing the parameter interval to a lower limit of the parameter interval according to a preset rule to obtain a plurality of reduced parameter intervals; Selecting a plurality of second pose parameters from a predetermined range centered at the first pose parameter corresponding to the maximum value in the first matching evaluation value at each reduced parameter interval; Second matching evaluation values ​​for characterizing the degree of matching of geographical elements are respectively obtained after the road image having the second posture parameter is matched with the corresponding point cloud data, and a maximum value of the second matching evaluation values ​​is obtained.

5. The method according to claim 1, reducing the parameter interval according to a preset rule, selecting a plurality of second posture parameters within a predetermined range centered on the first posture parameter corresponding to the maximum value in the first matching evaluation value with the reduced parameter interval, and determining the maximum value of the second matching evaluation value after the road image with the second posture parameter matches the geographic element in the point cloud data, comprising: Reducing the parameter interval according to a preset rule to obtain a reduced parameter interval, and selecting a second posture parameter from a predetermined range centered on the first posture parameter corresponding to the maximum value in the first matching evaluation value with the reduced parameter interval; Obtaining a second matching evaluation value after the road image having the second posture parameter matches the geographic element in the point cloud data; If the second match evaluation value obtained is greater than the first match evaluation value or greater than the second match evaluation value obtained last time, return to continue the step of reducing the parameter interval according to the preset rule to obtain a reduced parameter interval, until the second match evaluation value obtained is no greater than the second match evaluation value obtained last time, and the maximum value of the second match evaluation value is obtained.

6. The method according to claim 1, before respectively obtaining first matching evaluation values ​​after matching the road images having the first posture parameters with the point cloud data corresponding to the road images, further comprising: Determining an initial position and posture of the road image according to a shooting time of the road image, an inertial navigation observation value of a camera shooting the road image, and an inertial navigation initial calibration parameter interpolation of the camera; Spatial indexing is performed in the point cloud data according to the initial posture to determine the point cloud data corresponding to the road image.

7. A method for matching road images and point cloud data, comprising: Acquiring matching parameters of the road image according to the matching parameter determination method according to any one of claims 1 to 6; The road image is translated and / or rotated according to the matching parameters so that the road image matches the point cloud data.

8. A device for determining matching parameters of a road image and point cloud data, comprising: A first pose parameter selection module is used to select a first pose parameter from optional pose parameters of the road image at a predetermined parameter interval; A first matching evaluation value acquisition module, used to respectively acquire a first matching evaluation value for characterizing a matching degree of a geographic element after a road image having the first posture parameter is matched with corresponding point cloud data; A second matching evaluation value acquisition module is configured to reduce the parameter interval according to a preset rule, select a plurality of second posture parameters within a predetermined range centered on the first posture parameter corresponding to the maximum value in the first matching evaluation value with each reduced parameter interval, and determine the maximum value of the second matching evaluation value after the road image having the second posture parameter matches the geographic element in the point cloud data; A matching parameter determination module is used to determine the second posture parameter corresponding to the maximum value of the second matching evaluation value as the matching parameter.

9. A road image and point cloud data matching device, comprising: a matching parameter acquisition module, configured to acquire matching parameters of the road image according to the matching parameter determination device according to claim 8; A matching module is used to translate and / or rotate the road image according to the matching parameters so that the road image matches the point cloud data.

10. A road image and point cloud data matching system, comprising: Server, image acquisition equipment and point cloud acquisition equipment; The image acquisition device is used to acquire road images of the road; The point cloud acquisition device is used to collect point cloud data of the road; The server comprises the matching device as claimed in claim 9.

11. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for determining parameters of road image and point cloud data matching as described in any one of claims 1 to 6 and the method for matching road image and point cloud data as described in claim 7.

12. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for determining parameters of matching road images and point cloud data as described in any one of claims 1 to 6 and the method for matching road images and point cloud data as described in claim 7 are implemented.

Images