Positioning information determination device, vehicle, method, and program product

Abstract

This application discloses a positioning information determination device, vehicle, method, and program product, relating to the field of positioning technology, for quickly, robustly, and with high precision determining positioning information. This application obtains each first line segment contained in an image and its planar line features, and obtains each second line segment contained in point cloud data and its spatial line features. Each second line segment is projected onto the image, and the second line segment equation parameters of each projected second line segment are determined. For each second line segment, based on the standard deviation between the second line segment equation parameters of the second line segment and the first line segment equation parameters of each first line segment, a first matching line segment is determined. Based on the spatial line features of the second line segment and the planar line features of the first matching line segment, first comprehensive feature information of the first matching line segment is determined. Positioning information is determined based on the first comprehensive feature information, enabling quick, robust, and high-precision positioning information determination.

Description

Technical Field

[0001] This application relates to the field of positioning technology, and in particular to a positioning information determination device, vehicle, method, and program product. Background Technology

[0002] In a broad sense, odometry refers to the study of localization for vehicles, robots, and other similar devices. Generally, odometry methods can be divided into two main categories: LiDAR-based methods (hereinafter referred to as LiDAR odometry) and vision-based methods (hereinafter referred to as visual odometry). The basic idea of ​​LiDAR odometry is that LiDAR detects the distance to obstacles and then uses their reflection and other features, combined with prior information about the vehicle or robot, to estimate its position. Visual odometry, on the other hand, acquires images, extracts image features, and estimates the motion of the vehicle or robot based on these features, thereby achieving localization.

[0003] Comparative analysis reveals that lidar odometry offers advantages such as high accuracy, but suffers from drawbacks including large data volume, low frequency, and inability to recognize line features in color space. Visual odometry, on the other hand, boasts advantages such as higher frequency and the ability to recognize line features in color space, but suffers from low accuracy. Therefore, combining lidar and visual odometry to leverage their respective strengths and compensate for their weaknesses to achieve rapid, robust, and high-precision location information determination is a current research hotspot in the field of positioning technology. However, it also presents a significant technical challenge, as a satisfactory solution for achieving rapid, robust, and high-precision location information determination currently lacks a definitive solution. Summary of the Invention

[0004] This application provides a positioning information determination device, vehicle, method, and program product for quickly, robustly, and with high precision determining positioning information.

[0005] In a first aspect, this application provides a location information determining device, the device including a processing module, wherein:

[0006] The processing module is used to obtain the image to be processed and the point cloud data of the lidar at any statistical time.

[0007] Obtain each first line segment and its planar line features contained in the image to be processed, and determine the first line segment equation parameters of each first line segment; and obtain each second line segment and its spatial line features contained in the point cloud data, project each second line segment onto the image to be processed, and determine the second line segment equation parameters of each projected second line segment;

[0008] For each second line segment, determine the standard deviation between the second line segment equation parameters of the second line segment and the first line segment equation parameters of each first line segment. Based on each standard deviation, determine the first matching line segment that matches the second line segment. Based on the spatial line features of the second line segment and the obtained planar line features of the first matching line segment, determine the first comprehensive feature information of the first matching line segment. Based on the first comprehensive feature information of each first matching line segment, determine the positioning information.

[0009] In one possible implementation, the processing module is specifically used for:

[0010] Based on the standard deviations and the set standard deviation thresholds, the first set of line segments associated with the second line segment is determined.

[0011] Based on the distance between each first line segment in the first line segment set and the projected second line segment, a first matching line segment that matches the second line segment is determined.

[0012] In one possible implementation, the processing module is further configured to:

[0013] Along the same direction, configure the starting point and ending point of each first line segment in the first line segment set respectively;

[0014] If it is detected that the distance between the endpoint of the first matching line segment and the starting point of another first line segment is less than a first preset distance threshold, or the distance between the starting point of the first matching line segment and the endpoint of another first line segment is less than a first preset distance threshold, then the first matching line segment is merged with the other first line segment, and the first matching line segment is updated based on the merged line segment; and / or,

[0015] If the distance between the starting point of the first matching line segment and the starting point of another first line segment is less than a second set distance threshold, or the distance between the ending point of the first matching line segment and the ending point of another first line segment is less than a second set distance threshold, then the first matching line segment is merged with the other first line segment, and the first matching line segment is updated according to the merged line segment.

[0016] In one possible implementation, the processing module is further configured to:

[0017] Along the same direction, configure the starting point and ending point of each first line segment in the first line segment set respectively;

[0018] If it is detected that the distance between the endpoint of any line segment in the first line segment set and the starting point of any other first line segment is less than a first preset distance threshold, or the distance between the starting point of any line segment and the endpoint of any other first line segment is less than the first preset distance threshold, then that line segment is merged with the other first line segments into a single first line segment; and / or,

[0019] If it is detected that the distance between the starting point of any line segment in the first line segment set and the starting point of other first line segments is less than a second set distance threshold, or the distance between the ending point of any line segment and the ending point of other first line segments is less than a second set distance threshold, then the any line segment is merged with the other first line segments into a first line segment.

[0020] In one possible implementation, the processing module is further configured to:

[0021] For each remaining first line segment other than each set of first line segments in each first line segment contained in the image to be processed, the depth feature of the remaining first line segment is determined based on the point cloud data; and the second comprehensive feature information of the remaining first line segment is determined based on the depth feature and the planar line feature of the remaining first line segment.

[0022] Based on the first comprehensive feature information of each first matching line segment, the second comprehensive feature information of each remaining first line segment, and the established residual equation, an algorithm is constructed to determine the first sub-error corresponding to each first comprehensive feature information and the second sub-error corresponding to each second comprehensive feature information; based on each first sub-error and each second sub-error, the positioning information is determined.

[0023] In one possible implementation, the processing module is specifically used for:

[0024] Obtain the first preset certainty weight corresponding to the first preset comprehensive feature information and the second preset certainty weight corresponding to the second comprehensive feature information;

[0025] The positioning information is determined based on the first product of each first sub-error and the first preset certainty weight, and the second product of each second sub-error and the second preset certainty weight.

[0026] In one possible implementation, the first preset certainty weight is greater than the second preset certainty weight.

[0027] In one possible implementation, the processing module is further configured to: obtain each line segment and the spatial line features of each line segment contained in other point cloud data of the lidar at other statistical times;

[0028] Each line segment and its spatial line feature contained in the other point cloud data are transformed into the line segment and spatial line feature at any statistical time.

[0029] The transformed line segments and spatial line features of the line segments contained in the other point cloud data are used as the second line segments and spatial line features of the second line segments contained in the point cloud data.

[0030] Secondly, this application also provides a vehicle that includes a positioning information determining device as described in any of the first aspects.

[0031] Thirdly, this application also provides a method for determining location information, the method comprising:

[0032] Obtain the image to be processed and the point cloud data of the LiDAR at any given statistical time.

[0033] Obtain each first line segment and its planar line features contained in the image to be processed, and determine the first line segment equation parameters of each first line segment; and obtain each second line segment and its spatial line features contained in the point cloud data, project each second line segment onto the image to be processed, and determine the second line segment equation parameters of each projected second line segment;

[0034] For each second line segment, determine the standard deviation between the second line segment equation parameters of the second line segment and the first line segment equation parameters of each first line segment. Based on each standard deviation, determine the first matching line segment that matches the second line segment. Based on the spatial line features of the second line segment and the obtained planar line features of the first matching line segment, determine the first comprehensive feature information of the first matching line segment. Based on the first comprehensive feature information of each first matching line segment, determine the positioning information.

[0035] In one possible implementation, determining the first matching line segment based on each standard deviation includes:

[0036] Based on the standard deviations and the set standard deviation thresholds, the first set of line segments associated with the second line segment is determined.

[0037] Based on the distance between each first line segment in the first line segment set and the projected second line segment, a first matching line segment that matches the second line segment is determined.

[0038] In one possible implementation, after determining the first matching line segment that matches the second line segment, and before determining the first comprehensive feature information of the first matching line segment based on the spatial line features of the second line segment and the obtained planar line features of the first matching line segment, the method further includes:

[0039] Along the same direction, configure the starting point and ending point of each first line segment in the first line segment set respectively;

[0040] If it is detected that the distance between the endpoint of the first matching line segment and the starting point of another first line segment is less than a first preset distance threshold, or the distance between the starting point of the first matching line segment and the endpoint of another first line segment is less than a first preset distance threshold, then the first matching line segment is merged with the other first line segment, and the first matching line segment is updated based on the merged line segment; and / or,

[0041] If the distance between the starting point of the first matching line segment and the starting point of another first line segment is less than a second set distance threshold, or the distance between the ending point of the first matching line segment and the ending point of another first line segment is less than a second set distance threshold, then the first matching line segment is merged with the other first line segment, and the first matching line segment is updated according to the merged line segment.

[0042] In one possible implementation, after determining the first set of line segments associated with the second line segment and before determining the first matching line segment matching the second line segment based on the distance between each first line segment in the first set and the projected second line segment, the method further includes:

[0043] Along the same direction, configure the starting point and ending point of each first line segment in the first line segment set respectively;

[0044] If it is detected that the distance between the endpoint of any line segment in the first line segment set and the starting point of any other first line segment is less than a first preset distance threshold, or the distance between the starting point of any line segment and the endpoint of any other first line segment is less than the first preset distance threshold, then that line segment is merged with the other first line segments into a single first line segment; and / or,

[0045] If it is detected that the distance between the starting point of any line segment in the first line segment set and the starting point of other first line segments is less than a second set distance threshold, or the distance between the ending point of any line segment and the ending point of other first line segments is less than a second set distance threshold, then the any line segment is merged with the other first line segments into a first line segment.

[0046] In one possible implementation, after determining the first set of line segments associated with the second line segment and before determining the positioning information based on the first comprehensive feature information of each first matching line segment, the method further includes:

[0047] For each remaining first line segment other than each set of first line segments in each first line segment contained in the image to be processed, the depth feature of the remaining first line segment is determined based on the point cloud data; and the second comprehensive feature information of the remaining first line segment is determined based on the depth feature and the planar line feature of the remaining first line segment.

[0048] The determination of positioning information based on the first comprehensive feature information of each first matching line segment includes:

[0049] Based on the first comprehensive feature information of each first matching line segment, the second comprehensive feature information of each remaining first line segment, and the established residual equation, an algorithm is constructed to determine the first sub-error corresponding to each first comprehensive feature information and the second sub-error corresponding to each second comprehensive feature information; based on each first sub-error and each second sub-error, the positioning information is determined.

[0050] In one possible implementation, determining the positioning information based on each first sub-error and each second sub-error includes:

[0051] Obtain the first preset certainty weight corresponding to the first preset comprehensive feature information and the second preset certainty weight corresponding to the second comprehensive feature information;

[0052] The positioning information is determined based on the first product of each first sub-error and the first preset certainty weight, and the second product of each second sub-error and the second preset certainty weight.

[0053] In one possible implementation, after obtaining the image to be processed and the point cloud data of the lidar at any statistical moment, and before projecting each second line segment onto the image to be processed and determining the second line segment equation parameters of each projected second line segment, the method further includes:

[0054] Obtain each line segment and its spatial line characteristics from other point cloud data of the lidar at other statistical times;

[0055] Each line segment and its spatial line feature contained in the other point cloud data are transformed into the line segment and spatial line feature at any statistical time.

[0056] The transformed line segments and spatial line features of the line segments contained in the other point cloud data are used as the second line segments and spatial line features of the second line segments contained in the point cloud data.

[0057] Fourthly, this application also provides a computer program product, including a computer program / instructions that, when executed by a computer program / instruction processor, implement the method as described in any of the third aspects.

[0058] This application can obtain the image to be processed and the point cloud data of the lidar at any statistical moment; obtain each first line segment and its planar line features contained in the image to be processed, and determine the first line segment equation parameters of each first line segment; and obtain each second line segment and its spatial line features contained in the point cloud data, project each second line segment onto the image to be processed, and determine the second line segment equation parameters of each projected second line segment; for each second line segment, determine the standard deviation between the second line segment equation parameters of the second line segment and the first line segment equation parameters of each first line segment, and determine the first matching line segment that matches the second line segment based on each standard deviation; determine the first comprehensive feature information of the first matching line segment based on the spatial line features of the second line segment and the obtained planar line features of the first matching line segment, and determine the positioning information based on the first comprehensive feature information. The first comprehensive feature information of this application integrates the spatial line features of the second line segment and the planar line features of the first matching line segment that matches the second line segment. Based on this, the purpose of quickly, robustly, and with high precision in determining positioning information can be achieved. Attached Figure Description

[0059] To more clearly illustrate the implementation methods in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0060] Figure 1 A schematic diagram of a positioning information determination device provided in some embodiments is shown;

[0061] Figure 2A A schematic diagram of a disconnected line provided in some embodiments is shown;

[0062] Figure 2B A schematic diagram of a broken wire fusion provided by some embodiments is shown;

[0063] Figure 3A A schematic diagram of a similar line provided in some embodiments is shown;

[0064] Figure 3B A schematic diagram of nearline fusion provided by some embodiments is shown;

[0065] Figure 4 This diagram illustrates a visualization of endpoint depth recovery provided by some embodiments;

[0066] Figure 5 A schematic diagram of a first type of location information determination process provided in some embodiments is shown;

[0067] Figure 6A schematic diagram of a second location information determination process provided in some embodiments is shown;

[0068] Figure 7 A schematic diagram of a third location information determination process provided in some embodiments is shown;

[0069] Figure 8 A schematic diagram of a fourth location information determination process provided in some embodiments is shown. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this application clearer, a further detailed description of this application will be provided below with reference to the accompanying drawings. Obviously, the embodiments described in this application are merely some embodiments, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0071] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0072] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0073] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0074] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0076] To quickly, robustly, and accurately determine positioning information, this application provides a positioning information determination device, vehicle, method, and procedure.

[0077] All embodiments of this application comply with the relevant provisions of national laws and regulations regarding the acquisition, storage, use, and processing of data.

[0078] Figure 1 The diagram shows a location information determination device according to some embodiments, such as Figure 1 As shown, the device includes a processing module 11, wherein:

[0079] Processing module 11 is used to obtain the image to be processed and the point cloud data of the lidar at any statistical time.

[0080] Obtain each first line segment and its planar line features contained in the image to be processed, and determine the first line segment equation parameters of each first line segment; and obtain each second line segment and its spatial line features contained in the point cloud data, project each second line segment onto the image to be processed, and determine the second line segment equation parameters of each projected second line segment;

[0081] For each second line segment, determine the standard deviation between the second line segment equation parameters of the second line segment and the first line segment equation parameters of each first line segment. Based on each standard deviation, determine the first matching line segment that matches the second line segment. Based on the spatial line features of the second line segment and the obtained planar line features of the first matching line segment, determine the first comprehensive feature information of the first matching line segment. Based on the first comprehensive feature information of each first matching line segment, determine the positioning information.

[0082] In one possible implementation, the positioning information determining device provided in this application embodiment can be applied to a vehicle, such as an autonomous vehicle. Optionally, the positioning information determining device provided in this application embodiment can also be applied to an electronic device, such as a PC, mobile terminal, or server; additionally, the electronic device can be a device installed in a vehicle. Optionally, the positioning information determining device provided in this application embodiment can also be applied to an intelligent robot. For ease of understanding, the following example illustrates the application of the positioning information determining device provided in this application embodiment to a vehicle.

[0083] In one possible implementation, the processing module 11 in the positioning information determination device can obtain the image to be processed and the point cloud data of the lidar at any given statistical moment. Optionally, the processing module 11 may include an image acquisition submodule such as a camera, and the processing module 11 can obtain the image to be processed based on the image acquisition submodule. For example, during vehicle operation, the image acquisition submodule in the processing module 11 can acquire images of the vehicle's surrounding environment at a set frequency, and the processing module 11 can use the images acquired by the image acquisition submodule as the image to be processed obtained by the processing module 11.

[0084] Similarly, the processing module 11 may include a LiDAR acquisition submodule such as a multi-line LiDAR, and the processing module 11 can obtain point cloud data of the LiDAR based on the LiDAR acquisition submodule. For example, during vehicle operation, the LiDAR acquisition submodule in the processing module 11 can acquire point cloud data of the LiDAR in the vehicle's surrounding environment at a set frequency, and the processing module 11 can use the point cloud data acquired by the LiDAR acquisition submodule as the point cloud data obtained by the processing module 11.

[0085] It is worth noting that the LiDAR acquisition submodule and the image acquisition submodule can have a good shared field of view, meaning they can acquire point cloud data and images within the same field of view around the vehicle. Furthermore, the LiDAR acquisition submodule and the image acquisition submodule can use the same clock source, i.e., simultaneous clock source. The internal parameters of both the LiDAR acquisition submodule and the image acquisition submodule have been precisely calibrated, as have their external parameters. Existing technologies can be used to calibrate the internal and external parameters of the LiDAR acquisition submodule and the image acquisition submodule, which will not be elaborated upon here. This application does not impose specific limitations on the location of the LiDAR acquisition submodule and the image acquisition submodule; they can be flexibly set according to requirements. For example, a pinhole camera or similar image acquisition submodule and a multi-line LiDAR or similar LiDAR acquisition submodule can be installed at the head of the vehicle.

[0086] Optionally, the positioning information determining device may further include an acquisition module connected to the processing module 11. The acquisition module may include an image acquisition module such as a camera and a LiDAR acquisition module such as a multi-line LiDAR. For example, during vehicle operation, the acquisition module can acquire images of the vehicle's surrounding environment at a set frequency and send the acquired images to the processing module 11 as the images to be processed by the processing module 11; or the processing module 11 can acquire the images to be processed from the acquisition module at a set frequency and use them as the images to be processed by the processing module 11. Optionally, the processing module 11 can also receive images acquired by other image acquisition devices and use the received images as the images to be processed. Similarly, during vehicle operation, the acquisition module can acquire point cloud data of the vehicle's surrounding environment at a set frequency and send the acquired point cloud data to the processing module 11 as the point cloud data obtained by the processing module 11; or the processing module 11 can acquire point cloud data from the acquisition module at a set frequency and use it as the point cloud data obtained by the processing module 11. Optionally, the processing module 11 can also receive point cloud data acquired by other LiDAR acquisition devices and use the received point cloud data as the point cloud data obtained. This application does not specify the exact method by which the processing module 11 obtains the image and point cloud data to be processed, and can be flexibly set according to requirements. Similar to the above-mentioned lidar acquisition submodule and image acquisition submodule, the lidar acquisition module (lidar acquisition device) and the image acquisition module (image acquisition device) can also have a good common field of view, can use the same clock source, and the corresponding internal and external parameters have been accurately calibrated, which will not be described in detail here.

[0087] In one possible implementation, the processing module 11 can be a vehicle controller, etc. After obtaining the image to be processed and the point cloud data of the LiDAR at any statistical time, the processing module 11 can obtain each first line segment (for ease of description, the line segments contained in the image to be processed are referred to as first line segments) and the planar line features of each first line segment (for ease of description, the line features of the first line segment in the image are referred to as planar line features). For example, each first line segment and the planar line features of each first line segment in the image to be processed can be obtained based on the Line Segment Detector (LSD) algorithm or deep learning methods, which will not be elaborated here. Optionally, the newly obtained image to be processed at the current statistical time can be used... This means that the two endpoints of each obtained first line segment are respectively... and This indicates that the first line segment set is used. express, j is a positive integer. For ease of understanding, in this embodiment, keyframes are represented by F, lines or line segments by v, pixels in an image by u, and points in a point cloud (spatial points) by p. Superscripts I and L can be used to distinguish between image and point cloud data; for example, F... I F represents a keyframe of an image (the image to be processed). L Represents a point cloud keyframe (point cloud data), v I v L These are straight lines in the image or point cloud, respectively. For easy and accurate differentiation, the specific time when the image to be processed is obtained at the current statistical time is denoted as time j, and the time when the point cloud data is obtained at the current statistical time is denoted as time i. Time j and time i can be exactly the same or have some differences. When time j and time i are different, the point cloud data features at time i and the image features at time j can be aligned based on existing technologies, etc., which will not be elaborated here.

[0088] In one possible implementation, after obtaining each first line segment, the first line segment equation parameters for each first line segment can also be determined. For example, the slope 'a' of the first line segment in the image coordinate system can be calculated based on the endpoints of the first line segment, etc. j and intercept b j The parameters of the first line segment equation can be represented using the slope-intercept form, for example, the first line segment... We can use y = a j x+b j Represented by equations, that is in It indicates equivalent expressions or equivalence relations.

[0089] In one possible implementation, the newly acquired lidar point cloud data at the current statistical moment is used... This means that it is possible to obtain each second line segment (for ease of description, the line segments contained in the point cloud data are referred to as second line segments) and the spatial line features of each second line segment (for ease of description, the line features of the second line segments in the point cloud data are referred to as spatial line features) contained in the point cloud data. Optionally, when determining the second line segments contained in the point cloud data and the spatial line features of the second line segments, inflection points can be extracted by judging whether the curvature of each point in the point cloud (point-by-point curvature judgment) meets a preset curvature threshold. Then, points with curvature close to the inflection point are matched by a neighbor-nearest neighbor search method, thereby obtaining a set of points located on the same straight line. For example, when determining the curvature of each point, the average curvature of several (n') points including that point can be used as the curvature of that point. The curvature of the point is represented by c, and n' represents the number of points used to calculate the curvature. It can be expressed by the formula: The curvature of each point is determined. The curvature threshold is represented by c. n' and c can be flexibly set according to requirements. This application does not impose specific limitations on them. For example, n' can be 10, and the curvature threshold c can be 0.3, etc. Existing technologies can be used to determine the second line segment and the spatial line characteristics of the second line segment contained in the point cloud data, which will not be elaborated here.

[0090] To make it easier to understand, the two endpoints of each obtained second line segment are respectively... and This indicates that the second line segment set uses express, Where k > 0, and k is a positive integer. In one possible implementation, Plücker notation can be used to describe the second line segment. The linear features, namely in Let d represent the direction vector of the second line segment, and let d represent the cross product of the direction vector and the point on the second line segment closest to the origin. These will not be elaborated further here.

[0091] After obtaining the planar line features of each first line segment and the spatial line features of each second line segment, cross-modal correlation, alignment, and feature fusion can be performed on the planar and spatial line features. Specifically, the external parameters of each second line segment can be obtained first through the LiDAR acquisition submodule (LiDAR acquisition module, LiDAR acquisition device) and the image acquisition submodule (image acquisition module, image acquisition device). Point p included k The points u′ are projected onto the image to be processed. k This can be expressed as a formula: In this application, the projection operator is abstractly represented by π(·), w represents the global coordinate system (not involved in this formula), l represents the lidar coordinate system, and c represents the camera coordinate system. This indicates the transformation from the LiDAR coordinate system to the camera coordinate system, that is, the LiDAR coordinate system expressed in the camera coordinate system. It can also represent the external parameters of the camera and lidar.

[0092] In one possible implementation, after projecting the second line segment onto the image to be processed, the slope of the second line segment in the image coordinate system can be calculated based on the endpoints of the second line segment. and intercept The parameters of the second line segment equation can be represented using the slope-intercept form, such as the projected second line segment. It can be used Represented by equations, that is

[0093] In one possible implementation, superscripts I and L are used to distinguish the parameters of the image (video) and point cloud data, respectively, and the first line segment equation parameter a j It can also be used It means, b j It can also be used This means that the parameters of the first line segment equation corresponding to the image (video) can be organized into a first line segment equation parameter set. At the same time, the corresponding second line segment equation parameters of the point cloud data can be organized into a second line segment equation parameter set.

[0094] In one possible implementation, to perform cross-modal correlation of planar line features and spatial line features, for each second line segment, the standard deviation (also referred to as variance r) between the second line segment equation parameters of that second line segment and the first line segment equation parameters of each first line segment can be determined. For example, the standard deviation between the second line segment equation parameters of any second line segment and the first line segment equation parameters of any first line segment... Optionally, for each second line segment, a first matching line segment can be determined based on the standard deviation between the second line segment equation parameters of the second line segment and the first line segment equation parameters of each first line segment, thereby establishing an association between the second line segment and the first matching line segment, so as to realize cross-modal association of planar line features and spatial line features and form feature matching pairs.

[0095] In one possible implementation, when determining the first matching line segment that matches the second line segment based on the standard deviation between the second line segment equation parameters of the second line segment and the first line segment equation parameters of each first line segment for each second line segment, the set of first line segments associated with the second line segment can be determined firstly based on the standard deviations and a set standard deviation threshold. For example, for each second line segment, the first line segments whose standard deviation between their second line segment equation parameters and the second line segment equation parameters is less than the set standard deviation threshold can be identified as the set of first line segments associated with the second line segment. For ease of understanding, assume that the first line segment equation parameters of m first line segments are respectively associated with the k-th second line segment... If the standard deviation between the parameters of the second line segment equation is less than the set standard deviation threshold, then the second line segment can be determined to be related to the k-th second line segment. The associated first set of line segments is Right now Among them, the following can be used in this application This indicates a link or correspondence relationship; optionally, it can be derived from the first segment set. Among the m line segments, determine the connection with the k-th second line segment. The first matching line segment.

[0096] In one possible implementation, for each second line segment, the slope and other equation parameters of each line segment (first line segment) in the first line segment set associated with the second line segment can be considered to be the same as or similar to the slope and other equation parameters of the second line segment. When determining the first matching line segment that matches the second line segment based on the line segments in the first line segment set associated with the second line segment, the first matching line segment that matches the second line segment can be determined based on the distance between each first line segment in the first line segment set and the second line segment. For example, the distance between each first line segment in the first line segment set and the second line segment can be determined, and the first line segment closest to the second line segment can be determined as the first matching line segment that matches the second line segment.

[0097] Since this application targets each second line segment, it can first determine the first line segment set associated with the second line segment based on the standard deviation between the second line segment equation parameters of the second line segment and the first line segment equation parameters of each first line segment and the set standard deviation threshold. When determining the first matching line segment that matches the second line segment from the line segments contained in the first line segment set, the speed and accuracy of determining the first matching line segment can be improved, thereby achieving the purpose of determining positioning information quickly, robustly and with high precision.

[0098] In one possible implementation, considering that if the starting and ending points of each segment in the first segment set are arranged along the same direction, there might be a situation where the ending point of one segment is close to the starting point of another segment. These two segments can be considered as two broken lines (or disconnected lines) within the same longer segment. See [reference needed]. Figure 2A , Figure 2A The diagram illustrates a broken line according to some embodiments. Optionally, the two line segments (broken lines) can be merged to form a longer line segment, in order to improve the accuracy of positioning based on the more line features contained in the longer line segment.

[0099] Specifically, in one possible implementation, after determining the first matching line segment that is closest to the second line segment as the first matching line segment based on the distance between each first line segment and the second line segment in the first line segment set, it can also be determined whether there are two broken line segments in the first line segment set that belong to the same longer line segment as the first matching line segment. If so, the first matching line segment and the line segment are merged (also known as broken line splicing), and the longer line segment formed after the broken lines are merged is determined as the first matching line segment that matches the second line segment. In other words, after determining the first matching line segment that matches the second line segment based on the distance between each first line segment in the first line segment set and the second line segment, if it is found that the distance between the end point of the first matching line segment and the starting point of any other first line segment in the first line segment set is less than a first set distance threshold, then the first matching line segment and the other first line segment can be considered as two broken lines in the same longer line segment. The first matching line segment and the other first line segment can be merged (merged). Optionally, the two endpoints of the merged line segment can be determined based on the starting point of the first matching line segment and the ending point of the other first line segment. For example, the starting point of the reference first matching line segment can be used as the starting point of the merged line segment, and the ending point of the other first line segment can be determined as the ending point of the merged line segment. Then, the first matching line segment is updated according to the merged line segment, that is, the merged line segment is determined as the first matching line segment that matches the second line segment.

[0100] In one possible implementation, if the distance between the starting point of the first matching line segment and the ending point of any other first line segment in the first line segment set is less than a first set distance threshold, then the first matching line segment and the other first line segment can be considered as two broken lines in the same longer line segment. The first matching line segment and the other first line segment can be merged (merged). Optionally, the two endpoints of the merged line segment can be determined based on the ending point of the first matching line segment and the starting point of the other first line segment. For example, the ending point of the first matching line segment can be used as the ending point of the merged line segment, and the starting point of the other first line segment can be used as the starting point of the merged line segment. Then, the first matching line segment is updated according to the merged line segment, that is, the merged line segment is determined as the first matching line segment that matches the second line segment.

[0101] The first distance threshold can be flexibly set according to requirements, and this application does not impose specific limitations on it. For example, the first distance threshold can be 3 pixels, etc. In addition, the first matching line segment can be merged with one other first line segment or with multiple other first line segments, and this application does not impose specific limitations on it either.

[0102] For ease of understanding, the disconnection fusion process provided in this application will be explained and illustrated below through a specific embodiment. For example, see [link to relevant documentation]. Figure 2B , Figure 2B The diagram illustrates a broken line fusion method according to some embodiments, assuming the first matching line segment is... The endpoint of the first matching line segment and the first line segment in the first line segment set The distance between the starting points is less than a first set distance threshold, wherein the first matching line segment The starting point is The destination is First line segment The starting point is The destination is Then the first matching line segment can be With the first line segment Perform a merge (break line fusion), the starting point of the merged line segment is u. j 0 The destination is The merged line segment can be represented as Based on the line segments obtained after merging For the first matching line segment The update is performed, meaning the first matching line segment that matches the corresponding second line segment has endpoints as... The line segment.

[0103] In one possible implementation, see [reference] Figure 3A , Figure 3A The diagram illustrates a near-parallel line according to some embodiments. The line segments in the first line segment set can be considered as line segments with the same or similar slopes, and parallel or approximately parallel. Considering that there may be two line segments in the first line segment set whose starting points and ending points are close to each other (for ease of description, these are called near-parallel lines), optionally, these two line segments (near-parallel lines) can be merged to improve positioning accuracy. The near-parallel merging process is described in detail below.

[0104] Specifically, in one possible implementation, after determining the first matching line segment based on the distance between each first line segment and the second line segment in the first line segment set, for example, determining the first line segment closest to the second line segment as the first matching line segment, it can also be determined whether there are other first line segments in the first line segment set whose starting point is less than the starting point of the first matching line segment, or whether there are other first line segments in the first line segment set whose ending point is less than the ending point of the first matching line segment. If it is identified that the distance between the starting point of the first matching line segment and the starting point of any other first line segment is less than the second set distance threshold, or if it is identified that the distance between the ending point of the first matching line segment and the ending point of any other first line segment is less than the second set distance threshold, then the other first line segment and the first matching line segment can be considered as close lines, and the other first line segment and the first matching line segment can be merged (merged). Specifically, when performing close-line merging, the starting point of the merged line segment can be determined based on the starting point of the first matching line segment and the starting point of the other first line segment. For example, the starting point of the merged line segment can be determined by the center point (midpoint) between the starting point of the first matching line segment and the starting point of the other first line segment; or, any point on the line connecting the starting point of the first matching line segment and the starting point of the other first line segment can be determined by the starting point of the merged line segment. The choice can be made flexibly according to the requirements.

[0105] In addition, when merging other first line segments with the first matching line segment, the endpoint of the merged line segment can be determined based on the endpoint of the first matching line segment and the endpoint of the other first line segment. For example, the endpoint of the merged line segment can be determined as the center point (midpoint) between the endpoint of the first matching line segment and the endpoint of the other first line segment; or any point on the line connecting the endpoint of the first matching line segment and the endpoint of the other first line segment can be determined as the endpoint of the merged line segment. The choice can be made flexibly according to the requirements.

[0106] Once the start and end points of the merged line segment are determined, the merged line segment can be updated based on it. That is, the merged line segment is used as the first matching line segment to match the corresponding second line segment.

[0107] For ease of understanding, the nearline fusion process provided in this application will be explained and illustrated below through a specific embodiment. For example, see [link to relevant documentation]. Figure 3B , Figure 3B A schematic diagram of near-line fusion provided by some embodiments is shown, assuming that the first matching line segment is The starting point of the first matching line segment is the first line segment in the first line segment set (and other first line segments). The distance between the starting points is less than a second set distance threshold, wherein the first matching line segment The starting point is The destination is First line segment The starting point is The destination is Then the first matching line segment can be With the first line segment Perform a merging process (near-line fusion), and the starting point of the merged line segment can be the first matching line segment. The starting point With the first line segment The starting point The center point between them, i.e. The endpoint of the merged line segment can be the first matching line segment. The endpoint is With the first line segment The End The center point between them, i.e. The merged line segment can be represented as Based on the line segments obtained after merging For the first matching line segment The update is performed, meaning the first matching line segment that matches the corresponding second line segment has endpoints as... The line segment.

[0108] In one possible implementation, when performing near-line fusion, one segment from multiple near-line lines can be used as the fused (merged) segment. For example, a longer segment from multiple near-line lines can be used as the merged segment, and this longer segment can be used as the first matching segment to match the corresponding second segment. For instance, suppose the first matching segment... Other first line segments The distance between the starting points is less than the second set distance threshold, and the first matching line segment Other first line segments If they are similar lines, the first matching line segment can be... Or other first line segments As a merged line segment, for example, it can be used when the first matching line segment... Compared to other first line segments When the length is long, the first matching segment is used as the merged segment, and the first matching segment is still used. As the first matching line segment that matches the corresponding second line segment. It can also be used when other first line segments... If the length of the other first line segment is longer than the length of the first matching line segment, then the other first line segments will be... As the merged line segment, the other first line segments Update to the first matching line segment that matches the corresponding second line segment.

[0109] The second distance threshold can be flexibly set according to requirements. The second distance threshold can be the same as or different from the first distance threshold; this application does not specifically limit this. For example, the second distance threshold can be 2 pixels, etc. Furthermore, the first matching line segment can undergo the aforementioned near-line fusion process with one other first line segment, or it can undergo the aforementioned near-line fusion process with multiple other first line segments; this application also does not specifically limit this. Additionally, the first matching line segment can undergo the aforementioned discontinuity fusion process with one or more other first line segments while simultaneously undergoing the aforementioned near-line fusion process with one or more other first line segments. For example, the first matching line segment can undergo discontinuity fusion with other first line segment a to form an updated first matching line segment. The updated first matching line segment can then undergo near-line fusion with other first line segment b to form a further updated first matching line segment. This further updated first matching line segment can be used as the first matching line segment to match the corresponding second line segment. For example, the first matching segment can be merged with other first segment c via near-line fusion to form an updated first matching segment. This updated first matching segment can then be merged with other first segment d via discontinuity fusion to form a further updated first matching segment. This further updated first matching segment can then be used as the first matching segment to match the corresponding second segment. Alternatively, the first matching segment can be simultaneously merged via discontinuity fusion with other first segment e and via near-line fusion with other first segment f. The segment resulting from both discontinuity fusion and near-line fusion can then be used as the updated first matching segment, and this updated first matching segment can then be used as the first matching segment to match the corresponding second segment. This application does not specify the number of near-line fusions or the order in which discontinuity fusions are performed.

[0110] In another possible implementation, after determining the set of first segments associated with each second line segment, a process of discontinuity fusion and near-line fusion can be performed on the segments in the set of first segments. After the discontinuity fusion and near-line fusion are completed, a process of determining the first matching segment that matches the second line segment is performed based on the distance between each first segment in the set of first segments and the projected second line segment. Specifically, after determining the set of first segments associated with each second line segment, and before determining the first matching segment that matches the second line segment based on the distance between each first segment in the set of first segments and the projected second line segment, the start and end points of each first segment in the set of first segments can be configured along the same direction. If the distance between the endpoint of any line segment in the first line segment set and the starting point of any other first line segment in the first line segment set is less than a first set distance threshold, then the line segment and the other first line segment can be considered as two broken lines in the same longer line segment. The line segment and the other first line segment can be merged into a single first line segment, which is the same as the process of merging the first matching line segment with other first line segments in the above embodiment. Optionally, the two endpoints of the merged first line segment can be determined based on the starting point of the line segment and the endpoint of the other first line segment. For example, the starting point of the line segment can be used as the starting point of the merged first line segment, and the endpoint of the other first line segment can be used as the endpoint of the merged first line segment.

[0111] In one possible implementation, if the distance between the starting point of any line segment in the first line segment set and the ending point of any other first line segment in the first line segment set is less than a first set distance threshold, then the line segment and the other first line segment can be considered as two broken lines in the same longer line segment. The line segment and the other first line segment can be merged into a single first line segment. Optionally, the two endpoints of the merged first line segment can be determined based on the ending point of the line segment and the starting point of the other first line segment. For example, the ending point of the line segment can be used as the ending point of the merged first line segment, and the starting point of the other first line segment can be used as the starting point of the merged first line segment.

[0112] Similar to the process of merging the first matching line segment with other first line segments in the above embodiments, for each line segment, the line segment can be merged with one other first line segment or with multiple other first line segments. This application does not make any specific limitation in this regard.

[0113] In one possible implementation, similar to the process of near-line fusion of the first matching line segment with other first line segments in the above embodiments, if the distance between the starting point of any line segment in the first line segment set and the starting point of any other first line segment in the first line segment set is less than a second preset distance threshold, or if the distance between the ending point of any line segment in the first line segment set and the ending point of any other first line segment in the first line segment set is less than the second preset distance threshold, then the line segment and the other first line segment can be considered as similar lines, and the line segment and the other first line segment can be near-line fused (merged) into a single first line segment. Specifically, near-line fusion can be performed based on the starting point of the line segment and the starting point of the other first line segment to determine the starting point of the merged first line segment. For example, the center point (midpoint) between the starting point of the line segment and the starting point of the other first line segment can be determined as the starting point of the merged first line segment; alternatively, any point on the line connecting the starting point of the line segment and the starting point of the other first line segment can be determined as the starting point of the merged first line segment, which can be flexibly selected according to requirements.

[0114] In addition, when merging any line segment with other first line segments, the endpoint of the merged first line segment can be determined based on the endpoint of the line segment and the endpoint of the other first line segment. For example, the endpoint of the merged first line segment can be determined as the center point (midpoint) between the endpoint of the line segment and the endpoint of the other first line segment; or any point on the line connecting the endpoint of the line segment and the endpoint of the other first line segment can be determined as the endpoint of the merged first line segment. The choice can be made flexibly according to the requirements.

[0115] In one possible implementation, when performing near-line fusion, one segment from multiple near-line lines can be used as the segment after near-line fusion (merging). For example, one of the longer segments from multiple near-line lines can be used as the first segment after merging, which will not be elaborated further here.

[0116] Furthermore, any segment in the first segment set can undergo the aforementioned near-line fusion process with one other first segment in the first segment set, or with multiple other first segments; this application does not specifically limit this. Additionally, any segment in the first segment set can undergo the aforementioned discontinuity fusion process with one or more other first segments simultaneously with the aforementioned near-line fusion process with one or more other first segments; this application does not specifically limit the number of near-line fusions and the order in which they are performed.

[0117] In one possible implementation, after performing near-line fusion and discontinuity fusion on each line segment in the first line segment set, the distance between each first line segment and the projected second line segment can be determined based on each first line segment generated after near-line fusion and discontinuity fusion contained in the first line segment set. Based on this distance, the first matching line segment that matches the second line segment can be determined. This will not be elaborated further here.

[0118] Since this application can perform broken line fusion and near-line fusion on the line segments in the set of first line segments associated with the second line segment based on the spatial line features (point cloud line features) of the second line segment in the point cloud data, and determine the first matching line segment that matches the second line segment based on the line segments generated after broken line fusion and near-line fusion, the accuracy of the determined first matching line segment can be improved, thereby achieving the purpose of quickly, robustly and with high precision in determining the positioning information.

[0119] This allows us to determine the corresponding first matching line segment for each of the k second line segments from the j first line segments. Ultimately, we can obtain n pairs of correspondences between second line segments and first matching line segments that satisfy a single mapping relationship, resulting in n line feature pairs. Where n ≤ k and n ≤ j, the first matching line segment is... This means that the second line segment corresponding to the first matching line segment is used... The correspondence (mapping relationship) between the second line segment and the first matching line segment can be represented as follows:

[0120] In one possible implementation, after determining the first matching line segment that matches each second line segment, for each second line segment, a first comprehensive feature information of the first matching line segment can be determined based on the spatial line features of the second line segment and the planar line features of the first matching line segment that matches the second line segment. Optionally, since the point cloud data of the lidar has geometric meaning, the first comprehensive feature information of the first matching line segment may include the spatial line features of the corresponding second line segment. and the planar line features of the first matching line segment The spatial characteristics of the second line segment It can be considered as a line feature based on geometry (Baesd), or it can be considered as a planar line feature of the first matching line segment. If it is directly identified as a Geometry-Baesd line feature, the correspondence (mapping relationship) between the second line segment and the first matching line segment can be established. Convert to Right now in, This indicates a correspondence or link relationship. It indicates equivalent expressions or equivalence relations.

[0121] In one possible implementation, for ease of description, the first comprehensive feature information of the first matching line segment is used as... This indicates that the vehicle's positioning information can be determined based on the first comprehensive information of each first matching line segment. The positioning information may include vehicle position information and attitude information such as vehicle orientation; it can also be referred to as positioning and attitude determination information. Optionally, the positioning information can be determined using the first comprehensive feature information in a manner similar to that used for determining positioning information with existing LiDAR odometers, or it can be determined using the first comprehensive feature information in a manner similar to that used for determining positioning information with existing visual odometers; these details will not be elaborated further here. Since the first comprehensive feature information in this application integrates the spatial line features of the LiDAR point cloud data (second line segment) and the planar line features (such as color space line features) of the image (first line segment), even if the first comprehensive feature information is used to determine the positioning information using existing methods for determining positioning information, the goal of improving the positioning accuracy can still be achieved.

[0122] This application can obtain the image to be processed and the point cloud data of the lidar at any statistical moment; obtain each first line segment and its planar line features contained in the image to be processed, and determine the first line segment equation parameters of each first line segment; and obtain each second line segment and its spatial line features contained in the point cloud data, project each second line segment onto the image to be processed, and determine the second line segment equation parameters of each projected second line segment; for each second line segment, determine the standard deviation between the second line segment equation parameters of the second line segment and the first line segment equation parameters of each first line segment, and determine the first matching line segment that matches the second line segment based on each standard deviation; determine the first comprehensive feature information of the first matching line segment based on the spatial line features of the second line segment and the obtained planar line features of the first matching line segment, and determine the positioning information based on the first comprehensive feature information. The first comprehensive feature information of this application integrates the spatial line features of the second line segment and the planar line features of the first matching line segment that matches the second line segment. Based on this, the purpose of quickly, robustly, and with high precision in determining positioning information can be achieved.

[0123] In one possible implementation, for ease of description, each first line segment contained in the image to be processed is... In the first set of line segments, the line segments other than those in the first set of line segments are called the remaining first line segments (using the first set of line segments). (This is an example of a process involving multiple line segments). Based on the first line segment set, after performing broken line fusion and near-line fusion to determine the first matching line segment that matches the second line segment, other first line segments in the first matching set can be deleted. The remaining first line segments can then be... Consider it as the first matching line segment (can also be expressed as) In each first line segment The complement of can be expressed by the formula:

[0124] In one possible implementation, besides determining the first comprehensive feature information of the first matching line segment, for each remaining first line segment, since the remaining first line segments only have planar line features and no corresponding spatial line features, in order to improve the accuracy of the determined positioning information, the depth features of each remaining first line segment can be determined based on the point cloud data of the LiDAR. For example, the endpoint depth features of each remaining first line segment can be recovered from the point cloud data. After recovering the endpoint depth features, the Plücker expression of the straight line formed by the endpoints in space can be obtained, thereby obtaining the planar line features of the remaining first line segments. Matching depth features (also known as spatial line features) This yields the line feature matching pairs for the remaining first line segment:

[0125] The process of recovering the endpoint depth features of each remaining first line segment using point cloud data can be as follows: First, project the points contained in the point cloud onto the image to be processed to obtain the projected points u′. The projection method is the same as that described in the above embodiment, where the points contained in the second line segment are projected onto the image to be processed to obtain the projected points u′. k The process is the same and will not be repeated here. Then, the nearest neighbor method (such as KDtree) is used to find the three spatial projection point pairs (denoted by {u′0, u′1, u′2}) corresponding to the endpoint of the first line segment (also called the endpoint of the color-based line feature) and associate them with the corresponding spatial points, that is, the points in the point cloud (denoted by {p0, p1, p2}, also called depth points). Thus, the endpoint u of each remaining first line segment is obtained. c Established relationships: See Figure 4 , Figure 4 This diagram illustrates a visualization of endpoint depth recovery provided by some embodiments, for each endpoint u of the remaining first line segment. c The endpoint u can be projected back. c (Also known as visual features) and spatial point cloud points {p0, p1, p2} (depth points) are projected from Cartesian space onto a unit circle (also known as a unit sphere) centered at the camera. Finally, the endpoint depth features of each remaining first line segment reconstructed from the point cloud data can be based on the camera center O. cThe depth features are obtained by connecting the feature points to the point cloud data, where the plane formed by this connection and the three depth points intersects in a Cartesian coordinate system (Cartesian space). The process of recovering the endpoint depth features of each remaining first line segment from the point cloud data can employ existing techniques, which will not be elaborated upon here.

[0126] In one possible implementation, for each remaining first line segment, a second comprehensive feature information for that remaining first line segment can be determined based on its depth features and planar line features. For example, the second comprehensive feature information may include both the depth features and planar line features of the remaining first line segment. It is worth noting that since the spatial line features corresponding to the first matching line segment are directly extracted from the point cloud data (second line segment), their spatial line features have high accuracy, while the depth features corresponding to the remaining first line segments... This is generated by recovering depth from point cloud data. The depth features corresponding to the remaining first line segments have relatively low accuracy. The first and second comprehensive feature information can be classified and distinguished. The first comprehensive feature information of the first matching line segment is considered as a geometry-based line feature, while the second comprehensive feature information of the remaining first line segments is considered as a color-based line feature. Thus, the planar line features of each remaining first line segment can be obtained. Deep features Spatial Plücker expression (for ease of description, use...) The correspondence (mapping relationship) between (representations) can be expressed by the formula: At the same time, the planar line features of each first matching line segment were also obtained. spatial line characteristics Spatial Plücker expression (for ease of description, use...) The correspondence between (representations) can be expressed by the formula:

[0127] In one possible implementation, the processes involved in the above embodiments, such as line fusion, near-line fusion, determining the first matching line segment that matches the second line segment, determining the first comprehensive feature information of the first matching line segment, and determining the second comprehensive feature information of the remaining first line segments, are all based on first calculating a hypothetical current position and attitude of the vehicle (for convenience, it is called pose). The above processes of determining the first matching line segment that matches the second line segment, determining the first comprehensive feature information of the first matching line segment, and determining the second comprehensive feature information of the remaining first line segments can be performed first based on the hypothetical current pose of the vehicle. Then, based on the first comprehensive feature information of each first matching line segment and the second comprehensive feature information of each remaining first line segment, a more accurate current pose information of the vehicle can be obtained. Optionally, based on the obtained relatively accurate vehicle pose information, the aforementioned processes of line fusion, near-line fusion, determining the first matching line segment that matches the second line segment, determining the first comprehensive feature information of the first matching line segment, and determining the second comprehensive feature information of the remaining first line segments can be performed again. Then, based on the optimized first comprehensive feature information of each first matching line segment and the second comprehensive feature information of each remaining first line segment, a more accurate vehicle pose information can be obtained (for ease of description, the process of obtaining more accurate pose information is referred to as the optimization process). Based on this, the above optimization process can be performed a set number of times to obtain the final accurate pose information. The set number of times can be flexibly set according to needs, and this application does not make a specific limitation on it.

[0128] Specifically, the calculation assumes the vehicle's pose at time j. At this time, the vehicle's pose at the previous time step i can be obtained first. Furthermore, based on the uniform velocity model, the change in vehicle pose during the time interval between time i and time j (also known as the prior value of the pose increment) can be obtained. It can be based on the vehicle's pose at the previous time step i. and pose change To calculate the vehicle's pose at time j. For example, the vehicle's pose at the previous moment i can be used. Add pose change This gives the vehicle's pose at time j. Expressed as a formula:

[0129] In one possible implementation, the straight line expressed using the Plücker expression can be expressed by the formula: Complete the coordinate system transformation, which means combining the spatial line features of the second line segment contained in the point cloud data with the depth features of the remaining first line segment reconstructed from the point cloud data (for ease of description, we will use "spatial line features" and "depth features" interchangeably). (representation) is transformed into features in the camera coordinate system at time j. in, This indicates that the lidar coordinate system is being converted to the camera coordinate system. It can also represent the extrinsic parameters from the camera to the lidar. T = [Q|t]. T represents the pose. For ease of description or calculation, T can be decomposed into an orthogonal matrix representing the pose and a translation vector. Q represents the pose quantity (orthogonal matrix), and t represents the translation vector (also called displacement vector).

[0130] In one possible implementation, the spatial line features of the second line segment and the depth features of the remaining first line segment recovered based on point cloud data are converted into features in the camera coordinate system at time j. Then, the positioning information can be determined based on the first comprehensive feature information of each first matching line segment and the second comprehensive feature information of each remaining first line segment. Optionally, orthonormal representation can be used to parameterize the lines expressed by the Plücker expression to obtain normal vectors (direction vectors), etc. The first comprehensive feature information can then be... Projected onto an image (video), the feature u of the corresponding line segment endpoints in the image. g Perform generalized subtraction By obtaining the distance from the point to the line, etc., the error can be obtained (for ease of description, this is called the first sub-error corresponding to the first comprehensive feature). Simultaneously, the information from each second comprehensive feature can be... Projected onto the image, and corresponding to the feature u of the endpoints of the line segments in the image. c Perform generalized subtraction The distance from the point to the line is obtained, thus yielding the error (for ease of description, referred to as the second sub-error corresponding to the second comprehensive feature). The residual equation construction algorithm determining the positioning information is expressed by the following formula: Wherein, the first sub-error can be The second sub-error can be argmin represents the final localization information (pose) that minimizes the sum of the errors of the first and second sub-errors. Once the localization information is determined using this formula, one pose optimization process can be considered complete.

[0131] Since this application can classify the comprehensive feature information into Geometry-Based first comprehensive feature information and Color-Based second comprehensive feature information, and determine the positioning information based on the first comprehensive feature information and the second comprehensive feature information, the accuracy and robustness of the determined positioning information can be improved.

[0132] In one possible implementation, considering that the spatial line features corresponding to the first matching line segment are directly extracted from the point cloud data (second line segment), their spatial line features have high accuracy, while the depth features corresponding to the remaining first line segment... The depth is recovered from point cloud data. The accuracy of the depth features corresponding to the remaining first line segment is relatively low. To improve the accuracy of the determined positioning information, different certainty weights can be set for the first and second comprehensive feature information. For ease of description, the certainty weight corresponding to the first comprehensive feature information is called the first preset certainty weight. The degree of certainty weight corresponding to the second comprehensive feature information is called the second preset degree of certainty weight, and is expressed as... This indicates that, optionally, the first preset certainty weight of the first comprehensive feature information can be greater than the second preset certainty weight of the second comprehensive feature information. This application does not specifically limit the values ​​of the first preset certainty weight and the second preset certainty weight, and they can be flexibly set according to needs. For example, the first preset certainty weight... When the value is 1, the second preset certainty weight It can be 0.1.

[0133] In one possible implementation, positioning information can be determined based on the product of each first sub-error and the first preset certainty weight (referred to as the first product for convenience) and the product of each second sub-error and the second preset certainty weight (referred to as the second product for convenience). For example, it can be based on the formula: To determine the positioning information, the final positioning information (pose) should minimize the sum of the first and second products; this will not be elaborated further here. The first preset certainty weights are used. The second preset certainty weight is 1. Taking 0.1 as an example, we can use the formula: To determine location information.

[0134] Since this application can not only classify the comprehensive feature information into Geometry-Based first comprehensive feature information and Color-Based second comprehensive feature information, but also configure different certainty weights for the first comprehensive feature information and the second comprehensive feature information respectively, and determine the positioning information based on the first comprehensive feature information, the second comprehensive feature information and the corresponding certainty weights, the accuracy and robustness of the determined positioning information can be improved.

[0135] In one possible implementation, to improve the accuracy of positioning information, the image at time j can be used to determine positioning information not only in conjunction with the point cloud data at time i, but also by combining it with point cloud data added at other statistical times during the vehicle's movement, such as time k (referred to as "other point cloud data" for convenience). Specifically, after obtaining the other point cloud data, each line segment and its spatial line features contained within the other point cloud data can be obtained. The process of obtaining each line segment and its spatial line features contained within the other point cloud data is the same as the process of obtaining the point cloud data at time i described in the above embodiments, and will not be repeated here. For ease of description, the spatial line features of the line segments in the other point cloud data are referred to as... express.

[0136] In one possible implementation, each line segment and its spatial line features contained in other point cloud data can be used. Transform them respectively into line segments and spatial line characteristics at any of the above statistical times, such as time j. The transformed spatial line characteristics are used as follows: This indicates that the transformation method, similar to that described above, can combine the spatial line features of the second line segment contained in the point cloud data with the depth features of the remaining first line segment recovered from the point cloud data. Transformed into features in the camera coordinate system at time j The methods will not be elaborated here.

[0137] In one possible implementation, the transformed line segments and spatial line features can be... Add the transformed line segments and spatial line features to the second line segment and the spatial line features contained in the above point cloud data. As the second line segment and its spatial line features contained in the point cloud data, each second line segment is then projected onto the image to be processed, as described above, to determine the second line segment equation parameters for each projected second line segment. For each second line segment, the standard deviation between the second line segment equation parameters and the first line segment equation parameters of each first line segment is determined. Based on these standard deviations, the process of determining the first matching line segment that matches the second line segment is not elaborated here. Regarding the spatial line features of the second line segment at time k... The spatial line characteristics at time k can be obtained. The matching correspondence between the line features at time j and the line features at time j can be expressed by the formula:

[0138] In one possible implementation, the second line segment may include line segments contained in the point cloud data (also referred to as point cloud frames) at time i and line segments contained in the point cloud frames at time k. The positioning and pose information can be optimized by combining the matching correspondence between the image frame at time j and the point cloud frames at times i and k. For example, taking the first preset certainty weight corresponding to the first comprehensive feature information as 1 and the second preset certainty weight corresponding to the second comprehensive feature information as 0.1, it can be based on the formula: To obtain more accurate positioning information (pose information), in this formula, the first sub-error is... The second sub-error is The positioning information can be determined based on the first product of each first sub-error and a first preset certainty weight, and the second product of each second sub-error and a second preset certainty weight. This formula is similar to the formula already described above. The process for determining the positioning information is similar and will not be described in detail here. This can be considered a completion of an optimization process to obtain more accurate pose information. Similar to the above embodiments, the above optimization process can be repeated a set number of times to obtain the final accurate pose information, which will not be described in detail here.

[0139] Since this application can combine the features of multiple frames, such as image frames, point cloud frames, and point cloud frames from other statistical times, at any statistical time to determine the positioning information at any statistical time (such as the current statistical time), it can improve the accuracy and robustness of the determined positioning information.

[0140] To facilitate understanding, the positioning information determination process provided in this application will be described below through a specific embodiment. (See reference...) Figure 5 , Figure 5 The diagram illustrates a first type of location information determination process provided by some embodiments, which includes the following steps:

[0141] S501: Obtain the image to be processed and the point cloud data of the LiDAR at any given statistical time.

[0142] S502: Obtain each first line segment and its planar line features contained in the image to be processed, and determine the first line segment equation parameters of each first line segment; and obtain each second line segment and its spatial line features contained in the point cloud data, project each second line segment onto the image to be processed, and determine the second line segment equation parameters of each projected second line segment; for each second line segment, determine the standard deviation between the second line segment equation parameters of the second line segment and the first line segment equation parameters of each first line segment.

[0143] S503: For each second line segment, determine the first line segment set associated with that second line segment based on each standard deviation and the set standard deviation threshold.

[0144] S504: Along the same direction, configure the start and end points of each first line segment in the first line segment set; if it is detected that the distance between the end point of any line segment in the first line segment set and the start point of other first line segments is less than a first preset distance threshold, or the distance between the start point of any line segment and the end point of other first line segments is less than a first preset distance threshold, then merge that line segment with the other first line segments into one first line segment; and / or,

[0145] If it is detected that the distance between the starting point of any line segment in the first line segment set and the starting point of other first line segments is less than a second set distance threshold, or the distance between the ending point of any line segment and the ending point of other first line segments is less than a second set distance threshold, then the any line segment is merged with the other first line segments into a single first line segment.

[0146] S505: For each second line segment, based on the distance between each first line segment in the corresponding first line segment set and the projected second line segment, determine the first matching line segment that matches the second line segment.

[0147] S506: For each second line segment, based on the spatial line features of the second line segment and the planar line features of the obtained first matching line segment, determine the first comprehensive feature information of the first matching line segment; and for each remaining first line segment in each first line segment contained in the image to be processed, excluding each set of first line segments, determine the depth features of the remaining first line segment based on point cloud data; based on the depth features and the planar line features of the remaining first line segment, determine the second comprehensive feature information of the remaining first line segment. Based on the first comprehensive feature information of each first matching line segment, the second comprehensive feature information of each remaining first line segment, and the set residual equation construction algorithm, determine the first sub-error corresponding to each first comprehensive feature information, and the second sub-error corresponding to each second comprehensive feature information; based on each first sub-error and each second sub-error, determine the positioning information at any of the above statistical times.

[0148] In addition, after completing S506, steps S502-S506 can be repeated to improve the accuracy and robustness of the determined positioning information, which will not be elaborated here.

[0149] To facilitate understanding, the positioning information determination process provided in this application will be further illustrated below through a specific embodiment. (See reference...) Figure 6 , Figure 6 The diagram illustrates a second location information determination process provided in some embodiments, which includes the following steps:

[0150] S601: Obtain the image to be processed and the point cloud data of the LiDAR at any given statistical time.

[0151] S602: Obtain each first line segment and its planar line features contained in the image to be processed, and determine the first line segment equation parameters of each first line segment; and obtain each second line segment and its spatial line features contained in the point cloud data, project each second line segment onto the image to be processed, and determine the second line segment equation parameters of each projected second line segment; for each second line segment, determine the standard deviation between the second line segment equation parameters of the second line segment and the first line segment equation parameters of each first line segment.

[0152] S603: For each second line segment, based on each standard deviation and the set standard deviation threshold, determine the first line segment set associated with the second line segment.

[0153] S604: For each second line segment, based on the distance between each first line segment in the corresponding first line segment set and the projected second line segment, determine the first matching line segment that matches the second line segment.

[0154] S605: Based on the start and end points of each first line segment in the set of first line segments corresponding to each second line segment, configured along the same direction, if the distance between the end point of the first matching line segment and the start point of another first line segment is less than a first preset distance threshold, or the distance between the start point of the first matching line segment and the end point of another first line segment is less than a first preset distance threshold, then the first matching line segment is merged with the other first line segments, and the first matching line segment is updated according to the merged line segment; and / or,

[0155] If the distance between the starting point of the first matching line segment and the starting point of other first line segments is less than a second set distance threshold, or the distance between the ending point of the first matching line segment and the ending point of other first line segments is less than a second set distance threshold, then the first matching line segment is merged with the other first line segment, and the first matching line segment is updated according to the merged line segment.

[0156] S606: For each second line segment, based on the spatial line features of the second line segment and the planar line features of the obtained first matching line segment, determine the first comprehensive feature information of the first matching line segment; and for each remaining first line segment in each first line segment contained in the image to be processed, excluding each set of first line segments, determine the depth features of the remaining first line segment based on point cloud data; based on the depth features and the planar line features of the remaining first line segment, determine the second comprehensive feature information of the remaining first line segment. Based on the first comprehensive feature information of each first matching line segment, the second comprehensive feature information of each remaining first line segment, and the set residual equation construction algorithm, determine the first sub-error corresponding to each first comprehensive feature information and the second sub-error corresponding to each second comprehensive feature information; based on each first sub-error and each second sub-error, determine the positioning information at any of the above statistical times.

[0157] Furthermore, after completing S606, steps S602-S606 can be repeated to improve the accuracy and robustness of the determined positioning information, which will not be elaborated further here. Specifically, the operation of determining the start and end points of each first segment in the first segment set corresponding to each second line segment, configured along the same direction, can be performed immediately after determining the first segment set associated with the second line segment (step S603), or it can be performed after determining the first matching line segment that matches the second line segment (step S604). This can be set according to requirements, and this application does not impose specific limitations on it.

[0158] To facilitate understanding, the positioning information determination process provided in this application will be further illustrated below through a specific embodiment. (See reference...) Figure 7 , Figure 7 The diagram illustrates a third location information determination process provided in some embodiments, which includes the following steps:

[0159] S701: Obtain the image to be processed and the point cloud data of the LiDAR at any given statistical time.

[0160] S702: Obtain each first line segment and its planar line features contained in the image to be processed, and determine the first line segment equation parameters for each first line segment; obtain each second line segment and its spatial line features contained in the point cloud data; and obtain each line segment and its spatial line features contained in other point cloud data of the LiDAR at other statistical times, transform each line segment and its spatial line features contained in the other point cloud data into line segments and spatial line features at any of the above statistical times, and use the transformed line segments and their spatial line features as the second line segments and their spatial line features contained in the point cloud data at any of the above statistical times. Project each second line segment onto the image to be processed, and determine the second line segment equation parameters for each projected second line segment; for each second line segment, determine the standard deviation between the second line segment equation parameters of the second line segment and the first line segment equation parameters of each first line segment.

[0161] S703: For each second line segment, determine the set of first line segments associated with that second line segment based on each standard deviation and the set standard deviation threshold.

[0162] S704: Along the same direction, configure the start and end points of each first line segment in the first line segment set; if it is detected that the distance between the end point of any line segment in the first line segment set and the start point of other first line segments is less than a first preset distance threshold, or the distance between the start point of any line segment and the end point of other first line segments is less than a first preset distance threshold, then merge that line segment with the other first line segments into one first line segment; and / or,

[0163] If it is detected that the distance between the starting point of any line segment in the first line segment set and the starting point of other first line segments is less than a second set distance threshold, or the distance between the ending point of any line segment and the ending point of other first line segments is less than a second set distance threshold, then the any line segment is merged with the other first line segments into a single first line segment.

[0164] S705: For each second line segment, based on the distance between each first line segment in the corresponding first line segment set and the projected second line segment, determine the first matching line segment that matches the second line segment.

[0165] S706: For each second line segment, based on the spatial line features of the second line segment and the planar line features of the obtained first matching line segment, determine the first comprehensive feature information of the first matching line segment; and for each remaining first line segment in each first line segment contained in the image to be processed, excluding each set of first line segments, determine the depth features of the remaining first line segment based on point cloud data; based on the depth features and the planar line features of the remaining first line segment, determine the second comprehensive feature information of the remaining first line segment. Based on the first comprehensive feature information of each first matching line segment, the second comprehensive feature information of each remaining first line segment, and the set residual equation construction algorithm, determine the first sub-error corresponding to each first comprehensive feature information, and the second sub-error corresponding to each second comprehensive feature information; based on each first sub-error and each second sub-error, determine the positioning information at any of the above statistical times.

[0166] In addition, after completing S706, steps S702-S706 can be repeated to improve the accuracy and robustness of the determined positioning information, which will not be elaborated here.

[0167] Based on the same technical concept, this application also provides a vehicle, which may include the positioning information determining device in any of the above embodiments, and will not be described again here.

[0168] Based on the same technical concept, this application also provides a method for determining positioning information. This method can be applied to vehicles, electronic devices such as terminals and servers, and robots; this application does not specifically limit its application to these applications. (See also...) Figure 8 , Figure 8 The diagram illustrates a fourth location information determination process provided in some embodiments, which includes the following steps:

[0169] S801: Obtain the image to be processed and the point cloud data of the LiDAR at any given statistical time.

[0170] S802: Obtain each first line segment and the planar line features of each first line segment contained in the image to be processed, and determine the first line segment equation parameters of each first line segment; and obtain each second line segment and the spatial line features of each second line segment contained in the point cloud data, project each second line segment onto the image to be processed, and determine the second line segment equation parameters of each projected second line segment.

[0171] S803: For each second line segment, determine the standard deviation between the second line segment equation parameters of the second line segment and the first line segment equation parameters of each first line segment; based on each standard deviation, determine the first matching line segment that matches the second line segment; based on the spatial line features of the second line segment and the obtained planar line features of the first matching line segment, determine the first comprehensive feature information of the first matching line segment; based on the first comprehensive feature information of each first matching line segment, determine the positioning information.

[0172] In one possible implementation, determining the first matching line segment based on each standard deviation includes:

[0173] Based on the standard deviations and the set standard deviation thresholds, the first set of line segments associated with the second line segment is determined.

[0174] Based on the distance between each first line segment in the first line segment set and the projected second line segment, a first matching line segment that matches the second line segment is determined.

[0175] In one possible implementation, after determining the first matching line segment that matches the second line segment, and before determining the first comprehensive feature information of the first matching line segment based on the spatial line features of the second line segment and the obtained planar line features of the first matching line segment, the method further includes:

[0176] Along the same direction, configure the starting point and ending point of each first line segment in the first line segment set respectively;

[0177] If it is detected that the distance between the endpoint of the first matching line segment and the starting point of another first line segment is less than a first preset distance threshold, or the distance between the starting point of the first matching line segment and the endpoint of another first line segment is less than a first preset distance threshold, then the first matching line segment is merged with the other first line segment, and the first matching line segment is updated based on the merged line segment; and / or,

[0178] If the distance between the starting point of the first matching line segment and the starting point of another first line segment is less than a second set distance threshold, or the distance between the ending point of the first matching line segment and the ending point of another first line segment is less than a second set distance threshold, then the first matching line segment is merged with the other first line segment, and the first matching line segment is updated according to the merged line segment.

[0179] In one possible implementation, after determining the first set of line segments associated with the second line segment and before determining the first matching line segment matching the second line segment based on the distance between each first line segment in the first set and the projected second line segment, the method further includes:

[0180] Along the same direction, configure the starting point and ending point of each first line segment in the first line segment set respectively;

[0181] If it is detected that the distance between the endpoint of any line segment in the first line segment set and the starting point of any other first line segment is less than a first preset distance threshold, or the distance between the starting point of any line segment and the endpoint of any other first line segment is less than the first preset distance threshold, then that line segment is merged with the other first line segments into a single first line segment; and / or,

[0182] If it is detected that the distance between the starting point of any line segment in the first line segment set and the starting point of other first line segments is less than a second set distance threshold, or the distance between the ending point of any line segment and the ending point of other first line segments is less than a second set distance threshold, then the any line segment is merged with the other first line segments into a first line segment.

[0183] In one possible implementation, after determining the first set of line segments associated with the second line segment and before determining the positioning information based on the first comprehensive feature information of each first matching line segment, the method further includes:

[0184] For each remaining first line segment other than each set of first line segments in each first line segment contained in the image to be processed, the depth feature of the remaining first line segment is determined based on the point cloud data; and the second comprehensive feature information of the remaining first line segment is determined based on the depth feature and the planar line feature of the remaining first line segment.

[0185] The determination of positioning information based on the first comprehensive feature information of each first matching line segment includes:

[0186] Based on the first comprehensive feature information of each first matching line segment, the second comprehensive feature information of each remaining first line segment, and the established residual equation, an algorithm is constructed to determine the first sub-error corresponding to each first comprehensive feature information and the second sub-error corresponding to each second comprehensive feature information; based on each first sub-error and each second sub-error, the positioning information is determined.

[0187] In one possible implementation, determining the positioning information based on each first sub-error and each second sub-error includes:

[0188] Obtain the first preset certainty weight corresponding to the first preset comprehensive feature information and the second preset certainty weight corresponding to the second comprehensive feature information;

[0189] The positioning information is determined based on the first product of each first sub-error and the first preset certainty weight, and the second product of each second sub-error and the second preset certainty weight.

[0190] In one possible implementation, after obtaining the image to be processed and the point cloud data of the lidar at any statistical moment, and before projecting each second line segment onto the image to be processed and determining the second line segment equation parameters of each projected second line segment, the method further includes:

[0191] Obtain each line segment and its spatial line characteristics from other point cloud data of the lidar at other statistical times;

[0192] Each line segment and its spatial line feature contained in the other point cloud data are transformed into the line segment and spatial line feature at any statistical time.

[0193] The transformed line segments and spatial line features of the line segments contained in the other point cloud data are used as the second line segments and spatial line features of the second line segments contained in the point cloud data.

[0194] Based on the same technical concept, this application also provides a computer program product, which includes a computer program / instruction that, when executed by a processor, implements the positioning information determination method as described above.

[0195] The methods in this application can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in this application are performed, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, network equipment, user equipment, core network equipment, OAM, or other programmable devices.

[0196] The computer program or instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions may be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; or an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0197] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0198] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0199] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0200] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0201] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A positioning information determination apparatus characterized by comprising: The device includes a processing module, wherein: The processing module is used to obtain the image to be processed and the point cloud data of the lidar at any statistical time. Obtain each first line segment and its planar line features contained in the image to be processed, and determine the first line segment equation parameters of each first line segment; and obtain each second line segment and its spatial line features contained in the point cloud data, project each second line segment onto the image to be processed, and determine the second line segment equation parameters of each projected second line segment; For each second line segment, determine the standard deviation between the second line segment equation parameters of the second line segment and the first line segment equation parameters of each first line segment. Based on each standard deviation, determine the first matching line segment that matches the second line segment. Based on the spatial line features of the second line segment and the obtained planar line features of the first matching line segment, determine the first comprehensive feature information of the first matching line segment. Based on the first comprehensive feature information of each first matching line segment, determine the positioning information. Specifically, the processing module is used for: Based on the standard deviations and the set standard deviation thresholds, determine the first set of line segments associated with the second line segment; Based on the distance between each first line segment in the first line segment set and the projected second line segment, a first matching line segment that matches the second line segment is determined. The processing module is further configured to: For each remaining first line segment other than each set of first line segments in the image to be processed, the depth feature of the remaining first line segment is determined based on the point cloud data; and the second comprehensive feature information of the remaining first line segment is determined based on the depth feature and the planar line feature of the remaining first line segment. Based on the first comprehensive feature information of each first matching line segment, the second comprehensive feature information of each remaining first line segment, and the set residual equation, an algorithm is constructed to determine the first sub-error corresponding to each first comprehensive feature information and the second sub-error corresponding to each second comprehensive feature information; based on each first sub-error and each second sub-error, the positioning information is determined.

2. The apparatus according to claim 1, characterized in that, The processing module is further configured to: Along the same direction, configure the starting point and ending point of each first line segment in the first line segment set respectively; If it is detected that the distance between the endpoint of the first matching line segment and the starting point of another first line segment is less than a first preset distance threshold, or the distance between the starting point of the first matching line segment and the endpoint of another first line segment is less than a first preset distance threshold, then the first matching line segment is merged with the other first line segment, and the first matching line segment is updated based on the merged line segment; and / or, If the distance between the starting point of the first matching line segment and the starting point of another first line segment is less than a second set distance threshold, or the distance between the ending point of the first matching line segment and the ending point of another first line segment is less than a second set distance threshold, then the first matching line segment is merged with the other first line segment, and the first matching line segment is updated according to the merged line segment.

3. The apparatus according to claim 1, characterized in that, The processing module is further configured to: Along the same direction, configure the starting point and ending point of each first line segment in the first line segment set respectively; If it is identified that the distance between the end point of any line segment in the first line segment set and the starting point of other first line segments is less than a first set distance threshold, or the distance between the starting point of any line segment and the end point of other first line segments is less than a first set distance threshold, then the any line segment is merged with the other first line segments into a first line segment. And / or, If it is detected that the distance between the starting point of any line segment in the first line segment set and the starting point of other first line segments is less than a second set distance threshold, or the distance between the ending point of any line segment and the ending point of other first line segments is less than a second set distance threshold, then the any line segment is merged with the other first line segments into a first line segment.

4. The apparatus according to claim 1, characterized in that, The processing module is specifically used for: Obtain the first preset certainty weight corresponding to the first preset comprehensive feature information and the second preset certainty weight corresponding to the second comprehensive feature information; The positioning information is determined based on the first product of each first sub-error and the first preset certainty weight, and the second product of each second sub-error and the second preset certainty weight.

5. The apparatus according to any one of claims 1-4, characterized in that, The processing module is also used to: obtain each line segment and the spatial line features of each line segment contained in other point cloud data of the lidar at other statistical times; Each line segment and its spatial line feature contained in the other point cloud data are transformed into the line segment and spatial line feature at any statistical time. The transformed line segments and spatial line features of the line segments contained in the other point cloud data are used as the second line segments and spatial line features of the second line segments contained in the point cloud data.

6. A vehicle, characterized in that, The vehicle includes a positioning information determination device as described in any one of claims 1-5.

7. A method for determining location information, characterized in that, The method includes: Obtain the image to be processed and the point cloud data of the LiDAR at any given statistical time. Obtain each first line segment and its planar line features contained in the image to be processed, and determine the first line segment equation parameters of each first line segment; and obtain each second line segment and its spatial line features contained in the point cloud data, project each second line segment onto the image to be processed, and determine the second line segment equation parameters of each projected second line segment; For each second line segment, determine the standard deviation between the second line segment equation parameters of the second line segment and the first line segment equation parameters of each first line segment. Based on each standard deviation, determine the first matching line segment that matches the second line segment. Based on the spatial line features of the second line segment and the obtained planar line features of the first matching line segment, determine the first comprehensive feature information of the first matching line segment. Based on the first comprehensive feature information of each first matching line segment, determine the positioning information. The step of determining the first matching line segment based on each standard deviation includes: Based on the standard deviations and the set standard deviation thresholds, determine the first set of line segments associated with the second line segment; Based on the distance between each first line segment in the first line segment set and the projected second line segment, a first matching line segment that matches the second line segment is determined. After determining the first set of line segments associated with the second line segment, and before determining the positioning information based on the first comprehensive feature information of each first matching line segment, the method further includes: For each remaining first line segment other than each set of first line segments in the image to be processed, the depth feature of the remaining first line segment is determined based on the point cloud data; and the second comprehensive feature information of the remaining first line segment is determined based on the depth feature and the planar line feature of the remaining first line segment. The determination of positioning information based on the first comprehensive feature information of each first matching line segment includes: Based on the first comprehensive feature information of each first matching line segment, the second comprehensive feature information of each remaining first line segment, and the set residual equation, an algorithm is constructed to determine the first sub-error corresponding to each first comprehensive feature information and the second sub-error corresponding to each second comprehensive feature information; based on each first sub-error and each second sub-error, the positioning information is determined.

8. A computer program product comprising a computer program / instructions, characterized in that, The method of claim 7 is implemented when the computer program / instruction processor is executed.

Images