Method and system for two-stage camera-to-lidar alignment

Abstract

Systems and methods for generating alignment parameters for processing data associated with a vehicle. In one embodiment, a method includes receiving image data associated with an environment of the vehicle; receiving lidar data associated with the environment of the vehicle; processing, by a processor, the image data to determine data points associated with at least one vehicle identified within the image data; processing, by the processor, the lidar data to determine data points associated with at least one vehicle identified within the lidar data; selectively storing the data points in a data buffer based on at least one condition associated with a quality of the data points; processing, by the processor, the data points in the data buffer with a joint analysis method to generate alignment parameters between the lidar and the camera; and processing future data based on the alignment parameters.

Description

Technical Field

[0001] This technical field generally relates to computer vision, and more specifically to methods and systems for determining camera and lidar alignment information used in computer vision in vehicles. Background Technology

[0002] Modern vehicles are typically equipped with one or more optical cameras configured to provide image data that can be displayed to vehicle occupants and used to determine elements of the vehicle's environment. The image data can depict a virtual scene of the environment surrounding the vehicle. This virtual scene can be generated based on data from one or more cameras and data from one or more other sensors, such as lidar or radar. For example, the image data may be taken from different image sources located at different locations around the vehicle or from a single source rotated relative to the vehicle. Based on alignment information, the image data is evaluated and merged into a single viewpoint, such as a bird's-eye view. The methods for determining the alignment information can be computationally intensive, especially if performed in real time.

[0003] Therefore, it is desirable to provide an improved system and method for determining camera-LiDAR alignment information. Furthermore, other desirable features and characteristics of the invention will become apparent from the following detailed description and appended claims, in conjunction with the accompanying drawings and the foregoing technical and background information. Summary of the Invention

[0004] Systems and methods for generating alignment parameters for processing vehicle-related data. In one embodiment, a method includes: receiving image data associated with an environment of the vehicle; receiving lidar data associated with the environment of the vehicle; processing the image data by a processor to determine data points associated with at least one vehicle identified within the image data; processing the lidar data by the processor to determine data points associated with at least one vehicle identified within the lidar data; selectively storing the data points in a data buffer based on at least one condition associated with the quality of the data points; processing the data points in the data buffer by the processor using a joint analysis method to generate alignment parameters between the lidar and the camera; and processing future data based on the alignment parameters.

[0005] In various embodiments, processing the image data includes: processing the image data using machine learning methods to identify the edges and contours of vehicles within the image; and applying an inverse distance transform to the edges and contours.

[0006] In various embodiments, processing LiDAR data includes: applying a clustering method to detect objects within the scene; removing objects based on a filtering method; projecting the vehicle data into a coordinate system associated with the image data; and applying a convex hull method to the projected vehicle data to identify data points associated with the edges of the vehicles.

[0007] In various embodiments, the condition is associated with the handling of the vehicle.

[0008] In various embodiments, the condition is associated with the number of data points.

[0009] In various embodiments, the condition is associated with the distribution of data points between the left and right sides of the image data.

[0010] In various embodiments, the condition is associated with a road structure similarity index measure between a first image and a second image of the image data.

[0011] In various embodiments, the condition is associated with the number of vehicles in the image of the image data.

[0012] In various embodiments, the condition is associated with the vehicle-to-image ratio.

[0013] In various embodiments, the joint analysis method includes: generating multiple three-dimensional cube spaces with multiple random perturbation points; evaluating data point pairs of each perturbation point in each of the three-dimensional cube spaces to determine a score; and selecting a transformation matrix associated with a score greater than a threshold, wherein the alignment parameters are determined based on the transformation matrix.

[0014] In another embodiment, a computer-implemented system is provided for generating alignment parameters for processing vehicle-related data. The system includes: a data storage element including computer-readable instructions; and a processor configured to execute the computer-readable instructions, which control the processor to perform operations including: receiving image data associated with the vehicle's environment; receiving lidar data associated with the vehicle's environment; processing the image data to determine data points associated with at least one vehicle identified within the image data; processing the lidar data to determine data points associated with at least one vehicle identified within the lidar data; selectively storing the data points in a data buffer based on at least one condition associated with the quality of the data points; processing the data points in the data buffer using a joint analysis method to generate alignment parameters between the lidar and a camera; and processing future data based on the alignment parameters.

[0015] In various embodiments, processing the image data includes: processing the image data using machine learning methods to identify the edges and contours of vehicles within the image; and applying an inverse distance transform to the edges and contours.

[0016] In various embodiments, processing LiDAR data includes: applying a clustering method to detect objects within the scene; removing objects based on a filtering method; projecting the vehicle data into a coordinate system associated with the image data; and applying a convex hull method to the projected vehicle data to identify data points associated with the edges of the vehicles.

[0017] In various embodiments, the condition is associated with the handling of the vehicle.

[0018] In various embodiments, the condition is associated with the number of data points.

[0019] In various embodiments, the condition is associated with the distribution of data points between the left and right sides of the image data.

[0020] In various embodiments, the condition is associated with a road structure similarity index measure between a first image and a second image of the image data.

[0021] In various embodiments, the condition is associated with the number of vehicles in the image of the image data.

[0022] In various embodiments, the condition is associated with the vehicle image ratio.

[0023] In various embodiments, the joint analysis method includes: generating multiple three-dimensional cube spaces with multiple random perturbation points; evaluating data point pairs of each perturbation point in each of the three-dimensional cube spaces to determine a score; and selecting a transformation matrix associated with a score greater than a threshold, wherein the alignment parameters are determined based on the transformation matrix.

[0024] The present invention also includes the following solutions:

[0025] Option 1. A method for generating alignment parameters for processing data associated with a vehicle, the method comprising:

[0026] Receive image data associated with the environment of the vehicle;

[0027] Receive lidar data associated with the vehicle's environment;

[0028] The image data is processed by a processor to determine data points associated with at least one vehicle identified within the image data;

[0029] The processor processes the lidar data to determine data points associated with at least one vehicle identified within the lidar data;

[0030] Based on at least one condition associated with the quality of the data point, the data point is selectively stored in the data buffer;

[0031] The processor processes the data points in the data buffer using a joint analysis method to generate alignment parameters between the lidar and the camera; and

[0032] Future data is processed based on the alignment parameters.

[0033] Solution 2. The method according to Solution 1, wherein processing the image data includes:

[0034] The image data is processed using machine learning methods to identify the edges and contours of vehicles within the image; and

[0035] Apply the inverse distance transform to the edges and contours.

[0036] Option 3. The method according to Option 1, wherein processing the lidar data includes:

[0037] Clustering methods are used to detect objects within a scene;

[0038] Objects are removed based on filtering methods;

[0039] Projecting the vehicle data into a coordinate system associated with the image data; and

[0040] The convex hull method is applied to the projected vehicle data to identify data points associated with the edges of the vehicle.

[0041] Option 4. The method according to Option 1, wherein the condition is associated with the operation of the vehicle.

[0042] Option 5. The method according to Option 1, wherein the condition is associated with the number of data points.

[0043] Option 6. The method according to Option 1, wherein the condition is associated with the distribution of the data points between the left and right sides of the image data.

[0044] Option 7. According to the method of Option 1, wherein the condition is associated with a road structure similarity index measure between a first image and a second image of the image data.

[0045] Option 8. The method according to Option 1, wherein the condition is associated with the number of vehicles in the image of the image data.

[0046] Option 9. The method according to Option 1, wherein the condition is associated with the vehicle image ratio.

[0047] Option 10. The method according to Option 1, wherein the joint analysis method comprises:

[0048] Generate multiple three-dimensional cubic spaces with multiple random perturbation points;

[0049] Evaluate the data point pairs of each perturbation point in each of the three-dimensional cubic spaces to determine a score; and

[0050] Select a transformation matrix associated with scores greater than a threshold, wherein the alignment parameters are determined based on the transformation matrix.

[0051] Option 11. A computer-implemented system for generating alignment parameters for processing vehicle-related data, the system comprising:

[0052] Data storage element, including computer-readable instructions; and

[0053] A processor configured to execute the computer-readable instructions, the computer-readable instructions controlling the processor to perform operations, the operations including:

[0054] Receive image data associated with the environment of the vehicle;

[0055] Receive lidar data associated with the vehicle's environment;

[0056] The image data is processed to determine data points associated with at least one vehicle identified within the image data;

[0057] The lidar data is processed to determine data points associated with at least one vehicle identified within the lidar data;

[0058] Based on at least one condition associated with the quality of the data point, the data point is selectively stored in the data buffer;

[0059] The data points in the data buffer are processed using a joint analysis method to generate alignment parameters between the lidar and the camera; and

[0060] Future data is processed based on the alignment parameters.

[0061] Solution 12. The system according to Solution 11, wherein processing the image data includes:

[0062] The image data is processed using machine learning methods to identify the edges and contours of vehicles within the image; and

[0063] Apply the inverse distance transform to the edges and contours.

[0064] Option 13. The system according to Option 11, wherein processing the lidar data includes:

[0065] Clustering methods are used to detect objects within a scene;

[0066] Objects are removed based on filtering methods;

[0067] Projecting the vehicle data into a coordinate system associated with the image data; and

[0068] The convex hull method is applied to the projected vehicle data to identify data points associated with the edges of the vehicle.

[0069] Option 14. The system according to Option 11, wherein the condition is associated with the operation of the vehicle.

[0070] Option 15. The system according to Option 11, wherein the condition is associated with the number of data points.

[0071] Option 16. The system according to Option 11, wherein the condition is associated with the distribution of the data points between the left and right sides of the image of the image data.

[0072] Option 17. The system according to Option 11, wherein the condition is associated with a road structure similarity index measure between a first image and a second image of the image data.

[0073] Option 18. The system according to Option 11, wherein the condition is associated with the number of vehicles in the image of the image data.

[0074] Option 19. The system according to Option 11, wherein the condition is associated with the vehicle image ratio.

[0075] Option 20. The system according to Option 11, wherein the joint analysis method comprises:

[0076] Generate multiple three-dimensional cubic spaces with multiple random perturbation points;

[0077] Evaluate the data point pairs of each perturbation point in each of the three-dimensional cubic spaces to determine a score; and

[0078] Select a transformation matrix associated with scores greater than a threshold, wherein the alignment parameters are determined based on the transformation matrix. Attached Figure Description

[0079] Exemplary embodiments will now be described in conjunction with the following accompanying drawings, wherein the same reference numerals denote the same elements, and wherein:

[0080] Figure 1 This is a schematic diagram of a vehicle having a controller that implements functions for generating alignment information, according to various embodiments;

[0081] Figure 2 This is a data flow diagram illustrating the controller of a vehicle according to various embodiments; and

[0082] Figure 3 and Figure 4 This is a flowchart illustrating a method performed by a vehicle and a controller according to various embodiments. Detailed Implementation

[0083] The following specific embodiments are exemplary in nature and are not intended to limit application and use. Furthermore, it is not intended to be construed as being bound by any express or implied theory presented in the foregoing technical fields, background art, summary of the invention, or the following specific embodiments. As used herein, the term "module" individually or in any combination refers to any hardware, software, firmware, electronic control components, processing logic, and / or processor device, including but not limited to: application-specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated, or grouped) and memories executing one or more software or firmware programs, combinational logic circuits, and / or other suitable components providing the aforementioned functionality.

[0084] The embodiments of this disclosure can be described herein in terms of functional and / or logical block components and various processing steps. It should be understood that such block components can be implemented by any number of hardware, software, and / or firmware components configured to perform specified functions. For example, embodiments of this disclosure can employ various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, lookup tables, etc., which can perform various functions under the control of one or more microprocessors or other control devices. Furthermore, those skilled in the art will understand that embodiments of this disclosure can be practiced in conjunction with any number of systems, and the systems described herein are merely exemplary embodiments of this disclosure.

[0085] For the sake of brevity, conventional techniques related to signal processing, data transmission, signaling, control, and other functional aspects of the system (as well as the various operating components of the system) may not be described in detail herein. Furthermore, the connecting lines shown in the figures included herein are intended to represent example functional relationships and / or physical couplings between elements. It should be noted that many alternative or additional functional relationships or physical connections may exist in the embodiments of this disclosure.

[0086] refer to Figure 1 The illustration shows a vehicle 10 having a system 100 according to various embodiments. Typically, system 100 determines alignment information between different data sources of the vehicle. The alignment information can be used, for example, to process and / or generate image data from multiple data sources. The generated image data can be used, for example, to display a surround view of the vehicle's environment on a display 50 of vehicle 10. As will be understood, the alignment data can be used for other purposes, including but not limited to controlling the vehicle, and is not limited to the display example.

[0087] like Figure 1 As shown, vehicle 10 typically includes a chassis 12, a body 14, front wheels 16, and rear wheels 18. The body 14 is mounted on the chassis 12 and substantially surrounds the components of vehicle 10. The body 14 and chassis 12 may together form a frame. Wheels 16 and 18 are rotatably coupled to the chassis 12 near respective corners of the body 14.

[0088] In various embodiments, vehicle 10 is an autonomous vehicle. An autonomous vehicle is, for example, a vehicle automatically controlled to transport passengers from one location to another. In the illustrated embodiment, vehicle 10 is described as a passenger car, but it should be understood that any other means of transportation, including motorcycles, trucks, sports utility vehicles (SUVs), recreational vehicles (RVs), boats, aircraft, etc., may also be used. In exemplary embodiments, the autonomous vehicle is a Level 2 or higher automation system. A Level 2 automation system indicates “partial automation.” However, in other embodiments, the autonomous vehicle may be a so-called Level 3, Level 4, or Level 5 automation system. A Level 3 automation system indicates situational automation. A Level 4 system indicates “high automation,” which refers to the driving mode-specific performance of the automated driving system for all aspects of a dynamic driving task, even when a human driver does not properly respond to intervention requests. A Level 5 system (the fifth level of the system) indicates “full automation,” which refers to the full-time execution of the automated driving system for all aspects of a dynamic driving task under all road and environmental conditions that a human driver can manage.

[0089] However, it should be understood that vehicle 10 may also be a conventional vehicle without any autonomous driving capabilities. Vehicle 10 may implement the functions and methods for generating alignment information according to this disclosure.

[0090] As shown in the figure, vehicle 10 typically includes a propulsion system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, at least one controller 34, and a communication system 36. In various embodiments, the propulsion system 20 may include an internal combustion engine, an electric motor such as a traction motor, a fuel cell propulsion system, and / or combinations thereof. The transmission system 22 is configured to transmit power from the propulsion system 20 to wheels 16 and 18 according to a selectable speed ratio. According to various embodiments, the transmission system 22 may include a stepped-ratio automatic transmission, a continuously variable transmission (CVT), a manual transmission, or any other suitable transmission.

[0091] Braking system 26 is configured to provide braking torque to wheels 16 and 18. In various embodiments, braking system 26 may include friction brakes, brake-by-wire brakes, regenerative braking systems such as electric motors, and / or other suitable braking systems. Steering system 24 affects the position of wheels 16 and 18. Although described for illustrative purposes as including a steering wheel, in some embodiments conceived within the scope of this disclosure, steering system 24 may not include a steering wheel.

[0092] Sensor system 28 includes one or more sensing devices 40a-40n that sense observable conditions of the external and / or internal environment of vehicle 10. Sensing devices 40a-40n may include, but are not limited to, radar, lidar, global positioning system (GPS), optical cameras, thermal cameras, ultrasonic sensors, and / or other sensors. Sensing devices 40a-40n are also configured to sense observable conditions of vehicle 10. Sensing devices 40a-40n may include, but are not limited to, speed sensors, position sensors, inertial measurement sensors, temperature sensors, pressure sensors, etc.

[0093] The actuator system 30 includes one or more actuator devices 42a-42n that control one or more vehicle features, such as, but not limited to, the propulsion system 20, the transmission system 22, the steering system 24, and the braking system 26. In various embodiments, the vehicle features may also include interior and / or exterior vehicle features, such as, but not limited to, doors, trunk, and cabin features, such as air, music, lighting, etc. (not numbered).

[0094] Communication system 36 is configured to wirelessly transmit information to and from other entities 48, said entities being such as, but not limited to, other vehicles (“V2V” communication), infrastructure (“V2I” communication), remote systems and / or personal devices (regarding...). Figure 2 (Described in more detail). In an exemplary embodiment, communication system 36 is a wireless communication system configured to communicate using the IEEE 802.11 standard or via a wireless local area network (WLAN) using cellular data communication. However, additional or alternative communication methods, such as dedicated short-range communication (DSRC) channels, are also considered within the scope of this disclosure. A DSRC channel refers to a unidirectional or bidirectional short-to-medium-range wireless communication channel specifically designed for automotive use, along with the corresponding set of protocols and standards.

[0095] Data storage device 32 stores data for functions used to automatically control vehicle 10. In various embodiments, data storage device 32 stores a defined map of the navigable environment. The defined map may include various data in addition to the road data associated with it, including altitude, climate, lighting, etc. In various embodiments, the defined map may be generated by a remote system (see reference 10). Figure 2 (To be described in more detail) Predefined and obtained from it. For example, the defined map may be combined by a remote system and transmitted to vehicle 10 (wirelessly and / or via wire) and stored in data storage device 32. As can be understood, data storage device 32 may be part of controller 34, separate from controller 34, or part of controller 34 and separate system.

[0096] The controller 34 includes at least one processor 44 and a computer-readable storage device or medium 46. The processor 44 may be any custom or commercially available processor, central processing unit (CPU), graphics processing unit (GPU), auxiliary processor among several processors associated with the controller 34, semiconductor-based microprocessor (in the form of a microchip or chipset), macroprocessor, any combination thereof, or any means generally used for executing instructions. The computer-readable storage device or medium 46 may include, for example, volatile and non-volatile storage devices such as read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is persistent or non-volatile memory that can be used to store various operational variables when the processor 44 is powered off. The computer-readable storage device or medium 46 may be implemented using any of a variety of known storage devices, such as PROM (programmable read-only memory), EPROM (electrical PROM), EEPROM (electrically erasable PROM), flash memory, or any other electrical, magnetic, optical, or combined storage device capable of storing data, some of which represents executable instructions used by the controller 34 in controlling and performing functions of the vehicle 10.

[0097] The instructions may include one or more separate programs, each including an ordered list of executable instructions for implementing logical functions. When executed by processor 44, the instructions receive and process signals from sensor system 28, execute logic, calculations, methods, and / or algorithms for automatically controlling components of vehicle 10, and generate control signals based on the logic, calculations, methods, and / or algorithms to actuator system 30 for automatically controlling components of vehicle 10. Although Figure 1 Only one controller 34 is shown, but embodiments of vehicle 10 may include any number of controllers 34 that communicate via any suitable communication medium or combination of communication media and cooperate to process sensor signals, perform logic, calculations, methods and / or algorithms, and generate control signals to automatically control the features of vehicle 10.

[0098] In various embodiments, one or more instructions of controller 34 are implemented in system 100, and when executed by processor 44, process image data from at least one optical camera of sensor system 28 and image data from at least one lidar of sensor system 28 to extract data pairs. When executed by processor 44, the instructions use the data pairs to determine camera-to-lidar alignment information. The camera alignment information is then used to combine the image data for display within vehicle 10 or for other purposes.

[0099] It will be understood that controller 34 can otherwise interact with... Figure 1The embodiments shown differ. For example, controller 34 may be coupled to or may otherwise utilize one or more remote computer systems and / or other control systems, for example, as part of one or more of the aforementioned vehicle devices and systems. It will be understood that although this exemplary embodiment has been described in the context of a full-featured computer system, those skilled in the art will recognize that the mechanisms of this disclosure are capable of being distributed as a program product having one or more types of non-transient computer-readable signal-bearing media for storing a program and its instructions and performing its distribution, such as a non-transient computer-readable medium carrying a program and containing computer instructions stored therein for causing a computer processor (such as processor 44) to implement and execute the program. Such program products may take various forms, and this disclosure applies equally regardless of the specific type of computer-readable signal-bearing medium used to perform the distribution. Examples of signal-bearing media include recordable media such as floppy disks, hard disks, memory cards, and optical disks, and transmission media such as digital and analog communication links. It will be understood that cloud-based storage and / or other technologies may also be utilized in some embodiments. Similarly, it should be understood that the computer system of controller 34 may also be otherwise connected to Figure 1 The embodiments shown may differ; for example, the computer system of controller 34 may be coupled to or may otherwise utilize one or more remote computer systems and / or other control systems.

[0100] refer to Figure 2 And continue to refer to Figure 1 The data flow diagram illustrates various embodiments. Figure 1 The components of system 100. As will be understood, various embodiments of system 100 according to this disclosure may include any number of modules embedded within controller 34, which may be combined and / or further divided to similarly implement the systems and methods described herein. Furthermore, inputs to system 100 may be received from sensor system 28, from other control modules (not shown) associated with vehicle 10, and / or by… Figure 1Other submodules (not shown) within the controller 34 determine / model the data. Furthermore, the input may undergo preprocessing such as subsampling, noise reduction, normalization, feature extraction, and loss reduction. In various embodiments, the modules of system 100 are implemented to enable a two-stage method for generating camera-to-LiDAR alignment information. For example, the first-stage modules include an image data processing module 102, a LiDAR data processing module 104, and a data pair selection module 106. The image data processing module 102 processes image data 110 to generate vehicle point data 112. The image data 110 includes two-dimensional points sensed from the environment and generated by the camera. The LiDAR data processing module 104 processes LiDAR data to generate vehicle point data 116. The LiDAR data 114 includes a three-dimensional point cloud sensed from the environment and generated by the LiDAR. The data pair selection module 106 processes the vehicle point data 112, 116 to selectively store quality data pairs in a data buffer 118.

[0101] In another example, the second-stage module includes an alignment determination module 108. The alignment determination module 108 uses a joint analysis method to process data pairs from the data buffer 118 to determine alignment data 120 that defines the alignment information between the camera and the lidar.

[0102] refer to Figure 3 and Figure 4 And continue to refer to Figures 1-2 Flowcharts for methods 400 and 500 are provided for execution, as shown by... Figure 1 and Figure 2 The system 100 executes a two-stage method for determining lidar alignment to a camera. In various embodiments, method 400 is associated with a first stage, and method 500 is associated with a second stage. Typically, method 400 is executed before method 500. As will be understood from this disclosure, the order of operations within methods 400 and 500 is not limited to the following. Figure 3 and Figure 4 The methods 400 and 500 may be executed in the order shown, but may be executed in one or more applicable variations according to this disclosure. In various embodiments, methods 400 and 500 may be arranged to run based on one or more predetermined events, and / or may run continuously during operation of vehicle 10.

[0103] For details, please refer to the following: Figure 3 In one example, method 400 may begin at 405. At 410, image data 110 is processed to identify vehicle point data 112. For example, image data 110 is processed using a deep learning neural network or other machine learning methods to identify points associated with the edges and contours of vehicles within the image; and an inverse distance transform is performed on the edge and contour data to define a distance map of the identified points.

[0104] At 420, LiDAR data 114 is processed to determine vehicle point data 116. For example, LiDAR data 114 is processed to identify the outermost point of the vehicle within the 3D point cloud using clustering and / or filtering methods (e.g., geometric filtering based on the known geometry of the vehicle). LiDAR data processing module 104 then projects the LiDAR points associated with the identified object into a 2D camera coordinate system and processes the projected points using a convex hull method to find the outermost contour points representing the vehicle's edge.

[0105] Once vehicle point data 112 and 116 are determined, a condition value associated with the quality of the data points is determined at 430 and evaluated at 440. As can be understood, the quality of the data pair can be determined based on any number of conditions. In various embodiments, these conditions may be based on vehicle maneuvering, the number of edge points, the number of detected vehicles, symmetry in the image, a road structure similarity index (or road SSIM) measure, and a vehicle-to-image ratio (VIR) measure. For example, a vehicle making a sharp turn may amplify small synchronization differences between the lidar and the camera, resulting in poor alignment. Therefore, conditions used to monitor quality may include when the vehicle's yaw rate exceeds a threshold (e.g., five degrees per second).

[0106] In another example, the number of edge points in a LiDAR image at a specific distance can indicate quality. Therefore, quality status can include when the number of edge points in the LiDAR data is less than a threshold at a defined distance (e.g., 200 points at 30 meters). In another example, the distribution of edge points between the left and right sides of an image can indicate quality. Therefore, quality status can include when the distribution between the left and right sides is slightly equal.

[0107] In another example, the number of vehicles within an image can indicate quality. Therefore, quality status can include when the number of identified vehicles in an image is less than a threshold number (e.g., two vehicles). In another example, Road SSIM is used to detect a given input image. x Is it related to any image in the buffer? y (Previously accepted data pairs) are similar. This is useful for avoiding storing redundant data pairs of image frames accumulated when vehicle 10 stops. Therefore, the quality condition can include when the road SSIM is greater than a threshold (e.g., 0.7).

[0108] In another example, VIR can indicate a quantitative measurement of the existing vehicles in the camera's FOV. Therefore, quality status can include when VIR is less than a threshold (e.g., 0.1).

[0109] At 440, if any of these conditions are false, the data pair is excluded from data buffer 118, and processing continues at 410. If all these conditions are true at 440, the data pair is saved to data buffer 118 at 450. Once data buffer 118 has sufficient data at 460, the first phase of the data collection process is complete, and method 400 can proceed at 470.

[0110] For details, please refer to the following: Figure 4 In one example, method 500 could begin at 505. At 510, a 3D cube containing randomly perturbed points in a rotated 3D space is generated. Each cube is associated with a search level (e.g., search level one, search level two, search level three, and search level four). Each search level has a different possible search space to allow convergence. The points within the cube are uniformly random.

[0111] At position 520, a first cube with search level 1 and a perturbation of up to three degrees in the rotational search space evaluates the input data pairs to see if alignment can be improved and generates a score. This score is generated by evaluating the correlation of the projected LiDAR profile points on the range map of the image. Each cube generates 500 possible scores using 500 uniformly random points representing the rotational perturbation. Typically, the highest score corresponds to the optimal perturbation adjustment. The same first level (search level 1) is repeated three times to ensure stable convergence.

[0112] At points 530, 540, and 550, the same process is repeated for the remaining search cubes with different search levels. If the final score at 560 does not exceed the threshold, the analysis is repeated at 510. The final score is evaluated at 560. If the final score at 560 exceeds the threshold, the transformation matrix is ​​stored at 570 for future use. The transformation matrix is ​​then used to define the alignment data 120. Method 500 can then terminate at 580.

[0113] As can be understood, the methods and systems described herein thus improve the computational resources required to align a camera with a LiDAR, and therefore the claimed embodiments achieve an improvement in the field of computer vision.

[0114] While at least one exemplary embodiment has been presented in the foregoing detailed description, it should be understood that numerous variations exist. It should also be understood that one or more exemplary embodiments are merely examples and are not intended to limit the scope, applicability, or configuration of this disclosure in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient roadmap for implementing one or more exemplary embodiments. It should be understood that various changes may be made to the function and arrangement of elements without departing from the scope of this disclosure as set forth in the appended claims and their legal equivalents.

Claims

1. A method for generating alignment parameters for processing data associated with a vehicle, the method comprising: Receive image data from the camera that is associated with the environment of the vehicle; Receive lidar data associated with the vehicle's environment from the lidar; The image data is processed by a processor to determine image data points associated with at least one vehicle identified within the image data; The processor processes the lidar data to determine lidar data points associated with at least one vehicle identified within the lidar data; Based on at least one condition associated with the quality of the image data points and lidar data points, the image data points and lidar data points are selectively stored in a data buffer; The processor uses a joint analysis method to process the image data points and lidar data points in the data buffer to generate alignment parameters between the lidar and the camera; as well as Future data is processed based on the alignment parameters. The joint analysis method includes: Generate multiple three-dimensional cubic spaces with multiple random perturbation points; For each perturbation point in each of the three-dimensional cubic spaces, the point pairs of image data points and LiDAR data points are evaluated to determine a score regarding the correlation between the image data points and LiDAR data points; and Select a transformation matrix associated with scores greater than a threshold, wherein the alignment parameters are determined based on the transformation matrix.

2. The method according to claim 1, wherein, The processing of the image data includes: The image data is processed using machine learning methods to identify the edges and contours of vehicles within the image; and Apply the inverse distance transform to the edges and contours.

3. The method according to claim 1, wherein, The processing of the lidar data includes: Clustering methods are used to detect objects within a scene; Objects are removed based on filtering methods; Projecting the lidar data onto a coordinate system associated with the image data; and The convex hull method is applied to the projected LiDAR data to identify LiDAR data points associated with the edges of the vehicle.

4. The method according to claim 1, wherein, The condition is related to the operation of the vehicle.

5. The method according to claim 1, wherein, The status is related to the number of image data points and lidar data points.

6. The method according to claim 1, wherein, The condition is related to the distribution of the image data points between the left and right sides of the image data.

7. The method according to claim 1, wherein, The condition is associated with a road structure similarity index measure between the first and second images of the image data.

8. The method according to claim 1, wherein, The condition is associated with the number of vehicles in the image of the image data.

9. The method according to claim 1, wherein, The condition is related to the vehicle image ratio.

10. The method according to claim 1, wherein, Multiple random perturbation points include 500 random perturbation points.

11. A computer-implemented system for generating alignment parameters for processing vehicle-related data, the system comprising: Data storage element, which includes computer-readable instructions; as well as A processor configured to execute the computer-readable instructions, the computer-readable instructions controlling the processor to perform operations, the operations including: Receive image data from the camera that is associated with the environment of the vehicle; Receive lidar data associated with the vehicle's environment from the lidar; The image data is processed to determine image data points associated with at least one vehicle identified within the image data; The lidar data is processed to determine lidar data points associated with at least one vehicle identified within the lidar data; Based on at least one condition associated with the quality of the image data points and lidar data points, the image data points and lidar data points are selectively stored in a data buffer; The image data points and LiDAR data points in the data buffer are processed using a joint analysis method to generate alignment parameters between the LiDAR and the camera; and Future data is processed based on the alignment parameters. The joint analysis method includes: Generate multiple three-dimensional cubic spaces with multiple random perturbation points; For each perturbation point in each of the three-dimensional cubic spaces, the point pairs of image data points and LiDAR data points are evaluated to determine a score regarding the correlation between the image data points and LiDAR data points; and Select a transformation matrix associated with scores greater than a threshold, wherein the alignment parameters are determined based on the transformation matrix.

12. The system according to claim 11, wherein, Processing the image data includes: The image data is processed using machine learning methods to identify the edges and contours of vehicles within the image; and Apply the inverse distance transform to the edges and contours.

13. The system according to claim 11, wherein, The processing of the lidar data includes: Clustering methods are used to detect objects within a scene; Objects are removed based on filtering methods; Projecting the lidar data onto a coordinate system associated with the image data; and The convex hull method is applied to the projected LiDAR data to identify LiDAR data points associated with the edges of the vehicle.

14. The system according to claim 11, wherein, The condition is related to the operation of the vehicle.

15. The system according to claim 11, wherein, The status is related to the number of image data points and lidar data points.

16. The system according to claim 11, wherein, The condition is related to the distribution of the image data points between the left and right sides of the image data.

17. The system according to claim 11, wherein, The condition is associated with a road structure similarity index measure between the first and second images of the image data.

18. The system according to claim 11, wherein, The condition is associated with the number of vehicles in the image of the image data.

19. The system according to claim 11, wherein, The condition is related to the vehicle image ratio.

20. The system according to claim 11, wherein, Multiple random perturbation points include 500 random perturbation points.

Images