Alignment verification in vehicle-mounted sensors

Abstract

A system in a vehicle includes an image sensor to obtain an image in an image sensor coordinate system and a depth sensor to obtain a point cloud in a depth sensor coordinate system. Processing circuitry implements a neural network to determine a validation status of a transformation matrix that transforms the point cloud in the depth sensor coordinate system to a transformed point cloud in the image sensor coordinate system. The transformation matrix includes rotation parameters and translation parameters.

Description

Technical Field

[0001] This disclosure relates to alignment verification in vehicle-mounted sensors. Background Technology

[0002] Vehicles (e.g., cars, motorcycles, trucks, construction equipment) are increasingly using sensors and communication systems to enhance operation. For example, some sensors (e.g., inertial measurement units (IMUs), wheel angle sensors) provide information about the vehicle, while others (e.g., cameras, lidar systems, radar systems) provide information about the vehicle's surroundings. This information can facilitate semi-autonomous actions (e.g., adaptive cruise control, automatic braking) or autonomous operation, or it can facilitate alerts to the driver. When multiple sensors are used to obtain information about the vehicle's surroundings, each sensor has its own coordinate system. Alignment between these sensor coordinate systems can be helpful or necessary for accurate detection and can facilitate the fusion of sensor information. Therefore, alignment verification is expected to be provided in onboard sensors. Summary of the Invention

[0003] In one exemplary embodiment, the system in the vehicle includes an image sensor for acquiring an image in an image sensor coordinate system and a depth sensor for acquiring a point cloud in a depth sensor coordinate system. Processing circuitry implements a neural network to determine a verification state of a transformation matrix that transforms the point cloud in the depth sensor coordinate system to a transformed point cloud in the image sensor coordinate system. The transformation matrix includes rotation and translation parameters.

[0004] In addition to one or more features described herein, the verification state of the transformation matrix is ​​determined as a binary indication of alignment or misalignment.

[0005] In addition to one or more features described herein, the verification state of the transformation matrix is ​​determined to be one or more of the following: projection loss associated with the projection of the transformed point cloud onto the image plane of the image sensor, 3D loss associated with the transformed point cloud, and rotation and translation loss associated with rotation and translation parameters.

[0006] In addition to one or more features described herein, the image sensor is a camera, and the processing circuitry obtains an image representation based on one or more images.

[0007] In addition to one or more of the features described herein, the image represents an indication of the red, green, and blue (RGB) intensity levels of the one or more of the image or the image gradient magnitude of the one or more of the image.

[0008] In addition to one or more features described herein, the depth sensor is a radar system or lidar system, and the processing circuitry obtains a point cloud representation based on one or more point clouds.

[0009] In addition to one or more features described herein, a point cloud representation indicates the depth and intensity level of each point in one or more point clouds, the aggregate depth and aggregate intensity of stationary points of two or more point clouds, or the aggregate depth gradient magnitude and aggregate intensity gradient magnitude of two or more point clouds.

[0010] In addition to one or more features described herein, the processing circuitry trains the neural network based on image data from an aligned image sensor aligned with an aligned depth sensor and on transformed point cloud data obtained from the aligned depth sensor and transformed to the coordinate system of the aligned image sensor using an aligned transformation matrix.

[0011] In addition to one or more features described herein, the processing circuitry also generates training samples by perturbing one or more parameters of the rotation and translation parameters that constitute the transformation matrix of the alignment.

[0012] In addition to one or more features described in this paper, perturbations of one or more parameters are randomly selected for each training sample.

[0013] In another exemplary embodiment, a method in a vehicle includes: acquiring an image from an image sensor in an image sensor coordinate system, and acquiring a point cloud from a depth sensor in a depth sensor coordinate system. The method further includes implementing a neural network to determine a verification state of a transformation matrix that transforms the point cloud in the depth sensor coordinate system to the transformed point cloud in the image sensor coordinate system. The transformation matrix includes rotation and translation parameters.

[0014] In addition to one or more features described in this paper, the verification state of the transformation matrix is ​​determined as a binary indication of alignment or misalignment.

[0015] In addition to one or more features described herein, the validation state is determined as one or more of the following: projection loss associated with the projection of the transformed point cloud onto the image plane of the image sensor, 3D loss associated with the transformed point cloud, and rotation and translation loss associated with rotation and translation parameters.

[0016] In addition to one or more features described herein, the method also includes obtaining an image representation based on one or more images from an image sensor that serves as a camera.

[0017] In addition to one or more of the features described herein, the image represents an indication of the red, green, and blue (RGB) intensity levels of the one or more of the image or the image gradient magnitude of the one or more of the image.

[0018] In addition to one or more features described herein, the method also includes obtaining a point cloud representation based on one or more point clouds from a depth sensor, which is a radar system or a lidar system.

[0019] In addition to one or more features described herein, a point cloud representation indicates the depth and intensity level of each point in one or more point clouds, the aggregate depth and aggregate intensity of stationary points of two or more point clouds, or the aggregate depth gradient magnitude and aggregate intensity gradient magnitude of two or more point clouds.

[0020] In addition to one or more features described herein, the method also includes training the neural network based on image data from an aligned image sensor aligned with an aligned depth sensor and on transformed point cloud data obtained from the aligned depth sensor and transformed to the coordinate system of the aligned image sensor using an aligned transformation matrix.

[0021] In addition to one or more of the features described herein, the method also includes generating training samples by perturbing one or more parameters of the rotation parameters and translation parameters constituting the transformation matrix of the alignment.

[0022] In addition to one or more features described herein, the perturbation includes the amount of perturbation of one or more parameters randomly selected for each training sample.

[0023] The above-described features and advantages, as well as other features and advantages of this disclosure, will become apparent when taken in conjunction with the accompanying drawings and the following detailed description. Attached Figure Description

[0024] Other features, advantages, and details appear by way of example only in the following detailed specification, which refers to the accompanying drawings, wherein:

[0025] Figure 1 It is a block diagram of a vehicle including alignment verification between an image sensor and a depth sensor according to one or more embodiments;

[0026] Figure 2 This is a process flow for alignment verification according to one or more embodiments;

[0027] Figure 3 The process involved in obtaining alignment verification according to one or more embodiments is illustrated; and

[0028] Figure 4 This is a process flow for providing alignment verification in an onboard sensor according to one or more embodiments. Detailed Implementation

[0029] The following description is exemplary in nature only and is not intended to limit this disclosure, its application, or use. It should be understood that throughout the drawings, corresponding reference numerals denote the same or corresponding parts and features.

[0030] As mentioned earlier, more than one onboard sensor can be used to obtain information about the vehicle's surroundings. It should also be noted that in this case, alignment of one sensor with the other sensors may be desirable. The alignment process is the process of finding a transformation matrix that provides information from one sensor's coordinate system in the coordinate system of the other sensor. When the transformation matrix correctly provides information from one coordinate system in the other, the two sensors are considered correctly aligned. When the transformation matrix does not correctly provide information from one coordinate system in the other, the two coordinate systems, or more generally the sensors, are said to be misaligned. Existing methods involve a time-consuming process of manually determining alignment or determining alignment parameters when misalignment is suspected.

[0031] Implementations of the systems and methods detailed herein relate to alignment verification in automotive sensors. Specifically, for illustrative purposes, the verification of alignment between an image sensor (e.g., a camera) that acquires an image in one coordinate system and a depth sensor (e.g., a lidar system, radar system) that acquires a 3D point cloud in another coordinate system is described in detail. Alignment involves transforming (i.e., rotating and translating) the point cloud acquired by the depth sensor into the coordinate system of the image sensor, and then projecting it onto the image acquired by the image sensor based on the parameters of the image sensor. A neural network is trained to provide a rapid evaluation of alignment in the form of a binary indication (i.e., aligned or misaligned) or a measurement of alignment error. The error measurement (i.e., loss) may involve projection loss, 3D loss, or rotation and translation loss. Neural network-based verification helps to abandon the determination of alignment parameters unless the sensor is actually misaligned.

[0032] According to exemplary embodiments, Figure 1 It is a block diagram of a vehicle 100 that includes alignment verification between an image sensor 120 and a depth sensor 130. Figure 1The exemplary vehicle 100 is an automobile 101. The exemplary image sensor 120 may be a camera 125, and the exemplary depth sensor 130 may be a radar system 135 or a lidar system 140. Vehicle 100 includes a controller 110 that can control one or more aspects of the operation of vehicle 100. For example, controller 110 may obtain information from image sensor 120, depth sensor 130, or a combination of sensors to control autonomous operation or semi-autonomous actions of vehicle 100 (e.g., automatic braking, adaptive cruise control).

[0033] According to one or more embodiments, controller 110 can perform alignment verification, as described in detail. Performing alignment verification includes indicating verification status 240. Figure 2 The verification state 240 can simply be an indication of alignment or misalignment. Alternatively, the verification state 240 can be an indication of one or more error measurements. Based on the verification state 240 determined by the alignment verification process, the controller 110 can obtain alignment parameters or abandon additional alignment-related processing. The controller 110 may include processing circuitry, which may include application-specific integrated circuits (ASICs), electronic circuitry, a processor (shared, dedicated, or grouped) executing one or more software or firmware programs and memory, combinational logic circuitry, and / or other suitable components that provide the described functionality.

[0034] Figure 2 This is a process flow for alignment verification according to one or more embodiments. Figure 2 In the exemplary embodiment shown, image representation 210 and point cloud representation 220 are provided to neural network 230. Neural network 230 may be implemented by controller 110. Image representation 210 and point cloud representation 220 may be provided by image sensor 120 and depth sensor 130, respectively. Alternatively, image representation 210 and point cloud representation 220 may be generated by controller 110 by processing data obtained from image sensor 120 and depth sensor 130, respectively. Typically, point cloud representation 220 is in the coordinate system of image sensor 120. (See reference...) Figure 3 Further discussion on image representation 210 and point cloud representation 220.

[0035] The implementation of neural network 230 is not limited to any particular architecture. For example, each of image representation 210 and point cloud representation 220 can be encoded separately to generate separate feature maps. This can be followed by a fusion layer that connects the feature maps or forms correlation layers between the feature maps. Alternatively, image representation 210 and point cloud representation 220 can be stacked before implementing a standard backbone, allowing the neural network to learn a single set of parameters. The output of neural network 230 is a verification state 240 with a binary indication (e.g., aligned or misaligned) or an indication of alignment quality based on one or more error measurements. (Reference) Figure 4 Further discussion will be given on the training of verification state 240 and neural network 230.

[0036] Figure 3 The process involved in obtaining alignment verification according to one or more embodiments is illustrated. A point cloud 310 is shown in the coordinate system of the depth sensor 130. The point cloud 310 is three-dimensional, as shown. A transformation of the point cloud 310 based on rotation and translation (indicated as [R|T]) is used to obtain a transformed point cloud 320 in the coordinate system of the image sensor 120. Like the point cloud 310, the transformed point cloud 320 is three-dimensional. The projection of the three-dimensional transformed point cloud 320 onto the two-dimensional image plane of the image sensor 120 is based on known camera intrinsics (K) and lens distortion (D) parameters.

[0037] The result of the projection is Figure 3 The point cloud representation 220 is indicated as an example. Specifically, each point indicates {instantaneous depth, instantaneous point intensity}. Alternatively, the point cloud representation 220 can be generated by aggregation of results from two or more frames. Aggregation refers to the accumulation of point clouds over two or more frames (i.e., with two or more timestamps). A depth from any timestamp can be assigned to each point in the point cloud. The result indicates {aggregated depth, aggregated point intensity}. According to yet another alternative, gradient magnitudes can be computed on the aggregation result. The gradient magnitude of a parameter is a scalar quantity describing the local rate of change of the parameter. In this case, according to an alternative embodiment, the point cloud representation 220 can indicate {aggregated depth gradient magnitude, aggregated point intensity gradient magnitude}. In the case of aggregation, points in the point cloud 310 that relate to a moving object (e.g., a pedestrian, another vehicle) are removed. The point cloud representation 220 according to one example is one of the inputs to the neural network 230.

[0038] Figure 3The image 330 acquired by the image sensor 120 is also indicated. For example, image 330 is two-dimensional in the x, y plane. Image representation 210 can be image 330 itself, for example, indicating the intensity of red, green, and blue {R, G, B}. Alternatively, image representation 210 can be the image gradient magnitude obtained as g(a, b) as indicated. Image representation 210 according to one example is another input to neural network 230. The input to neural network 230 can be a stack of image representation 210 and a stack of point cloud representation 220. Image representation 210 and point cloud representation 220 are obtained independently. That is, for example, image representation 210 can be an indication of {R, B, G} of image 330, while point cloud representation 220 indicates {aggregated depth gradient magnitude, aggregated point intensity gradient magnitude}. Neural network 230 uses image representation 210 and point cloud representation 220 to provide verification state 240.

[0039] Figure 4 This is a process flow of a method 400 for providing alignment verification in vehicle-mounted sensors 120, 130 according to one or more embodiments. For example, Figure 4 The process shown can be executed by the processing circuitry of controller 110. At block 410, training the neural network 230 includes several processes. Alignment of sensors 120 and 130 is achieved. That is, image sensor 120 and depth sensor 130 are aligned such that rotation parameters (yaw, pitch, roll) and translation parameters (x, y, z) correctly transform the point cloud 310 obtained by depth sensor 130 to the coordinate system of image sensor 120. The six parameters {yaw, pitch, roll, x, y, z} constituting the transformation matrix of aligned sensors 120 and 130 represent the ground reality.

[0040] As part of the training, at box 410, binary alignments (i.e., good alignment, bad alignment) are randomly selected following a Bernoulli distribution with an alignment probability p = 0.5. Training samples are then generated by perturbing one or more of the six parameters according to the selected binary alignment. If a good alignment is randomly selected, one or more of the six parameters vary in a uniform distribution between g1 and g2, and if a bad alignment is randomly selected, one or more of the six parameters vary in a uniform distribution between b1 and b2, where 0 ≤ g1 ≤ g2 ≤ b1 ≤ b2. The perturbation of each parameter can be chosen independently within the specified uniform distribution. Furthermore, some of the six parameters may not be perturbed at all. The neural network 230 is trained using the multiple training samples generated as described. Once the neural network 230 is trained, it can provide a validation state 240 based on the inputs of the image representation 210 and the point cloud representation 220.

[0041] At box 420, one or more images 330 are acquired from image sensor 120, and one or more point clouds are acquired from depth sensor 130 (e.g., radar system, LIDAR system). At box 430, based on reference... Figure 3 One of the exemplary methods discussed is to generate image representation 210 and point cloud representation 220. For example, image representation 210 may be a stack of {R, G, B} indications from multiple images 330, while point cloud representation 220 may be an indication of {aggregate depth, aggregate point intensity}. At box 440, determining the verification state 240 is performed by a trained neural network 230.

[0042] As previously mentioned, the verification state 240 can be a binary indication (i.e., aligned or misaligned) or it can provide a measurement of alignment error. The error measurement (i.e., the loss) can involve projection loss, 3D loss, or rotation and translation loss. Each of these is interpreted by referring to the training samples (i.e., the training point cloud representation 220 generated by perturbations of one or more of the six parameters {yaw, pitch, roll, x, y, z}). Both projection loss and 3D loss are indications of the distance between perfectly aligned points and the perturbated point cloud representation 220.

[0043] The projection loss is an indicator of the distance s on the two-dimensional image plane, and is given by the following equation:

[0044]

[0045] [R|T] is the rotation and translation of the aligned sensors 120, 130 (i.e., the six parameters obtained for the aligned sensors 120, 130 as part of the process at box 410). It is a rotation and translation caused by the disturbance. X i P(Y) is the three-dimensional (homogeneous) coordinate of each point i out of a total of N points in the point cloud 310 of the coordinate system of the depth sensor 130. P(Y) is the projection result of a given point Y in the three-dimensional coordinate system of the image sensor 120 (i.e., the two-dimensional coordinate in the image plane).

[0046] The 3D loss is an indicator of the distance s in 3D (i.e., unprojected) and is given by the following equation:

[0047]

[0048] Rotation and translation losses can also be calculated in detail. Alignment rotation R and disturbance rotation... The difference rotation matrix between them is Based on the composition of the rotation matrix, Based on Rodrigues's rotation formula, the rotation matrix can be represented as the rotation axis (i.e., direction) and the angle θ (i.e., quantity). Specifically,

[0049]

[0050] Using weighting factors α and β, such that α≥0, β≥0, and α+β=1, the rotation and translation losses are given by the following equation:

[0051]

[0052] While the foregoing disclosure has been described with reference to exemplary embodiments, those skilled in the art will understand that various changes can be made and elements can be substituted with equivalents without departing from its scope. Furthermore, many modifications can be made to adapt particular situations or materials to the teachings of this disclosure without departing from the basic scope of this disclosure. Therefore, this disclosure is not intended to be limited to the specific embodiments disclosed, but will include all embodiments falling within its scope.

Claims

1. A system in a vehicle, comprising: An image sensor, configured to acquire an image in an image sensor coordinate system; A depth sensor, configured to acquire a point cloud in a depth sensor coordinate system; as well as Processing circuitry configured to implement a neural network to determine a verification state of a transformation matrix, the transformation matrix being configured to transform the point cloud in the depth sensor coordinate system into a transformed point cloud in the image sensor coordinate system, the transformation matrix including rotation and translation parameters. The processing circuit is configured to train the neural network based on image data from an aligned image sensor aligned with an aligned depth sensor, and based on transformed point cloud data obtained from the aligned depth sensor and transformed to the coordinate system of the aligned image sensor using an aligned transformation matrix. The processing circuit is configured to generate training samples by perturbing one or more parameters of the rotation and translation parameters constituting the aligned transformation matrix, and to randomly select the perturbation amount of one or more parameters for each training sample.

2. The system of claim 1, wherein, The verification state of the transformation matrix is ​​determined as a binary indication of alignment or misalignment.

3. The system of claim 1, wherein, The verification state of the transformation matrix is ​​determined to be one or more of the following: projection loss associated with the projection of the transformed point cloud onto the image plane of the image sensor, 3D loss associated with the transformed point cloud, and rotation and translation loss associated with the rotation and translation parameters.

4. The system according to claim 1, wherein, The image sensor is a camera, and the processing circuitry obtains an image representation based on one or more of the images, the image representation indicating the red, green, and blue (RGB) intensity levels of the one or more of the images or the image gradient magnitude of the one or more of the images. The depth sensor is a radar system or a lidar system, and the processing circuitry obtains a point cloud representation based on one or more of the point clouds, and the point cloud representation indicating the depth and intensity level of each point in one or more point clouds, the aggregated depth and aggregated intensity of stationary points of two or more point clouds, or the aggregated depth gradient magnitude and aggregated intensity gradient magnitude of two or more point clouds.

5. A method in a vehicle, comprising: The processing circuit obtains the image from the image sensor in the image sensor coordinate system; The processing circuit obtains a point cloud from the depth sensor in the depth sensor coordinate system. as well as The processing circuit implements a neural network to determine the verification state of a transformation matrix, which is configured to transform the point cloud in the depth sensor coordinate system into a transformed point cloud in the image sensor coordinate system. The transformation matrix includes rotation and translation parameters. The neural network is trained based on image data from an aligned image sensor aligned with an aligned depth sensor and on transformed point cloud data obtained from the aligned depth sensor and transformed to the coordinate system of the aligned image using an aligned transformation matrix. Training samples are generated by perturbing one or more parameters of the rotation and translation parameters constituting the aligned transformation matrix, wherein the perturbation includes a perturbation amount of the one or more parameters randomly selected for each of the training samples.

6. The method of claim 5, wherein determining the verification state of the transformation matrix is ​​as a binary indication of alignment or misalignment.

7. The method according to claim 5, wherein, The verification state is determined as one or more of the following: projection loss associated with the projection of the transformed point cloud onto the image plane of the image sensor, 3D loss associated with the transformed point cloud, and rotation and translation loss associated with the rotation and translation parameters.

8. The method according to claim 5, further comprising: An image representation is obtained based on one or more images from the image sensor, which is a camera, wherein the image representation indicates the red, green, and blue (RGB) intensity levels of the one or more images or the image gradient magnitude of the one or more images; and a point cloud representation is obtained based on one or more point clouds from the point cloud, which is a depth sensor, which is a radar system or a lidar system, wherein the point cloud representation indicates the depth and intensity level of each point in the one or more point clouds, the aggregated depth and aggregated intensity of two or more stationary points in the point cloud, or the aggregated depth gradient magnitude and aggregated intensity gradient magnitude of the two or more points in the point cloud.

Images