Computing system
By combining camera and lidar technology in autonomous vehicles, using cube-shaped markers to calculate calibration information, the problem of difficulty in calibrating the three-dimensional coordinate system in the prior art is solved, and more efficient sensor fusion and vehicle automatic driving are achieved. Security.
Patent Information
- Application Number
- CN202310610222.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-26
- Publication Date
- 2024-11-26
AI Technical Summary
The prior art is difficult to calibrate the object marks detected by left and right radars and cameras in three-dimensional coordinate systems, especially in autonomous vehicles, resulting in limited accuracy and accuracy of sensor fusion.
A computing system was designed to use a combination of camera and lidar to realize the alignment of two-dimensional and three-dimensional coordinate systems by setting cube-shaped markers outside the vehicle, and using camera images and lidar point cloud data to calculate calibration information. . The specific steps include: the camera detects the object position as a two-dimensional coordinate, the lidar detects the object position as a three-dimensional coordinate, and calculates the calibration information through the coordinate axis calibration processing unit, record and store it for subsequent use.
Effectively correlate the object marks detected by the camera and the lidar, improve the accuracy of sensor fusion processing and the safety of vehicle autonomous driving, and ensure the stable operation of the vehicle under different sensor states.
Smart Images

Figure CN119024309A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a computing system for computing calibration information for matching object targets respectively detected by a plurality of sensors. Background Art
[0002] There is a known technology called sensor fusion, which uses cameras and radars to detect objects outside the vehicle (see Patent Document 1). In sensor fusion, in order to match the objects detected by each sensor, the coordinate system of one sensor needs to be calibrated with the coordinate system of another sensor.
[0003] In the prior art, the same target is detected by left and right radars and cameras, and the direction of the detection is determined in a two-dimensional coordinate system. Therefore, it is difficult to determine the sensor of a three-dimensional coordinate system such as a LiDAR (Light Detection And Ranging).
[0004] Prior art literature
[0005] Patent Literature
[0006] Patent document 1: Japanese Patent No. 6943678. Summary of the invention
[0007] A computing system of a technical solution of the present invention is a computing system for calculating calibration information of a vehicle in which a first sensor and a second sensor are set at specified positions, wherein the first sensor uses an image sensor to obtain two-dimensional image information, and the second sensor irradiates electromagnetic waves to multiple detection points to obtain three-dimensional point cloud data including distance information of each detection point, and comprises: a target, which is arranged outside the vehicle and has a cubic shape with at least one vertex formed by three adjacent square faces; a first calculation unit, which calculates the position of the vertex of the target as a first position on a two-dimensional coordinate system based on image information of a field of view including the target obtained by the first sensor; a second calculation unit, which calculates the position of the vertex of the target as a second position on a three-dimensional coordinate system based on distance information of each of multiple detection points in the field of view including the target obtained by the second sensor; and a third calculation unit, which calculates calibration information based on the first position and the second position. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] The objects, features and advantages of the present invention will be further clarified through the following description of the embodiments in conjunction with the accompanying drawings.
[0009] Figure 1 It is a block diagram schematically showing the overall structure of a vehicle control system.
[0010] Figure 2FIG. 1 is a schematic diagram illustrating the positional relationship between a vehicle and a target when the calibration information is calculated by the calculation system.
[0011] Figure 3A This is a diagram for explaining how to calculate the coordinates (Xw, Yw, Zw) of the vertices of a target based on point cloud data obtained by a laser radar.
[0012] Figure 3B The diagram explains how to calculate the coordinates (I, J) of the vertices of a target in a camera image captured by a camera.
[0013] Figure 4 This is a flowchart illustrating the flow of a calculation process of calibration information performed by a calculation system. DETAILED DESCRIPTION
[0014] A computing system according to an embodiment of the present invention calculates calibration information of a two-dimensional coordinate system of a camera and a three-dimensional coordinate system of a laser radar in order to match an object detected by a camera as a first sensor of a vehicle with an object detected by a laser radar as a second sensor of the vehicle. Such a computing system is described in detail.
[0015] The computing system of the embodiment can be applied to both vehicles with an autonomous driving function, i.e., autonomous driving vehicles, and manually driven vehicles without an autonomous driving function. Manually driven vehicles include vehicles with a driving assistance function. The following describes an example of applying the computing system to an autonomous driving vehicle.
[0016] It should be noted that a vehicle to which the computing system according to the embodiment is applied is sometimes referred to as a host vehicle, as distinguished from other vehicles.
[0017] The vehicle may be any of an engine vehicle having an internal combustion engine (engine) as a driving source, an electric vehicle having a driving motor as a driving source, and a hybrid vehicle having an engine and a driving motor as driving sources. The vehicle (autonomous driving vehicle) can be driven not only in an automatic driving mode that does not require a driver's driving operation, but also in a manual driving mode (using a driving assistance function) in which a driver performs a driving operation.
[0018] <About the outline structure of autonomous driving>
[0019] First, the general structure of autonomous driving will be described. Figure 1 1 is a block diagram schematically showing the overall structure of a vehicle control system 100 constituting a computing system of an embodiment. Figure 1As shown, the vehicle control system 100 mainly includes an automatic driving ECU (Electronic Control Unit) 10 and a plurality of external sensors (a camera 1 as a first sensor, a laser radar 2 as a second sensor) communicatively connected to the automatic driving ECU 10 via a CAN (Controller Area Network) communication line, etc., a sensor group 3, an input / output device 4, a positioning unit 5, a map database 6, a navigation device 7, a communication unit 8, and an actuator AC for driving.
[0020] <Camera>
[0021] The camera 1 includes an imaging element such as a CMOS (Complementary Metal Oxide Semiconductor) sensor, and repeatedly captures images of objects around the vehicle at a predetermined frame rate. The imaging element is also called an image sensor.
[0022] The camera 1 is installed at a predetermined position, for example, at the front of the vehicle, and captures the space in front of the vehicle. In the embodiment, the position on the camera image obtained by the camera 1 is represented by two-dimensional coordinates (I, J). When the lateral (horizontal) direction of the camera image is the I direction and the longitudinal (vertical) direction is the J direction, the position of the object in the vehicle width direction can be obtained using the position in the I direction on the camera image, and the position of the object in the height direction and the travel direction can be obtained using the position in the J direction. In other words, the position information of the object can be obtained based on the image information of the camera 1.
[0023] <LiDAR>
[0024] The laser radar 2 detects objects around the vehicle by irradiating a laser as an electromagnetic wave to a plurality of detection points (also called measurement points) set in, for example, the field of view, and receiving reflected light from each detection point. The reflected light is the light irradiated from the laser source and reflected back from a certain point on the surface of the object, and includes the distance information from the laser source to the point, the intensity information of the reflected light, and the relative speed information between the laser source and the point.
[0025] The laser radar 2 repeatedly detects the position of objects in the field of view at a predetermined frame rate. The field of view of the laser radar 2 is set within the shooting area of the camera 1. Therefore, when the object photographed by the camera 1 is the same as the object detected by the laser radar 2, the position and speed of the object can be derived by performing fusion processing.
[0026] In the embodiment, the positions of detection points on the surface of the object included in the field of view of the laser radar 2 are represented by coordinates (Xw, Yw, Zw) in three-dimensional space. The data of the plurality of detection points detected by the laser radar 2 is also referred to as point cloud data.
[0027] <Sensor Group>
[0028] The sensor group 3 is a general term for a plurality of sensors that detect the driving state of the vehicle. The sensor group 3 includes, for example, a vehicle speed sensor that detects the driving speed (vehicle speed) of the vehicle, an acceleration sensor that detects the acceleration of the front-rear direction and the left-right direction of the vehicle, a gyro sensor that detects the rotation and direction change of the vehicle as an angular velocity, and a rotation speed sensor that detects the rotation speed of the driving source. Sensors that detect the driving operation of the driver in the manual driving mode, such as the operation of the accelerator pedal, the operation of the brake pedal, the operation of the steering wheel, etc., may also be included in the sensor group 3.
[0029] <Input and output devices>
[0030] The input / output device 4 is a general term for devices for inputting commands from the driver and outputting information to the driver. For example, the input / output device 4 includes: various switches for the driver to input various commands by operating the operating member, a microphone for the driver to input commands by voice, a display that provides information to the driver by displaying images, a speaker that provides information to the driver by sound, etc.
[0031] <Positioning unit>
[0032] The positioning unit (GNSS (Global Navigation Satellite System) unit) 5 has a positioning sensor that receives a positioning signal sent from a positioning satellite. The positioning sensor can also be included in the sensor group 3. The positioning satellite is an artificial satellite such as a GPS satellite and a quasi-zenith satellite. The positioning unit 5 uses the positioning information received by the positioning sensor to measure the current position (latitude, longitude, altitude) of the vehicle.
[0033] <Map Database>
[0034] The map database 6 is a device that stores general map information used in the navigation device 7 and is composed of, for example, a hard disk or a semiconductor device. The map information includes road position information, road shape information (curvature, etc.), and intersection and fork position information.
[0035] It should be noted that the map information stored in the map database 6 is different from the high-precision map information stored in the storage unit 14 of the automatic driving ECU.
[0036] <Navigation device>
[0037] The navigation device 7 searches for a target path on the road leading to the destination input by the driver and guides the driver along the target path. The input of the destination and the guidance along the target path are performed through the input / output device 4. The target path is calculated based on the current position of the vehicle measured by the positioning unit 5 and the map information stored in the map database 6. The current position of the vehicle can also be measured using the detection values of the camera 1 and the laser radar 2, and the target path can also be calculated based on the current position and the high-precision map information stored in the storage unit 14.
[0038] <Communication unit>
[0039] The communication unit 8 communicates with various servers (not shown) through a network including a wireless communication network represented by the Internet, a mobile phone network, etc., and obtains map information, driving history information, traffic information, etc. from the server regularly or at any time. The network includes not only a public wireless communication network, but also a closed communication network set up in each specified management area, such as a wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc. The obtained map information is output to the map database 6 and the storage unit 14, and the map information is updated.
[0040] <Actuator>
[0041] The actuator AC is a driving actuator for controlling the driving of the vehicle. When the driving source is an engine, the actuator AC includes a throttle actuator for adjusting the opening of the throttle valve of the engine (throttle opening). When the driving source is a driving motor, the driving motor is included in the actuator AC. The brake actuator that operates the braking device of the vehicle and the steering actuator that drives the steering device are also included in the actuator AC.
[0042] <Autonomous driving ECU>
[0043] The automatic driving ECU 10 is composed of a computer having a CPU (microprocessor) and other computing units, a ROM (Read Only Memory), a DDR (Double Data Rate) RAM (Random Access Memory) and other storage units 14, and an I / O (input / output) interface and other peripheral circuits not shown. It should be noted that multiple ECUs with different functions, such as an engine control ECU, a driving motor control ECU, and a brake device ECU, can be separately set, but for convenience, Figure 1 The automatic driving ECU 10 is shown as a collection of these ECUs.
[0044] The automatic driving ECU 10 has an outside recognition unit 11 , a control processing unit 12 , a coordinate axis calibration processing unit 13 , and a storage unit 14 as functional structures.
[0045] (External Identification Department)
[0046] The external recognition unit 11 recognizes the external conditions around the vehicle based on the signals from the camera 1 and the laser radar 2. For example, it recognizes the position, speed, acceleration of surrounding vehicles (vehicles in front, vehicles behind) traveling around the vehicle, the position of surrounding vehicles parked or parked around the vehicle, and the position and status of other objects. Other objects include: signs, traffic lights, road markings or stop lines, buildings, guardrails, telephone poles, billboards, pedestrians, bicycles, tunnel entrances, etc. The status of other objects includes: the color of the traffic light (red, green, yellow), the moving speed and direction of pedestrians and bicycles, etc.
[0047] Objects that are detected by the camera 1 and the laser radar 2 are called target objects. Target objects include both people and objects, as well as both moving objects and stationary objects. The external recognition unit 11 comprehensively processes (fusion processing) the detection data of the camera 1 and the laser radar 2 as external sensors, determines whether to process the target objects detected by each sensor as the same target object, and derives the position data of the target object. Generally, the laser radar 2 is superior to the camera 1 in terms of measurement accuracy, and the camera 1 is superior to the laser radar 2 in terms of close-range resolution. The characteristics of these sensors can be appropriately combined through fusion processing. For example, regarding the position information of the preceding vehicle used in vehicle control such as automatic braking (such as collision mitigation braking) and ACC (Adaptive Cruise Control), information based on the detection data of the camera 1 or information based on the detection data of the laser radar 2 can be used.
[0048] The external recognition unit 11 includes a sensor fusion processing unit 111. The sensor fusion processing unit 111 performs the above-mentioned fusion processing. For example, by using the calibration information stored in the storage unit 14, the coordinates of the target object in the three-dimensional coordinate system (Xw, Yw, Zw) detected by the laser radar 2 are projected onto the camera image (two-dimensional coordinate system (I, J)) obtained by the camera 1. When the coordinates (I, J) of the target object reflected in the camera image and the coordinates (I, J) of the target object projected on the camera image are consistent through this projection, it is determined that the targets detected by the camera 1 and the laser radar 2 can be processed as the same target object, and the targets detected by the camera 1 and the laser radar 2 are associated (also called establishing association).
[0049] It should be noted that the calibration information is calculated by the coordinate axis calibration processing unit 13 described later.
[0050] (Control Processing Unit)
[0051] The control processing unit 12 includes a vehicle position recognition unit 121 , an action plan generation unit 122 , and a travel control unit 123 .
[0052] The vehicle position recognition unit 121 recognizes the position of the vehicle on the map (vehicle position) based on the position information of the vehicle obtained by the positioning unit 5 and the map information of the map database 6. The vehicle position can also be recognized using the map information stored in the storage unit 14 and the surrounding information of the vehicle detected by the camera 1 and the laser radar 2 as external sensors, thereby being able to recognize the vehicle position with high accuracy. It should be noted that when the vehicle position can be measured by an external sensor on or beside the road, the vehicle position can also be recognized by communicating with the sensor via the communication unit 8.
[0053] The vehicle position recognition unit 121 can also calculate the azimuth of the direction photographed by the camera 1 based on the output of the gyro sensor constituting the sensor group 3 .
[0054] The action plan generation unit 122 generates a driving trajectory (target trajectory) of the vehicle from the current time point to a predetermined time based on, for example, the target path calculated by the navigation device 7, the map information stored in the storage unit 14, the vehicle position identified by the vehicle position recognition unit 121, and the external conditions (the position of the object, etc.) identified by the external recognition unit 11. When there are multiple trajectories as candidates for the target trajectory on the target path, the action plan generation unit 122 selects the best trajectory that complies with the law and meets the criteria of efficient and safe driving, and uses the selected trajectory as the target trajectory. Then, the action plan generation unit 122 generates an action plan corresponding to the generated target trajectory. The action plan generation unit 122 generates various action plans corresponding to overtaking driving for overtaking a preceding vehicle, lane change driving for changing the driving lane, following driving for following a preceding vehicle, lane keeping driving for keeping the lane without deviating from the driving lane, constant speed driving, deceleration driving, or acceleration driving. When generating the target trajectory, the action plan generation unit 122 first determines the driving mode and generates the target trajectory according to the driving mode.
[0055] In the automatic driving mode, the driving control unit 123 controls each actuator AC so that the host vehicle travels along the target trajectory generated by the action plan generation unit 122. More specifically, the driving control unit 123 calculates the required driving force for obtaining the target acceleration per unit time calculated by the action plan generation unit 122, taking into account the driving resistance determined by the road slope, etc. in the automatic driving mode. Then, for example, the actuator AC is feedback-controlled so that the actual acceleration detected by the sensor group 3 becomes the target acceleration. That is, the actuator AC is controlled so that the host vehicle travels at the target vehicle speed and the target acceleration.
[0056] It should be noted that, in the manual driving mode, the driving control unit 123 controls each actuator AC based on a driving instruction (steering operation, etc.) from the driver acquired by the sensor group 3 .
[0057] (Coordinate axis calibration processing unit)
[0058] The coordinate axis calibration processing unit 13 calculates calibration information used in the sensor fusion processing unit 111. The calibration information is used as a conversion coefficient for converting from the three-dimensional coordinate system (Xw, Yw, Zw) showing the position of the target object detected by the laser radar 2 to the two-dimensional coordinate system (I, J) of the camera image obtained by the camera 1.
[0059] In the embodiment, the following equation (1) is used as a mathematical expression representing the relationship between the two-dimensional coordinate system (I, J) of the camera image and the world coordinate system (Xw, Yw, Zw).
[0060]
[0061] The left side of equation (1) corresponds to the coordinate system (I, J) of the camera image (image on the image sensor), and the fourth matrix on the right side corresponds to the world coordinate system (Xw, Yw, Zw).
[0062] The first matrix on the right is a matrix that uses homogeneous coordinates to represent internal parameters that take into account the influence of the imaging element (lens, image sensor, etc.) of the camera 1. Symbol S, symbol a, symbol Ti, and symbol Tj correspond to the tilt angle, aspect ratio, image center in the I direction, and image center in the J direction, respectively.
[0063] The second matrix on the right corresponds to the perspective projection model of the pinhole camera, and the symbol f corresponds to the focal length of the lens of camera 1.
[0064] The third matrix on the right is a matrix that uses homogeneous coordinates to represent external parameters that take into account the effects of the installation height and installation angle (rotation about the Xw axis, Yw axis, and Zw axis) of the camera 1 relative to the vehicle. Symbols R11, R12, R13, R21, R22, R23, R31, R32, and R33 correspond to rotation matrices, and symbols tx, ty, and tz correspond to translation parameters.
[0065] When the product of the first matrix to the third matrix on the right side of the above formula (1) is rewritten as a matrix with 3 rows and 4 columns, the following formula (2) is obtained.
[0066]
[0067] According to the above formula (2), the following formulas (3) to (5) are obtained.
[0068] hI=C 11 X w +C 12 Y w +C 13 Z w +C 14 (3)
[0069] hJ=C 21 X w +C 22 Y w +C 23 Z w +C 24 (4)
[0070] h=C 31 X w +C 32 Y w +C 33 Z w +C 34 (5)
[0071] Furthermore, the following equations (6) and (7) are obtained based on the above equations (3) to (5).
[0072] C 31 X w I+C 32 Y w I+C 33 Z w I+I=C 11 X w +C 12 Y w +C 13 Z w +C 14 (6)
[0073] C 31 Xw J+C 32 Y w J+C 33 Z w J+J=C 21 X w +C 22 Y w +C 23 Z w +C 24 (7)
[0074] The coordinate axis calibration processing unit 13 causes the camera 1 and the laser radar 2 to detect the position of the target object used to calculate the calibration information. In more detail, while changing the position of the target object relative to the camera 1 and the laser radar 2 (i.e., relative to the vehicle), the target object position is detected n times. The coordinate axis calibration processing unit 13 substitutes the coordinates (I, J) of the target object obtained by the camera 1 through n detections and the three-dimensional coordinates (Xw, Yw, Zw) of the target object obtained by the laser radar 2 into equations (6) and (7), respectively, to obtain 2n equations. Then, the 2n equations are solved to obtain parameters C11, C12, C13, C14, C21, C22, C23, C24, C31, C32, C33 (these can also be called axis alignment calibration data).
[0075] In the embodiment, the calculation system 200 is constituted by the vehicle V equipped with the above-mentioned vehicle control system 100 and the target Tgt. Figure 2 2 is a schematic diagram showing the positional relationship between an example vehicle V and a target Tgt when the calibration information is calculated by the calculation system 200. In the embodiment, for example, a target Tgt consisting of a cube with a side length of 2 m is used as an object for calculating the calibration information. The target Tgt is supported at a height of 1.5 m from the ground, for example.
[0076] The target Tgt is placed at 11 positions of positions P1 to P11, for example. As a specific example, four positions P1 to P4 at intervals of 5 m in the left-right direction at a position 10 m from the front end of the vehicle V in the depth direction, four positions P5 to P8 at intervals of 5 m in the left-right direction at a position 30 m from the front end of the vehicle V in the depth direction, and three positions P9 to P11 at intervals of 5 m in the left-right direction at a position 60 m from the front end of the vehicle V in the depth direction are placed as the placement positions.
[0077] It should be noted that, in the embodiment, the target Tgt is placed at 11 positions that are moved in parallel in the depth direction and the left-right direction, but the number of the positions P at which the target is placed is at least the number of the above-mentioned parameters C11, C12, C13, C14, C21, C22, C23, C24, C31, C32, and C33, that is, 11 or more. The target Tgt may be placed at 11 positions at the same time, but may be placed in sequence one position at a time.
[0078] In addition, an example is described in which the position P of the loading target Tgt is set to a position 10m, 30m, and 60m away from the front end of the vehicle V in the depth direction, but the position P of the loading target Tgt may be determined according to the distance at which the sensor fusion processing unit 111 performs the fusion processing when the vehicle V actually travels. That is, the position P of the loading target Tgt in the depth direction may be a position 5m, 10m, and 15m away from the front end of the vehicle V, or a position 5m, 10m, and 20m away from the front end of the vehicle V.
[0079] Furthermore, the positions of the loading targets Tgt in the left-right direction may be arranged at intervals of 3 m or 6 m instead of at intervals of 5 m.
[0080] In addition, the height from the ground for supporting the target Tgt may be 1m or 2m. Furthermore, the side length of one side of the cube constituting the target Tgt may not be 2m. For example, when the farthest position P in the depth direction where the target Tgt is placed is within 15m from the front end of the vehicle V, the side length of one side of the cube constituting the target Tgt may be set to be less than 1m.
[0081] Figure 3A 1 is a diagram for explaining the calculation of the coordinates (Xw, Yw, Zw) of the vertex Vtx of the target Tgt obtained based on the point cloud data obtained by the laser radar 2. The coordinate axis calibration processing unit 13 obtains the equations of the planes Pa, Pb and Pc based on the distance information of the plurality of detection points DP included in each of the three adjacent square planes Pa, Pb and Pc in the cube, and calculates the coordinates (Xw, Yw, Zw) of the intersection of the planes Pa, Pb and Pc (in other words, the vertex Vtx of the target Tgt) based on the normal vectors VNa, VNb and VNc of these planes Pa, Pb and Pc.
[0082] Figure 3B1 is a diagram for explaining the calculation of the coordinates (I, J) of the vertex Vtx of the target Tgt in the camera image captured by the camera 1. The coordinate axis calibration processing unit 13 obtains an edge image from which straight line portions are extracted by performing, for example, Hough transformation on the camera image data. The coordinate axis calibration processing unit 13 extracts line segments of the planes Pa, Pb, and Pc that surround three adjacent squares on the cube from the edge image, and calculates the coordinates (I, J) of the intersection of the intersecting straight lines La, Lb, and Lc in the line segments (in other words, the vertex Vtx of the target Tgt).
[0083] The coordinate axis calibration processing unit 13 performs the calculation processing of the calibration information when a maintenance worker or the like instructs to start the calibration processing through the input / output device 4 or the like. The maintenance worker instructs to start the calibration processing at a manufacturing plant of the vehicle V equipped with the vehicle control system 100, a service plant of a sales company that sells the vehicle V equipped with the vehicle control system 100, or the like.
[0084] The coordinate axis calibration processing unit 13 records parameters C11 , C12 , C13 , C14 , C21 , C22 , C23 , C24 , C31 , C32 , and C33 , which are calibration information calculated by the above-mentioned calibration processing, in the storage unit 14 .
[0085] It should be noted that the calculation of the calibration information and the recording in the storage unit 14 may be performed, for example, during the periodic inspection of the vehicle V, when the camera 1 is replaced, when the laser radar 2 is replaced, and the like.
[0086] (Storage Department)
[0087] The storage unit 14 stores the above-mentioned calibration information.
[0088] The storage unit 14 may also store high-precision road map information. The road map information includes: road location information, road shape (curvature, etc.) information, road slope information, intersection and fork location information, lane number information, lane width, and location information of each lane (lane center position, lane boundary information), landmark information (signals, signs, buildings, etc.) as marks on the map, and road surface contour information such as road surface bumps. Landmark information (landmark information) includes landmark shape (contour), characteristics, location, and other information.
[0089] <Vehicle Control>
[0090] The automatic driving ECU 10 constituting the vehicle control system 100 controls each actuator AC using information about objects that are associated as the same object among the objects detected by the camera 1 and the laser radar 2. For example, among the information about objects detected by the camera 1 and the laser radar 2, information based on the detection data of one party is used for vehicle control such as automatic braking and ACC. By configuring in this way, even if the state of one of the camera 1 and the laser radar 2 deteriorates, it is possible to improve redundancy and ensure safety by using information obtained based on the detection data of the other party for vehicle control.
[0091] <Explanation of the flow chart>
[0092] Figure 4 The flowchart is a flowchart for explaining the flow of the calibration information calculation process performed by the automatic driving ECU 10 in the calculation system 200 according to the calculation process program. When the operator instructs to start the calibration process via the input / output device 4 or the like, the automatic driving ECU 10 executes Figure 4 Processing of examples.
[0093] In step S10 , the automatic driving ECU 10 uses the coordinate axis calibration processing unit 13 to obtain the three-dimensional coordinates (Xw, Yw, Zw) of the vertex Vtx of the target Tgt based on the point cloud data detected by the laser radar 2 , and proceeds to step S20 .
[0094] In step S20, the automatic driving ECU 10 uses the coordinate axis calibration processing unit 13 to obtain the two-dimensional coordinates (I, J) of the intersection of the planes Pa, Pb and Pc of the target Tgt (in other words, the vertex Vtx) based on the camera image captured by the camera 1, and enters step S30.
[0095] In step S30, the automatic driving ECU 10 determines whether the acquired coordinate groups have reached a prescribed number. When the coordinate groups acquired by the camera 1 and the laser radar 2 have reached a prescribed number (e.g., 11), the automatic driving ECU 10 determines that step S30 is positive (S30: Yes) and proceeds to step S40. When the coordinate groups acquired by the camera 1 and the laser radar 2 are less than the prescribed number, the automatic driving ECU 10 determines that step S30 is negative (S30: No) and returns to step S10. When returning to step S10, the coordinate axis calibration processing unit 13 repeatedly executes the processing of steps S10 and S20 in a state where the target Tgt has moved to a different position.
[0096] In step S40 , the automatic driving ECU 10 calculates parameters C11 , C12 , C13 , C14 , C21 , C22 , C23 , C24 , C31 , C32 , and C33 as calibration information using the coordinate axis calibration processing unit 13 , and proceeds to step S50 .
[0097] In step S50, the automatic driving ECU 10 records the calibration information in the storage unit 14, and ends. Figure 4 processing.
[0098] According to the above-described embodiment, the following effects can be achieved.
[0099] (1) The computing system 200 calculates calibration information of a vehicle V which is set at a specified position using a camera 1 as a first sensor and a lidar 2 as a second sensor, wherein the first sensor uses an image sensor to obtain a camera image as two-dimensional image information, and the second sensor irradiates a laser as an electromagnetic wave to a plurality of detection points DP, thereby obtaining three-dimensional point cloud data including distance information of each detection point DP. The computing system 200 also includes: a target Tgt, which is arranged outside the vehicle V and has a cubic shape with at least one vertex Vtx formed by three adjacent square-shaped planes Pa, Pb and Pc; a coordinate axis calibration processing unit 13, which serves as a first computing unit and calculates the position (I, J) of the vertex Vtx of the target Tgt as a first position on a two-dimensional coordinate system based on a camera image of the field of view including the target Tgt obtained by the camera 1; the coordinate axis calibration processing unit 13, which serves as a second computing unit and calculates the position (Xw, Yw, Zw) of the vertex Vtx of the target Tgt as a second position on a three-dimensional coordinate system based on respective distance information of a plurality of detection points DP in the field of view including the target Tgt obtained by the laser radar 2; and a coordinate axis calibration processing unit 13, which serves as a third computing unit and calculates calibration information based on the first position and the second position.
[0100] Due to this configuration, in the vehicle V, it is possible to achieve efficient alignment between the two-dimensional coordinate axes in the camera image obtained by the camera 1 and the coordinate axes of the three-dimensional point cloud data obtained by the laser radar 2. That is, it is possible to appropriately perform calibration of the coordinate systems of sensors at different times. Therefore, it is possible to effectively associate the object targets detected by the camera 1 and the laser radar 2.
[0101] (2) In the calculation system 200 of (1) above, the coordinate axis calibration processing unit 13 as the third calculation unit calculates a conversion coefficient for converting the second position into the first position as the calibration information.
[0102] Because of this structure, by using the calibration information calculated for each vehicle V, the position of the target object in the three-dimensional coordinate system (Xw, Yw, Zw) detected by the laser radar 2 of the vehicle V can be projected onto the camera image obtained by the camera 1 of the vehicle V.
[0103] (3) The calculation system 200 of (1) above includes the automatic driving ECU 10 as a control unit that records the calibration information calculated by the coordinate axis calibration processing unit 13 in the storage unit 14 included in the vehicle V.
[0104] With this configuration, in the vehicle V, the calibration information calculated by the coordinate axis calibration processing unit 13 can be repeatedly used.
[0105] (4) In the algorithm system 200 of (3) above, the calibration information recorded in the storage unit 14 can be used for sensor fusion processing performed by the external recognition unit 11 (sensor fusion processing unit 111) as an external recognition device of the vehicle V, which is a comprehensive detection processing based on the camera image obtained by the camera 1 and the point cloud data obtained by the laser radar 2.
[0106] Due to this configuration, the efficiency of the sensor fusion processing in the vehicle V can be improved.
[0107] (5) In the calculation system 200 of (1) above, the coordinate axis calibration processing unit 13 as the first calculation unit calculates the first position based on the edge information of the target Tgt obtained from the camera image acquired by the camera 1 .
[0108] With this configuration, the position (I, J) of the vertex Vtx of the target Tgt in the camera image can be appropriately calculated.
[0109] (6) In the computing system 200 of (1) above, the coordinate axis calibration processing unit 13, which serves as the second computing unit, calculates the second position based on the normal vectors VNa, VNb and VNc, which are the normal information of the three square-shaped surfaces Pa, Pb and Pc obtained from the point cloud data obtained by the laser radar 2.
[0110] Because of this configuration, the coordinates (Xw, Yw, Zw) of the vertex Vtx of the target Tgt can be appropriately calculated based on the point cloud data acquired by the laser radar 2.
[0111] The above-described embodiment can be modified in various forms. Modifications will be described below.
[0112] (Variant 1)
[0113] As an example, the shape of the target Tgt may be not limited to a cube but a rectangular parallelepiped, and may be not limited to a hexahedron but other polyhedrons.
[0114] (Variant 2)
[0115] In the above description, an example is described in which the position of one target Tgt is moved in sequence and placed at 11 positions and measured 11 times during the calibration process. Alternatively, multiple target Tgts may be used to reduce the number of measurements. For example, four target Tgts are placed at positions P1 to P4 for the first measurement, four target Tgts are placed at positions P5 to P8 for the second measurement, and three target Tgts are placed at positions P9 to P11 for the third measurement.
[0116] By adopting the second modification, the number of measurements for obtaining a predetermined number (eg, 11) of coordinate groups acquired by the camera 1 and the laser radar 2 is reduced, thereby shortening the time required for the entire calibration process.
[0117] The above description is merely an example, and the above embodiment and modified examples do not limit the present invention unless the features of the present invention are impaired. One or more of the above embodiment and modified examples may be arbitrarily combined, and the modified examples may be combined with each other.
[0118] According to the present invention, it is possible to appropriately perform calibration of the coordinate system of the sensor at different times.
[0119] The present invention has been described above in conjunction with the preferred embodiments, but it should be understood by those skilled in the art that various modifications and changes can be made without departing from the scope of the disclosure of the claims set forth below.
Claims
1. A computing system, a computing system (200) for calculating calibration information of a vehicle having a first sensor (1) and a second sensor (2) arranged at a predetermined position, wherein the first sensor (1) uses an image sensor to obtain two-dimensional image information, and the second sensor (2) irradiates electromagnetic waves to a plurality of detection points to obtain three-dimensional point cloud data including distance information of each detection point, characterized in that: The computing system (200) comprises: a target (Tgt) disposed outside the vehicle and having a cubic shape with at least one vertex formed by three adjacent square-shaped faces; a first calculation unit that calculates the position of the vertex of the target (Tgt) as a first position on a two-dimensional coordinate system based on image information of a field of view including the target (Tgt) acquired by the first sensor (1); a second calculation unit, which calculates the position of the vertex of the target (Tgt) as a second position on the three-dimensional coordinate system based on the distance information of each of the plurality of detection points in the field of view including the target (Tgt) obtained by the second sensor (2); as well as A third calculation unit is configured to calculate the calibration information according to the first position and the second position.
2. The computing system according to claim 1, characterized in that The third calculation section calculates a conversion coefficient for converting the second position into the first position as the calibration information.
3. The computing system according to claim 1, characterized in that Also available: A control unit (10) records the calibration information calculated by the third calculation unit in a storage unit (14) provided in the vehicle.
4. The computing system according to claim 3, characterized in that: The calibration information recorded by the control unit (10) in the storage unit (14) is used for comprehensive detection processing performed by the vehicle's external recognition device (11) based on the image information obtained by the first sensor (1) and the point cloud data obtained by the second sensor (2).
5. The computing system according to any one of claims 1 to 4, characterized in that: The first calculation unit calculates the first position based on edge information of the target (Tgt) obtained from the image information acquired by the first sensor (1).
6. The computing system according to any one of claims 1 to 4, characterized in that: The second calculation unit calculates the second position based on normal information of three faces of the square shape obtained from the point cloud data acquired by the second sensor (2).
