In-vehicle camera system

Abstract

We provide an in-vehicle camera system that can optimize camera calibration. [Solution] The external image acquisition unit (onboard camera) 41 acquires vehicle type information from the information of the preceding vehicle captured by the onboard camera, and based on the vehicle height, vehicle width, and distance to the preceding vehicle, the mounting position of the external image acquisition unit 41 is determined and the onboard camera is calibrated. This allows the onboard camera to be properly calibrated regardless of the road being traveled on.

Description

[Technical Field]

[0001] This invention relates to an in-vehicle camera system. In particular, this invention relates to an improvement in the technology for calibrating in-vehicle cameras. [Background technology]

[0002] Conventionally, as disclosed in Patent Document 1, for example, an in-vehicle camera system is known that uses an in-vehicle camera (hereinafter sometimes simply referred to as a camera) to capture images of the area around a vehicle for use in assisting vehicle operation.

[0003] In this type of system, the camera mounting height, mounting position (e.g., horizontal camera mounting position), and mounting angle are given as design values ​​for each vehicle, and assuming that these values ​​are obtained accurately (as per the design values), the system estimates the positional relationship of surrounding objects, lane markings, and preceding vehicles based on the images captured by the camera. Furthermore, Patent Document 1 discloses a method for calibrating the on-board camera by detecting two lane division lines that are assumed to be parallel to each other and determining their degree of parallelism, thereby enabling accurate lane marking detection. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 10-40499 [Overview of the project] [Problems that the invention aims to solve]

[0005] However, if the vehicle's ride height is modified, the camera's mounting height, position, and angle may differ from the design values. Furthermore, other factors may also cause these positions and angles to deviate from the design values. As mentioned above, the in-vehicle camera system is designed to estimate the positional relationship with surrounding objects, lane markings, and preceding vehicles, assuming that these positions and angles are accurately obtained according to the design values. Therefore, if these positions and angles deviate from the design values, it may result in inaccurate acquisition of external information by the camera, potentially leading to diagnostic errors, unnecessary activation or failure of safety systems, and ultimately compromising the system's reliability.

[0006] Furthermore, while the in-vehicle camera system disclosed in Patent Document 1 can perform proper position correction (correction of deviations from design values) on roads where the road structure is legally defined and well-maintained, such as expressways, it cannot be used on other types of roads. Therefore, there has been a need for an in-vehicle camera system that can recognize deviations in position and angle from design values ​​and perform proper camera calibration regardless of the type of road being traveled on.

[0007] This invention has been made in view of the above, and its objective is to provide an in-vehicle camera system that can optimize the calibration of the camera. [Means for solving the problem]

[0008] The present invention provides a solution for achieving the above objectives, which is based on an in-vehicle camera system capable of calibration. This in-vehicle camera system is characterized by acquiring vehicle type information of other vehicles from information of other vehicles captured by the in-vehicle camera, and determining the mounting position of the in-vehicle camera on the own vehicle and performing calibration of the in-vehicle camera based on at least one of the following pieces of information: vehicle height, vehicle width, and distance to the other vehicle.

[0009] This specific feature allows the mounting position of the on-board camera to be determined using information from other vehicles, and the on-board camera is then calibrated. In other words, the mounting position of the on-board camera can be determined without using road information (such as road image information), so the on-board camera can be properly calibrated regardless of the road being traveled on. [Effects of the Invention]

[0010] In this invention, vehicle type information of other vehicles is obtained from information captured by an in-vehicle camera, and the mounting position of the vehicle's in-vehicle camera is determined and the in-vehicle camera is calibrated based on at least one of the following pieces of information: vehicle height, vehicle width, and distance to the other vehicle. Therefore, the in-vehicle camera can be properly calibrated regardless of the road being traveled on. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram showing the schematic configuration of an in-vehicle camera system according to an embodiment. [Figure 2] Figure 2(a) shows the horizontal camera coordinate system perpendicular to the direction of vehicle travel, and Figure 2(b) shows the coordinate system of the camera image of the preceding vehicle captured by the on-board camera. [Figure 3] This is a flowchart illustrating the process of updating camera mounting parameters. [Modes for carrying out the invention]

[0012] The embodiments of the present invention will be described below with reference to the drawings. In this embodiment, the present invention will be described in the case where it is applied as an in-vehicle camera system capable of calibrating a camera that photographs the front of the vehicle (in-vehicle camera). In other words, the case in which the other vehicle referred to in the present invention is the preceding vehicle will be described. The present invention can also be applied as an in-vehicle camera system capable of calibrating other cameras (cameras that photograph the rear of the vehicle). In this case, the other vehicle referred to in the present invention will be the following vehicle.

[0013] Figure 1 is a block diagram illustrating the schematic configuration of the in-vehicle camera system 1 according to this embodiment. The in-vehicle camera system 1 includes an in-vehicle system 2 mounted on a vehicle and a data center server (hereinafter simply referred to as "server") 3 installed in a data center. The in-vehicle system 2 and the server 3 can communicate with each other via a network. Although Figure 1 shows the communication status between the in-vehicle system 2 of one vehicle and the server 3, the server 3 can communicate with in-vehicle systems 2 of multiple vehicles.

[0014] The in-vehicle system 2 consists of a surrounding environment recognition device 4, an in-vehicle equipment database 5, and an information processing device (camera system ECU) 6.

[0015] The surrounding environment recognition device 4 includes an external image acquisition unit 41, such as an in-vehicle camera, and an external distance acquisition unit 42, such as a millimeter-wave radar.

[0016] The external image acquisition unit 41 includes an imaging device and an image analysis device. As the imaging device, for example, a CCD (Charge Coupled Device), a CMOS (Complementary Metal-Oxide Semiconductor), etc. can be applied. The imaging device is installed at the front of the vehicle (the host vehicle). The imaging device captures an image of the front of the vehicle at a predetermined frame rate to acquire image data, and transmits the image data to the image analysis device. The image analysis device recognizes, in addition to the preceding vehicle, lane dividing lines, curbs, median strips, etc. that demarcate the driving lane, and transmits the recognition result to the information processing device 6. In the following description, the external image acquisition unit 41 may also be referred to as an in-vehicle camera.

[0017] The external distance acquisition unit 42 includes a transceiver and a signal processing unit. The transceiver emits radio waves in the millimeter wave band (hereinafter referred to as "millimeter waves") forward of the vehicle, and receives the millimeter waves (reflected waves) reflected by an object located within the radiation range. The signal processing unit recognizes the distance between the vehicle and the object (e.g., the preceding vehicle), the relative position (direction) of the object with respect to the vehicle, etc. based on the time from when the transceiver emits the millimeter waves until the reflected waves are received, and physical quantities related to the transmitted millimeter waves and the received reflected waves, and transmits the recognition result to the information processing device 6. Incidentally, the external distance acquisition unit 42 may be a lidar (LIDAR: Laser Imaging Detection And Ranging).

[0018] The in-vehicle device database 5 stores information on various devices mounted on the vehicle. In particular, the in-vehicle device database 5 stores the mounting height, mounting position, mounting angle, etc. of the external image acquisition unit 41 in the vehicle interior (hereinafter, these values or values defined as correlated with these values will be collectively referred to as camera mounting parameters). These camera mounting parameters are stored as predefined design values before the "update process of camera mounting parameters" described later is performed. When the "update process of camera mounting parameters" is performed and the camera mounting parameters are changed, the stored information in the in-vehicle device database 5 will be updated with the changed camera mounting parameters. The "update process of camera mounting parameters" will be described later.

[0019] The information processing device 6 includes a processor such as a CPU (Central Processing Unit), a ROM (Read-Only Memory) that stores a control program, a RAM (Random-Access Memory) that temporarily stores data, and input / output ports, etc.

[0020] As functional units realized by the control program, the information processing device 6 includes an information reception unit 61, a vehicle type discrimination unit 62, an information transmission unit 63, a parameter calculation unit 64, a parameter correction value storage unit 65, and a camera mounting parameter determination unit 66.

[0021] The information receiving unit 61 is capable of receiving various types of information, including recognition results from the external image acquisition unit 41 (especially image information of the preceding vehicle), recognition results from the external distance acquisition unit 42 (especially distance information to the preceding vehicle), and camera mounting parameters from the in-vehicle equipment database 5. The information receiving unit 61 is also capable of receiving information from the server 3 (height data, length data, width data, etc. of the preceding vehicle). This information from the server 3 is transmitted in response to a request for preceding vehicle information (a request for preceding vehicle information corresponding to the type of preceding vehicle determined by the image information of the preceding vehicle from the external image acquisition unit 41) transmitted from the information processing device 6.

[0022] The vehicle type identification unit 62 identifies the vehicle type of the preceding vehicle based on the image information of the preceding vehicle received as a recognition result from the external image acquisition unit 41. For example, the information processing device 6 stores image information for multiple types of vehicles (for example, image information viewed from the rear), and the vehicle type identification unit 62 performs a matching process between the image information of the preceding vehicle received from the external image acquisition unit 41 and the stored image information for each vehicle to identify the vehicle type of the preceding vehicle. However, the method for identifying the vehicle type of the preceding vehicle is not limited to this. For example, the vehicle type of the preceding vehicle may be identified through communication with the preceding vehicle. The result of this vehicle type identification is output to the information transmission unit 63.

[0023] The information transmission unit 63 transmits the vehicle identification result information input from the vehicle identification unit 62 to the server 3 (transmits a request signal for preceding vehicle information). The server 3 is equipped with a vehicle database 31, which stores information (vehicle height data, vehicle length data, vehicle width data) for multiple types of vehicles. When the server 3 receives the vehicle identification result information, if there is a vehicle type in the vehicle database 31 that matches the received information, it transmits the vehicle information (vehicle height data, vehicle length data, vehicle width data) for that vehicle type to the information receiving unit 61.

[0024] Parameter calculation unit 64 calibrates the external image acquisition unit 41 based on each piece of information received by the information reception unit 61, and calculates the current camera mounting parameters of the external image acquisition unit 41 (hereinafter sometimes referred to as calculated parameters). Then, this parameter calculation unit 64 transmits information on the calculated parameters (more specifically, correction values described later) to the parameter correction value storage unit 65. FIG. 2 is a diagram for explaining the calculation operation of the calculated parameters. FIG. 2(a) shows a camera coordinate system in the horizontal direction orthogonal to the traveling direction of the vehicle (the host vehicle V1 and the preceding vehicle V2), and FIG. 2(b) is a diagram showing the coordinate system of the camera image of the preceding vehicle V2 taken by the external image acquisition unit (in-vehicle camera) 41. As shown in FIG. 2(a), when the preceding vehicle V2 exists in front of the vehicle (the host vehicle) V1, the external image acquisition unit 41 captures the rear of the preceding vehicle V2, and the external distance acquisition unit 42 measures the distance V dN to the preceding vehicle V2. As shown in FIG. 2(b), camera coordinates (V W , V h , V d ) are configured, and the positions (V xN , V yN ) of N points (N = 1 to 3) in the camera image in the camera coordinate system (V WN , V hN , V dN ) are calculated. Then, calibration of the external image acquisition unit 41 is performed using the positions of the N points in the camera image and the positions in the camera coordinate system, and the current camera mounting parameters of the external image acquisition unit 41 are calculated as calculated parameters. In FIG. 2(b), as the N points in the camera image, the center position of the roof of the preceding vehicle V2 (corresponding to the uppermost position of the preceding vehicle V2 in the camera image; N = 1), the rightmost position of the preceding vehicle V2 in the camera image (N = 2), and the leftmost position of the preceding vehicle V2 in the camera image (N = 3) are shown as target positions respectively.

[0025] The parameter correction value storage unit 65 stores the difference between the calculated parameters received from the parameter calculation unit 64 and the camera mounting parameters currently stored in the in-vehicle equipment database 5 as a correction value at predetermined intervals. Specifically, for example, if modifications such as adjusting the vehicle's ride height are made, and at least one of the camera's mounting height, mounting position, or mounting angle deviates from the design value, the correction value will increase in proportion to that deviation. If the correction value increases even though no modifications such as adjusting the vehicle's ride height have been made, this is, for example, due to the influence of external disturbances, and the correction value will be obtained as small in subsequent operations. On the other hand, if modifications such as adjusting the vehicle's ride height have been made, the state in which the correction value is large will continue. In this way, the parameter correction value storage unit 65 stores (stores) the correction values ​​calculated at predetermined intervals.

[0026] The camera mounting parameter determination unit 66 uses the correction values ​​stored in the parameter correction value storage unit 65 to determine whether or not the camera mounting parameters stored in the in-vehicle equipment database 5 should be updated. If it determines that the parameters should be updated, it stores the updated mounting parameters in the in-vehicle equipment database 5. Specifically, when the number of correction values ​​calculated by the parameter calculation unit 64 and stored in the parameter correction value storage unit 65 is the Mth vehicle of the preceding vehicle V2, it calculates the variation in those correction values. If this variation is above a predetermined threshold, it is determined that the reliability of the calculated parameters is low, and therefore the camera mounting parameters should not be updated. Also, if the number of correction values ​​has not reached the Mth vehicle of the preceding vehicle V2, it is determined that the camera mounting parameters should not be updated due to the small sample size. On the other hand, if the number of correction values ​​reaches the Mth vehicle V2, the variation in the correction values ​​is less than a predetermined threshold, and the difference between the value obtained by adding a representative value (e.g., average value) of the correction values ​​(the value of the camera mounting parameter with the correction values ​​added) and the camera mounting parameter currently stored in the in-vehicle equipment database 5 is greater than or equal to a predetermined threshold, then it is determined that the camera mounting parameter should be updated, and the updated mounting parameter is transmitted to the in-vehicle equipment database 5 for storage.

[0027] Next, the calibration process for the in-vehicle camera in the in-vehicle camera system 1 configured as described above will be explained. Figure 3 is a flowchart illustrating the calibration process for the in-vehicle camera. This flowchart is executed repeatedly when the vehicle's start switch is ON.

[0028] First, in step ST1, when the camera correction mode is turned ON, in step ST2, camera mounting parameters (initial camera coordinate values) are obtained from the in-vehicle equipment database 5. The timing for turning on the camera correction mode can be set arbitrarily. For example, the camera correction mode can be turned ON each time the vehicle's start switch is turned ON, and after the calibration process of the in-vehicle camera is performed, the camera correction mode can be turned OFF until the vehicle's start switch is turned OFF. In step ST3, when the vehicle (own vehicle) V1 is moving, in step ST4, it is determined whether or not the vehicle V1 has stopped. This determination is made based on a signal from a vehicle speed sensor (not shown). If the vehicle V1 is moving and the determination in step ST4 is NO, the system waits for the vehicle V1 to stop.

[0029] On the other hand, if vehicle V1 stops and a YES determination is made in step ST4, the process moves to step ST5, where it is determined whether or not a preceding vehicle V2 exists. This determination is made based on the recognition results from the external image acquisition unit 41 (in particular, the image information of the preceding vehicle V2). If no preceding vehicle V2 exists and a NO determination is made in step ST5, the process returns to step ST4.

[0030] On the other hand, if a preceding vehicle V2 exists and the result in step ST5 is YES, the process moves to step ST6 to determine the vehicle type of the preceding vehicle V2. This determination is performed by the vehicle type determination unit 62. In step ST7, vehicle height data, vehicle length data, and vehicle width data of the preceding vehicle V2 are acquired. Specifically, the information of the vehicle type determination result is sent to the server 3, and if the server 3 has stored information about the vehicle type corresponding to the received information, it will send the information about that vehicle type (vehicle height data, vehicle length data, vehicle width data) to the information receiving unit 61.

[0031] In step ST8, the external image acquisition unit 41 is calibrated as described above, and the current camera mounting parameters of the external image acquisition unit 41 are calculated as calculated parameters. In step ST9, the difference between the calculated parameters received from the parameter calculation unit 64 and the camera mounting parameters currently stored in the in-vehicle equipment database 5 is accumulated as a correction value.

[0032] In step ST10, it is determined whether the number of correction values ​​stored in the parameter correction value storage unit 65 belongs to the Mth vehicle of the preceding vehicle V2. If it does not belong to the Mth vehicle and the determination in step ST10 is NO, the process returns to step ST4.

[0033] On the other hand, if the number of correction values ​​stored in the parameter correction value storage unit 65 is that of the Mth vehicle V2, and a YES determination is made in step ST10, the process moves to step ST11 to determine whether the variation in the correction values ​​is within a predetermined threshold d1. If the variation in the correction values ​​exceeds the predetermined threshold d1, and a NO determination is made in step ST11, the process returns to step ST4.

[0034] On the other hand, if the variation in the correction value is within a predetermined threshold d1 and a YES determination is made in step ST11, the process moves to step ST12, where it is determined whether the difference between the value obtained by adding a representative value (e.g., the average value) of the correction value (the value of the camera mounting parameter with the correction value added) and the camera mounting parameter currently stored in the in-vehicle equipment database 5 is greater than or equal to a predetermined threshold d2. If this difference is less than the predetermined threshold d2 and a NO determination is made in step ST12, the process returns to step ST4.

[0035] On the other hand, if the difference is greater than or equal to a predetermined threshold d2 and a YES determination is made in step ST12, the process moves to step ST13, where the camera mounting parameters are updated with the correction value added, and these updated mounting parameters are sent to the in-vehicle equipment database 5 for storage. At this point, the cumulative value of the correction value (the cumulative value used to determine whether the number of samples has reached M) is reset. The above operations are repeated.

[0036] As described above, in this embodiment, vehicle type information of the preceding vehicle V2 is obtained from the information of the preceding vehicle V2 captured by the external image acquisition unit 41, and the mounting position of the external image acquisition unit 41 of the vehicle V1 is determined and the external image acquisition unit 41 is calibrated based on the vehicle height, vehicle width, and distance to the preceding vehicle V2. Therefore, the external image acquisition unit 41 can be properly calibrated regardless of the road being traveled on.

[0037] Furthermore, the present invention is not limited to the embodiments described above, and all modifications and applications are possible within the scope of the claims and equivalents thereof.

[0038] For example, in the above embodiment, when vehicle V1 stops, it is determined whether or not a preceding vehicle V2 is present, and the "camera mounting parameter update process" is performed on the condition that the preceding vehicle V2 is present. The present invention is not limited to this, and the "camera mounting parameter update process" may also be performed on the condition that the same preceding vehicle V2 is present ahead for a predetermined time while vehicle V1 is in motion (especially in situations where distance measurement can be performed with a certain degree of consistency). Alternatively, by employing a surrounding environment recognition device 4 that can recognize the surrounding environment (especially the preceding vehicle V2) with higher accuracy than in this case, the "camera mounting parameter update process" may be performed regardless of whether the vehicle is stopped or in motion.

[0039] Furthermore, in the above embodiment, vehicle height data, vehicle length data, and vehicle width data were given as examples of information for identifying the preceding vehicle V2. However, other information such as the vertical and horizontal dimensions of glass (rear window, etc.) and the distance between side mirrors may also be used to identify the type of vehicle of the preceding vehicle V2. In addition, when applied to an in-vehicle camera system that performs calibration of a camera that photographs the rear of the vehicle, the type of vehicle of the following vehicle will be identified, so the type of vehicle of the following vehicle may be identified using information on the vertical and horizontal dimensions of the windshield of the following vehicle. [Industrial applicability]

[0040] This invention is applicable to an in-vehicle camera system for performing calibration of an in-vehicle camera. [Explanation of Symbols]

[0041] 1…In-vehicle camera system 4…Surrounding environment recognition device 41…External image acquisition unit (in-vehicle camera) 6...Information processing unit 62...Vehicle type identification unit 64...Parameter calculation unit V1...Own vehicle (vehicle) V2…Preceding vehicle

Claims

[Claim 1] An in-vehicle camera system capable of calibrating the in-vehicle camera, An in-vehicle camera system characterized by having the following configuration: obtaining vehicle type information of other vehicles from information of other vehicles captured by the in-vehicle camera, determining the mounting position of the in-vehicle camera on the own vehicle and performing calibration of the in-vehicle camera based on at least one of the following pieces of information: vehicle height, vehicle width, and distance to the other vehicle.

Images