Object detection device and calibration method of object detection device

Abstract

To provide an object detection device capable of being used in various cameras 10 and executing calibration with high accuracy.SOLUTION: An object detection device 1 includes a camera 10 for acquiring an image 11 in front of a vehicle, a distance measuring sensor 20 for acquiring point group data 21 in front of the vehicle, and a correction unit 30 for executing calibration on the basis of the image 11 and the point group data 21. The correction unit 30 sets a bounding box 12 in the image 11, detects a target 3 located in the bounding box 12, acquires a deviation amount L and a direction D to a recognition area 12A of a point group 22 on the condition that the point group 22 corresponding to the target 3 is deviated from the recognition area 12A surrounded by the bounding box 12, and executes calibration of at least one of the camera 10 and the distance measuring sensor 20 on the basis of the deviation amount L and the direction D.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an object detection device for a vehicle or the like and a method for calibrating the object detection device. [Background technology]

[0002] Conventionally, technologies such as automatic driving control technology and collision avoidance support technology have been developed for various vehicles. For vehicles that employ these technologies, driving control devices using various imaging devices (stereo cameras, monocular cameras, etc.), radar devices, sensors (sonar devices), etc. are known (for example, Patent Document 1). The driving control device described in the above Patent Document 1 is said to control the driving of the vehicle using image data acquired by a stereo camera and three-dimensional point cloud data acquired by a LiDAR (corresponding to a distance measuring sensor). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-157096 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the driving control device described in the above Patent Document 1 is capable of acquiring depth information using both a stereo camera and a LiDAR (corresponding to a distance measuring sensor), and both are expensive. In other words, the driving control device described in the above Patent Document 1 has an expensive and redundant configuration. Therefore, there is a demand for an object detection device that uses an inexpensive monocular camera instead of a stereo camera.

[0005] However, in the object detection device used in the driving control device, for example, image shifts may occur due to reasons such as position shifts caused by the installation accuracy when the imaging device is installed on the vehicle or aging. As a result, there is a concern that malfunctions may occur in the driving control of the vehicle performed using image information acquired by the imaging device. For this reason, the driving control device described in the above-mentioned Patent Document 1 discloses a technique for calibrating a stereo camera. However, the driving control device described in the above-mentioned Patent Document 1 discloses a calibration technique related to a stereo camera having depth information, and there is a problem that it cannot be applied to the calibration of a monocular camera that does not have depth information.

[0006] Therefore, an object of the present invention is to provide an object detection device and a calibration method for an object detection device that can be used in various cameras including a monocular camera and that can perform calibration with high accuracy. [Means for solving the problem]

[0007] (1) The object detection device of the present invention provided to solve the above-mentioned problems is an object detection device in a vehicle, comprising: a camera that captures an image of the area in front of the vehicle, a ranging sensor that captures point cloud data of the area in front of the vehicle, and a correction unit that performs calibration based on information about the image and the point cloud data acquired by the camera and the ranging sensor, wherein the correction unit sets a bounding box in the image and detects a specified target located within the bounding box, and, under the condition that, in the point cloud data acquired by the ranging sensor, a specified point cloud corresponding to the target is shifted with respect to a recognition area surrounded by the bounding box, acquires the amount of deviation and direction of the point cloud with respect to the recognition area, and performs calibration of either the camera or the ranging sensor or both based on the amount of deviation and the direction.

[0008] The above-mentioned object detection device is configured to detect a predetermined target located within a bounding box set in an image captured by a camera in the correction unit. The above-mentioned object detection device is configured to recognize a predetermined point cloud corresponding to a target from point cloud data including three-dimensional information (also referred to as depth information) acquired by a distance measurement sensor, and acquire a shift amount and direction of the point cloud with respect to the recognition area on the condition that the point cloud is shifted with respect to the recognition area surrounded by the bounding box. In other words, the above-mentioned object detection device can recognize a point cloud corresponding to a target on the same plane as the bounding box from the three-dimensional point cloud data on a plane formed by the bounding box set in the two-dimensional image, and acquire a shift amount and direction of the point cloud with respect to the recognition area. Here, for example, a LiDAR (Light Detection And Ranging) can be used as the distance measurement sensor. The above-mentioned object detection device can perform calibration of either one or both of the camera and the distance measurement sensor based on the acquired shift amount and direction. It is preferable that the above-mentioned detection of the target by the camera and the distance measurement sensor is performed in a state where the vehicle is stopped. Moreover, it is preferable that the target is one that is in a stationary state.

[0009] In this way, the above-mentioned object detection device can perform calibration based on the two-dimensional image captured by the camera and the three-dimensional point cloud data measured by the distance measuring sensor. That is, the above-mentioned object detection device can perform calibration with high accuracy even if the image captured by the camera does not contain three-dimensional information. Therefore, the above-mentioned object detection device can reduce the cost of the camera. In addition, the above-mentioned object detection device can use various cameras capable of acquiring two-dimensional images, regardless of whether they are stereo cameras or monocular cameras, thereby improving versatility.

[0010] (2) In the object detection device of the present invention described above, the correction unit may detect a plurality of targets and obtain a plurality of deviation amounts and directions corresponding to each of the plurality of targets, and the calibration may be performed based on the plurality of deviation amounts and directions.

[0011] The above-described object detection device can perform calibration based on a plurality of deviation amounts and directions for a plurality of targets, thereby improving the accuracy of the calibration.

[0012] (3) The object detection device of the present invention described above may be characterized in that the camera is a monocular camera.

[0013] The above-mentioned object detection device is expected to reduce costs by using a monocular camera as the camera. In addition, the above-mentioned object detection device can unify the acquisition of depth information into a distance measuring sensor, thereby suppressing redundancy of the device due to duplication of the depth information acquisition function.

[0014] (4) In the object detection device of the present invention described above, the correction unit may be characterized in that it sets the bounding box based on position coordinates of the target estimated using AI analysis from an image acquired by the camera.

[0015] By configuring the above-mentioned object detection device in this way, it is possible to estimate and detect the target object by analysis using AI. Therefore, the above-mentioned object detection device can set a bounding box for the optimal target object. This allows the above-mentioned object detection device to perform calibration with high accuracy. Here, in the analysis using AI, for example, it is preferable to estimate as the target object an object that has a flat surface and is easy to detect, such as a road sign.

[0016] (5) In the object detection device of the present invention described above, the image and the point cloud data are acquired when the vehicle is stopped, and the correction unit obtains the predetermined point cloud corresponding to the target by projecting the point cloud data onto the image using a projection matrix acquired in advance by performing the calibration.

[0017] The object detection device described above can easily obtain a predetermined point cloud corresponding to a target object by using a projection matrix obtained in advance, by using the projection matrix obtained in advance. That is, the object detection device described above can easily extract point cloud data on a two-dimensional same plane set as a bounding box from three-dimensional point cloud data by using a projection matrix prepared in advance.

[0018] (6) The object detection device of the present invention described above may be characterized in that the target object includes at least one of a road sign, a signboard, and another stopped vehicle different from the vehicle.

[0019] The object detection device described above, having such a configuration, can acquire images and point cloud data based on the stopped target, and can perform calibration with high accuracy using the images and point cloud data. Note that targets used for calibration are preferably road signs, signboards, etc. with flat surfaces. By forming the target surfaces flat in this way, distance measurement errors for the target can be reduced, and calibration can be performed with high accuracy.

[0020] (7) The object detection device of the present invention described above may be characterized in that a plurality of bounding boxes are set for each of a plurality of targets, and calibration is not performed in the correction unit under the condition that at least a portion of the plurality of bounding boxes overlap with each other.

[0021] The above-described object detection device can eliminate overlapping bounding boxes by using such a configuration, and can therefore eliminate images and point cloud data in which the bounding boxes overlap and the target cannot be accurately recognized, thereby enabling accurate calibration.

[0022] (8) A method for calibrating an object detection device of the present invention, which is provided to solve the above-mentioned problems, uses the above-mentioned object detection device and includes a vehicle stopped state determination step of determining whether the vehicle is stopped or not, a point cloud data acquisition step of acquiring the point cloud data by the distance measurement sensor, on condition that it is determined that the vehicle is stopped in the vehicle stopped state determination step, an image acquisition step of acquiring the image by the camera, a bounding box presence / absence determination step of determining the presence or absence of the bounding box in the image by analyzing the image, a coordinate conversion step of converting the position coordinates of the bounding box into distance measurement coordinates in the distance measurement sensor, on condition that it is determined that the bounding box is present in the bounding box presence / absence determination step, and a coordinate conversion step of converting the position coordinates of the bounding box into distance measurement coordinates in the distance measurement sensor. the method for calculating a point cloud located on the same plane from the point cloud data located within a recognition area; a point cloud position determination step for determining whether or not the point cloud on the same plane exists outside the recognition area; a deviation amount / direction calculation step for calculating a deviation amount and direction of the point cloud with respect to the recognition area on the condition that the point cloud on the same plane exists outside the recognition area in the point cloud position determination step; a correction coefficient calculation step for calculating a correction coefficient for the camera or the distance measuring sensor based on the deviation amount and direction calculated in the deviation amount / direction calculation step; a correction coefficient setting step for storing the correction coefficient calculated in the correction coefficient calculation step in the correction unit; and a calibration execution step for executing the calibration based on the correction coefficient set in the correction coefficient setting step.

[0023] In the above-mentioned calibration method for the object detection device, the point cloud data acquisition step and the image acquisition step are executed on the condition that the vehicle is determined to be in a stopped state in the vehicle stop state determination step. Therefore, the processing load of the acquired point cloud data and images can be reduced, and the detection error of the target can be reduced. In addition, in the coordinate conversion step, the position coordinates of the bounding box are converted into the distance measurement coordinates (also called LiDAR coordinates) in the distance measurement sensor. That is, the position coordinates corresponding to the two-dimensional plane of the bounding box are converted into the distance measurement coordinates. As a result, in the point cloud calculation step, point cloud data located on the plane of the bounding box is calculated. In other words, of the three-dimensional point cloud data acquired by the distance measurement sensor, two-dimensional point cloud data on the same plane as the bounding box is calculated. In addition, based on the calculated point cloud data, the deviation amount / direction calculation step, the correction coefficient calculation step, and the correction coefficient setting step are executed, and the calibration is executed in the calibration execution step based on the calculated correction coefficient.

[0024] In this way, the above-mentioned calibration method can calculate the correction coefficient by replacing the three-dimensional point cloud data acquired by the distance measurement sensor with two-dimensional point cloud data, so that even if the image captured by the camera does not contain three-dimensional information, the calibration method can perform calibration with high accuracy. Therefore, the above-mentioned calibration method can reduce the cost of the camera. In addition, the above-mentioned calibration method can use various cameras capable of acquiring two-dimensional images, regardless of whether they are stereo cameras or monocular cameras, and therefore improves versatility. Effect of the Invention

[0025] According to the present invention, it is possible to provide an object detection device and a calibration method for an object detection device that can be used in various cameras including a monocular camera and that can perform calibration with high accuracy. [Brief description of the drawings]

[0026] [Figure 1]1 is a schematic configuration diagram of an object detection device according to an embodiment of the present invention; [Diagram 2] 1 is an explanatory diagram showing a state in which a point cloud is shifted outside a bounding box in an object detection device of the present invention; [Diagram 3] FIG. 10 is an explanatory diagram showing a state in which a point cloud falls within a bounding box after calibration is performed in the object detection device of the present invention; [Figure 4] 4 is a flow diagram illustrating an embodiment of a calibration method for an object detection device of the present invention. [Diagram 5] This is a continuation of the flow diagram in Figure 4. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] Hereinafter, the details of an object detection device 1 according to an embodiment of the present invention will be described with reference to the attached Figs. 1 to 5. These figures are schematic diagrams, and the sizes are not necessarily drawn to exact proportions. In addition, in the figures, similar components are indicated by the same reference numerals. Also, please note that in Figs. 2 and 3, the point cloud data 21 and the point cloud 22 are drawn exaggerated compared to their actual sizes and pitches in order to facilitate understanding. Also, please note that in Figs. 2 and 3, the point cloud data 21 and the point cloud 22 are only those corresponding to the target 3 in order to facilitate understanding.

[0028] As shown in Fig. 1, the object detection device 1 is mounted on a vehicle 2. The object detection device 1 includes a camera 10, a distance measurement sensor 20, a correction unit 30, etc. An object (corresponding to a target 3 (see Figs. 2 and 3)) detected by the object detection device 1 is used, for example, for automatic driving control of the vehicle 2 or a collision avoidance support system.

[0029] In this embodiment, for example, a monocular camera is used as the camera 10. The camera 10 is attached, for example, to the upper inside part of the windshield 2A of the vehicle 2. The camera 10 can capture an image of the front of the vehicle 2 to obtain a two-dimensional image 11 (see FIG. 2). The camera 10 can capture an image of the front side of the vehicle 2 at a predetermined angle of view. In this embodiment, the camera 10 captures and obtains the image 11 when the vehicle 2 is stopped. Here, the image 11 captured by the camera 10 includes a target 3, which will be described later. Note that the camera 10 is not limited to a monocular camera, and various types of cameras such as a stereo camera can be used.

[0030] In this embodiment, the distance measurement sensor 20 is, for example, a LiDAR (Light Detection And Ranging). The LiDAR is a remote sensing (sensing from a remote position using a sensor) method that irradiates an object with near-infrared light, visible light, or ultraviolet light, captures the reflected light with an optical sensor, and measures the distance. The distance measurement sensor 20 is attached, for example, to the upper inside part of the windshield 2A of the vehicle 2. The distance measurement sensor 20 can acquire point cloud data 21 (see Figs. 2 and 3) in front of the vehicle 2. In this embodiment, the distance measurement sensor 20 acquires the point cloud data 21 when the vehicle 2 is stopped. The distance measurement sensor 20 is provided, for example, adjacent to the camera 10. Here, the point cloud data 21 acquired by the distance measurement sensor 20 includes three-dimensional information (also referred to as depth information).

[0031] The correction unit 30 is formed as a computer capable of executing various calculations necessary for image processing and calibration. The correction unit 30 includes a storage means (memory, HDD, SDD, etc.) for storing correction coefficients related to calibration and conversion tables and conversion equations necessary for correction and the like. The correction unit 30 can perform calibration of either or both of the camera 10 and the distance measurement sensor 20 (the camera 10 in this embodiment) based on information related to the image 11 acquired by the camera 10 and the point cloud data 21 acquired by the distance measurement sensor 20. As shown in FIG. 2 and FIG. 3, the correction unit 30 is configured to set a bounding box 12 in the image 11. Specifically, the correction unit 30 is configured to set the bounding box 12 based on the position coordinates of a predetermined target 3 estimated from the image 11 acquired by the camera 10 using an analysis by AI.

[0032] In this embodiment, two targets 3, 3, a road sign 3A and another vehicle 3B that is stopped, are estimated by analysis using AI, and a bounding box 12 is set for each of the two targets 3, 3. That is, in this embodiment, two bounding boxes 12, 12 are set.

[0033] In this way, the above-described object detection device 1 can infer and detect the target object 3 by analysis using AI. Therefore, the above-described object detection device 1 can set the bounding box 12 for the optimal target object 3. This allows the above-described object detection device 1 to perform calibration with high accuracy. Here, in the analysis using AI, it is preferable to infer that the target object 3 is an object that has a flat surface and is easy to detect, such as a road sign 3A.

[0034] The bounding box 12 is formed in a rectangular shape, forming a two-dimensional plane. A recognition area 12A is set within the range surrounded by the bounding box 12.

[0035] The correction unit 30 can detect the targets 3, 3 located within the bounding box 12 (recognition area 12A). The correction unit 30 obtains a predetermined point cloud 22 corresponding to the target 3 by projecting the point cloud data 21 onto the image 11 using a projection matrix acquired by performing calibration in advance. Specifically, the correction unit 30 can obtain a projection matrix for projecting the point cloud data 21 onto the same plane as the bounding box 12 (two-dimensional image 11) by performing calibration, which will be described later, on the point cloud data 21 including three-dimensional information. In this way, the correction unit 30 can easily obtain the predetermined point cloud 22 corresponding to the target 3 existing on the same plane as the bounding box 12 by projecting the three-dimensional point cloud data 21 onto the image 11 using the projection matrix acquired in advance. That is, the above-mentioned object detection device 1 can easily extract the two-dimensional point cloud data 21 on the same plane set as the bounding box 12 from the three-dimensional point cloud data 21 by using a projection matrix prepared in advance.

[0036] The correction unit 30 acquires the amount of deviation L and direction D of the point cloud 22 relative to the recognition area 12A, under the condition that a specific point cloud 22 corresponding to the target 3 in the point cloud data 21 acquired by the distance measurement sensor 20 is shifted relative to the recognition area 12A surrounded by the bounding box 12, and can perform calibration of either the camera 10 or the distance measurement sensor 20 or both (in this embodiment, the camera 10) based on the amount of deviation L and direction D.

[0037] As shown in FIG. 2, for example, when the point cloud 22 corresponding to the targets 3, 3 (road sign 3A, other vehicle 3B) is shifted to the left (-D direction) by the shift amount L, the calibration is performed by correcting the imaging position of the camera 10 to return to the left (-D direction) by the shift amount L. That is, in this embodiment, the position of the camera 10 is corrected by the difference based on the shift amount L. When the calibration is performed, the position shift of the camera 10 is corrected. Thereby, as shown in FIG. 3, the position shift of the point cloud 22, 22 corresponding to the targets 3, 3 relative to the bounding boxes 12, 12 is corrected. Note that the calibration may be performed by correcting the position coordinates of the targets 3, 3 using a correction coefficient calculated based on the shift amount L and the direction D. In this embodiment, the calibration is performed by moving the imaging position of the camera 10, but the calibration can be performed by moving the imaging position of the distance measuring sensor 20 or by moving the imaging positions of both the camera 10 and the distance measuring sensor 20. That is, the calibration can be performed by moving the imaging positions of the camera 10 and the distance measuring sensor 20 relative to each other.

[0038] 2 illustrates an example in which the point cloud 22 is shifted in the horizontal direction (direction D) by a shift amount L for ease of understanding, but in reality, various types of shifts may occur, such as shifts in various directions, such as diagonal directions, or shifts with different tendencies for each of the multiple bounding boxes 12. In such cases, correction coefficients may be calculated using, for example, a preset conversion table or a predetermined conversion formula, rather than simply performing correction based on the difference as described above.

[0039] Furthermore, in the case where a plurality of bounding boxes 12, 12 are set for each of a plurality of targets 3, 3 as in this embodiment, the correction unit 30 can be configured not to perform calibration on condition that at least a portion of the bounding boxes 12, 12 overlap with one another. In other words, when at least a portion of the bounding boxes 12, 12 overlap with one another, the point clouds 22, 22 corresponding to the targets 3, 3 cannot be accurately acquired, and therefore, calibration is not performed.

[0040] In this way, the above-described object detection device 1 can eliminate overlapping bounding boxes 12, 12. As a result, the above-described object detection device 1 can eliminate images 11 and point cloud data 21 in cases where the bounding boxes 12, 12 overlap and the targets 3, 3 cannot be accurately recognized, thereby enabling accurate calibration.

[0041] The above is the configuration of the object detection device 1 according to one embodiment of the present invention, and next, a calibration method for the object detection device 1 of the present invention will be described in detail below with reference to the flow diagrams of Figures 4 and 5. Note that in Figures 4 and 5, the bounding box 12 is abbreviated to BBOX.

[0042] As shown in Fig. 4, when calibration of the object detection device 1 is started, it is determined whether or not the vehicle 2 is stopped (step S10, vehicle stopped state determination step). If it is determined that the vehicle 2 is not stopped, the process returns to step S10. On the other hand, if it is determined in step S10 that the vehicle 2 is stopped, the distance measurement sensor 20 acquires the point cloud data 21 (step S11, point cloud data acquisition step). That is, in the point cloud data acquisition step, the distance measurement sensor 20 acquires the point cloud data 21 on the condition that it is determined that the vehicle 2 is in a stopped state in the vehicle stopped state determination step (step S10).

[0043] Next, the image 11 is acquired by the camera 10 (step S12, image acquisition step). When the image 11 is acquired in step S12, image analysis (in this embodiment, analysis by AI) is performed (step S13).

[0044] Next, it is determined whether or not a bounding box 12 exists in the image 11 (step S14, bounding box presence / absence determination step). That is, the bounding box presence / absence determination step is intended to determine the presence or absence of a bounding box 12 in the image 11 by analyzing the image 11.

[0045] If it is determined in step S14 that the bounding box 12 does not exist, the process returns to step S10. On the other hand, if it is determined that the bounding box 12 exists, the bounding box 12 in the camera coordinates of the camera 10 is converted into ranging coordinates (LiDAR coordinates) in the ranging sensor 20 (step S15, coordinate conversion step). In other words, the camera coordinates related to the two-dimensional bounding box 12 are converted into three-dimensional ranging coordinates. That is, the coordinate conversion step converts the position coordinates of the bounding box 12 into ranging coordinates in the ranging sensor 20 on the condition that it is determined in the bounding box existence determination step (step S14) that the bounding box 12 exists.

[0046] Next, from the point cloud 22 (point cloud data 21) located within the bounding box 12, the point cloud 22 (point cloud data 21) on the same plane as the bounding box 12 is calculated (step S16, point cloud calculation step). That is, the point cloud calculation step is intended to calculate the point cloud 22 corresponding to the target 3 located on the same plane from the point cloud data 21 located within the recognition area 12A of the bounding box 12. When the processing of step S16 is completed, it is determined whether or not the point cloud 22 on the same plane exists outside the bounding box 12 (step S17, point cloud position determination step). That is, the point cloud position determination step is intended to determine whether or not the point cloud 22 on the same plane exists outside the recognition area 12A.

[0047] If it is determined in step S17 that the point cloud 22 does not exist outside the bounding box 12, the process returns to step S10. That is, it is determined that calibration is not necessary. On the other hand, if it is determined in step S17 that the coplanar point cloud 22 exists outside the bounding box 12, the deviation amount L and direction D of the coplanar point cloud 22 from the bounding box 12 are calculated as shown in FIG. 5 (step S18, deviation amount / direction calculation step). That is, the deviation amount / direction calculation step calculates the deviation amount L and direction D of the point cloud 22 with respect to the recognition area 12A on the condition that the coplanar point cloud 22 exists outside the recognition area 12A in the point cloud position determination step (step S17).

[0048] Next, a correction coefficient for calibrating the camera 10 or the distance measurement sensor 20 is calculated based on the deviation amount L and the direction D (step S19, correction coefficient calculation step). Once the correction coefficient is calculated in step S19, the correction coefficient is stored (set) in the correction unit 30 (step S20, correction coefficient setting step). Once the correction coefficient is stored in the correction unit 30, calibration is performed based on the correction coefficient (step S21, calibration execution step). Calibration can be performed on either or both of the camera 10 and the distance measurement sensor 20, as described above. Once step S21 has been executed, the process returns to step S10. Note that the process may be terminated after step S21 has been executed.

[0049] The above is one embodiment of the calibration method for the object detection device 1 of the present invention. Next, the effects of the object detection device 1 and the calibration method for the object detection device 1 of the present invention will be described in detail.

[0050] The above-described object detection device 1 can perform calibration based on the two-dimensional image 11 captured by the camera 10 and the three-dimensional point cloud data 21 measured by the distance measuring sensor 20. That is, the above-described object detection device 1 can perform calibration with high accuracy even if the image 11 captured by the camera 10 does not include three-dimensional information. Therefore, the above-described object detection device 1 can reduce the cost of the camera 10. In addition, the above-described object detection device 1 can use various cameras capable of acquiring the two-dimensional image 11, regardless of whether they are stereo cameras or monocular cameras, thereby improving versatility. In this way, the object detection device 1 of the present invention can be used with various cameras 10 including monocular cameras, and can perform calibration with high accuracy.

[0051] Furthermore, in the above-described object detection device 1, the correction unit 30 detects a plurality of targets 3 and acquires a plurality of deviation amounts L corresponding to each of the plurality of targets 3, and calibration is performed based on the plurality of deviation amounts L. This allows the above-described object detection device 1 to perform calibration based on the plurality of deviation amounts L and directions D for each of the plurality of targets 3, thereby improving the accuracy of the calibration.

[0052] Moreover, the object detection device 1 described above is expected to reduce costs by configuring the camera 10 with a monocular camera. Moreover, the object detection device 1 described above can unify the acquisition of depth information into the distance measuring sensor 20, thereby suppressing redundancy of the device due to duplication of the depth information acquisition function.

[0053] Furthermore, in the above-mentioned object detection device 1, the target 3 includes at least one of a road sign 3A, a signboard, and another vehicle 3B that is stopped and different from the vehicle 2. Therefore, the above-mentioned object detection device 1 can acquire an image 11 and point cloud data 21 based on the stopped target 3, and can perform calibration with high accuracy using the image 11 and the point cloud data 21. Note that it is desirable for the target 3 used for calibration to have a flat surface, such as a road sign 3A or a signboard. Since the surface of the target 3 is formed flat in this way, distance measurement errors for the target 3 can be reduced, and calibration can be performed with high accuracy.

[0054] According to the above-described calibration method of the object detection device 1, the point cloud data acquisition step and the image acquisition step are executed on the condition that the vehicle 2 is determined to be in a stopped state in the vehicle stop state determination step. Therefore, the processing load of the acquired point cloud data 21 and the image 11 can be reduced, and the detection error of the target 3 can be reduced. In addition, in the coordinate conversion step, the position coordinates of the bounding box 12 are converted into the distance measurement coordinates (LiDAR coordinates) in the distance measurement sensor 20. That is, the position coordinates corresponding to the two-dimensional plane of the bounding box 12 are converted into the distance measurement coordinates. As a result, in the point cloud calculation step, the point cloud data 21 located on the plane of the bounding box 12 can be calculated. In other words, of the three-dimensional point cloud data 21 acquired by the distance measurement sensor 20, the two-dimensional point cloud data 21 on the same plane as the bounding box 12 is calculated. In addition, based on the calculated point cloud data 21, the deviation amount / direction calculation step, the correction coefficient calculation step, and the correction coefficient setting step are executed, and the calibration is executed in the calibration execution step based on the calculated correction coefficient.

[0055] In this way, the above-mentioned calibration method can calculate the correction coefficient by replacing the three-dimensional point cloud data 21 acquired by the distance measurement sensor 20 with the two-dimensional point cloud data 21, so that even if the image 11 captured by the camera 10 does not contain three-dimensional information, the calibration method can perform calibration with high accuracy. Therefore, the above-mentioned calibration method can reduce the cost of the camera 10. Furthermore, the above-mentioned calibration method can use various cameras capable of acquiring the two-dimensional image 11, regardless of whether they are stereo cameras or monocular cameras, and therefore versatility is improved. Therefore, the calibration method of the object detection device 1 of the present invention can be used in various cameras 10 including monocular cameras, and can perform calibration with high accuracy.

[0056] The above is one embodiment of the object detection device 1 and the calibration method for the object detection device 1 of the present invention, but the object detection device 1 and the calibration method for the object detection device 1 of the present invention are not limited to the embodiment described above and can be modified in various ways.

[0057] In this embodiment, the camera 10 is configured as a monocular camera, but various cameras can be used for the camera 10. For example, the camera 10 may be configured as a stereo camera. From the viewpoint of cost reduction, it is desirable that the camera 10 is configured as a monocular camera. Furthermore, the camera 10 can be used regardless of color or monochrome, and as long as it can recognize the target 3, cameras with various resolutions can be used. Furthermore, in this embodiment, the distance measurement sensor 20 is configured as a LiDAR, but as the distance measurement sensor 20, various means capable of acquiring three-dimensional information (depth information) can be used. Furthermore, as long as the resolution detected by the distance measurement sensor 20 can detect the amount of deviation L and the direction D of the target 3 relative to the bounding box 12, various resolutions can be used. Furthermore, in this embodiment, the correction unit 30 is provided independently of the camera 10 and the distance measurement sensor 20, but the correction unit 30 may be built into the camera 10 and the distance measurement sensor 20, or integrated into a control device mounted on the vehicle 2.

[0058] In this embodiment, a plurality of bounding boxes 12 are set for each of a plurality of targets 3, and the shift amount L and the direction D for each of the plurality of targets 3 are acquired, but the number of bounding boxes 12 may be one or more. In this embodiment, the bounding boxes 12 are set based on the position information of the targets 3 estimated from the image 11 using an analysis by AI, but the targets 3 can be extracted using a means other than AI. For example, the bounding boxes 12 may be set based on the position information of a standard target 3. In this embodiment, the bounding boxes 12 are rectangular, but the bounding boxes 12 can be changed to various shapes and sizes according to the shape of the targets 3, etc., within a range in which the targets 3 can be detected. In this embodiment, the calibration in the case where the point cloud 22 is shifted in the horizontal direction is exemplified, but the calibration can be performed in response to various shift amounts L and directions D of the point cloud 22.

[0059] In the present embodiment, the image 11 and the point cloud data 21 relating to the target 3 are acquired when the vehicle 2 is stopped, but the image 11 and the point cloud data 21 may be acquired while the vehicle 2 is traveling, as long as the point cloud data 21 on the plane formed by the bounding box 12 can be acquired. In the present embodiment, the point cloud data 21 is projected onto the image 11 using a projection matrix to obtain a predetermined point cloud 22 corresponding to the target 3, but the projection matrix may be used as necessary, and the point cloud data 21 may be projected onto the image 11 using a conversion table or conversion formula other than the projection matrix. In the present embodiment, the point cloud data 21 is projected onto the plane formed by the bounding box 12, but the point cloud data 21 on the plane formed by the bounding box 12 may be extracted from the point cloud data 21.

[0060] In this embodiment, the target 3 includes at least one of a road sign 3A, a signboard, and another stopped vehicle 3B different from the vehicle 2, but the target 3 is not limited to these and can be various objects. Note that it is preferable to use a flat target 3 with few irregularities on the surface in terms of reducing the burden of image processing and improving the accuracy of calibration.

[0061] In addition, in this embodiment, calibration is not performed in the correction unit 30 on the condition that at least a portion of the bounding boxes 12 overlap with one another, but the present invention is not limited to this. For example, even if at least a portion of the bounding boxes 12 overlap with one another, calibration may be performed as long as it is possible to extract the image 11 of the target 3 in each bounding box 12 and the point cloud 22 corresponding to the target 3.

[0062] The above are various embodiments and modifications of the object detection device and the calibration method for the object detection device according to the present invention. However, the present invention is not limited to those exemplified in the above-mentioned embodiments and modifications, and it will be easily understood by those skilled in the art that there are other embodiments based on the teachings and spirit of the present invention without departing from the scope of the claims. [Industrial Applicability]

[0063] The object detection device of the present invention can be used for cameras and distance measuring sensors (LiDAR) in automatic driving control systems and collision avoidance support systems in various vehicles. Also, the calibration method for the object detection device of the present invention can be used for calibrating cameras and distance measuring sensors in automatic driving control systems and collision avoidance support systems in various vehicles. [Explanation of symbols]

[0064] 1: Object detection device 2: Vehicle 3:Target 3A: Road sign (target) 3B: Other vehicle (target) 10: Camera 11:Image 12: Bounding box 20: Distance sensor (LiDAR) 21: Point cloud data 22: Point cloud 30: Correction section

Claims

1. An object detection device for a vehicle, comprising: A camera for capturing an image of a scene ahead of the vehicle; A distance measurement sensor for acquiring point cloud data in front of the vehicle; a correction unit that performs calibration based on information about the images and the point cloud data acquired by the camera and the distance measuring sensor; Equipped with The correction unit is an object detection device that sets a bounding box within the image, detects a predetermined target object located within the bounding box, and, under the condition that a predetermined point cloud corresponding to the target object in the point cloud data acquired by the ranging sensor is shifted with respect to a recognition area surrounded by the bounding box, acquires an amount of deviation and direction of the point cloud with respect to the recognition area, and performs calibration of either the camera or the ranging sensor or both based on the amount of deviation and direction.

2. The correction unit detects a plurality of the targets and acquires a plurality of deviation amounts and directions corresponding to each of the plurality of the targets, The object detection device according to claim 1 , wherein the calibration is performed based on a plurality of the deviation amounts and the directions.

3. Using the object detection device according to claim 1 or 2, a vehicle stop state determination step of determining whether or not the vehicle is stopped; a point cloud data acquisition step of acquiring the point cloud data by the distance measuring sensor on condition that it is determined that the vehicle is in a stopped state in the vehicle stopped state determination step; an image capturing step of capturing the image by the camera; a bounding box presence / absence determination step of determining the presence or absence of the bounding box in the image by analyzing the image; a coordinate conversion step of converting a position coordinate of the bounding box into a distance measurement coordinate of the distance measurement sensor, on condition that it is determined in the bounding box presence / absence determination step that the bounding box is present; a point cloud calculation step of calculating the point cloud located on the same plane from the point cloud data located within the recognition area of ​​the bounding box; a point cloud position determination step of determining whether or not the point cloud on the same plane exists outside the recognition area; a displacement amount / direction calculation step of calculating a displacement amount and a direction of the point cloud with respect to the recognition area, on the condition that the point cloud on the same plane exists outside the recognition area in the point cloud position determination step; a correction coefficient calculation step of calculating a correction coefficient for the camera or the distance measuring sensor based on the amount of deviation and the direction of deviation calculated in the amount of deviation and direction calculation step; a correction coefficient setting step of storing the correction coefficient calculated in the correction coefficient calculation step in the correction unit; a calibration execution step of executing the calibration based on the correction coefficient set in the correction coefficient setting step; A calibration method for an object detection device, comprising the steps of:

Images