Object detection device and object detection system

Abstract

To provide an object detection device that can accurately estimate an object position from a 2D-BBOX.SOLUTION: An object detection device comprises: an object extraction unit that extracts an object from an image and outputs a rectangle inscribed on and surrounding the object; a direction calculation unit that calculates a direction of the extracted object on the image; and a bottom face area calculation unit that calculates a bottom face area of the object on the image and a bottom face area of the object on the real-world coordinates, by using the width of the rectangle output from the object extraction unit and the direction of the object on the image calculated by the direction calculation unit.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present application relates to an object detection device and an object detection system. [Background technology]

[0002] In recent years, there has been active development of autonomous driving technology for automobiles. To realize autonomous driving, the installation of roadside units that detect objects within an area and report information about the detected objects to vehicles, people, dynamic maps, etc. is being considered. Roadside units are equipped with sensors such as LiDAR (Light Detection And Ranging) and cameras, and detect objects using each sensor, calculate and report information such as the position and type of the detected object.

[0003] For example, when considering a situation where an autonomous vehicle is overtaking another vehicle, the autonomous vehicle needs to obtain the object's presence area with high positional accuracy (about 0.1 to 1 m) from the object information detected by the roadside unit. Specifically, the presence area refers to a rectangular parallelepiped on a dynamic map that has information on "position," "length, width, height," and "direction." When "height" information is not important, the presence area can be interpreted as the bottom area, which refers to a rectangle on a dynamic map that has information on "position," "length, width," and "direction."

[0004] Furthermore, when an object is detected from image information acquired by a roadside device, it is necessary to calculate the position of the detected object in the real world. Generally, an object is detected in an image as a two-dimensional rectangle (2D bounding box: hereinafter referred to as 2D-BBOX), and then the coordinates of any position in the detected 2D-BBOX are converted into real-world coordinates using extrinsic parameters or a homography matrix of the camera, thereby calculating the position in the real world. For example, Patent Document 1 proposes a method for improving position accuracy by calculating real-world coordinates after correcting image misalignment due to camera shake using matching with a template image.

[0005] Furthermore, Non-Patent Document 1 describes a method for using a neural network model to output a three-dimensional rectangular parallelepiped (3D bounding box: hereinafter referred to as 3D-BBOX) in order to estimate the size and orientation of an object in an image. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2022-34034 [Non-patent literature]

[0007] [Non-Patent Document 1] Peixuan Li, Huaici Zhao, Pengfei Liu, Feidao Cao”RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving” arXiv preprint arXiv:2001.03343,2020 Summary of the Invention [Problem to be solved by the invention]

[0008] In the method described in Patent Document 1, when calculating the object position from a 2D-BBOX, the center position or the position of the bottom center of the 2D-BBOX is used as the representative position of the object and converted onto real-world coordinates. However, the actual center position of an object changes depending on the orientation of the object in the image, and the object position converted from the image using the above method does not reflect the actual center position of the object, resulting in poor position estimation accuracy.

[0009] Furthermore, the method described in Non-Patent Document 1 uses a neural network capable of outputting a 3D-BBOX, and therefore requires a large amount of annotation data for a 3D rectangular parallelepiped to train the neural network. There are many existing technologies for calculating a 2D-BBOX, and even if annotation is required, it can be implemented at low cost. However, a problem with the 3D-BBOX is that it requires high costs for annotation and learning.

[0010] The present application discloses a technique for solving the above-mentioned problems, and aims to provide an object detection device that can estimate the object position from a 2D-BBOX with high accuracy. [Means for solving the problem]

[0011] The object detection device disclosed in the present application comprises: An object detection device that extracts an object from an image acquired by an imaging unit and calculates a position of the object in real world coordinates, Extracting the object from the image; Surrounding the object an object extraction unit that outputs a rectangle; a direction calculation unit that calculates a direction of the object on the image extracted by the object extraction unit; a bottom surface area calculation unit that calculates a bottom surface area of ​​the object on the image and in real world coordinates using the width of the rectangle output from the object extraction unit and the direction of the object on the image calculated by the direction calculation unit, The bottom surface region includes the position, size, and direction of the object on the image and in real world coordinates, respectively. [Effects of the Invention]

[0012] According to the present application, the bottom surface area of ​​an object can be estimated with high accuracy using 2D-BBOX from a captured image, and the object position can be estimated with high accuracy. [Brief explanation of the drawings]

[0013] [Figure 1]1 is a block diagram showing a configuration of an object detection system according to a first embodiment. [Figure 2] 1 is a functional block diagram showing a configuration of an object detection device according to a first embodiment. [Figure 3] 1A and 1B are diagrams illustrating an example of the arrangement of roadside units and the detection ranges of the roadside units. [Figure 4] FIG. 10 is a diagram showing an example in which an object is extracted from an image acquired by a roadside device. [Figure 5] 5A and 5B are diagrams for explaining an example of conversion between image coordinates and real-world coordinates, in which FIG. 5A shows an example of an acquired image and FIG. 5B shows an example of a dynamic map. [Figure 6] FIG. 10 is a diagram for explaining a method for calculating a bottom surface area in image coordinates. [Figure 7] 7A and 7B are diagrams for explaining a method of calculating the bottom surface area in image coordinates to the bottom surface area in real world coordinates, with FIG. 7A showing image coordinates and FIG. 7B showing real world coordinates. [Figure 8] 8A and 8B are diagrams showing an example of calculating the bottom surface area of ​​an object, where FIG. 8A is a diagram shown on an image, and FIG. 8B is a diagram shown on a dynamic map. [Figure 9] 9A and 9B are diagrams illustrating how the bottom surface area changes depending on the direction of the object. FIG. 9A shows an example in which the direction of the object is at an angle θ1, and FIG. 9B shows an example in which the direction of the object is at an angle θ2. [Figure 10] These figures are for explaining the length conditions of an object, and FIG. 10A is a diagram showing the bottom area of ​​an object with a length of Lpix1 and a width of Wpix1, and FIG. 10B is a diagram showing the bottom area of ​​an object with a length of Lpix2 and a width of Wpix2. [Figure 11] 4 is a flowchart illustrating the procedure for object detection in the object detection device according to the first embodiment. [Figure 12] FIG. 10 is a functional block diagram showing the configuration of an object detection device according to a second embodiment. [Figure 13] FIG. 11 is a functional block diagram showing the configuration of an object detection device according to a third embodiment. [Figure 14] FIG. 10 is a diagram for explaining an object direction map. [Figure 15]11 is a flowchart illustrating the procedure for object detection in the object detection device according to the third embodiment. [Figure 16] FIG. 10 is a functional block diagram showing the configuration of an object detection device according to a fourth embodiment. [Figure 17] FIG. 10 is a diagram illustrating an example of an object direction table. [Figure 18] FIG. 1 is a diagram illustrating a hardware configuration of an object detection system and an object detection device according to first to fourth embodiments. [Figure 19] FIG. 10 is a diagram illustrating another example of the hardware configuration of the object detection system and the object detection device according to the first to fourth embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0014] Embodiment 1 The object detection system and object detection device according to the first embodiment will be described below with reference to the drawings. <Configuration of object detection system> 1 is a block diagram showing the configuration of an object detection system according to embodiment 1. Object detection system 10 includes an imaging unit 100 that captures images using a camera or the like mounted on a roadside device, an object detection device 200 that extracts objects from the images captured by imaging unit 100 and calculates the position, size, and facing direction (angle, attitude) of the objects, and a storage unit 300 that stores information required for processing by object detection device 200. Storage unit 300 is composed of one or more of any storage devices, such as a RAM (Random Access Memory) and a ROM (Read Only Memory). Storage unit 300 also stores information such as the images captured by imaging unit 100 and the position, size, and direction of the objects calculated by object detection device 200.

[0015] <Configuration of object detection device 200> Fig. 2 is a functional block diagram showing the configuration of object detection device 200 according to embodiment 1. In Fig. 2, object detection device 200 includes object extraction unit 201 that receives an image captured by imaging unit 100 and extracts an object from the image, direction calculation unit 202 that calculates the direction in which the object extracted by object extraction unit 201 is facing on the image, bottom area calculation unit 203 that calculates the bottom area of ​​the object using the object area extracted by object extraction unit 201 and object direction information obtained from direction calculation unit 202, and output unit 204 that distributes the bottom area of ​​the object calculated by bottom area calculation unit 203. Each of these components will be described in detail below.

[0016] The imaging unit 100 transmits camera images (hereinafter simply referred to as "images") captured by a camera mounted on the roadside unit RU to the object extraction unit 201. Images are typically captured at intervals of several to 30 fps and transmitted via any communication method such as a USB (Universal Serial Bus) / LAN (Local Area Network) cable or wireless communication.

[0017] Here, an image captured by the imaging unit 100 will be described. FIG. 3 is a diagram showing an example of the arrangement of roadside units RU equipped in the imaging unit 100 and the detection range of the roadside unit RU. As shown in FIG. 3, the roadside unit RU is installed near an intersection, and the camera is installed at a certain height (e.g., 6 m) above the road, looking down diagonally from above the object so as to cover the desired detection range. Here, the detection range captured by the camera is a radial area including the center of the intersection, and is the area hatched with dots. Furthermore, the objects to be detected are objects that need to be detected for autonomous driving, such as vehicles and people. The detectable objects and the detection accuracy depend on the algorithm, model, etc. used in the object extraction unit 201. In FIG. 3, at a certain time, the roadside unit RU captures two vehicles VE1 and VE2 heading toward the intersection within its imaging range. The following explanation will be given using an example of detecting the position of vehicle VE1 of the two vehicles VE1 and VE2.

[0018] The object extraction unit 201 acquires an image captured by the image capture unit 100 and extracts an object from the image using known techniques such as pattern matching, neural networks, and background subtraction. In contact with the object The image captured by the image capturing unit 100 is generally subjected to processes such as scaling and normalization in accordance with the object extraction algorithm and object extraction model used in the object extraction unit 201. When a neural network or the like is used, the object type is also generally output at the same time, but this is not necessarily required in this embodiment. 4 is a diagram showing an example of an object extracted from an image acquired by a roadside unit RU by the object extraction unit 201. The dotted rectangle surrounding the vehicle VE1, which is the extracted object in the diagram, is the 2D-BBOX.

[0019] The direction calculation unit 202 calculates the direction in which the object is facing (in the case of a vehicle, the front is facing) for the object extracted by the object extraction unit 201. Since the correspondence between image coordinates on the image and real-world coordinates is known in advance in the roadside unit RU, directions on the image coordinates and directions on the real-world coordinates can be converted into each other.

[0020] Here, the conversion between image coordinates and real-world coordinates will be explained using FIG. 5. In FIG. 5, FIG. 5A shows an acquired image, and FIG. 5B shows, for example, a dynamic map. Coordinates on the image are image coordinates, and coordinates on the dynamic map are real-world coordinates. The dynamic map is a map that is referenced when a vehicle is driving autonomously, and includes obstacle information and the like. Since it is assumed that the outputs of object detection system 10 and object detection device 200 according to this embodiment will be applied to autonomous driving, the target of real-world coordinates is the dynamic map.

[0021] As shown in Figure 5A, image coordinates are generally defined in pixel (pix) units with the upper left corner of the image as the origin, with the right direction being the positive x-axis and the downward direction being the positive y-axis. As shown in Figure 5B, real-world coordinates are generally defined as the x-axis being longitude, the y-axis being latitude, and the z-axis being height, or as a coordinate system in meters with an appropriate position as the origin, the east direction being the positive x-axis, the north direction being the positive y-axis, and the height direction being the positive z-axis.

[0022] Since the camera of the roadside unit RU is fixed, by creating a conversion formula between image coordinates and real-world coordinates in advance, mutual conversion is possible as long as the height in real-world coordinates is within the same plane. For example, for a point on the ground (height = 0), if the four sets of image coordinates (a, b, c, d) in Figure 5A correspond to the set of real-world coordinates (A, B, C, D) in Figure 5B, then it is possible to find the homography matrix M that converts from image coordinates to real-world coordinates. The inverse matrix M of the homography matrix M is -1 Using the above, it is also possible to convert from real-world coordinates to image coordinates. Although a similar conversion can be achieved by using a camera extrinsic parameter matrix that defines the camera's orientation (rotation, translation) instead of the homography matrix, the following explanation will be given using the homography matrix as an example.

[0023] The direction calculation unit 202 calculates the direction of the object on the real world coordinates based on the above-mentioned conversion between the image coordinates and the real world coordinates. The method of defining the direction may be arbitrary, and for example, on the image coordinates, the x-axis direction of the image may be 0°, counterclockwise rotation may be positive, and the range of direction may be 0 to 360°, and on the real world coordinates, the east side (x-axis direction) may be 0°, and the direction rotating from east to north (y-axis direction) may be positive, and the range of direction may be 0 to 360°.

[0024] The direction can also be calculated by any method, for example, using the history of the movement direction of the 2D-BBOX. In that case, the direction is calculated by taking an arbitrary position of the 2D-BBOX, for example the center of the base, calculating the difference in coordinates between frames, and using the direction of the difference vector as the direction of the object in the image. Also, known image processing algorithms such as optical flow and direction estimation using a neural network may be used to calculate the direction. Furthermore, if information from other sensors such as LiDAR or millimeter-wave radar is available, the direction determined from those sensors may be used. Also, image information obtained from other cameras installed in different locations may be used for calculation. Furthermore, if the extracted object is a vehicle, information from the vehicle's GNSS (Global Navigation Satellite System) sensor or speed information may be used if available.

[0025] The bottom surface area calculation unit 203 calculates the bottom surface area of ​​the object using information about the rectangular 2D-BBOX acquired from the object extraction unit 201 and information about the direction of the object on the image calculated by the direction calculation unit 202. Fig. 6 is a diagram for explaining a method for calculating the bottom area of ​​an object in image coordinates, and Fig. 7 is a diagram for explaining a method for calculating the bottom area in real world coordinates from the bottom area in image coordinates. In Fig. 7, Fig. 7A shows the image coordinates, and Fig. 7B shows the real world coordinates. In each figure, the physical quantities are defined as follows: wbbox: 2D - width of BBOX hbbox: 2D - height of BBOX Lpix: Vertical width of the object on the image Wpix: width of the object in the image ratio_pix: Ratio of the object's height and width on the image (Lpix / Wpix) θ: angle between Lpix and the x-axis φ: angle between Wpix and x-axis

[0026] Next, the procedure for calculating the bottom surface area will be described. 1) In Fig. 7A, it is assumed that an object has an angle θ with the x-axis at the center of the bottom edge of the 2D-BBOX (coordinates ((x0 + x1) / 2, y1)). The angle θ is an angle that indicates the direction of the object on the image estimated by the direction calculation unit 202. 2) In Figure 7A, create Ltmp and W'tmp. Ltmp is a vector that extends from the center of the bottom edge of the 2D-BBOX at an angle θ for an arbitrary length (for example, half the height of the 2D-BBOX), where the arbitrary length is, for example, half the vertical length of the 2D-BBOX. W'tmp is a vector that extends from the center of the bottom edge of the 2D-BBOX at an arbitrary angle by a length of Ltmp / ratio_w. The arbitrary angle is, for example, 90°-θ. Here, ratio_w is set in advance. 3) Using the homography matrix M, the vectors Ltmp and W'tmp are transformed into vectors Ltmp_w and W'tmp_w in the real world coordinate system, respectively. 4) In the real world coordinate system of FIG. 7B, the vector W'tmp_w is rotated so as to be perpendicular to the vector Ltmp_w, and is adjusted by enlarging or reducing it so that its length becomes 1 / ratio_w of Ltmp_w, and the result is defined as the vector Wtmp_w. |Wtmp_w|=|Ltmp_w| / ratio_w As described above, ratio_w, which is the ratio of the vertical width to the horizontal width of the object in the real world coordinate system, is set in advance. 5) The vector Wtmp_w is converted into the inverse matrix M of the homography matrix. -1 and let the result be the vector Wtmp. 6) In FIG. 7A, the ratio of the lengths of vectors Ltmp and Wtmp is ratio_pix, and the angle between vector Wtmp and the x-axis is angle φ. 7) In Figure 6, 2D-BBOX is the In contact with the object It is represented by a rectangle with sides parallel to the x-axis that surrounds the object. Therefore, the width wbbox of 2D-BBOX can be expressed by formula (1). By using formula (1) and solving for the width Wpix of the object on the image, the width Wpix of the object and the height Lpix of the object on the image (Lpix = Wpix × ratio_pix) can be obtained.

[0027]

number

[0028] The output unit 204 outputs the bottom area of ​​the detected object calculated by the bottom area calculation unit 203. Fig. 8 is a diagram showing an example of the bottom area of ​​an object calculated by the bottom area calculation unit 203. Fig. 8A shows the bottom area BAp as a dashed rectangle in the image, and Fig. 8B shows the bottom area BAw as a dashed rectangle in the dynamic map. The two can be converted to each other by performing homography transformation / inverse transformation on each vertex of the area. Note that the output format is not limited to such a rectangle and can be any format; for example, it may be output as "the position of each of the four vertices in real world coordinates" or "the position of the object center (center of the bottom area) in real world coordinates, as well as width, height, and direction".

[0029] The conditions for calculating the bottom area described above will now be explained. FIG. 9 is a diagram illustrating how the bottom area changes depending on the direction of the object. FIG. 9A is an example of an object oriented at an angle θ1 on the image, showing a bottom area BAp1, and FIG. 9B is a diagram illustrating an example of an object oriented at an angle θ2 on the image, showing a bottom area BAp2. FIG. 10 is a diagram illustrating the conditions related to the length of the object. In FIG. 10, FIG. 10A is an example of a bottom area when an object is positioned at the center of the bottom edge of a 2D-BBOX, forming an angle θ with the x-axis, and FIG. 10B is an example of a bottom area when an object is positioned away from the center of the bottom edge of a 2D-BBOX, forming an angle θ with the x-axis. In other words, FIGS. 9A and 9B in FIG. 9 are examples of the same vehicle but in different directions, and FIGS. 10A and 10B in FIG. 10 are examples of vehicles of different sizes (length, width) oriented in the same direction.

[0030] Center coordinates are usually coordinates that indicate the center of an object's position, but which position in a 2D-BBOX on an image is optimal as the center coordinate of the object varies depending on the object's position, size, orientation, and camera position. For example, when transforming to real-world coordinates using the center of the 2D-BBOX as the object center, the orientation of the object in Figure 9B is at an angle θ2, which is smaller than the angle θ1 of the object in Figure 9A, and the bottom area cannot be obtained with high positional accuracy using a normal homography matrix. Furthermore, for objects with the same angle θ shown in Figures 10A and 10B, they cannot be distinguished from each other in homography matrix transformation without certain conditions.

[0031] In this embodiment, by using the angle θ that the object forms with the x-axis at the center of the bottom edge of the 2D-BBOX and the ratio_w (=Ltmp_w / Wtmp_w) between the vector Wtmp_w and the vector Ltmp_w in real-world coordinates, the bottom region can be estimated with high accuracy in transformation using a homography matrix, thereby improving the accuracy of the object's position. That is, by using the "object direction θ" on the image, the "2D-BBOX width Wbbox," and the "aspect ratio of the object in real-world coordinates" as conditions, Equation (1) can be solved. Furthermore, even in the examples of FIGS. 9A and 9B in FIG. 9 showing the same vehicle but in different directions, and the examples of FIGS. 10A and 10B in FIG. 10 showing vehicles of different sizes (length, width) facing the same direction, by using these conditions, the bottom region can be accurately estimated and the object's position can also be accurately calculated.

[0032] In the image, if the angle θ that the object makes with the x-axis is near the center of the bottom edge of the 2D-BBOX, the angle φ and the ratio ratio_pix of the vertical width to horizontal width of the object on the image will remain almost unchanged. Furthermore, when the object to be detected is considered to be various vehicles, for example, the vertical length of a truck and that of a passenger car differ greatly, but it is assumed that the aspect ratio does not differ greatly, so in this embodiment, a ratio_w is set in advance as the aspect ratio. This condition is not limited to the aspect ratio, and may be the vertical or horizontal length of the object in real world coordinates.

[0033] For example, it is assumed that the "vertical length Lw of an object in real world coordinates" is known or set. 1) Vectors Ltmp and W'tmp are transformed into vectors Ltmp_w and W'tmp_w in the real world coordinate system using the homography matrix M, respectively, and then the length of vector Ltmp_w is multiplied by Lw / |Ltmp_w|. 2) Rotate the vector W'tmp_w so that it is orthogonal to the vector Ltmp_w, and define the result as the vector Wtmp_w. 3) Vector Ltmp_w and vector Wtmp_w are converted into the inverse matrix M of the homography matrix. -1 These are converted into vectors Ltmp_w_pix and Wtmp_w_pix, respectively. 4) Translate the vector Ltmp_w_pix so that its tip touches the 2D-BBOX, and set this as the vertical width Lpix of the object on the image. Translate the vector Wtmp_w_pix by the same amount as the vector Ltmp_w_pix, then adjust the length so that its tip touches the 2D-BBOX, and scale this to set this as the horizontal width Wpix of the object on the image. Similarly, when the "width Ww of an object in real-world coordinates" is known or set, the height Lpix and width Wpix of the object on the image can be calculated. These conditions are related to the "length of the object."

[0034] <Operation of Object Detection Device 200> Next, the procedure for object detection in object detection device 200 according to embodiment 1 will be described using the flowchart in Fig. 11. The processing in the flowchart in Fig. 11 is executed repeatedly. Each step in Fig. 11 will be described in association with each functional unit shown in the functional block diagram of object detection device 200 in Fig. 2.

[0035] First, in step S101, the object extraction unit 201 acquires, by the imaging unit 100, an image captured by a camera mounted on the roadside unit RU.

[0036] Next, in step S102, the object extraction unit 201 extracts an object from the image acquired from the image capture unit 100, In contact with the object Outputs the enclosing 2D BBOX.

[0037] Next, in step S103, the direction calculation unit 202 uses the 2D-BBOX output from the object extraction unit 201 to calculate the direction of the object on the image.

[0038] Next, in step S104, the bottom area calculation unit 203 uses the 2D-BBOX output from the object extraction unit 201 and the object direction calculated by the direction calculation unit 202 to calculate the bottom area of ​​the object on the image and on the dynamic map.

[0039] Finally, the output unit 204 outputs the bottom surface area of ​​the object calculated by the bottom surface area calculation unit 203. Through the above operations, object detection device 200 detects an object from an image acquired by the camera of roadside unit RU, and outputs information on the bottom area of ​​the object, including the position, size (width, length), and orientation of the object.

[0040] As described above, according to the first embodiment, the object detection device 200 extracts an object from an image acquired by the imaging unit 100, In contact with the object The object detection device 200 includes an object extraction unit 201 that outputs a 2D-BBOX, which is a surrounding rectangle, as an object extraction unit 201; a direction calculation unit 202 that calculates the direction θ of the object on the image extracted by the object extraction unit 201; and a bottom area calculation unit 203 that calculates the bottom area of ​​the object on the image and in real-world coordinates using the width of the 2D-BBOX and the direction θ of the object on the image calculated by the direction calculation unit 202. In this configuration, transformation is performed using a homography matrix using the direction θ of the object on the image, so it is also possible to deal with changes in the center position depending on the object's direction. Therefore, the bottom area of ​​the object on the image and in real-world coordinates can be calculated more accurately than with conventional methods, and an object detection device 200 can be obtained that can estimate the position, size (width, length), and direction of the object with high accuracy.

[0041] In addition, the bottom surface area calculation unit 203 calculates the "vertical length of the object" in real world coordinates. 、 The calculation is performed using the condition of "object length," which is either "the horizontal length of the object" or "the aspect ratio of the object," so it is possible to distinguish between vehicles of different sizes in the same direction and calculate the bottom area on the image of the object and the bottom area on the real-world coordinates.

[0042] Embodiment 2 The object detection system and object detection device according to the second embodiment will be described below with reference to the drawings. The configuration of the object detection system according to the second embodiment is the same as that of the first embodiment shown in FIG. 1, and therefore a description thereof will be omitted.

[0043] <Configuration of object detection device 200> Fig. 12 is a functional block diagram showing the configuration of object detection device 200 according to Embodiment 2. In Fig. 12, object detection device 200 includes a type determination unit 201a in object extraction unit 201 in the configuration of Fig. 2 of Embodiment 1. The object extraction unit 201 extracts an object from an image acquired by the imaging unit 100. In contact with the object The object is output as a surrounding 2D-BBOX, and the type of the object is determined by the type determination unit 201a. Here, determining the type of object means, for example, distinguishing between a standard vehicle, a large vehicle such as a truck, a two-wheeled vehicle, a person, etc. The type determination unit 201a performs type determination using existing technology such as an object detection model using a neural network. Note that the trained model used for type determination may be stored in the storage unit 300 provided in the object detection system 10 and read out when type determination is performed.

[0044] As in the first embodiment, direction calculation section 202 calculates the direction of the object extracted by object extraction section 201 on the image.

[0045] As in the first embodiment, bottom surface area calculation unit 203 calculates the bottom surface area of ​​the object using the 2D-BBOX of the object extracted by object extraction unit 201 and the direction information of the object calculated by direction calculation unit 202. At this time, the bottom surface area is calculated using the "object length" and "object aspect ratio" according to the type of object determined by type determination unit 201a. For example, the aspect ratio is 3:1 for a standard vehicle, 4:1 for a large vehicle, and 1:1 for a person. The vertical length may be 3 m for a standard vehicle, 8 m for a large vehicle, and 1 m for a person, etc. Data associated with these types is stored in memory unit 300 provided in object detection system 10, read out by bottom surface area calculation unit 203, and used to calculate the bottom surface area.

[0046] The output unit 204 outputs information about the bottom surface area of ​​the object, including the position, size (width, length) and direction of the object calculated by the bottom surface area calculation unit 203.

[0047] As described above, according to the second embodiment, the same effects as those of the first embodiment can be achieved. Furthermore, since the object extraction unit 201 includes the type determination unit 201a, it is possible to extract an object from an image acquired by the imaging unit 100. In contact with the object The object is output as a surrounding 2D-BBOX, and the type of the object can be determined by type determination unit 201a. This allows the bottom area to be calculated using the "length of the object" or the "aspect ratio of the object" based on the type of object determined by bottom area calculation unit 203, making it possible to calculate the bottom area with higher accuracy and improving the accuracy of the estimated object position.

[0048] Embodiment 3 The object detection system and object detection device according to the third embodiment will be described below with reference to the drawings. The configuration of the object detection system according to the third embodiment is the same as that of the first embodiment shown in FIG. 1, and therefore a description thereof will be omitted.

[0049] <Configuration of object detection device 200> Fig. 13 is a functional block diagram showing the configuration of object detection device 200 according to Embodiment 3. In Fig. 13, object detection device 200 includes object direction map 202a in the configuration of Fig. 2 in Embodiment 1. Direction calculation section 202 calculates the direction of an object by referring to object direction map 202a.

[0050] Fig. 14 is a diagram for explaining object direction map 202a. Object direction map 202a defines the direction of an object according to the position of the object in real world coordinates. As shown in Fig. 14, for example, the direction is defined as 90° on a road extending north-south, and 180° on a road extending east-west. In the third embodiment, an arbitrary position in 2D-BBOX is used as the center coordinate of the object, and is converted into a position in real world coordinates using matrix M. After that, the direction of the object according to that position is obtained from this map, thereby determining the direction of the object.

[0051] <Operation of Object Detection Device 200> Next, the procedure for object detection in object detection device 200 according to Embodiment 3 will be described with reference to the flowchart in Fig. 15. Note that the processing in the flowchart in Fig. 15 is repeatedly executed.

[0052] As in the first embodiment, first, in step S201, the object extraction unit 201 acquires an image captured by a camera mounted on the roadside unit RU using the imaging unit 100, and in step S202, the object extraction unit 201 extracts an object from the image, In contact with the object Output as a surrounding 2D BBOX.

[0053] Next, in step S203, the direction calculation unit 202 uses the 2D-BBOX output from the object extraction unit 201, and performs conversion from image coordinates to real-world coordinates by setting an arbitrary position, such as the center position of the bottom edge of the 2D-BBOX, as described in embodiment 1.

[0054] In step S204, the object direction at the position on the real world coordinate after the transformation is obtained from the object direction map 202a. The obtained direction on the real world coordinate is converted into a direction on the image coordinate, and then output to the bottom area calculation unit 203.

[0055] In step S205, the bottom surface area calculation unit 203 uses the 2D-BBOX output from the object extraction unit 201 and the object direction output from the direction calculation unit 202 to calculate the bottom surface area of ​​the object on the image and on the dynamic map. The output unit 204 outputs the bottom surface area of ​​the object calculated by the bottom surface area calculation unit 203 .

[0056] As in the first and second embodiments, it is also possible to calculate the direction of an object without using the object direction map 202a. However, it is possible to use the direction acquired from the object direction map 202a when the reliability of the object direction calculated by another method is low, or to use the direction acquired from the object direction map 202a only for a part of the detection area. Specifically, when the amount of change in the 2D-BBOX position over time is small and equal to or less than a preset threshold, or when the lane is narrow and the vehicle direction is limited, using the direction acquired from the object direction map 202a according to the third embodiment as the object direction will improve the accuracy of calculating the bottom area. By using them appropriately in this way, the accuracy of object detection is improved.

[0057] As described above, according to the third embodiment, the same effects as those of the first embodiment can be achieved. Furthermore, since the object detection device 200 is provided with the object direction map 202a, when the reliability of the object direction calculated by another method is low, it is possible to use the object direction map 202a to complement the calculated reliability. This makes it possible to calculate the bottom surface area with higher accuracy, improving the positional accuracy of the detected object.

[0058] Embodiment 4 An object detection system and an object detection device according to the fourth embodiment will be described below with reference to the drawings. The configuration of the object detection system according to the fourth embodiment is the same as that of the first embodiment shown in FIG. 1, and therefore a description thereof will be omitted.

[0059] <Configuration of object detection device 200> 16 is a functional block diagram showing the configuration of object detection device 200 according to embodiment 4. In FIG. 16, object detection device 200 is 3 13, an object direction table 202b is provided instead of the object direction map 202a. The direction calculation unit 202 calculates the direction of an object by referring to the object direction table 202b.

[0060] The object direction table 202b defines the direction of an object according to the aspect ratio of the 2D-BBOX. As shown in Figures 9A and 9B in Figure 9, it is expected that the aspect ratio of the 2D-BBOX will change depending on the position and direction of the object for the same object. Therefore, a correspondence table of object directions according to object type, position, and aspect ratio is created in advance and tabulated.

[0061] FIG. 17 is a diagram showing an example of the object direction table 202b. Information on the type, center coordinate on the image coordinate system, aspect ratio of the 2D-BBOX, angle on the real world coordinate system, and angle on the image coordinate system can be tabulated. Here, an example is shown in which a standard vehicle, center coordinate on the image coordinate system (x1, y1), aspect ratio of the 2D-BBOX of 3:1, angle on the real world coordinate system of 30°, and angle on the image coordinate system of 170° are used. By providing such an object direction table 202b, it becomes possible to estimate the direction of an object even if, for example, the object extracted by the object extraction unit 201 is a stationary object and the time-series history of the 2D-BBOX position cannot be used for direction calculation.

[0062] However, even if the bottom surface area is the same, if the height of the object changes, the vertical width hbbox of the 2D-BBOX changes, and the aspect ratio also changes, so this method can only be applied between objects with the same width, length, and height. Therefore, this method is effective when it is possible to assume that "all vehicles are the same size for each type." Specifically, if all transport vehicles in a factory have the same model number, they will be extracted as objects with the same width, length, and height, and the object direction table 202b can be used.

[0063] Furthermore, because this aspect ratio does not allow for a distinction between directions that are symmetrical with respect to the y-axis on the image coordinate system (10° and 170°, 60° and 120°, etc.), which of these is the true direction can be estimated separately from the history of the 2D-BBOX position, or two types of bottom surface information can be output as is without making a distinction.When estimating from the history of the 2D-BBOX position, for example, when the aspect ratio is 3:2, if the 2D-BBOX position is moving to the upper right, it can be determined to be 70°, and if it is moving to the upper left, it can be determined to be 110°.

[0064] As in the first and second embodiments, it is possible to calculate the direction of an object without using object direction table 202b. However, as in the third embodiment, it is possible to use the direction obtained from object direction table 202b when the reliability of the object direction calculated by another method is low, or to use the direction obtained from object direction table 202b only for a part of the detection area.

[0065] As described above, according to the fourth embodiment, the same effects as those of the first embodiment can be achieved. Furthermore, since the object detection device 200 is provided with the object direction table 202b, when the detection targets are objects with the same width, length, and height, and when the reliability of the object direction calculated by another method is low, Object Orientation Table 202b This makes it possible to calculate the bottom surface area with higher accuracy, improving the positional accuracy of the detected object.

[0066] Each functional unit of the object detection system 10 and the object detection device 200 in the above-described first to fourth embodiments is configured with the hardware configuration illustrated in FIG. 18, that is, an arithmetic processing circuit 1001, a storage device 1002 including a ROM (Read Only Memory) storing a program that executes the function of each functional unit and a RAM (Random Access Memory) that stores each data of the execution results of each functional unit, which are the results of calculations performed by the program, and an input / output circuit 1003. Image information from the imaging unit 100 is input to the input / output circuit 1003, and the image information is stored in the storage device 1002. In addition, the output of the object detection device 200 is applied to an autonomous driving system, and is therefore output to, for example, an autonomous driving vehicle or a control system.

[0067] Furthermore, each functional unit of object detection system 10 and object detection device 200 in the above-described first to fourth embodiments may have the hardware configuration illustrated in Fig. 19, that is, a configuration in which communication circuit 1004 is further provided in addition to the configuration in Fig. 18. Signals can be exchanged between imaging unit 100, object detection device 200, and storage unit 300 via wired or wireless means. The communication circuit 1004 includes, as a communication module, a wide-area communication unit and a short-range communication unit. The wide-area communication unit conforms to a predetermined wide-area wireless communication standard, such as LTE (Long Term Evolution), 4G, or 5G (5th Generation; fifth-generation mobile communication system). The short-range communication unit may use, for example, DSRC (Dedicated Short Range Communications).

[0068] A processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor) is applied to the arithmetic processing circuit 1001. Dedicated hardware may also be applied to the arithmetic processing circuit 1001. When the arithmetic processing circuit 1001 is dedicated hardware, the arithmetic processing circuit 1001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination of these.

[0069] Furthermore, each functional unit of the object detection system 10 and the object detection device 200 may be realized by a separate arithmetic processing circuit, or they may be collectively realized by a single arithmetic processing circuit. Furthermore, each functional unit of the object detection system 10 and the object detection device 200 can realize each of the above-mentioned functions by hardware, software, etc., or a combination of these, with some functions being realized by dedicated hardware processing circuits and other functions being realized by software.

[0070] <Other embodiments> When the object detection system 10 including the object detection device 200 shown in any of the first to fourth embodiments is applied to an autonomous driving system, the object position can be detected with high accuracy from the image acquired by the roadside unit RU and reflected in a dynamic map, so that the traveling vehicle can avoid obstacles in a planned manner. to It has the following effects.

[0071] Furthermore, the autonomous driving system to which the above-described object detection system 10 and object detection device 200 are applied is not limited to automobiles, but can be applied to various other mobile bodies. For example, the system can be applied to autonomous mobile bodies such as an in-building mobile robot that inspects the inside of a building, a line inspection robot, and personal mobility.

[0072] Although the present application describes various exemplary embodiments and examples, the various features, aspects, and functions described in one or more embodiments are not limited to application to a particular embodiment, but may be applied to the embodiments alone or in various combinations. Therefore, countless variations not illustrated are conceivable within the scope of the technology disclosed in the present specification, including, for example, cases where at least one component is modified, added, or omitted, and cases where at least one component is extracted and combined with components of another embodiment.

[0073] Various aspects of the present disclosure are summarized below as appendices.

[0074] (Appendix 1) An object detection device that extracts an object from an image acquired by an imaging unit and calculates a position of the object in real world coordinates, extracting the object from the image; In contact with the object an object extraction unit that outputs an enclosing rectangle; a direction calculation unit that calculates a direction of the object on the image extracted by the object extraction unit; a bottom surface area calculation unit that calculates a bottom surface area of ​​the object on the image and in real world coordinates using the width of the rectangle output from the object extraction unit and the direction of the object on the image calculated by the direction calculation unit, An object detection device, wherein the bottom surface region includes the position, size and orientation of the object on the image and in real world coordinates, respectively. (Appendix 2) The object detection device described in Appendix 1, wherein the bottom area calculation unit further calculates the bottom area of ​​the object on the image and the bottom area in real world coordinates using any of the width, length, and width-to-length ratio of the object on real world coordinates. (Appendix 3) The object extraction unit includes a type determination unit that determines the type of the extracted object, In contact with the object outputting the surrounding rectangle and the type of the object determined by the type determination unit; An object detection device as described in Appendix 2, wherein the bottom area calculation unit calculates the bottom area of ​​the object on the image and the bottom area of ​​the object in real world coordinates using any of the width, length, and width-to-length ratio of the object in real world coordinates based on the type of the object determined by the type determination unit. (Appendix 4) an object direction map in which object directions are defined according to positions in real world coordinates; 4. The object detection device according to claim 1, wherein the direction calculation unit calculates the direction of the object extracted by the object extraction unit on the image using the object direction map. (Appendix 5) an object direction table in which directions on the image and on real world coordinates are defined according to the aspect ratio of the rectangle; 4. The object detection device according to claim 1, wherein the direction calculation unit calculates the direction of the object extracted by the object extraction unit on the image using the object direction table. (Appendix 6) An object detection device according to any one of Supplementary Notes 1 to 5; An object detection system comprising the imaging unit. (Appendix 7) 7. The object detection system according to claim 6, wherein the imaging unit comprises a roadside unit equipped with a camera. [Explanation of symbols]

[0075] 10: object detection system, 100: imaging unit, 200: object detection device, 201: object extraction unit, 201a: type determination unit, 202: direction calculation unit, 202a: object direction map, 202b: object direction table, 203: bottom area calculation unit, 204: output unit, 300: memory unit, 1001: arithmetic processing circuit, 1002: storage device, 1003: input / output circuit, 1004: communication circuit, BAp, BAw, BAp1, BAp2: bottom area, RU: roadside unit, VE1, VE2: vehicle.

Claims

1. An object detection device that extracts an object from an image acquired by an imaging unit and calculates a position of the object in real world coordinates, an object extraction unit that extracts the object from the image and outputs a rectangle that contacts and surrounds the object; a direction calculation unit that calculates a direction of the object on the image extracted by the object extraction unit; a bottom surface area calculation unit that calculates a bottom surface area of ​​the object on the image and in real world coordinates using the width of the rectangle output from the object extraction unit and the direction of the object on the image calculated by the direction calculation unit, An object detection device, wherein the bottom surface region includes the position, size and orientation of the object on the image and in real world coordinates, respectively.

2. The object detection device according to claim 1 , wherein the bottom area calculation unit further calculates the bottom area of ​​the object on the image and the bottom area on the real world coordinates using any one of the width, length, and width-to-length ratio of the object on the real world coordinates.

3. the object extraction unit includes a type determination unit that determines the type of the extracted object, and outputs a rectangle that contacts and surrounds the object, as well as the type of the object determined by the type determination unit; 3. The object detection device according to claim 2, wherein the bottom area calculation unit calculates the bottom area of ​​the object on the image and the bottom area of ​​the object on the real world coordinates using any of the width, length, and width-to-length ratio of the object on the real world coordinates based on the type of the object determined by the type determination unit.

4. an object direction map in which object directions are defined according to positions in real world coordinates; The object detection device according to claim 1 , wherein the direction calculation unit calculates the direction of the object extracted by the object extraction unit in the image by using the object direction map.

5. an object direction table in which directions on the image and on real world coordinates are defined according to the aspect ratio of the rectangle; The object detection device according to claim 1 , wherein the direction calculation unit calculates the direction of the object extracted by the object extraction unit in the image by using the object direction table.

6. The object detection device according to any one of claims 1 to 5; An object detection system comprising the imaging unit.

7. The object detection system according to claim 6 , wherein the imaging unit comprises a roadside unit equipped with a camera.

Images