Object detection device and object detection system

Abstract

To provide an object detection device that can accurately estimate an object position from a 2D-BBOX.SOLUTION: An object detection device comprises: an object extraction unit that extracts an object from an image and outputs a rectangle inscribed on and surrounding the object; a direction calculation unit that calculates a direction of the extracted object on the image; and a bottom face area calculation unit that calculates a bottom face area of the object on the image and a bottom face area of the object on the real-world coordinates, by using the width of the rectangle output from the object extraction unit and the direction of the object on the image calculated by the direction calculation unit.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present application relates to an object detection device and an object detection system. [Background technology]

[0002] In recent years, there has been active development of autonomous driving technology for automobiles. In order to realize autonomous driving, the installation of roadside units that detect objects within an area and report the detected object information to vehicles, people, dynamic maps, etc. is being considered. The roadside units are equipped with sensors such as LiDAR (Light Detection And Ranging) and cameras, and detect objects from each sensor, calculate and report information such as the position and type of the detected object.

[0003] For example, when considering a situation where an autonomous vehicle is overtaking another vehicle, the autonomous vehicle needs to obtain the object's presence area with high positional accuracy (about 0.1 to 1 m) from the object information detected by the roadside unit. The presence area specifically refers to a rectangular prism on a dynamic map that has information on "position," "length, width, height," and "direction." When "height" information is not important, the presence area is interpreted as the bottom area, which refers to a rectangle on a dynamic map that has information on "position," "length, width," and "direction."

[0004] Furthermore, when an object is detected from image information acquired by a roadside unit, it is necessary to calculate the position of the detected object in the real world. In general, after detecting an object on an image as a two-dimensional rectangle (2D bounding box: hereinafter referred to as 2D-BBOX), the coordinates of any position of the detected 2D-BBOX are converted into coordinates in the real world using the external parameters or homography matrix of the camera, so that the position in the real world can be calculated. For example, Patent Document 1 proposes a method of improving position accuracy by calculating real-world coordinates after correcting image misalignment caused by camera shaking using matching with a template image.

[0005] Furthermore, Non-Patent Document 1 describes a method of using a neural network model to output a three-dimensional rectangular box (3D bounding box: hereinafter referred to as 3D-BBOX) in order to estimate the size and orientation of an object in an image. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2022-34034 A [Non-patent literature]

[0007] [Non-Patent Document 1] Peixuan Li, Huaici Zhao, Pengfei Liu, Feidao Cao”RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving” arXiv preprint arXiv:2001.03343,2020 Summary of the Invention [Problem to be solved by the invention]

[0008] In the method described in Patent Document 1, when calculating an object position from a 2D-BBOX, the center position or the bottom center position of the 2D-BBOX is used as the representative position of the object and is converted onto real world coordinates. However, the actual center position of an object changes depending on the direction of the object in the image, and the object position converted from the image by the above method does not reflect the actual center position of the object, resulting in poor position estimation accuracy.

[0009] Furthermore, in the method described in Non-Patent Document 1, a neural network capable of outputting a 3D-BBOX is used, and therefore a large amount of annotation data for a 3D rectangular parallelepiped is required for training the neural network. There are many existing technologies for calculating a 2D-BBOX, and even if annotation is required, it can be implemented at low cost. However, there is a problem with the 3D-BBOX in that annotation and learning require high costs.

[0010] The present application discloses a technique for solving the above-mentioned problems, and has an object to provide an object detection device capable of estimating an object position from a 2D-BBOX with high accuracy. [Means for solving the problem]

[0011] The object detection device disclosed in the present application is An object detection device that extracts an object from an image acquired by an imaging unit and calculates a position of the object in a real world coordinate system, an object extraction unit that extracts the object from the image and outputs a rectangle that inscribes and surrounds the object; a direction calculation unit that calculates a direction of the object on the image extracted by the object extraction unit; a bottom surface area calculation unit that calculates a bottom surface area of ​​the object on the image and on real world coordinates using the width of the rectangle output from the object extraction unit and the direction of the object on the image calculated by the direction calculation unit, The bottom surface region includes the position, size and orientation of the object on the image and in real world coordinates, respectively. Effect of the Invention

[0012] According to the present application, the bottom area of ​​an object can be estimated with high accuracy using 2D-BBOX from a captured image, and the object position can be estimated with high accuracy. [Brief description of the drawings]

[0013] [Figure 1]1 is a block diagram showing a configuration of an object detection system according to a first embodiment. [Diagram 2] 1 is a functional block diagram showing a configuration of an object detection device according to a first embodiment. [Diagram 3] 1 is a diagram showing an example of the arrangement of roadside units and the detection ranges of the roadside units; [Figure 4] FIG. 13 is a diagram showing an example in which an object is extracted from an image captured by a roadside unit. [Diagram 5] FIG. 5 is a diagram for explaining an example of conversion between image coordinates and real world coordinates, in which FIG. 5A is a diagram showing an example of an acquired image, and FIG. 5B is a diagram showing an example of a dynamic map. [Figure 6] 11 is a diagram for explaining a method of calculating a bottom surface area in image coordinates. FIG. [Figure 7] FIG. 7A is a diagram for explaining a method of calculating the bottom area in image coordinates to the bottom area in real world coordinates, and FIG. 7A is a diagram showing image coordinates, and FIG. 5B is a diagram showing real world coordinates. [Figure 8] 8A and 8B are diagrams showing an example of calculation of the bottom area of ​​an object, where FIG. 8A is a diagram shown on an image, and FIG. 5B is a diagram shown on a dynamic map. [Figure 9] 9A and 9B are diagrams for explaining how the bottom area changes depending on the direction of an object. FIG. 9A shows an example in which the direction of the object is at an angle θ1, and FIG. 9B shows an example in which the direction of the object is at an angle θ2. [Figure 10] FIG. 10A is a diagram illustrating the bottom area of ​​an object having a length Lpix1 and a width Wpix1, and FIG. 9B is a diagram illustrating the bottom area of ​​an object having a length Lpix2 and a width Wpix2. [Figure 11] 4 is a flowchart for explaining a procedure for object detection in the object detection device according to the first embodiment. [Figure 12] FIG. 11 is a functional block diagram showing a configuration of an object detection device according to a second embodiment. [Figure 13] FIG. 11 is a functional block diagram showing a configuration of an object detection device according to a third embodiment. [Figure 14] FIG. 13 is a diagram for explaining an object direction map. [Figure 15]11 is a flowchart illustrating a procedure for object detection in the object detection device according to the third embodiment. [Figure 16] FIG. 11 is a functional block diagram showing a configuration of an object detection device according to a fourth embodiment. [Figure 17] FIG. 13 is a diagram illustrating an example of an object direction table. [Figure 18] FIG. 1 is a diagram showing a hardware configuration of an object detection system and an object detection device according to first to fourth embodiments. [Figure 19] FIG. 13 is a diagram showing another example of the hardware configuration of the object detection system and the object detection device according to the first to fourth embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Embodiment 1 The object detection system and the object detection device according to the first embodiment will be described below with reference to the drawings. <Object detection system configuration> Fig. 1 is a block diagram showing a configuration of an object detection system according to the first embodiment. The object detection system 10 includes an imaging unit 100 that captures an image using a camera or the like mounted on a roadside unit, an object detection device 200 that extracts an object from an image captured by the imaging unit 100 and calculates the position, size, and facing direction (angle, attitude) of the object, and a storage unit 300 that stores information required for processing by the object detection device 200. The storage unit 300 is composed of one or more of any storage devices such as a RAM (Random Access Memory) and a ROM (Read Only Memory). The storage unit 300 also stores information such as the image captured by the imaging unit 100 and the position, size, and direction of the object calculated by the object detection device 200.

[0015] <Configuration of Object Detection Device 200> Fig. 2 is a functional block diagram showing a configuration of object detection device 200 according to embodiment 1. In Fig. 2, object detection device 200 includes object extraction unit 201 that receives an image captured by imaging unit 100 and extracts an object from the image, direction calculation unit 202 that calculates a direction in which the object extracted by object extraction unit 201 faces on the image, bottom surface area calculation unit 203 that calculates a bottom surface area of ​​the object using the object area extracted by object extraction unit 201 and object direction information obtained from direction calculation unit 202, and output unit 204 that distributes the bottom surface area of ​​the object calculated by bottom surface area calculation unit 203. Each part will be described in detail below.

[0016] The imaging unit 100 transmits camera images (hereinafter simply referred to as "images") captured by a camera mounted on the roadside unit RU to the object extraction unit 201. Images are usually captured at intervals of about several fps to 30 fps, and transmitted via any communication method such as a Universal Serial Bus (USB) / Local Area Network (LAN) cable or wireless communication.

[0017] Here, an image captured by the imaging unit 100 will be described. FIG. 3 is a diagram showing an example of the arrangement of the roadside unit RU equipped in the imaging unit 100 and the detection range of the roadside unit RU. As shown in FIG. 3, the roadside unit RU installed near an intersection has a camera installed at a certain height (for example, 6 m) above the road, looking down diagonally from above the object so as to cover the range to be detected. Here, the detection range captured by the camera is a radial area including the center of the intersection, and is an area hatched with dots. Furthermore, the object to be detected is an object that needs to be detected for automatic driving, such as a vehicle and a person. The detectable object and the detection accuracy depend on the algorithm and model used in the object extraction unit 201. Note that in FIG. 3, at a certain time, the roadside unit RU captures two vehicles VE1 and VE2 heading toward the intersection in its imaging range. In the following description, the position detection of the vehicle VE1 out of the two vehicles VE1 and VE2 will be described as an example.

[0018] The object extraction unit 201 acquires an image captured by the imaging unit 100, and outputs a rectangle inscribed and surrounding an object in the image using known techniques such as pattern matching, neural networks, and background subtraction. Here, the image acquired from the imaging unit 100 is generally subjected to processing such as enlargement / reduction and normalization according to the object extraction algorithm and object extraction model used in the object extraction unit 201. Furthermore, when a neural network or the like is used, the object type is generally also output at the same time, but this is not necessarily required in the present embodiment. 4 is a diagram showing an example of an object extracted from an image acquired by a roadside unit RU by the object extraction unit 201. The dotted rectangle surrounding the vehicle VE1, which is the extracted object in the diagram, is the 2D-BBOX.

[0019] The direction calculation unit 202 calculates the direction in which the object is facing (in the case of a vehicle, the front is facing) with respect to the object extracted by the object extraction unit 201. Since the correspondence between the image coordinates on the image and the real world coordinates is known in advance in the roadside unit RU, the direction on the image coordinates and the direction on the real world coordinates can be converted into each other.

[0020] Here, the conversion between image coordinates and real world coordinates will be described with reference to Fig. 5. In Fig. 5, Fig. 5A shows an acquired image, and Fig. 5B shows, for example, a dynamic map. Coordinates on the image are image coordinates, and coordinates on the dynamic map are real world coordinates. The dynamic map is a map that is referred to when a vehicle is driven autonomously, and includes obstacle information and the like. Since it is assumed that the outputs of object detection system 10 and object detection device 200 according to this embodiment are applied to autonomous driving, the target of real world coordinates is the dynamic map.

[0021] As shown in Figure 5A, image coordinates are generally defined in pixel units with the top left corner of the image as the origin, the right direction as the positive direction of the x-axis, and the downward direction as the positive direction of the y-axis. As shown in Figure 5B, real world coordinates are generally defined as the x-axis as longitude, the y-axis as latitude, and the z-axis as height, or as a coordinate system in meters with an appropriate position as the origin, the east direction as the positive direction of the x-axis, the north direction as the positive direction of the y-axis, and the height direction as the positive direction of the z-axis.

[0022] Since the camera of the roadside unit RU is fixed, by creating a conversion formula between image coordinates and real-world coordinates in advance, mutual conversion is possible as long as the height in real-world coordinates is within the same plane. For example, for a point on the ground (height = 0), if the four sets of image coordinates (a, b, c, d) in Figure 5A correspond to the set of real-world coordinates (A, B, C, D) in Figure 5B, then it is possible to obtain the homography matrix M that converts from image coordinates to real-world coordinates. The inverse matrix M of the homography matrix M -1 Using the homography matrix, it is possible to convert from real world coordinates to image coordinates. A similar conversion can be realized by using a camera extrinsic parameter matrix that specifies the camera's orientation (rotation, translation) instead of the homography matrix, but the following explanation will be given using the homography matrix as an example.

[0023] The direction calculation unit 202 calculates the direction of the object on the real world coordinates based on the above-mentioned conversion between the image coordinates and the real world coordinates. The method of defining the direction may be arbitrary, and for example, the direction may be defined in the range of 0 to 360°, with the x-axis direction of the image being 0° on the image coordinates and counterclockwise being positive, and in the range of 0 to 360°, with the east side (x-axis direction) being 0° on the real world coordinates and the direction rotating from east to north (y-axis direction) being positive.

[0024] The direction can also be calculated by any method, for example, the history of the movement direction of the 2D-BBOX. In that case, the direction is calculated by taking an arbitrary position of the 2D-BBOX, for example the center of the bottom edge, calculating the difference in coordinates between frames, and determining the direction of the difference vector as the direction of the object in the image. It may also be calculated by using known image processing algorithms, such as optical flow or direction estimation using a neural network. Furthermore, if information from other sensors such as LiDAR or millimeter wave radar is available, the direction obtained from the sensors may be used. Also, the direction may be calculated using image information obtained from other cameras installed in different locations. Furthermore, if the extracted object is a vehicle, information from the vehicle's GNSS (Global Navigation Satellite System) sensor or speed information may be used if available.

[0025] A bottom surface area calculation unit 203 calculates the bottom surface area of ​​the object using information on the rectangular 2D-BBOX acquired from the object extraction unit 201 and information on the direction of the object on the image calculated by the direction calculation unit 202. Fig. 6 is a diagram for explaining a method for calculating the bottom area of ​​an object in image coordinates, and Fig. 7 is a diagram for explaining a method for calculating the bottom area in real world coordinates from the bottom area in image coordinates. In Fig. 7, Fig. 7A shows the image coordinates, and Fig. 7B shows the real world coordinates. In each figure, the physical quantities are defined as follows: wbbox: 2D - width of BBOX hbbox: 2D - Height of BBOX Lpix: The vertical width of the object in the image Wpix: The width of the object in the image ratio_pix: Ratio of the object's vertical and horizontal width in the image (Lpix / Wpix) θ: Angle between Lpix and the x-axis φ: Angle between Wpix and the x-axis

[0026] Next, the procedure for calculating the bottom surface area will be described. 7A, it is assumed that an object has an angle θ with the x-axis at the center of the bottom edge of the 2D-BBOX (coordinates ((x0+x1) / 2, y1)). The angle θ is an angle indicating the direction of the object on the image estimated by the direction calculation unit 202. 2) In Fig. 7A, create Ltmp and W'tmp. Ltmp is a vector that is extended from the center of the bottom edge of the 2D-BBOX by an arbitrary length (for example, half the height of the 2D-BBOX) at an angle θ, and the arbitrary length is, for example, half the vertical length of the 2D-BBOX. W'tmp is a vector that is extended from the center of the bottom edge of the 2D-BBOX by an arbitrary angle by the length Ltmp / ratio_w. The arbitrary angle is, for example, 90°-θ. Here, ratio_w is set in advance. 3) Vector Ltmp and vector W'tmp are transformed into vectors Ltmp_w and W'tmp_w in the real world coordinate system, respectively, using the homography matrix M. 4) In the real world coordinate system of FIG. 7B, the vector W'tmp_w is rotated so as to be perpendicular to the vector Ltmp_w, and is adjusted by enlarging or reducing it so that its length is 1 / ratio_w of Ltmp_w, and the resulting vector is set as the vector Wtmp_w. That is, |Wtmp_w|=|Ltmp_w| / ratio_w As described above, ratio_w, which is the ratio between the vertical width and the horizontal width of an object in real world coordinates, is set in advance. 5) Vector Wtmp_w is the inverse matrix M -1 and let the result be the vector Wtmp. 6) In FIG. 7A, the ratio of the lengths of vectors Ltmp and Wtmp is ratio_pix, and the angle between vector Wtmp and the x-axis is angle φ. 7) In Figure 6, the 2D-BBOX is represented by a rectangle with sides parallel to the x-axis that inscribes and surrounds the object in the image. Therefore, the width wbbox of the 2D-BBOX can be expressed by equation (1). By solving equation (1) for the width Wpix of the object in the image, the width Wpix of the object and the height Lpix of the object in the image (Lpix = Wpix × ratio_pix) can be obtained.

[0027]

number

[0028] The output unit 204 outputs the bottom area of ​​the detected object calculated by the bottom area calculation unit 203. FIG. 8 is a diagram showing an example of the bottom area of ​​an object calculated by the bottom area calculation unit 203. FIG. 8A shows the bottom area BAp as a dashed rectangle in the image, and FIG. 8B shows the bottom area BAw as a dashed rectangle in the dynamic map. The two can be converted to each other by performing homography transformation / inverse transformation on each vertex of the area. Note that the output format is not limited to such a rectangle and is arbitrary, and may be output, for example, as "the position of each of the four vertices in the real world coordinate system" or "the position of the object center (center of the bottom area) in the real world coordinate system, and the width, height, and direction", etc.

[0029] The conditions for the calculation of the bottom surface area described above will be explained. FIG. 9 is a diagram for explaining that the bottom surface area changes depending on the direction of the object, and FIG. 9A is an example of an object in the direction of angle θ1 on the image, showing the bottom surface area BAp1, and FIG. 9B is a diagram showing an example of an object in the direction of angle θ2 on the image, showing the bottom surface area BAp2. FIG. 10 is a diagram for explaining the conditions regarding the length of the object. In FIG. 10, FIG. 10A is an example showing the bottom surface area when the object has an angle θ with the x-axis at the center of the lower end of the 2D-BBOX, and FIG. 10B is an example of the bottom surface area when the object has an angle θ with the x-axis at a position away from the center of the lower end of the 2D-BBOX. In other words, FIG. 9A and FIG. 9B in FIG. 9 are examples of the same vehicle but in different directions, and FIG. 10A and FIG. 10B in FIG. 10 are examples of vehicles with different sizes (length, width) in the same direction.

[0030] Although the center coordinates are usually coordinates indicating the center of an object position, which position in the 2D-BBOX on the image is optimal as the center coordinate of the object changes depending on the object's position, size, direction, and camera position. For example, when transforming into real world coordinates with the center of the 2D-BBOX as the object center, the direction of the object in FIG. 9B is at an angle θ2, which is smaller than the angle θ1 of the object in FIG. 9A, and the bottom area cannot be obtained with high positional accuracy in transformation using a normal homography matrix. In addition, for objects having the same angle θ shown in FIG. 10A and FIG. 10B, they cannot be distinguished from each other in transformation using a homography matrix unless some conditions are met.

[0031] In this embodiment, the bottom area can be estimated with high accuracy in the transformation by the homography matrix by using the angle θ that the object makes with the x-axis at the center of the bottom end of the 2D-BBOX and the ratio_w (=Ltmp_w / Wtmp_w) between the vector Wtmp_w and the vector Ltmp_w on the real world coordinates. That is, the formula (1) can be solved by using the "direction θ of the object" on the image, the "width Wbbox of the 2D-BBOX" and the "aspect ratio of the object in the real world coordinates" as conditions. Furthermore, by using these conditions, it is possible to accurately estimate the bottom area and accurately calculate the position of the object even in the example of the same vehicle in different directions shown in FIG. 9A and FIG. 9B, and the example of the vehicles in FIG. 10A and FIG. 10B with different sizes (length, width) in the same direction.

[0032] In addition, in the image, if the angle θ that the object makes with the x-axis is near the center of the bottom edge of the 2D-BBOX, the angle φ and the ratio ratio_pix of the vertical width to horizontal width of the object on the image will remain almost unchanged. In addition, when the object to be detected is considered to be various vehicles, for example, the vertical length of a truck and that of a passenger car are significantly different, but the aspect ratio is assumed to be similar, so in this embodiment, the ratio_w is set in advance as the aspect ratio. This condition is not limited to the aspect ratio, and may be the vertical or horizontal length of the object in real world coordinates.

[0033] For example, it is assumed that the "vertical length Lw of an object in real world coordinates" is known or set. 1) Vector Ltmp and vector W'tmp are transformed into vectors Ltmp_w and W'tmp_w in the real world coordinate system, respectively, using homography matrix M, and then the length of vector Ltmp_w is multiplied by Lw / |Ltmp_w|. 2) Rotate vector W'tmp_w so that it is orthogonal to vector Ltmp_w, and set the result as vector Wtmp_w. 3) Vector Ltmp_w and vector Wtmp_w are transformed into the inverse matrix M -1 These are converted into vectors Ltmp_w_pix and Wtmp_w_pix, respectively. 4) Translate the vector Ltmp_w_pix so that its tip touches the 2D-BBOX, and let this be the vertical width Lpix of the object on the image. Translate the vector Wtmp_w_pix the same amount as the vector Ltmp_w_pix, then adjust the length so that its tip touches the 2D-BBOX, and scale this to let this be the horizontal width Wpix of the object on the image. Similarly, when the "lateral length Ww of an object in real world coordinates" is known or set, the vertical width Lpix and horizontal width Wpix of the object on the image can be found. These conditions concern the "length of the object."

[0034] <Operation of Object Detection Device 200> Next, the procedure of object detection in object detection device 200 according to embodiment 1 will be described with reference to the flowchart in Fig. 11. Note that the process of the flowchart in Fig. 11 is executed repeatedly. Each step in Fig. 11 will be described in association with each functional unit shown in the functional block diagram of object detection device 200 in Fig. 2.

[0035] First, in step S101, the object extraction unit 201 acquires, by the imaging unit 100, an image captured by a camera mounted on the roadside unit RU.

[0036] Next, in step S102, the object extraction unit 201 extracts an object from the image acquired from the imaging unit 100, and outputs a 2D-BBOX that inscribes and surrounds the object.

[0037] Next, in step S103, the direction calculation unit 202 uses the 2D-BBOX output from the object extraction unit 201 to calculate the direction of the object on the image.

[0038] Next, in step S104, the bottom area calculation unit 203 uses the 2D-BBOX output from the object extraction unit 201 and the object direction calculated by the direction calculation unit 202 to calculate the bottom area of ​​the object on the image and on the dynamic map.

[0039] Finally, the output unit 204 outputs the bottom surface region of the object calculated by the bottom surface region calculation unit 203 . Through the above operations, object detection device 200 detects an object from an image captured by the camera of roadside unit RU, and outputs information on the bottom area of ​​the object, including the object's position, size (width, length), and orientation.

[0040] As described above, according to the first embodiment, the object detection device 200 includes an object extraction unit 201 that extracts an object from an image acquired by the imaging unit 100 and outputs the object as a 2D-BBOX, which is a rectangle inscribed and surrounding the object; a direction calculation unit 202 that calculates a direction θ of the object on the image extracted by the object extraction unit 201; and a bottom area calculation unit 203 that calculates a bottom area of ​​the object on the image and a bottom area on the real world coordinates using the width of the 2D-BBOX and the direction θ of the object on the image calculated by the direction calculation unit 202. In this configuration, since the transformation is performed by the homography matrix using the direction θ of the object on the image, it is also possible to cope with a change in the center position according to the direction of the object. Therefore, it is possible to calculate the bottom area of ​​the object on the image and on the real world coordinates with higher accuracy than the conventional method, and therefore it is possible to obtain an object detection device 200 that can estimate the position, size (width, length) and direction of the object with high accuracy.

[0041] In addition, the bottom surface area calculation unit 203 performs calculations using the condition of "object length", which is either "object length" in real world coordinates, "object width" or "object aspect ratio", so that it is possible to distinguish between vehicles with different sizes in the same direction and calculate the bottom surface area on the image of the object and the bottom surface area on the real world coordinates.

[0042] Embodiment 2 The object detection system and the object detection device according to the second embodiment will be described below with reference to the drawings. The configuration of the object detection system according to the second embodiment is similar to that of the first embodiment shown in FIG. 1, and therefore a description thereof will be omitted.

[0043] <Configuration of Object Detection Device 200> Fig. 12 is a functional block diagram showing a configuration of object detection device 200 according to embodiment 2. In Fig. 12, object detection device 200 includes type determination unit 201a in object extraction unit 201 in the configuration of Fig. 2 of embodiment 1. The object extraction unit 201 extracts an object from an image acquired by the imaging unit 100, outputs it as a 2D-BBOX inscribed and surrounding the object, and determines the type of the object by the type determination unit 201a. Here, determining the type of the object means, for example, distinguishing between a normal vehicle, a large vehicle such as a truck, a two-wheeled vehicle, a person, etc. The type determination unit 201a performs type determination using existing technology such as an object detection model using a neural network. Note that the learned model, etc. used for type determination may be stored in the storage unit 300 provided in the object detection system 10, and read out when type determination is performed.

[0044] As in the first embodiment, direction calculation section 202 calculates the direction of the object extracted by object extraction section 201 on the image.

[0045] In the bottom surface area calculation unit 203, the bottom surface area of ​​the object is calculated using the 2D-BBOX of the object extracted by the object extraction unit 201 and the direction information of the object calculated by the direction calculation unit 202, as in the first embodiment. At this time, the bottom surface area is calculated using the "length of the object" and "aspect ratio of the object" according to the type of the object determined by the type determination unit 201a. For example, the aspect ratio is 3:1 for a normal vehicle, 4:1 for a large vehicle, and 1:1 for a person. It is also possible to use a vertical length of 3m for a normal vehicle, a vertical length of 8m for a large vehicle, and a vertical length of 1m for a person. The data associated with these types is stored in the storage unit 300 provided in the object detection system 10, read out by the bottom surface area calculation unit 203, and used to calculate the bottom surface area.

[0046] The output unit 204 outputs information on the bottom area of ​​the object, including the position, size (width, length) and direction of the object calculated by the bottom area calculation unit 203 .

[0047] As described above, according to the second embodiment, the same effects as those of the first embodiment are achieved. Furthermore, since the object extraction unit 201 includes the type determination unit 201a, it is possible to extract an object from an image acquired by the imaging unit 100 and output it as a 2D-BBOX inscribed and surrounding the object, and to determine the type of the object by the type determination unit 201a. As a result, the bottom area is calculated using the "length of the object" or the "aspect ratio of the object" based on the type of object determined by the bottom area calculation unit 203, so that it is possible to calculate the bottom area with higher accuracy, and the accuracy of the estimated object position is improved.

[0048] Embodiment 3 The object detection system and the object detection device according to the third embodiment will be described below with reference to the drawings. The configuration of the object detection system according to the third embodiment is similar to that of the first embodiment shown in FIG. 1, and therefore a description thereof will be omitted.

[0049] <Configuration of Object Detection Device 200> Fig. 13 is a functional block diagram showing the configuration of object detection device 200 according to embodiment 3. In Fig. 13, object detection device 200 includes object direction map 202a in the configuration of Fig. 2 of embodiment 1. Direction calculation section 202 calculates the direction of an object by referring to object direction map 202a.

[0050] Fig. 14 is a diagram for explaining object direction map 202a. Object direction map 202a defines the direction of an object according to the position of the object in real world coordinates. As shown in Fig. 14, for example, it is defined as 90° on a road extending north-south, and 180° on a road extending east-west. In the third embodiment, an arbitrary position of 2D-BBOX is set as the center coordinate of the object, and is converted to a position in real world coordinates by matrix M, and then the direction of the object according to the position is obtained from this map, thereby determining the direction of the object.

[0051] <Operation of Object Detection Device 200> Next, a procedure for object detection in object detection device 200 according to the third embodiment will be described with reference to the flowchart in Fig. 15. Note that the process of the flowchart in Fig. 15 is repeatedly executed.

[0052] As in the first embodiment, first, in step S201, the object extraction unit 201 acquires an image captured by a camera mounted on the roadside unit RU using the imaging unit 100, and in step S202, the object extraction unit 201 extracts an object from the image and outputs it as a 2D-BBOX that inscribes and surrounds the object.

[0053] Next, in step S203, the direction calculation unit 202 uses the 2D-BBOX output from the object extraction unit 201, and performs conversion from image coordinates to real-world coordinates by setting an arbitrary position, such as the center position of the bottom end of the 2D-BBOX, as described in embodiment 1.

[0054] In step S204, the object direction at the position on the transformed real world coordinates is obtained from the object direction map 202a. The obtained direction on the real world coordinates is converted into a direction on the image coordinates, and then output to the bottom area calculation unit 203.

[0055] In step S205, the bottom area calculation unit 203 uses the 2D-BBOX output from the object extraction unit 201 and the object direction output from the direction calculation unit 202 to calculate the bottom area of ​​the object on the image and on the dynamic map. The output unit 204 outputs the bottom surface area of ​​the object calculated by the bottom surface area calculation unit 203 .

[0056] As in the first and second embodiments, it is possible to calculate the direction of an object without using the object direction map 202a. However, it is possible to use the direction acquired from the object direction map 202a when the reliability of the object direction calculated by another method is low, or to use the direction acquired from the object direction map 202a only for a part of the detection area. Specifically, when the time-series change in the 2D-BBOX position is small and equal to or less than a preset threshold, or when the lane is narrow and the vehicle direction is limited, the calculation accuracy of the bottom area is higher when the direction acquired from the object direction map 202a according to the third embodiment is used as the object direction. By using them in this way, the accuracy of object detection is improved.

[0057] As described above, according to the third embodiment, the same effects as those of the first embodiment can be achieved. Furthermore, since the object detection device 200 includes the object direction map 202a, when the reliability of the object direction calculated by another method is low, the object direction map 202a can be used to complement the reliability. This makes it possible to calculate the bottom area with higher accuracy, improving the positional accuracy of the detected object.

[0058] Embodiment 4 An object detection system and an object detection device according to the fourth embodiment will be described below with reference to the drawings. The configuration of the object detection system according to the fourth embodiment is similar to that of the first embodiment shown in FIG. 1, and therefore a description thereof will be omitted.

[0059] <Configuration of Object Detection Device 200> Fig. 16 is a functional block diagram showing the configuration of object detection device 200 according to embodiment 4. In Fig. 16, object detection device 200 includes object direction table 202b instead of object direction map 202a in the configuration of Fig. 13 of embodiment 4. Direction calculation section 202 calculates the direction of an object by referring to object direction table 202b.

[0060] The object direction table 202b defines the direction of an object according to the aspect ratio of the 2D-BBOX. As shown in Figs. 9A and 9B in Fig. 9, it is assumed that the aspect ratio of the 2D-BBOX changes depending on the position and direction of the object for the same object. Therefore, a correspondence table of object directions according to object type, position, and aspect ratio is created in advance and tabulated.

[0061] FIG. 17 is a diagram showing an example of the object direction table 202b. Information on the type, center coordinates on the image coordinates, aspect ratio of the 2D-BBOX, angle on the real world coordinates, and angle on the image coordinates may be tabulated. Here, an example is shown in which a normal vehicle, center coordinates on the image coordinates (x1, y1), aspect ratio of the 2D-BBOX is 3:1, angle on the real world coordinates is 30°, and angle on the image coordinates is 170°. If such an object direction table 202b is provided, for example, even if the object extracted by the object extraction unit 201 is a stationary object and the time series history of the 2D-BBOX position cannot be used for direction calculation, it becomes possible to estimate the direction of the object.

[0062] However, even if the bottom area is the same, if the height of the object changes, the vertical width hbbox of the 2D-BBOX changes, and the aspect ratio also changes, so this method can only be applied between objects with the same width, length, and height. Therefore, this method is effective when it is possible to assume that "vehicles of all types are the same size." Specifically, if all transport vehicles in a factory have the same model number, they are extracted as objects with the same width, length, and height, and the object direction table 202b can be used.

[0063] Also, because this aspect ratio makes it impossible to distinguish between directions that are symmetrical with respect to the y-axis on the image coordinates (10° and 170°, 60° and 120°, etc.), which of these is the true direction can be estimated separately from the history of the 2D-BBOX position, or two types of bottom surface information can be output without making a distinction.When estimating from the history of the 2D-BBOX position, for example, when the aspect ratio is 3:2, if the 2D-BBOX position moves to the upper right, it can be determined to be 70°, and if it moves to the upper left, it can be determined to be 110°.

[0064] It is possible to calculate the object direction without using the object direction table 202b as in the first and second embodiments. However, as in the third embodiment, it is possible to use a direction obtained from the object direction table 202b when the reliability of the object direction calculated by another method is low, or to use a direction obtained from the object direction table 202b only for a part of the detection area.

[0065] As described above, according to the fourth embodiment, the same effects as those of the first embodiment are achieved. Furthermore, since the object detection device 200 is provided with the object direction table 202b, when objects of the same width, length, and height are to be detected, it is possible to complement the object direction calculated by another method using the object direction map 202a when the reliability of the object direction is low. This makes it possible to calculate the bottom area with higher accuracy, improving the positional accuracy of the detected object.

[0066] Each functional unit of object detection system 10 and object detection device 200 in the above-mentioned embodiments 1 to 4 is configured with the hardware configuration illustrated in FIG. 18, that is, an arithmetic processing circuit 1001, a memory device 1002 including a ROM (Read Only Memory) in which a program that executes the functions of each functional unit is stored and a RAM (Random Access Memory) that saves each data of the execution results of each functional unit, which are the results of calculations performed by the program, and an input / output circuit 1003. Image information from the imaging unit 100 is input to the input / output circuit 1003, and the image information is stored in the storage device 1002. In addition, the output of the object detection device 200 is applied to an autonomous driving system, and is therefore output to, for example, an autonomous driving vehicle or a control system.

[0067] Furthermore, each functional unit of object detection system 10 and object detection device 200 in the above-described first to fourth embodiments may have a hardware configuration as illustrated in Fig. 19, that is, a configuration in which a communication circuit 1004 is further provided in addition to the configuration in Fig. 18. Signals can be exchanged between imaging unit 100, object detection device 200, and storage unit 300 in a wired or wireless manner. The communication circuit 1004 includes, as a communication module, a wide-area communication unit and a short-range communication unit. The wide-area communication unit uses a predetermined wide-area wireless communication standard, such as LTE (Long Term Evolution), 4G, or 5G (5th Generation; 5th generation mobile communication system). The short-range communication unit may use, for example, DSRC (Dedicated Short Range Communications).

[0068] A processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor) is applied to the arithmetic processing circuit 1001. Dedicated hardware may also be applied to the arithmetic processing circuit 1001. When the arithmetic processing circuit 1001 is dedicated hardware, the arithmetic processing circuit 1001 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination of these.

[0069] Furthermore, each functional unit of the object detection system 10 and the object detection device 200 may be realized by a separate arithmetic processing circuit, or they may be collectively realized by a single arithmetic processing circuit. Furthermore, each functional unit of the object detection system 10 and the object detection device 200 can realize the above-mentioned functions by hardware, software, etc., or a combination of these, with some functions being realized by a calculation processing circuit as dedicated hardware, and other functions being realized by software.

[0070] <Other embodiments> When the object detection system 10 equipped with the object detection device 200 shown in any of the first to fourth embodiments is applied to an autonomous driving system, it becomes possible to detect the object position with high accuracy from the images acquired by the roadside unit RU and reflect the detected object position in a dynamic map, thereby enabling a traveling vehicle to avoid obstacles in a planned manner.

[0071] Furthermore, the automatic driving system to which the above-mentioned object detection system 10 and object detection device 200 are applied is not limited to automobiles, but can be applied to various other moving bodies. For example, the object detection system 10 and object detection device 200 can be applied to automatically traveling moving bodies such as an in-building mobile robot that inspects the inside of a building, a line inspection robot, and personal mobility.

[0072] Although the present application describes various exemplary embodiments and examples, the various features, aspects, and functions described in one or more embodiments are not limited to application to a particular embodiment, but may be applied to the embodiments alone or in various combinations. Therefore, countless modifications not exemplified are assumed within the scope of the technology disclosed in the present specification, including, for example, modifying, adding, or omitting at least one component, and further, extracting at least one component and combining it with a component of another embodiment.

[0073] Various aspects of the present disclosure are summarized below as appendices.

[0074] (Appendix 1) An object detection device that extracts an object from an image acquired by an imaging unit and calculates a position of the object in a real world coordinate system, an object extraction unit that extracts the object from the image and outputs a rectangle that inscribes and surrounds the object; a direction calculation unit that calculates a direction of the object on the image extracted by the object extraction unit; a bottom surface area calculation unit that calculates a bottom surface area of ​​the object on the image and on real world coordinates using the width of the rectangle output from the object extraction unit and the direction of the object on the image calculated by the direction calculation unit, An object detection device, wherein the bottom area includes the position, size and orientation of the object on the image and in real world coordinates, respectively. (Appendix 2) The object detection device described in Appendix 1, wherein the bottom area calculation unit further calculates the bottom area of ​​the object on the image and the bottom area in real world coordinates using any of the width, length, and ratio of width to length of the object in real world coordinates. (Appendix 3) the object extraction unit includes a type determination unit that determines a type of the extracted object, outputs a rectangle that inscribes and surrounds the object, and outputs the type of the object determined by the type determination unit; The object detection device described in Appendix 2, wherein the bottom area calculation unit calculates the bottom area of ​​the object on the image and the bottom area in real world coordinates using any of the width, length, and ratio of width to length of the object in real world coordinates based on the type of the object determined by the type determination unit. (Appendix 4) An object direction map in which an object direction is defined according to a position in a real world coordinate system is further provided, 4. The object detection device according to claim 1, wherein the direction calculation unit calculates a direction of the object on the image extracted by the object extraction unit by using the object direction map. (Appendix 5) an object direction table in which directions on the image and on real world coordinates are defined according to the aspect ratio of the rectangle; 4. The object detection device according to claim 1, wherein the direction calculation unit calculates a direction of the object on the image extracted by the object extraction unit by using the object direction table. (Appendix 6) An object detection device according to any one of appendix 1 to 5; An object detection system comprising the imaging unit. (Appendix 7) 7. The object detection system of claim 6, wherein the imaging unit includes a roadside unit equipped with a camera. [Explanation of symbols]

[0075] 10: object detection system, 100: imaging unit, 200: object detection device, 201: object extraction unit, 201a: type determination unit, 202: direction calculation unit, 202a: object direction map, 202b: object direction table, 203: bottom area calculation unit, 204: output unit, 300: memory unit, 1001: arithmetic processing circuit, 1002: storage device, 1003: input / output circuit, 1004: communication circuit, BAp, BAw, BAp1, BAp2: bottom area, RU: roadside unit, VE1, VE2: vehicles.

Claims

1. An object detection device that extracts an object from an image acquired by an imaging unit and calculates a position of the object in a real world coordinate system, an object extraction unit that extracts the object from the image and outputs a rectangle that is adjacent to and surrounds the object; a direction calculation unit that calculates a direction of the object on the image extracted by the object extraction unit; a bottom surface area calculation unit that calculates a bottom surface area of ​​the object on the image and on real world coordinates using the width of the rectangle output from the object extraction unit and the direction of the object on the image calculated by the direction calculation unit, An object detection device, wherein the bottom area includes the position, size and orientation of the object on the image and in real world coordinates, respectively.

2. The object detection device according to claim 1 , wherein the bottom area calculation unit further calculates the bottom area of ​​the object on the image and the bottom area in the real world coordinates using any one of the width, length, and ratio of the width and length of the object in real world coordinates.

3. the object extraction unit includes a type determination unit that determines a type of the extracted object, outputs a rectangle that is in contact with and surrounds the object, and outputs the type of the object determined by the type determination unit; 3. The object detection device according to claim 2, wherein the bottom area calculation unit calculates the bottom area of ​​the object on the image and the bottom area of ​​the object in real world coordinates using any one of the width, length, and ratio of width to length of the object in real world coordinates based on the type of the object determined by the type determination unit.

4. An object direction map in which an object direction is defined according to a position in a real world coordinate system is further provided, The object detection device according to claim 1 , wherein the direction calculation unit calculates the direction of the object in the image extracted by the object extraction unit by using the object direction map.

5. an object direction table in which directions on the image and on real world coordinates are defined according to the aspect ratio of the rectangle; The object detection device according to claim 1 , wherein the direction calculation unit calculates the direction of the object in the image extracted by the object extraction unit by using the object direction table.

6. An object detection device according to any one of claims 1 to 5; An object detection system comprising the imaging unit.

7. The object detection system according to claim 6 , wherein the imaging unit comprises a roadside unit equipped with a camera.

Images