Object recognition system, object recognition device, object recognition method, and program

Abstract

This invention provides an object recognition system, object recognition device, object recognition method, and program that can perform object recognition even when it is difficult to perform object recognition using only images. [Solution] The object recognition system comprises: an image recognition unit that performs image recognition on image data captured from a target area and generates an image bounding box; a point cloud clustering unit that performs clustering on point cloud data measured from the target area and generates a point cloud bounding box; an association unit that associates the image bounding box with the point cloud bounding box; and an output unit that outputs the image bounding box associated with the point cloud bounding box as an object recognition result based on the distance of the point cloud bounding boxes.

Description

Technical Field

[0001] The present disclosure relates to an object recognition system, an object recognition device, an object recognition method, and a program.

Background Art

[0002] Techniques for performing object recognition using images captured by a camera are utilized. For example, as a related technique, Patent Document 1 is known. In Patent Document 1, it is described that in an object recognition device, the distance to an object is obtained from images captured by two cameras, and clustering is performed based on the obtained distance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In related techniques such as Patent Document 1, it is premised on using images captured by a camera. However, with only images captured by a camera, it may be difficult to perform object recognition, for example, when an object is far away.

[0005] In view of such problems, one object of the present disclosure is to provide an object recognition system, an object recognition device, an object recognition method, and a program capable of performing object recognition even when it is difficult to perform object recognition only with images.

Means for Solving the Problems

[0006] An object recognition system according to one aspect of this disclosure includes: an image recognition unit that performs image recognition on image data captured from a target area and generates an image bounding box; a point cloud clustering unit that performs clustering on point cloud data measured from the target area and generates a point cloud bounding box; an association unit that associates the image bounding box with the point cloud bounding box; and an output unit that outputs the image bounding box associated with the point cloud bounding box as an object recognition result based on the distance of the point cloud bounding box.

[0007] An object recognition device according to one aspect of the present disclosure includes: an image recognition unit that performs image recognition on image data captured from a target area to generate an image bounding box; a point cloud clustering unit that performs clustering on point cloud data measured from the target area to generate a point cloud bounding box; an association unit that associates the image bounding box with the point cloud bounding box; and an output unit that outputs the image bounding box associated with the point cloud bounding box as an object recognition result based on the distance of the point cloud bounding box.

[0008] An object recognition method according to one aspect of the present disclosure includes: performing image recognition on image data captured from a target area to generate an image bounding box; performing clustering on point cloud data measured from the target area to generate a point cloud bounding box; associating the image bounding box with the point cloud bounding box; and outputting the image bounding box associated with the point cloud bounding box as an object recognition result based on the distance of the point cloud bounding box.

[0009] A program according to one aspect of this disclosure is a program for causing a computer to perform the following processes: performing image recognition on image data captured from a target area to generate an image bounding box; performing clustering on point cloud data measured from the target area to generate a point cloud bounding box; associating the image bounding box with the point cloud bounding box; and outputting the image bounding box associated with the point cloud bounding box as an object recognition result based on the distance of the point cloud bounding box. [Effects of the Invention]

[0010] According to this disclosure, object recognition can be performed even when it is difficult to perform object recognition using only images. [Brief explanation of the drawing]

[0011] [Figure 1] This diagram shows an example configuration of an object recognition system according to several embodiments. [Figure 2] This is a configuration diagram showing an example of the configuration of an object recognition device according to several embodiments. [Figure 3] This flowchart shows examples of object recognition methods according to several embodiments. [Figure 4] This is a configuration diagram showing an example of the configuration of an infrastructure coordination system according to several embodiments. [Figure 5] This is a configuration diagram showing an example of the configuration of an infrastructure-cooperative server according to several embodiments. [Figure 6] This flowchart shows examples of the operation of an infrastructure-cooperative server according to several embodiments. [Figure 7] This is a configuration diagram showing an example of the configuration of an infrastructure-cooperative server according to several embodiments. [Figure 8] This is a configuration diagram showing an example of the configuration of an infrastructure coordinating server according to several embodiments. [Figure 9]It is a configuration diagram showing a configuration example of the hardware of a computer according to some embodiments.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments will be described with reference to the drawings. In each drawing, the same elements are denoted by the same reference numerals, and redundant descriptions will be omitted as necessary.

[0013] (Embodiment 1) First, Embodiment 1 will be described. In this embodiment, an overview of some embodiments will be described.

[0014] FIG. 1 shows a configuration example of an object recognition system 10 according to some embodiments. For example, the object recognition system 10 is an infrastructure cooperation (road-vehicle cooperation) system that processes data from a plurality of sensors on the roadside (road side) or vehicle side, but it may also be other systems that process data from a plurality of sensors. For example, the sensors include a 2D camera and LiDAR (Light Detection And Ranging), but may also include other sensors.

[0015] In the example of FIG. 1, the object recognition system 10 includes an image recognition unit 11, a point cloud clustering unit 12, a matching unit 13, and an output unit 14.

[0016] The image recognition unit 11 performs image recognition on the image data obtained by photographing the target area and generates an image bounding box. For example, the image recognition unit 11 acquires the image data captured by the camera and performs image recognition on the acquired image data. The image recognition unit 11 recognizes the objects in the image data by image recognition and generates an image bounding box, which is the area containing the objects.

[0017] The point cloud clustering unit 12 performs clustering on the point cloud data obtained by measuring the target area to generate a point cloud bounding box. For example, the point cloud clustering unit 12 acquires point cloud data from a LiDAR that has measured the same area as the area photographed by the camera, and clusters the acquired point cloud data. The point cloud clustering unit 12 performs clustering based on the feature amounts of the point cloud data, and generates a point cloud bounding box, which is an area including the point cloud of the cluster.

[0018] The association unit 13 associates the image bounding box generated by image recognition of the image data with the point cloud bounding box generated by clustering of the point cloud data. For example, the association unit 13 associates the image bounding box with the point cloud bounding box by matching between the image bounding box and the point cloud bounding box or by IoU (Intersection over Union).

[0019] The output unit 14 outputs the associated bounding box as an object recognition result. The output unit 14 selects a bounding box to be output from a plurality of bounding boxes. For example, as the bounding box to be output, the output unit 14 selects an image bounding box associated with the point cloud bounding box.

[0020] The output unit 14 outputs, as an object recognition result, an image bounding box associated with the point cloud bounding box based on the distance of the point cloud bounding box. For example, the output unit 14 may determine a selection threshold based on the distance of the point cloud bounding box, and select the image bounding box to be output based on the determined selection threshold. The selection threshold may be a threshold for determining the confidence score, which is the result of image recognition of the image bounding box, or may be a threshold for determining the IoU when the image bounding box overlaps with other image bounding boxes.

[0021] The output unit 14 may select the image bounding box to output based on the velocity of the point cloud bounding box and the illuminance of the target area. In other words, the output unit 14 may determine a selection threshold based on the velocity of the point cloud bounding box and the illuminance of the target area.

[0022] The matching unit 13 may also associate the image bounding box with the point cloud bounding box based on the distance between the point cloud bounding boxes. For example, the matching unit 13 may determine a matching threshold based on the distance between the point cloud bounding boxes and associate the image bounding box with the point cloud bounding box based on the determined matching threshold. The matching threshold may be a threshold for determining the matching rate between the image bounding box and the point cloud bounding box, or a threshold for determining the IoU.

[0023] The matching unit 13 may associate the image bounding box with the point cloud bounding box based on the velocity of the point cloud bounding box and the illuminance of the target area. In other words, the matching unit 13 may determine a matching threshold based on the velocity of the point cloud bounding box and the illuminance of the target area.

[0024] The object recognition system 10 may be composed of one device or multiple devices. Figure 2 shows an example configuration of an object recognition device 20 according to several embodiments. In the example in Figure 2, the object recognition device 20 includes the image recognition unit 11, point cloud clustering unit 12, correspondence unit 13, and output unit 14 shown in Figure 1. For example, part or all of the object recognition system 10 or the object recognition device 20 may be placed on an edge device installed at the edge, or on a cloud server installed in the cloud.

[0025] Figure 3 shows examples of object recognition methods according to several embodiments. For example, the object recognition methods according to some embodiments may be performed by the object recognition system 10 in Figure 1 or the object recognition device 20 in Figure 2.

[0026] In the example shown in Figure 3, first, the image recognition unit 11 performs image recognition on the image data captured from the target area and generates an image bounding box (S11). Then, the point cloud clustering unit 12 performs clustering on the point cloud data measured from the target area and generates a point cloud bounding box (S12).

[0027] Next, the mapping unit 13 maps the generated image bounding box to the generated point cloud bounding box (S13). Then, the output unit 14 outputs the image bounding box mapped to the point cloud bounding box as the object recognition result, based on the distance of the point cloud bounding box (S14).

[0028] In related technologies, performing object recognition using only camera images presented a problem: objects could not be recognized at long distances or in low-light environments, resulting in degraded object recognition accuracy. For example, when recognizing and tracking objects from images, object recognition is not possible at long distances or in low-light environments, preventing tracking until the object is close or in a high-light environment. A similar problem exists when objects are moving at high speed. Since tracking is possible once an object has been recognized, this embodiment enables earlier object recognition even at long distances or in low-light environments.

[0029] Therefore, in this embodiment, an image bounding box generated by image recognition of image data is associated with a point cloud bounding box generated by clustering point cloud data, and the object recognition result is output from the associated image bounding box. For example, the image bounding box to be output is selected based on the distance of the point cloud bounding box. A selection threshold for selecting the image bounding box to be output may be determined based on the distance of the point cloud bounding box. The selection threshold may also be determined based on illuminance or velocity. This makes it possible to output an image bounding box as an object recognition result and perform object recognition even in distant or low-light environments. For example, when an object moves from far away to near, object recognition can be performed earlier.

[0030] The following embodiments will describe specific examples of Embodiment 1.

[0031] (Embodiment 2) Next, Embodiment 2 will be described. In this embodiment, an example will be described in which the threshold for determining object recognition is adjusted based on the distance to the object.

[0032] Figure 4 shows an example configuration of the infrastructure coordination system 1 according to several embodiments. For example, the infrastructure coordination system 1 is a system that supports the safety of vehicle operation by combining and analyzing data collected from multiple sensors placed on the roadside or on vehicles. The infrastructure coordination system 1 may be a remote monitoring system such as an ITS system (Intelligence Transport System) that monitors roads and vehicles, or a remote control system that controls vehicles according to the monitoring results. Vehicles may include automobiles, motorcycles, heavy machinery such as forklifts, trains, robots, drones, etc.

[0033] In the example shown in Figure 4, the infrastructure coordination system 1 includes a cloud server 100a, an MEC 100b, multiple sensors 200, and a base station 300. For example, either the cloud server 100a or the MEC 100b constitutes the infrastructure coordination server 100.

[0034] Multiple sensors 200, MEC 100b, and base station 300 are located on the road side and vehicle side (also called the road-vehicle side), while the cloud server 100a is located on the cloud side. For example, the cloud server 100a is located in a data center or similar location far from the road-vehicle side. For example, the road-vehicle side is the edge side relative to the cloud.

[0035] Multiple sensors 200 and base station 300 are connected via network NW1 for communication. Network NW1 is a wireless network such as 4G, LTE (Long Term Evolution), local 5G / 5G, other generations of mobile communication, or Wi-Fi. For example, network NW1 could be a DSRC (Dedicated Short Range Communication) network for ITS systems, or a V2X (Vehicle to Everything) network connecting vehicles to everything. V2X could be LTE-V2X (Long Term Evolution-V2X), NR-V2X (New Radio V2X), C-V2X (Cellular V2X), etc. Note that network NW1 is not limited to a wireless network; it may also be a wired network.

[0036] The base station 300 and the MEC100b are connected in a way that enables communication using any communication method. It can also be said that the sensor 200 and the MEC100b are connected in a way that enables communication via the base station 300. The base station 300 and the MEC100b may be a single device. For example, the base station 300 may have the functions of the MEC100b.

[0037] The base station 300 and the cloud server 100a are connected via network NW2, enabling communication. Network NW2 includes, for example, core networks such as 5GC (5th Generation Core network) and EPC (Evolved Packet Core), as well as the internet. Note that network NW2 is not limited to a wired network; it may also be a wireless network. It can also be said that the sensor 200 and the cloud server 100a are connected via base station 300 and network NW2, enabling communication.

[0038] Sensor 200 may be included in or connected to a terminal device connected to network NW1. Sensor 200 or the terminal device may be installed on the roadside or mounted on a vehicle. For example, sensor 200 or the terminal device may be a roadside unit (RSU) installed on the roadside or an on-board unit (OBU) mounted on a vehicle. Sensor 200 transmits the measured sensor data to MEC100b or cloud server 100a.

[0039] Sensor 200 includes different types of sensors. For example, sensor 200 includes a camera 201 that captures 2D images (video) and a LiDAR 202 that generates point cloud data. It is not limited to LiDAR 202; it may also include 3D sensors capable of acquiring 3D information, such as radar or SfM (Structure From Motion).

[0040] The video data generated by camera 201 includes multiple time-series images, i.e., frames. The point cloud data generated by LiDAR 202 includes coordinate information in three-dimensional space obtained from reflected light from objects at each point within the measurement range measured by the LiDAR. The coordinate information indicates the depth or three-dimensional position of the point in three-dimensional space. The point cloud data may include not only the coordinates at each point, but also the reflectance of light at each point, etc.

[0041] Base station 300 is a base station device for network NW1 and also a relay device that relays communication between sensor 200 and MEC100b or cloud server 100a. For example, base station 300 may be a local 5G base station, a 5G gNB (next generation node B), an LTE eNB (evolved node B), a wireless LAN access point, or any other relay device.

[0042] MEC (Multi-access Edge Computing) 100b is an edge server installed at the edge of the system. MEC 100b may be one or more physical computers, or a virtual computer built on any virtualization platform. MEC 100b may process sensor data, including video data and point cloud data, received from sensor 200, and send the processed data to cloud server 100a, or control sensor 200 as needed.

[0043] The cloud server 100a is a server located on the cloud side. The cloud server 100a may be one or more physical servers, or it may be a virtualized server built on any virtualization platform. The cloud server 100a processes sensor data, including video data and point cloud data, received from the sensor 200, and may control the sensor 200 as needed.

[0044] The infrastructure coordination server 100, which consists of cloud server 100a or MEC 100b, monitors the situation around the vehicle by analyzing and recognizing sensor data, including road and vehicle-side video data and point cloud data, and controls the vehicle as needed. The infrastructure coordination server 100 includes object recognition functions (object recognition device), tracking, and prediction functions. For example, the infrastructure coordination server 100 performs object recognition from video data and point cloud data, tracks the recognized vehicle to predict its movement, and predicts the situation of pedestrians, fallen objects, animals, etc., around the moving vehicle.

[0045] The infrastructure coordination server 100 may feed back the predicted results to the vehicle's sensors 200. The infrastructure coordination server 100 may also transmit predicted information about the area around the vehicle, or transmit control information to control the vehicle's movement. For example, it can provide information about obstacles in blind spots that the vehicle cannot recognize. Even when the vehicle cannot recognize its surroundings at night or in bad weather, it can safely control its automatic driving. For example, even when detection using camera images is difficult, such as at a distance, it can utilize point cloud data from LiDAR to provide various vehicle position information and hazard information.

[0046] Figure 5 shows an example configuration of the infrastructure coordinating server 100 according to several embodiments. Figure 5 shows an example configuration of the object recognition function (object recognition device) in the infrastructure coordinating server 100. Note that the configuration in Figure 5 is just one example, and other configurations are also acceptable as long as they enable the operation according to several embodiments. For example, some functions of the infrastructure coordinating server 100 may be located in sensors, terminals connected to sensors, or other devices.

[0047] In the example shown in Figure 5, the infrastructure coordinating server 100 includes a point cloud acquisition unit 101, an image acquisition unit 102, a point cloud clustering unit 103, an image recognition unit 104, a data synchronization unit 105, a bounding box synchronization unit 106, a synchronization threshold setting unit 107, a bounding box selection unit 108, and a selection threshold setting unit 109.

[0048] The point cloud acquisition unit 101 acquires point cloud data measured by the LiDAR 202 from the LiDAR 202. The image acquisition unit 102 acquires image data captured by the camera 201 from the camera 201. The point cloud acquisition unit 101 and the image acquisition unit 102 acquire point cloud data and image data from the LiDAR 202 and camera 201, which measure the same area.

[0049] The point cloud clustering unit 103 performs clustering on the point cloud data acquired by the point cloud acquisition unit 101. The point cloud clustering unit 103 clusters the point cloud based on features and generates a point cloud bounding box. Features used for clustering include, for example, shape, velocity, and point cloud density. The point cloud clustering unit 103 clusters the point cloud data using a clustering engine that employs machine learning such as deep learning, and outputs a 3D rectangular region containing the clustered clusters as a point cloud bounding box. In addition, clustering in the point cloud clustering unit 103 may be performed by grouping features with similar Euclidean distances together. The point cloud bounding box is assigned the clustering results, such as position (3D coordinates including distance) and velocity. The distance is the distance (depth) from the LiDAR to the object.

[0050] The image recognition unit 104 performs image recognition on the image data acquired by the image acquisition unit 102. The image recognition unit 104 recognizes objects and generates image bounding boxes through image recognition. The image recognition unit 104 recognizes objects using an image recognition (object recognition) engine that employs machine learning such as deep learning, and outputs a two-dimensional rectangular region containing the recognized object as an image bounding box. The image recognition unit 104 outputs all image bounding boxes that were generated by the image recognition engine. The image bounding boxes are assigned information such as the object type, position (two-dimensional coordinates), and confidence score, which are the results of the image recognition.

[0051] The data synchronization unit 105 synchronizes the time and position of the point cloud data and image data. The data synchronization unit 105 synchronizes point cloud data and image data measured and captured in the same area at the same time. The synchronization can be performed by directly synchronizing the point cloud data and image data, or it can be performed after the bounding box is generated. That is, the point cloud clustering unit 103 and the image recognition unit 104 may generate point cloud bounding boxes and image bounding boxes from point cloud data and image data measured and captured in the same area at the same time, or they may synchronize point cloud bounding boxes and image bounding boxes for the same area at the same time using the point cloud bounding boxes and image bounding boxes generated by the point cloud clustering unit 103 and the image recognition unit 104.

[0052] The bounding box synchronization unit 106 associates and synchronizes the point cloud bounding box generated by the point cloud clustering unit 103 with the image bounding box generated by the image recognition unit 104. That is, it associates point cloud bounding boxes and image bounding boxes that may represent the same object. For example, the bounding box synchronization unit 106 projects the point cloud bounding box onto the same two-dimensional plane as the image data, matches the projected two-dimensional point cloud bounding box with the image bounding box, and associates the point cloud bounding box with the image bounding box based on the IoU of the matched point cloud bounding box and image bounding box. The bounding box synchronization unit 106 may select and associate point cloud bounding boxes and image bounding boxes whose matching rate is higher than a set value (synchronization threshold). The bounding box synchronization unit 106 may also select and associate point cloud bounding boxes and image bounding boxes whose IoU is greater than a set value (synchronization threshold). For example, the associated bounding box (image bounding box) may be assigned velocity and position information obtained by the point cloud clustering unit 103 and object recognition information (such as confidence score) obtained by the image recognition unit 104.

[0053] The synchronization threshold setting unit 107 sets a synchronization threshold (correspondence threshold) for the bounding box synchronization unit 106 to determine the correspondence between the point cloud bounding box and the image bounding box. The bounding box synchronization unit 106 and the synchronization threshold setting unit 107 may constitute a correspondence unit. For example, the synchronization threshold setting unit 107 sets the synchronization threshold based on the distance of the point cloud bounding box. For example, the synchronization threshold according to the distance may be determined using a table that associates distance with the synchronization threshold to be set, or the synchronization threshold according to the distance may be determined using a learning model that has learned the relationship between distance and the synchronization threshold to be set. The synchronization threshold may be a threshold for determining the matching rate, or a threshold for determining the IoU. For example, when the distance of the point cloud bounding box is far, that is, when determining the correspondence between a distant point cloud bounding box and an image bounding box, the synchronization threshold for determining the matching rate or the synchronization threshold for determining the IoU is set lower. This makes it possible to detect and start tracking even distant objects early. If the camera is fixed, the distance to objects may not change across different areas of the image, so a synchronization threshold based on distance may be set for each area within the image. Alternatively, if the point cloud data includes reflectivity, a synchronization threshold may be set based on reflectivity.

[0054] The bounding box selection unit 108 selects a bounding box to output as a recognition result from the bounding boxes (image bounding boxes to which point cloud bounding boxes have been associated) associated by the bounding box synchronization unit 106. The selected bounding box may be output to a function that uses the object recognition result (such as tracking) or to another device. For example, the selected bounding box may be displayed on a display device. For example, the bounding box selection unit 108 may select a bounding box to output based on the confidence score of the bounding box, or based on the IoU of the bounding box. The bounding box selection unit 108 may also select and output a bounding box whose confidence score is greater than a set value (selection threshold). When multiple bounding boxes overlap, the bounding box selection unit 108 determines the IoU, and if the IoU is greater than a set value (selection threshold), it may select one or all of the overlapping bounding boxes.

[0055] The selection threshold setting unit 109 sets a selection threshold for determining the bounding box output by the bounding box selection unit 108. The bounding box selection unit 108 and the selection threshold setting unit 109 may constitute an output unit. For example, the selection threshold setting unit 109 sets a selection threshold based on the distance of the bounding box. For example, a table relating distance to the selection threshold to be set may be used to determine the selection threshold according to the distance, or a learning model that has learned the relationship between distance and the selection threshold to be set may be used to determine the selection threshold according to the distance. The selection threshold may be a threshold for determining the confidence score, or a threshold for determining the IoU. For example, if the distance of the bounding box is far, the selection threshold for determining the confidence score or the selection threshold for determining the IoU may be set lower. This allows for early detection and tracking of distant objects. In addition, similar to the synchronization threshold, if the camera is a fixed camera, a distance-based selection threshold may be set for each area in the image. Also, if the point cloud data includes reflectance, the synchronization threshold may be set based on the reflectance.

[0056] Figure 6 shows examples of the operation of the infrastructure coordinating server 100 according to several embodiments. In the example in Figure 6, first, the infrastructure coordinating server 100 acquires image data (S101) and performs image recognition on the acquired image data (S102). For example, the image acquisition unit 102 acquires image data from the camera 201. The image recognition unit 104 performs image recognition on the acquired image data to recognize objects and generate image bounding boxes.

[0057] Furthermore, the infrastructure coordinating server 100 acquires point cloud data (S103) and performs clustering on the acquired point cloud data (S104). For example, the point cloud acquisition unit 101 acquires point cloud data from the LiDAR 202. The point cloud clustering unit 103 performs clustering on the acquired point cloud data based on its features and generates a point cloud bounding box. Note that steps S101-S102 and S103-S104 may be executed simultaneously, or one may be executed first.

[0058] Next, the infrastructure coordinating server 100 sets a synchronization threshold based on the distance of the generated point cloud bounding boxes (S105). For example, the distance of the point cloud bounding boxes is assigned to the point cloud bounding boxes in clustering. For example, the synchronization threshold setting unit 107 sets a lower synchronization threshold if the distance of the point cloud bounding boxes is farther than a predetermined value. The synchronization threshold setting unit 107 may set either a synchronization threshold for determining the matching rate, a synchronization threshold for determining the IoU, or both.

[0059] Next, the infrastructure coordination server 100 associates the generated point cloud bounding boxes with the generated image bounding boxes using the configured synchronization threshold (S106). For example, the bounding box synchronization unit 106 may perform matching between the point cloud bounding boxes and image bounding boxes and associate the point cloud bounding boxes and image bounding boxes that match with a matching rate higher than the synchronization threshold. Alternatively, the bounding box synchronization unit 106 may calculate the IoU of the matched point cloud bounding boxes and image bounding boxes and associate the point cloud bounding boxes and image bounding boxes that overlap with an IoU greater than the synchronization threshold.

[0060] Next, the infrastructure coordinating server 100 sets a selection threshold based on the distance of the point cloud bounding boxes (S107). For example, the distance of the point cloud bounding boxes is assigned to the point cloud bounding boxes in clustering. The distance may also be the distance assigned to the associated bounding box (image bounding box). For example, the selection threshold setting unit 109 sets a lower selection threshold if the distance of the point cloud bounding boxes is farther than a predetermined value. The selection threshold setting unit 109 may set either a selection threshold for determining the confidence score and a selection threshold for determining the IoU, or both.

[0061] Next, the infrastructure coordinating server 100 selects the bounding boxes to output using the configured selection threshold (S108). For example, the bounding box selection unit 108 may select and output the bounding box if the confidence score of the image bounding box associated with the point cloud bounding box is greater than the selection threshold. If multiple image bounding boxes overlap, the bounding box selection unit 108 calculates the IoU, and if the calculated IoU is greater than the selection threshold, it may select and output one or all of the overlapping bounding boxes.

[0062] As described above, in this embodiment, when recognizing vehicles, people, etc. in 2D images, LiDAR point clouds are clustered based on movement speed, point cloud density, shape, etc., and for distant objects (clusters), recognition results can be displayed even for bounding boxes with low image recognition confidence scores and IoU. By varying the threshold levels of the confidence score and IoU displayed in the recognition results according to the distance of the distant object, it becomes possible to prioritize tracking over recognition accuracy for distant objects, thereby improving safety in autonomous driving and other applications. When tracking is performed, once recognition is performed, tracking can begin early even if the recognition confidence level is low. Furthermore, when the distance to the object approaches, the threshold level can be adjusted again, enabling early recognition at a distance without reducing recognition accuracy at close range. The same effect can also be obtained by adjusting the threshold level for mapping image bounding boxes and point cloud bounding boxes.

[0063] (Embodiment 3) Next, Embodiment 3 will be described. In this embodiment, an example will be described in which the threshold for determining object recognition is adjusted based on the velocity of the object.

[0064] Figure 7 shows an example configuration of the infrastructure coordinating server 100 according to several embodiments. In the example in Figure 7, the infrastructure coordinating server 100 includes a speed calculation unit 110 in addition to the configuration in Figure 5. Otherwise, it is the same as in Embodiment 2.

[0065] The velocity calculation unit 110 calculates the velocity of the point cloud bounding boxes generated by the point cloud clustering unit 103. For example, past point cloud bounding boxes are stored in the memory unit, and the movement velocity is calculated from the amount of movement and time of the point cloud bounding boxes. The velocity calculation unit 110 may also obtain the movement velocity from the clustering results of the point cloud bounding boxes.

[0066] In this example, the synchronization threshold setting unit 107 sets the synchronization threshold based on the velocity of the point cloud bounding box. The synchronization threshold setting unit 107 may also set the synchronization threshold based on the distance and velocity of the point cloud bounding box, or it may set the synchronization threshold based only on the velocity of the point cloud bounding box. For example, if the velocity of the point cloud bounding box is high, the synchronization threshold for determining the matching rate or the synchronization threshold for determining the IoU is set lower. This allows for early detection of fast-moving objects and initiation of tracking.

[0067] Furthermore, the selection threshold setting unit 109 sets the selection threshold based on the velocity of the point cloud bounding box. The selection threshold setting unit 109 may set the selection threshold based on the distance and velocity of the point cloud bounding box, or it may set the selection threshold based only on the velocity of the point cloud bounding box. For example, if the velocity of the point cloud bounding box is high, the selection threshold for determining the confidence score or the selection threshold for determining the IoU is set lower. This makes it possible to detect fast-moving objects early and start tracking.

[0068] Thus, the threshold level can be varied according to the object's movement speed. For example, the threshold level can be kept unchanged for stationary clusters (objects) at a distance, or for slow-moving clusters such as pedestrians, while the threshold can be changed only for high-speed moving clusters that are considered to pose a high risk. This allows for the early detection of high-risk, high-speed moving objects.

[0069] (Embodiment 4) Next, Embodiment 4 will be described. In this embodiment, an example will be described in which the threshold for determining object recognition is adjusted based on the object's measurement environment.

[0070] Figure 8 shows an example configuration of the infrastructure coordinating server 100 according to several embodiments. In the example in Figure 8, the infrastructure coordinating server 100 includes an environmental information acquisition unit 111 in addition to the configuration in Figure 5. The rest is the same as in embodiments 2 and 3.

[0071] The environmental information acquisition unit 111 acquires environmental information related to the measurement environment of the camera 201. For example, the environmental information is the illuminance around the camera 201. The environmental information is not limited to illuminance; it may also include the time of shooting and the weather. For example, the environmental information acquisition unit 111 may acquire environmental information from the camera 201, or from other devices. For example, it may acquire illuminance from an illuminance sensor installed near the camera 201. It may also acquire the time of shooting along with the image data from the camera 201. It may also acquire the weather from a server or the like that manages the weather at each location.

[0072] In this example, for instance, the synchronization threshold setting unit 107 sets the synchronization threshold based on environmental information. The synchronization threshold setting unit 107 may set the synchronization threshold based on the distance (velocity) of the point cloud bounding box and environmental information, or it may set the synchronization threshold based solely on environmental information. For example, in low illumination (dark) conditions, the synchronization threshold for determining the matching rate or the synchronization threshold for determining the IoU is set lower. Similarly, the synchronization threshold may be set lower in cases of excessive illumination (such as car headlights), severe weather, or at night. This allows for early detection and tracking to begin even in dark environments. It also helps to suppress the influence of car headlights and the like.

[0073] Furthermore, the selection threshold setting unit 109 sets the selection threshold based on environmental information. The selection threshold setting unit 109 may set the selection threshold based on the distance (velocity) of the point cloud bounding box and environmental information, or it may set the selection threshold based on environmental information alone. For example, when the illumination is low, the selection threshold for determining the confidence score or the selection threshold for determining the IoU is set lower. The same selection threshold may be set in the case of excessive illumination, bad weather, or at night. This makes it possible to detect and start tracking even in dark environments. It also makes it possible to suppress the influence of car lights, etc.

[0074] Thus, the threshold level can be varied depending on environmental factors such as illuminance, time of day, and weather. For example, the threshold level can be lowered from a short distance during low light conditions such as severe weather or at night. This allows tracking of hazardous objects to be prioritized over recognition accuracy.

[0075] This disclosure is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention. For example, in the embodiments described above, the threshold is determined based on distance, etc., but the comparison target (confidence score, etc.) for the threshold may be adjusted based on distance, etc.

[0076] Each configuration in the above-described embodiment is composed of hardware, software, or both, and may consist of one piece of hardware or software, or multiple pieces of hardware or software. Functions (processing) such as sensors and infrastructure coordinating servers may be realized by a computer 30 having a processor 31 such as a CPU (Central Processing Unit) and a memory 32 as a storage device, as shown in Figure 9. For example, a program for performing the method in the embodiment may be stored in the memory 32, and each function may be realized by executing the program stored in the memory 32 with the processor 31.

[0077] The above program, when loaded into a computer, includes a set of instructions (or software code) for causing the computer to perform one or more of the functions described in the embodiments. The program may be stored on a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable medium or a communication medium that includes electrically, optically, acoustically or otherwise propagating signals.

[0078] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be made as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0079] Each drawing is merely illustrative to illustrate one or more embodiments. Each drawing may be associated with one or more other embodiments rather than with only one specific embodiment. As those skilled in the art will understand, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings, for example, to create embodiments not explicitly shown or described. Not all features or steps shown in any one drawing to illustrate an exemplary embodiment are necessarily required, and some features or steps may be omitted. The order of steps shown in any of the drawings may be changed as appropriate.

[0080] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0081] (Note 1) An image recognition unit performs image recognition on image data captured from the target area to generate an image bounding box, A point cloud clustering unit performs clustering on the point cloud data measured in the target area to generate a point cloud bounding box, A correspondence unit that associates the image bounding box with the point cloud bounding box, An output unit outputs the image bounding box associated with the point cloud bounding box as an object recognition result, based on the distance of the point cloud bounding box. An object recognition system equipped with the following features. (Note 2) The output unit determines a selection threshold based on the distance of the point cloud bounding box, and selects the image bounding box to be output based on the determined selection threshold. The object recognition system described in Appendix 1. (Note 3) The selection threshold is either a confidence score, which is the image recognition result of the image bounding box, or a threshold for determining the IoU when the image bounding box overlaps with another image bounding box. The object recognition system described in Appendix 2. (Note 4) The output unit selects the image bounding box to be output based on the velocity of the point cloud bounding box. An object recognition system as described in any one of the items 1 to 3 of the appendix. (Note 5) The output unit selects the image bounding box to be output based on the illuminance of the target area. An object recognition system as described in any one of the items 1 to 3 of the appendix. (Note 6) The matching unit matches the image bounding box and the point cloud bounding box based on the distance of the point cloud bounding box. An object recognition system as described in any one of the items 1 to 3 of the appendix. (Note 7) The matching unit determines a matching threshold based on the distance of the point cloud bounding boxes, and matches the image bounding box with the point cloud bounding box based on the determined matching threshold. The object recognition system described in Appendix 6. (Note 8) The aforementioned correspondence threshold is a threshold for determining the matching rate between the image bounding box and the point cloud bounding box, or the IoU. The object recognition system described in Appendix 7. (Note 9) The matching unit matches the image bounding box and the point cloud bounding box based on the velocity of the point cloud bounding box. An object recognition system as described in any one of the items 1 to 3 of the appendix. (Note 10) The matching unit matches the image bounding box and the point cloud bounding box based on the illuminance of the target area. An object recognition system as described in any one of the items 1 to 3 of the appendix. (Note 11) An image recognition unit performs image recognition on image data captured from the target area to generate an image bounding box, A point cloud clustering unit performs clustering on the point cloud data measured in the target area to generate a point cloud bounding box, A correspondence unit that associates the image bounding box with the point cloud bounding box, An output unit outputs the image bounding box associated with the point cloud bounding box as an object recognition result, based on the distance of the point cloud bounding box. An object recognition device equipped with the following features. (Note 12) This involves performing image recognition on image data captured from the target area to generate an image bounding box, and The point cloud data measured in the aforementioned target region is subjected to clustering to generate a point cloud bounding box, The process involves associating the aforementioned image bounding box with the aforementioned point cloud bounding box, Based on the distance of the point cloud bounding box, the image bounding box associated with the point cloud bounding box is output as the object recognition result. An object recognition method that includes [a specific method]. (Note 13) This involves performing image recognition on image data captured from the target area to generate an image bounding box, and The point cloud data measured in the aforementioned target region is subjected to clustering to generate a point cloud bounding box, The process involves associating the aforementioned image bounding box with the aforementioned point cloud bounding box, Based on the distance of the point cloud bounding box, the image bounding box associated with the point cloud bounding box is output as the object recognition result. A program that causes a computer to perform a process that includes [a specific action].

[0082] Some or all of the elements (e.g., configuration and function) described in Appendices 2 to 10 that are subordinate to Appendice 1 (Object Recognition System) may also be subordinate to Appendices 11 (Object Recognition Device), 12 (Object Recognition Method), and 13 (Program) in the same way as those described in Appendices 2 to 10. Some or all of the elements described in any appendice may be applied to various hardware, software, recording means, systems, and methods for recording software. [Explanation of symbols]

[0083] 1. Infrastructure Cooperative System 10 Object Recognition Systems 11 Image Recognition Unit 12-point cloud clustering section 13 Correspondence section 14 Output section 20 Object recognition device 30 Computers 31 processors 32 memory 100 Infrastructure Cooperative Servers 100a Cloud Server 100b MEC 101 Point cloud acquisition part 102 Image acquisition unit 103 Point Cloud Clustering Section 104 Image Recognition Unit 105 Data Synchronization Unit 106 Bounding Box Synchronization Unit 107 Synchronization threshold setting section 108 Bounding Box Selection Section 109 Selection threshold setting section 110 Speed ​​calculation section 111 Environmental Information Acquisition Department 200 sensors 201 Camera 202 LiDAR 300 base stations

Claims

1. An image recognition unit performs image recognition on image data captured from the target area to generate an image bounding box, A point cloud clustering unit performs clustering on the point cloud data measured in the target area to generate a point cloud bounding box, A correspondence unit that associates the image bounding box with the point cloud bounding box, An output unit outputs the image bounding box associated with the point cloud bounding box as an object recognition result, based on the distance of the point cloud bounding box. An object recognition system equipped with the following features.

2. The output unit determines a selection threshold based on the distance of the point cloud bounding box, and selects the image bounding box to be output based on the determined selection threshold. The object recognition system according to claim 1.

3. The selection threshold is either a confidence score, which is the image recognition result of the image bounding box, or a threshold for determining the IoU when the image bounding box overlaps with another image bounding box. The object recognition system according to claim 2.

4. The output unit selects the image bounding box to be output based on the velocity of the point cloud bounding box. The object recognition system according to any one of claims 1 to 3.

5. The output unit selects the image bounding box to be output based on the illuminance of the target area. The object recognition system according to any one of claims 1 to 3.

6. The matching unit matches the image bounding box and the point cloud bounding box based on the distance of the point cloud bounding box. The object recognition system according to any one of claims 1 to 3.

7. The matching unit determines a matching threshold based on the distance of the point cloud bounding boxes, and matches the image bounding box with the point cloud bounding box based on the determined matching threshold. The object recognition system according to claim 6.

8. An image recognition unit performs image recognition on image data captured from the target area to generate an image bounding box, A point cloud clustering unit performs clustering on the point cloud data measured in the target area to generate a point cloud bounding box, A correspondence unit that associates the image bounding box with the point cloud bounding box, An output unit outputs the image bounding box associated with the point cloud bounding box as an object recognition result, based on the distance of the point cloud bounding box. An object recognition device equipped with the following features.

9. This involves performing image recognition on image data captured from the target area to generate an image bounding box, and The point cloud data measured in the aforementioned target region is subjected to clustering to generate a point cloud bounding box, The process involves associating the aforementioned image bounding box with the aforementioned point cloud bounding box, Based on the distance of the point cloud bounding box, the image bounding box associated with the point cloud bounding box is output as the object recognition result. An object recognition method that includes [a specific method].

10. This involves performing image recognition on image data captured from the target area to generate an image bounding box, and The point cloud data measured in the aforementioned target region is subjected to clustering to generate a point cloud bounding box, The process involves associating the aforementioned image bounding box with the aforementioned point cloud bounding box, Based on the distance of the point cloud bounding box, the image bounding box associated with the point cloud bounding box is output as the object recognition result. A program that causes a computer to perform a process that includes [a specific action].

Images