Object recognition device and method for setting a region of interest

Abstract

Provided is an object recognition device for vehicles comprising: an object recognition unit that processes at least one of point cloud data output by a LiDAR for observing the external environment of a vehicle and image data output by a camera and recognizes an object; a region-of-interest determination unit that determines a region of interest in which at least one of the LiDAR and the camera performs observation at a high resolution; and a region-of-interest setting unit that outputs information regarding the determined region of interest, wherein the region-of-interest determination unit determines priority on the basis of the reliability of a plurality of target candidates recognized by the object recognition unit and the characteristics of the plurality of target candidates in the point cloud data and the image data, then sets the region of interest so as to include a target candidate with high priority among the plurality of target candidates.

Description

Technical Field

[0001] The present invention relates to an object recognition device, and particularly to a method for determining a region of interest.

Background Art

[0002] In realizing an autonomous vehicle, it is necessary to grasp the positions and movements of objects such as surrounding vehicles and pedestrians, and objects are detected and recognized using image data acquired by a camera and point cloud data acquired by LiDAR.

[0003] Objects that appear small because they are far away or relatively small objects (e.g., falling objects) have less data volume of the detected and recognized objects, and are more difficult to recognize than objects in the vicinity. Therefore, there is a sensor having a function of setting a range called a region of interest where the resolution is higher than others for a specific region, and the amount of acquirable data increases.

[0004] In order to acquire data of a target that is far away or small at a high resolution, it is necessary to control the region of interest according to the movement of the target so that the target always exists within the region of interest even if the target is moving.

[0005] As background art in this field, there is Patent Document 1 (Japanese Patent Application Laid-Open No. 2021-196939). Patent Document 1 describes an in-vehicle sensor system for observing the situation around a vehicle, including a first sensor that observes a predetermined range around the vehicle at a first resolution, a high-precision observation target specifying means for specifying a high-precision observation target that is a target detected within the predetermined range observed by the first sensor and is to be observed at a second resolution higher than the first resolution, a target future existence region predicting means for predicting the range of a target future existence region that is a region where the high-precision observation target may exist after its specification, a second sensor that observes the range of the target future existence region at the second resolution, and a target information output means for outputting information on the high-precision observation target observed by the second sensor.

Prior Art Documents

[0006] [Patent Document 1] Japanese Patent Publication No. 2021-196939 [Overview of the project] [Problems that the invention aims to solve]

[0007] In the system described in Patent Document 1, a coarse-gauge sensor observes the area around the vehicle, and a fine-gauge sensor is used to identify an object to be observed with higher resolution. Furthermore, the system predicts the area in which the object will be located in the future. However, because the target to be observed with high precision or the vehicle itself may move, the target may deviate from the range observed by the fine-gauge sensor (region of interest). Therefore, the system predicts the area in which the target will be located after the target moves and moves the region of interest so that the target enters the region of interest for observation.

[0008] By shifting the region of interest in response to the movement of the target or the vehicle itself, the target can be observed with high resolution at all times. However, the ability to track a moving target suggests that the target is clearly recognized as an object, thus reducing the need for high-resolution observation.

[0009] Therefore, the present invention aims to more effectively define the region of interest by determining the region of interest for target candidates detected and recognized within the field of view in each frame, setting priorities as needed to obtain more detailed information, and determining the region of interest to include high-priority target candidates, thereby ensuring that targets to be observed in high resolution are included within the region of interest. [Means for solving the problem]

[0010] A typical example of the invention disclosed in this application is as follows: an object recognition device for a vehicle, comprising: an object recognition unit that processes at least one of point cloud data output by a LiDAR that observes the external environment of the vehicle and image data output by a camera to recognize an object; a region of interest determination unit that determines a region of interest observed at high resolution by at least one of the LiDAR and the camera; and a region of interest setting unit that outputs information of the determined region of interest, wherein the region of interest determination unit processes a plurality of target candidates recognized by the object recognition unit. Confidence level, which indicates the likelihood of object recognition. The method is characterized by determining a priority based on the characteristics of the multiple target candidates in the point cloud data and the image data, and determining the region of interest to include the target candidates with the highest priority among the multiple target candidates. [Effects of the Invention]

[0011] According to one aspect of the present invention, it is possible to improve the understanding of the surrounding environment by detecting and recognizing objects. Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments. [Brief explanation of the drawing]

[0012] [Figure 1] This is a block diagram showing the configuration of the electronic control device of Embodiment 1 of the present invention. [Figure 2] This is a flowchart of the process for determining the area of ​​interest. [Figure 3] This is a flowchart of the process for calculating the target priority. [Figure 4] This is a flowchart of the process for determining the area of ​​interest. [Figure 5] This figure shows an example of a table structure that defines the points used to calculate target priority. [Figure 6] This figure shows an example of setting a priority for an object's target. [Figure 7] This diagram shows an example of setting point priorities for calculating priority. [Figure 8] This diagram shows an example of setting priority levels for roads along a travel route. [Figure 9] This is a diagram showing an example before changing the region of interest for the target of an object. [Figure 10] This is a diagram showing an example after changing the region of interest for the target of an object. [Figure 11] This is a diagram showing an example after further changing the region of interest for the target of an object. [Figure 12] This is a diagram showing an example of setting the region of interest for a distant road. [Figure 13] This is a diagram showing an example of setting the region of interest for a forked road. [Figure 14] This is a diagram showing an example of setting the region of interest for the road at the left turn destination. [Figure 15] This is a diagram showing an example of coordinating a side monitoring sensor and a front monitoring sensor.

Mode for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0014] <Example 1> FIG. 1 is a block diagram showing the configuration of the object recognition device according to Example 1 of the present invention.

[0015] The object recognition device of Example 1 is implemented as a function of an electronic control unit mounted on a vehicle. The object recognition device detects and recognizes objects using image data output from sensors (e.g., LiDAR 101, cameras 104 and 106). The object recognition device of Example 1 detects and recognizes objects using point cloud data output from LiDAR 101 and image data output from low-resolution camera 104. The object recognition device also sets a high-resolution region of interest (ROI) within the field of view of the sensor and acquires more data in the region of interest than in other areas. The object recognition device then calculates a priority for target candidates according to the need to acquire data in high definition, sets a region of interest frame so that high-priority targets are within the region of interest, sets the set region of interest on LiDAR 101, or has a function to extract high-resolution data of the corresponding range from the image data output from high-resolution camera 106.

[0016] The electronic control unit on which the object recognition device is implemented is a control unit having an arithmetic unit, a memory device, and a communication interface. The arithmetic unit is a processor (e.g., a microcontroller) that executes programs stored in the memory device. The arithmetic unit operates as a functional unit that provides various functions by executing a predetermined program. The memory device includes a non-volatile memory area and a volatile memory area. The non-volatile memory area includes a program area that stores programs executed by the arithmetic unit and a data area that temporarily stores data used by the arithmetic unit when executing programs. The volatile memory area stores data used by the arithmetic unit when executing programs. The communication interface connects to other electronic control units and sensors (LiDAR 101, cameras 104, 106, etc.) via a network such as CAN or Ethernet.

[0017] The LiDAR 101 sends the acquired point cloud data to the point cloud data processing unit 102. The point cloud data processing unit 102 performs processing to convert the angle and distance data of the point cloud output from the LiDAR 101 into a Cartesian coordinate system, and processing to identify the point cloud of road surface parts and three-dimensional objects, and sends the processing results to the three-dimensional object detection and recognition processing unit 103. The three-dimensional object detection and recognition processing unit 103 performs processing to detect and recognize targets of road surface and objects, and stores the detection and recognition results in the target data storage unit 110.

[0018] Furthermore, the low-resolution camera 104 sends the acquired image data to the image data processing unit 105. The high-resolution camera 106 also sends the acquired high-resolution image data to the region of interest data extraction unit 107. The region of interest data extraction unit 107 extracts data within the region of interest received from the region of interest setting unit 115 from the high-resolution image data and sends it to the image data processing unit 105.

[0019] The image data processing unit 105 integrates image data acquired from the low-resolution camera 104 with high-resolution image data within the region of interest setting unit 115, and sends the integrated data to the segment division processing unit 108 and the object recognition processing unit 109. The segment division processing unit 108 divides the image data into semantic regions such as vehicles and road surfaces, and sends the processing results to the position matching unit 111. The object recognition processing unit 109 recognizes objects from the image data and sends the recognition results to the position matching unit 111.

[0020] The position matching unit 111 compares the target detection and recognition results extracted by the target data storage unit 110 from the point cloud data obtained by the LiDAR 101 with the results of the segment division processing unit 108 dividing the image data from the low-resolution camera 104 into semantic regions (segment regions) and the results of the object recognition processing unit 109 recognizing objects from the image data obtained by the low-resolution camera 104. For targets where the point cloud data and image data of the same object can be associated, the position and size of the object and the road surface are integrated and corrected, and the data in the target data storage unit 110 is updated. The position matching unit 111 also stores the data of targets that could not be associated with point cloud data and were recognized only from image data in the target data storage unit 110.

[0021] The target priority calculation unit 112 takes in target candidate information stored in the target data storage unit 110, calculates priority based on the reliability, location, and appearance of the target candidates, and determines the direction of travel at intersections and junctions based on the road data of the map data provided by the map / driving route information provision unit 113 and the positional relationship of each target, calculates the priority of target candidates such as roads and road surface areas, and sends the calculated priority to the area of ​​interest determination unit 114.

[0022] The region of interest determination unit 114 sets the region of interest so that more high-priority candidates are included within the region of interest, based on the priority of each target calculated by the target priority calculation unit 112, and sends the information of the region of interest to the region of interest setting unit 115.

[0023] The region of interest setting unit 115 sets the information of the set region of interest in the LiDAR 101 and the region of interest data extraction unit 107.

[0024] In the above explanation, we showed an example of processing three types of data from LiDAR 101, low-resolution camera 104, and high-resolution camera 106. However, it is also possible to process data with different resolutions from LiDAR 101, or image data with different resolutions from the low-resolution camera 104 and high-resolution camera 106.

[0025] Next, we will explain how to determine the region of interest by referring to the flowchart in Figure 2.

[0026] Step 201: The point cloud data processing unit 102 aggregates one frame of point cloud data acquired by the LiDAR 101 and performs preprocessing such as conversion to a Cartesian coordinate system.

[0027] Step 202: The 3D object detection and recognition processing unit 103 uses information such as the low reflectance intensity and height of the laser light at each point to distinguish between the point cloud of the road surface and the point cloud of the 3D object, and recognizes the object using recognition technology such as AI for the point cloud of the 3D object. The 3D object detection and recognition processing unit 103 also understands the road structure, such as the width and curvature of the lanes, based on the point cloud data of the road surface. The 3D object detection and recognition processing unit 103 then stores each result in the target data storage unit 110.

[0028] Step 203: The image data processing unit 105 performs preprocessing such as integrating the image data acquired by the low-resolution camera 104 with the high-precision image data sent from the region of interest data extraction unit 107.

[0029] Step 204: The segment division processing unit 108 divides the image data into segment regions, which are semantic areas such as vehicles and road surfaces. The object recognition processing unit 109 then recognizes objects from the image data using recognition technology such as AI.

[0030] Step 205: The position matching unit 111 compares the recognition results of the divided segment regions with the recognition results obtained from the target data storage unit 110 using point cloud data. For targets where the corresponding point cloud data and image data of the same object can be associated, the position and size of the object and the road surface are integrated and corrected, and the data in the target data storage unit 110 is updated. For targets that cannot be associated with point cloud data and are recognized only from image data, the data is stored in the target data storage unit 110.

[0031] It is recommended that steps 201-202 and steps 203-205 be executed in parallel.

[0032] Step 206: The target priority calculation unit 112 calculates the priority of the object detected and recognized by the 3D object detection and recognition processing unit 103. Details of the process in step 206 will be described later with reference to Figure 3.

[0033] Step 207: The region of interest determination unit 114 determines the region of interest based on the target information for which the target priority calculation unit 112 has calculated the priority. Details of the process in step 207 will be described later with reference to Figure 4.

[0034] Step 208: The region of interest setting unit 115 sets the region of interest updated by the region of interest determination unit 114 to the LiDAR 101.

[0035] Step 209: The Region of Interest Setting Unit 115 sends the Region of Interest updated by the Region of Interest Determination Unit 114 to the Region of Interest Data Extraction Unit 107. The Region of Interest Data Extraction Unit 107 extracts data within the Region of Interest received from the Region of Interest Setting Unit 115 from the high-resolution image data and sends it to the Image Data Processing Unit 105.

[0036] Next, referring to the flowchart in Figure 3, we will explain how the target priority calculation unit 112 determines the priority of each target.

[0037] Step 301: The target priority calculation unit 112 repeats the subsequent processing for the number of target candidates and calculates the priority of the target candidates.

[0038] Step 302: The target priority calculation unit 112 refers to the table in Figure 5 and assigns points to target candidates according to the distance to the target candidate.

[0039] Step 303: The target priority calculation unit 112 refers to the table in Figure 5 and assigns points to target candidates according to the percentage of target candidates that are visible.

[0040] Step 304: The target priority calculation unit 112 refers to the table in Figure 5 and assigns points to target candidates according to the confidence level at which they were recognized.

[0041] Step 305: The target priority calculation unit 112 refers to the table in Figure 5 and assigns points to target candidates according to their target candidates and movement speed.

[0042] Step 306: The target priority calculation unit 112 refers to the table in Figure 5 and assigns points to target candidates according to the positional relationship between the target candidate and the lane.

[0043] Step 307: The target priority calculation unit 112 determines whether the target candidate is a pedestrian. If it is a pedestrian, the unit proceeds to step 307; otherwise, it proceeds to step 308.

[0044] Step 308: The target priority calculation unit 112 refers to the table in Figure 5 and assigns points to target candidates according to the positional relationship between the target pedestrian and the road.

[0045] Step 309: The target priority calculation unit 112 totals the points awarded in each step and determines the priority of each target candidate.

[0046] Next, referring to the flowchart in Figure 4, we will explain how the region of interest determination unit 114 determines the region of interest based on the priority of each target.

[0047] Step 401: If the location of each target candidate is not on or near the vehicle's driving route, the region of interest determination unit 114 excludes that target candidate from the region of interest setting.

[0048] Step 402: The region of interest determination unit 114 determines whether the highest priority of the target candidates is below a predetermined threshold. If it is below the threshold, the unit proceeds to step 408; otherwise, the unit proceeds to step 403.

[0049] Step 403: The region of interest determination unit 114 repeatedly processes a predetermined number of target candidates in order of priority, and determines whether the target candidate falls within the region of interest.

[0050] Step 404: The region of interest determination unit 114 determines whether the target candidate is within the region of interest. If it is, the unit returns to step 404. If the target candidate is not within the region of interest, the unit proceeds to step 406.

[0051] Step 405: The region of interest determination unit 114 determines whether the target within the region of interest will move outside the region of interest when the region of interest is moved so that the target candidate enters the region of interest. If the target will move outside the region of interest, the unit returns to step 404 without changing the region of interest. If the target does not move outside the region of interest, the unit proceeds to step 406.

[0052] Step 406: The region of interest determination unit 114 modifies the region of interest so that the target added in step 406 falls within the region of interest frame.

[0053] Step 407: The region of interest determination unit 114 has completed confirming whether or not the region of interest range needs to be modified for a given target candidate, so it prepares to set the updated region of interest from step 208 to the LiDAR 101, terminates the loop, and returns to step 404.

[0054] Step 408: The region of interest determination unit 114 determines a region of interest and detects distant objects on the driving route with high resolution, since there are no target candidates for which to determine a region of interest and acquire detailed data.

[0055] The region of interest determination unit 114 usually determines one region of interest, but it may determine multiple regions of interest.

[0056] In the process shown in Figure 3, for example, as shown in Figure 5, priorities are set so that objects that require high-resolution detection have higher points, depending on the location and situation of the target. The priority is then determined by the total points, and the object is detected at high resolution using the region of interest. In other words, objects that are difficult to recognize are given higher priority, the region of interest is determined, and the object is accurately recognized.

[0057] Distance points are set so that the further the distance to the target candidate, the higher the points. Percentage points are set according to the percentage of the target candidate that is visible, so that the more of the target candidate is hidden by other objects, the higher the points. If the percentage cannot be calculated and is unknown, a high point is also set. Confidence points are set according to the probability (e.g., 1 to 0) output when the object recognition model recognizes an object, so that the lower the confidence, the higher the points. High-resolution data is used to improve recognition accuracy. Also, if object recognition is not possible and confidence cannot be calculated, the points are set to the highest. Position points are set according to the degree to which the road and lane being traveled have an influence on the vehicle, so that the greater the influence on the vehicle, the higher the points. Also, if the driving lane cannot be identified, the points are set to the highest because there is a high need to identify the driving lane by increasing the amount of data with high resolution. Speed ​​points are set according to the speed difference with the target candidate, and are set higher when the distance between the vehicle and the target is approaching and lower when the distance is moving away. Pedestrian points are set when the target candidate is a pedestrian. They are set to be higher depending on the need to pay attention to pedestrians, and are set to be higher even if the target is off the road, as it is necessary to determine if the target is about to cross if they are near a crosswalk.

[0058] Note that the point setting method and values ​​shown in Figure 5 are just examples. You may set more levels than the four levels from 0 to 3, add other criteria for awarding points, or delete some items.

[0059] Given the target candidates shown in Figure 6, assigning points using the table shown in Figure 5 results in the points for each target, the total points, and priority being shown in Figure 7. Figure 6 shows the bounding boxes corresponding to the position and size of each target candidate.

[0060] Regarding object 601, which is located on the road surface at a distance, it receives 3 distance points because it is more than 100m away, 3 proportion points because it is unclear whether the entire object is visible from a distance, 3 reliability points because it is not recognized as an object and its reliability is unknown, 3 position points because it is located at a distance and it is unclear which lane it is on, 2 speed points because it is a stationary object, and 0 pedestrian points because object 601 is not a pedestrian. These points are added together to make a total of 14 points.

[0061] Regarding the oncoming vehicle 602 in the distance, the distance is 70m to 100m, so it has 2 distance points; the entire vehicle is visible, so it has 0 proportion points; the reliability points are 2; it is traveling in the oncoming lane, so it has 1 position point; it is approaching the vehicle, so it has 3 speed points; and the pedestrian points are 0. These points are added together, resulting in a total of 8 points.

[0062] Regarding vehicle 603, which is traveling in the same direction, the distance is 70m to 100m, so it has 2 distance points; the entire vehicle is visible, so it has 0 proportion points; the reliability points are 2; it is traveling in an adjacent lane, so it has 1 position point; it is traveling at a speed similar to the current vehicle, so it has 1 speed point; and vehicle 603 is not a pedestrian, so it has 0 pedestrian points. These points are added together, resulting in a total of 6 points.

[0063] Vehicle 604, which is partially obscured by vehicle 607, has a distance of 40m to 70m, so it receives 1 distance point; 2 proportion points because it is partially obscured by the preceding vehicle 607 and only part of the vehicle is visible; 3 reliability points because it is partially visible and therefore unreliable; 2 position points because it is traveling in the same lane; 1 speed point because it is traveling at a similar speed to the vehicle in question; and 0 pedestrian points because vehicle 604 is not a pedestrian. These points are added together to arrive at a total of 9 points.

[0064] Pedestrian 605 is located between 40m and 70m away, so it receives 1 distance point; it is visible in its entirety, so it receives 0 percentage points; it receives 2 reliability points; it is not on the road being traveled, so it receives 0 position points; it is stationary, so it receives 2 speed points; and it is near the crossing, so it receives 2 pedestrian points. These points are added together to give a total of 7 points.

[0065] The nearby two-wheeled vehicle 606 (bicycle or motorcycle) has a distance of 0 because it is less than 40m away, a proportion point of 0 because it is fully visible, a reliability point of 1 because it is traveling in an adjacent lane, a speed point of 3 because it is approaching at a slower speed than the vehicle, and a pedestrian point of 0 because the two-wheeled vehicle 606 is not a pedestrian. These points are added together to make a total of 5 points.

[0066] The preceding vehicle 607 has a distance of 0 points because it is less than 40m away, a proportion point of 0 because it is fully visible, a reliability point of 0, a position point of 2 because it is traveling in the same lane, a speed point of 1 because it is traveling at a similar speed to the vehicle in question, and a pedestrian point of 0 because the preceding vehicle 607 is not a pedestrian. These points are added together to make a total of 3 points.

[0067] Vehicle 608, which is outside the field of view and partially cut off, has a distance of 0 points because it is less than 40m away, a proportion of 1 point because part of the target is outside the field of view, a confidence of 1 point because it is traveling in an adjacent lane, a position of 1 point because it is moving away from the vehicle, a speed of 0 points because it is moving backward from the vehicle, and a pedestrian of 0 points because vehicle 608 is not a pedestrian. These points are added together to make a total of 3 points.

[0068] Figure 7 shows the priority set from highest to lowest point total. However, since the preceding vehicle 607 is highly reliable and clearly recognizable, and vehicle 608 is partially outside the field of view and is moving away from the field of view and will eventually go out of view as well, there is no need to determine a region of interest and acquire higher resolution data. Therefore, regardless of the number of points, it is excluded from setting a region of interest and no priority is set.

[0069] Note that the priority order shown in Figure 7 is just one example. You may choose to assign priority to the total points without excluding preceding vehicles 607 or 608, or you may exclude targets with points below a certain level, or targets below a certain rank.

[0070] Regarding road priorities, as shown in Figure 8, for example, if the travel route is straight, it is good to give higher priority to the area near the road surface at the far end of the road 801; if the travel route is a road branching off to the left, it is good to give higher priority to the branching road 802; and if turning left at an intersection, it is good to give higher priority to the destination 803. In other words, according to the travel route, it is good to give higher priority to determining the area of ​​interest in the direction of the travel destination.

[0071] If a travel route is set, the region of interest should be determined by the direction of travel, such as the branching point 802 or the left turn 803. If traveling straight or if no route is set, the region of interest should be determined according to the priority of target candidates such as objects. Alternatively, other criteria may be used to control whether to prioritize targets such as objects or roads, or targets and roads may be integrated to control priority.

[0072] Next, we will explain how to set specific areas of interest based on the movement of vehicles and other objects.

[0073] As shown in Figure 9, there are two vehicles with higher priority, vehicle 901 and motorcycle 902, and the region of interest 911 is set to include both. At this time, oncoming vehicle 803 has not yet been recognized, so no priority has been set for it.

[0074] At the next timing when determining the region of interest, the vehicle itself and the target candidates move to the state shown in Figure 10, and when the priority is updated, the region of interest 1011 is provisionally set so that the oncoming vehicle 903, which has the highest priority, is located near the center.

[0075] To ensure that the next highest priority vehicle, 901, also falls within the region of interest, we tentatively set the region of interest 1012 so that both vehicles are located near the center.

[0076] If the motorcycle 902, which has the next highest priority, is moved to the position of region of interest 1013 in order to include it within the region of interest frame, the oncoming vehicle 903, which has an even higher priority, will move out of the region of interest. Therefore, the region of interest is not tentatively set to position 1013, and the position of region of interest 1012 is maintained.

[0077] In the example shown in Figure 10, since there are no other targets with higher priority, the region of interest 1012 is determined, and the set region of interest 1012 is transferred to the LiDAR 101 and the region of interest data extraction unit 107.

[0078] Furthermore, at the next time the region of interest frame is set, when the object reaches the state shown in Figure 11 and the priority is updated, the region of interest 1111 is provisionally set so that the falling object 904, which has the highest priority, is located near the center.

[0079] To ensure that the next highest priority vehicle, oncoming vehicle 903, also falls within the region of interest, we tentatively set the region of interest 1112 so that both vehicles are located near the center.

[0080] If the region of interest is moved to position 1113 to include the motorcycle 902, which has the next highest priority, then the fallen object 904 and the oncoming vehicle 903, which have higher priority, will be moved out of the region of interest. Therefore, the region of interest is not tentatively set to position 1113, and the position of the region of interest 1112 is maintained.

[0081] Furthermore, since the preceding vehicle 901 can be reliably recognized as it approaches, the need for high-resolution detection is reduced, resulting in a lower priority and no impact on the setting of the region of interest.

[0082] In the example shown in Figure 11, since there are no other targets with higher priority, the region of interest 1112 is determined, and the set region of interest 1112 is transferred to the LiDAR 101 and the region of interest data extraction unit 107.

[0083] In setting the priority for the road portion, as shown in Figure 12, low-priority objects 1202 are excluded from the region of interest. If there are no high-priority objects within the field of view, the region of interest 1201 is set to the distant road surface area 801, and distant fallen objects, vehicles, etc. are detected with high resolution.

[0084] As shown in Figure 13, if there are no high-priority objects in the field of view and the set driving path leads to the left fork, the region of interest 1301 is set to the area near the road surface 802 far away from the road branching off to the left, rather than far away from the straight road, and fallen objects, vehicles, etc. on the road being driven are detected with high resolution.

[0085] As shown in Figure 14, if the driving route is set to turn left at an intersection, the region of interest 1401 is set near the road surface 803 of the road to be turned left, and objects such as fallen objects and vehicles on the road being driven are detected with high resolution.

[0086] In this case, even if there is a high-priority target (for example, oncoming vehicle 1402) on a route that the vehicle will not travel, it will not affect the vehicle's movement, so it will be excluded from the area of ​​interest.

[0087] If there are no high-priority targets, the region of interest will be determined by default to be the area near the road surface at a distance (801). However, if the vehicle speed is slow, there is little need to detect distant objects, so the region of interest function may be temporarily suspended, and only normal detection will be performed to reduce the processing load and power consumption.

[0088] In this case, the priority of target candidates is determined based on the results of detection and recognition at normal resolution. However, if a high-priority target appears, or if the vehicle's speed exceeds a predetermined speed, it is advisable to restart the setting of the region of interest.

[0089] Furthermore, if there is a function to expand and contract the area of ​​interest, you can expand the area of ​​interest to include more targets, or conversely, if the targets are concentrated near the center, you can narrow the area of ​​interest to include the targets.

[0090] As described above, according to Embodiment 1 of the present invention, a high priority is set for targets that require high-resolution data acquisition (such as unclear objects), and the region of interest is set so that high-priority targets are always included. This increases the amount of data from targets that require high-resolution data acquisition, allowing for clearer observation. As a result, objects can be accurately detected and recognized, and the understanding of the surrounding situation can be improved.

[0091] <Example 2> In Example 1, an example was shown in which the LiDAR 101 and low-resolution camera 104 were installed in front. However, as shown in Figure 15, if sensors 1501 are also installed on the sides, it is preferable to hand over the information to the side sensors 1501 when the target moves out of the field of view. For example, when determining a region of interest near the road surface 1502 in the distance on a curved road, the forward-monitoring sensor 1503 cannot detect the target within region 1502 even if it determines the region of interest at the right edge of its field of view because the road surface 1502 is outside its field of view 1504. At this time, region 1502 is within the field of view 1505 of the side-monitoring sensor 1501, so the forward sensor 1503 transmits information about region 1502 to the side sensor 1501, and the side sensor 1501 uses the information acquired from the forward sensor 1503 to determine the region of interest.

[0092] Furthermore, the types of sensors 1501 and 1503 do not matter; combinations of different types of sensors, such as a LiDAR and a camera, or combinations of cameras are also acceptable.

[0093] <Example 3> In Example 1, the processing when LiDAR 101 and low-resolution camera 104 are combined was described. However, in order to reduce devices and costs, the low-resolution camera 104, high-resolution camera 106, and related processing may be omitted, and the priority of target candidates may be calculated from the recognition results of the point cloud data acquired by LiDAR 101, and the region of interest may be determined. Alternatively, LiDAR 101 and related processing may be omitted, and the priority of target candidates may be calculated from the recognition results of the image data acquired by the low-resolution camera 104 and high-resolution camera 106, and the region of interest may be determined.

[0094] Furthermore, while the example shown uses a LiDAR 101 and a low-resolution camera 104, other sensors such as radar or sonar may be used instead of the LiDAR 101 and low-resolution camera 104, or similar sensors with different specifications and performance, such as resolution, may be used.

Claims

1. A vehicle object recognition device, An object recognition unit processes at least one of the point cloud data output by a LiDAR that observes the external environment of a vehicle and the image data output by a camera to recognize an object, A region of interest determination unit that determines a region of interest to be observed at high resolution by at least one of the LiDAR and the camera, The system includes a region of interest setting unit that outputs information about the determined region of interest, The aforementioned region of interest determination unit, Prioritization is determined based on the confidence level, which is the likelihood of object recognition for the multiple target candidates recognized by the object recognition unit, and the characteristics of the multiple target candidates in the point cloud data and the image data. An object recognition device characterized by determining the region of interest to include target candidates with high priority among the aforementioned plurality of target candidates.

2. The object recognition device according to claim 1, The object recognition device is characterized in that the region of interest setting unit outputs information of the determined region of interest to the LiDAR.

3. The object recognition device according to claim 1, The object recognition unit, A first object recognition unit that processes the point cloud data output by the LiDAR, A second object recognition unit that processes image data output by the aforementioned camera, An object recognition device characterized by comprising a comparison unit that compares the recognition result from the first object recognition unit with the recognition result from the second object recognition unit.

4. The object recognition device according to claim 1, The aforementioned camera includes a high-resolution camera and a low-resolution camera. The object recognition unit, A region of interest data extraction unit extracts the portion of the region of interest from the image observed by the high-resolution camera, The system includes an image data processing unit that integrates the image of the portion of the region of interest that has been cut out with the image observed by the low-resolution camera, The object recognition device is characterized in that the region of interest setting unit outputs information of the determined region of interest to the region of interest data extraction unit.

5. The object recognition device according to claim 1, The object recognition device is characterized in that the region of interest determination unit has a priority calculation unit that calculates the priority using at least one of the following as characteristics of the plurality of target candidates: the distance to the target candidate, the percentage of the target candidate that is visible, the confidence level when the target candidate is recognized, the positional relationship between the target candidate and the road, and the speed difference between the target candidate and the vehicle.

6. The object recognition device according to claim 5, The object recognition device is characterized in that the priority calculation unit calculates the priority such that the priority increases as the distance to the target candidate increases.

7. The object recognition device according to claim 5, The object recognition device is characterized in that the priority calculation unit calculates the priority such that the priority increases when the proportion of the target candidate that is visible is small.

8. The object recognition device according to claim 5, The object recognition device is characterized in that the priority calculation unit calculates the priority such that the lower the confidence level of recognizing the target candidate, the higher the priority.

9. The object recognition device according to claim 5, The priority calculation unit is characterized in that it calculates the priority such that the priority is higher if the target candidate is on the vehicle's driving lane.

10. The object recognition device according to claim 5, The object recognition device is characterized in that the priority calculation unit calculates the priority such that the priority is higher if the speed difference of the target candidate is close to that of the vehicle itself.

11. The object recognition device according to claim 1, The object recognition device is characterized in that, if there are no target candidates to be included in the region of interest, the region of interest determination unit determines the region of interest on the road surface of the travel path.

12. A method for defining a region of interest performed by an object recognition device for vehicles, The object recognition device comprises a arithmetic unit that performs predetermined calculation processing and a storage device connected to the arithmetic unit. The above method for setting the region of interest is: The aforementioned computing device processes at least one of the point cloud data output by the LiDAR that observes the external environment of the vehicle and the image data output by the camera to perform an object recognition procedure for recognizing an object. The computing device performs a region of interest determination procedure to determine a region of interest observed at high resolution by at least one of the LiDAR and the camera, The computing device includes a procedure for setting a region of interest that outputs information about the determined region of interest, In the above procedure for determining the region of interest, The computing device determines a priority based on the confidence level, which is the likelihood of object recognition of the multiple target candidates recognized in the object recognition procedure, and the characteristics of the multiple target candidates in the point cloud data and the image data. A method for setting a region of interest, characterized in that the computing device determines the region of interest so that it includes target candidates with a high priority among the plurality of target candidates.

Images