Target Recognition System

Abstract

To provide a target recognition system capable of determining whether or not an infrastructure sensor has an abnormality with a simple configuration. [Solution] The target recognition system includes an infrastructure sensor, a coordinate conversion unit that performs coordinate conversion on information obtained by the infrastructure sensor, a target recognition unit that recognizes targets in images captured by the infrastructure sensor, and a communication unit that transmits information on the targets recognized by the target recognition unit to a user terminal. The target recognition system also includes an azimuth angle calculation unit that calculates the driving azimuth angle of a vehicle traveling on a road among the targets recognized by the target recognition unit, and a map acquisition unit that acquires map information including a road azimuth angle that is the azimuth angle of a road included in the imaging range of the infrastructure sensor. The target recognition system also includes an angle calculation unit that calculates an azimuth angle difference that is the difference between the driving azimuth angle and the road azimuth angle, and an abnormality determination unit that determines whether the installation status of the infrastructure sensor is normal or abnormal based on the azimuth angle difference calculated by the angle calculation unit.

Description

[Technical Field]

[0001] The disclosure herein relates to a target recognition system. [Background technology]

[0002] Patent Document 1 discloses a camera system that automatically corrects the angle of view of a surveillance camera when the angle of view shifts. The contents of the prior art document are incorporated by reference as an explanation of the technical elements in this specification. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-219628 Summary of the Invention [Problem to be solved by the invention]

[0004] In the configuration of the prior art document, the amount of deviation of the reference point is detected based on a reference image captured of the detection area when the camera is installed and the image captured by the camera, and the view angle deviation is corrected. Here, the reference point is exemplified as a fixed reference point that is not affected by weather, season, etc., such as a pillar or an installed object. However, if the position of the fixed object set as the reference point changes, the amount of deviation cannot be properly detected. Furthermore, if a fixed object that can be used as a reference point is separately prepared and both the camera and the fixed object are maintained, the maintenance costs increase compared to maintaining only the camera. In the above-mentioned perspectives and other perspectives not mentioned, further improvements are required in target recognition systems.

[0005] One disclosed object is to provide a target recognition system that can determine whether or not an abnormality exists in an infrastructure sensor with a simple configuration. [Means for solving the problem]

[0006] The target recognition system disclosed herein includes an infrastructure sensor (31, 231) installed at a position where it can capture an image of a road; a coordinate conversion unit (51) that converts coordinates of information obtained by the infrastructure sensor; a target recognition unit (52) that recognizes a target in an image captured by an infrastructure sensor; a communication unit (21) that transmits information about the target recognized by the target recognition unit to a user terminal (91); an azimuth angle calculation unit (53) that calculates a traveling azimuth angle of a traveling vehicle (81, 82) traveling on a road among the targets recognized by the target recognition unit; a map acquisition unit (23) that acquires map information including a road azimuth angle, which is an azimuth angle of a road included in an imaging range of the infrastructure sensor; an angle calculation unit (54) that calculates an azimuth angle difference that is the difference between a traveling azimuth angle and a road azimuth angle; The system includes an abnormality determination unit (55) that determines whether the installation state of the infrastructure sensor is normal or abnormal based on the azimuth angle difference calculated by the angle calculation unit.

[0007] The disclosed target recognition system includes an abnormality determination unit that determines whether the installation state of the infrastructure sensor is normal or abnormal based on the azimuth angle difference calculated by the angle calculation unit. Here, the azimuth angle difference is the difference between the traveling azimuth angle obtained from the recognition of a traveling vehicle and the road azimuth angle included in the map information. Therefore, it is not necessary to recognize targets other than traveling vehicles when determining whether an infrastructure sensor is abnormal. Therefore, it is possible to provide a target recognition system that can determine whether an infrastructure sensor is abnormal with a simple configuration.

[0008] The various aspects disclosed in this specification employ different technical means to achieve their respective objectives. The reference numerals in parentheses in the claims and in this section are intended to exemplify correspondences with the following embodiments and are not intended to limit the technical scope. The objectives, features, and advantages disclosed in this specification will become more apparent by reference to the following detailed description and the accompanying drawings. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 2 is a block diagram relating to target recognition control. [Figure 2] 4 is a flowchart relating to target recognition control. [Figure 3] 3 is a flowchart relating to the process of step S110 in FIG. 2. [Figure 4] FIG. 2 is an explanatory diagram for explaining a road azimuth angle. [Figure 5] FIG. 2 is an explanatory diagram for explaining a traveling azimuth angle. [Figure 6] FIG. 10 is a block diagram relating to target recognition control according to a second embodiment. [Figure 7] 10 is a flowchart relating to target recognition control according to a second embodiment. [Figure 8] 8 is a flowchart relating to the process of step S210 in FIG. 7. [Figure 9] FIG. 10 is a diagram illustrating a calculation formula used to calculate a correction amount. DETAILED DESCRIPTION OF THE INVENTION

[0010] Several embodiments will be described with reference to the drawings. In several embodiments, functionally and / or structurally corresponding and / or associated parts may be designated by the same reference numeral or by different reference numerals in the hundreds or more places. For corresponding and / or associated parts, reference may be made to the descriptions of other embodiments.

[0011] (First embodiment) 1, the target recognition system 10 is a system that recognizes targets based on information acquired by infrastructure sensors 31 and transmits the recognition results to an external terminal 90. For example, the target recognition system 10 can be used as an accident prevention system at intersections. In this case, driving can be assisted by, for example, notifying the driver of the presence of targets located in the driver's blind spot at an intersection.

[0012] The infrastructure sensor 31 is a sensor installed on the roadside at an intersection. The infrastructure sensor 31 is installed in a position where it can capture images of multiple roads that intersect with each other. The infrastructure sensor 31 can be an imaging device such as a camera device, LiDAR, or millimeter-wave radar.

[0013] The target object recognition system 10 includes a coordinate conversion unit 51, a target object recognition unit 52, and an azimuth angle calculation unit 53. The coordinate conversion unit 51 converts the coordinates of the information acquired by the infrastructure sensor 31 from local coordinates to global coordinates. The local coordinates are a coordinate system whose origin is the installation position of the infrastructure sensor 31. The global coordinates are a coordinate system whose origin is, for example, the center position of an intersection. The coordinate conversion is calculated using initial values ​​of the yaw angle, pitch angle, and roll angle corresponding to the installation angle of the infrastructure sensor 31.

[0014] The target recognition unit 52 recognizes targets included in the captured image acquired by the infrastructure sensor 31. As targets, it is possible to recognize moving objects such as traveling vehicles and pedestrians, and fixed objects such as white lines and guardrails. When the captured image includes multiple targets, the target recognition unit 52 recognizes each of the multiple targets.

[0015] The azimuth angle calculation unit 53 calculates the traveling azimuth angle of the traveling vehicle recognized by the target recognition unit 52. In other words, by acquiring position information for the same traveling vehicle at different times, the trajectory followed by the traveling vehicle is calculated, and the traveling azimuth angle is calculated. The traveling vehicle is not limited to a four-wheeled vehicle, and any mobile object traveling along a road, such as a motorcycle or a moped, can be used. When there are multiple traveling vehicles, the azimuth angle calculation unit 53 calculates the traveling azimuth angle for each of the multiple traveling vehicles.

[0016] The target recognition system 10 includes a communication unit 21 and a map acquisition unit 23. The communication unit 21 communicates with an external terminal 90. Here, the external terminal 90 includes a user terminal 91 and a maintenance terminal 92. The user terminal 91 is a terminal including an application that uses the target recognition results of the target recognition system 10. The user terminal 91 is, for example, a terminal mounted on a vehicle, and can be a terminal including an application that issues an alert to prevent an accident or activates an automatic brake. The user terminal 91 is, for example, a mobile terminal carried by a pedestrian, and can be a terminal including an application that notifies the pedestrian of an approaching vehicle. The communication unit 21 transmits the target recognition results to the user terminal 91.

[0017] The maintenance terminal 92 is a terminal for performing maintenance and inspection to check whether the target recognition system 10 is functioning normally. The communication unit 21 transmits to the maintenance terminal 92 the determination result of the abnormality determination unit 55, which will be described later.

[0018] The map acquisition unit 23 acquires map information about the vicinity of the installation position of the infrastructure sensor 31 from the map database 22 through communication via the communication unit 21. The map information includes information about the extension direction of roads included in the imaging range of the infrastructure sensor 31. More specifically, the map information includes information about road azimuth angles indicating the azimuth angles of straight portions of the roads imaged by the infrastructure sensor 31. For example, assume that the east azimuth angle is 0°, the north azimuth angle is 90°, the west azimuth angle is 180°, and the south azimuth angle is 270°. In this case, if the road extends straight in the east-west direction, a road azimuth angle of 0° or 180° is obtained. If the road extends in a direction rotated 1° counterclockwise from the east-west direction, a road azimuth angle of 1° or 181° is obtained.

[0019] The target object recognition system 10 includes an angle calculation unit 54 and an abnormality determination unit 55. The angle calculation unit 54 calculates an azimuth angle difference based on the traveling azimuth angle calculated by the azimuth angle calculation unit 53 and the road azimuth angle of the map information acquired by the map acquisition unit 23. The azimuth angle difference is the difference in angle between the traveling azimuth angle and the road azimuth angle. When the azimuth angle difference is 0°, the traveling azimuth angle and the road azimuth angle match, and it can be said that the traveling trajectory and the road are parallel. On the other hand, when the azimuth angle difference is not 0°, it can be said that the traveling trajectory and the road are not parallel. However, the extension direction of the road and the trajectory of a vehicle traveling on that road should be parallel. Therefore, a state in which the azimuth angle difference is not 0° can be said to be a state in which it is highly likely that the traveling azimuth angle has been calculated incorrectly. In other words, it can be said to be a state in which it is highly likely that the installation angle of the infrastructure sensor 31 has changed, and the yaw angle, roll angle, and pitch angle used in coordinate transformation are no longer appropriate values.

[0020] The abnormality determination unit 55 determines whether the installation state of the infrastructure sensor 31 is normal or abnormal. A normal state is a state in which the entire target imaging range is included in the imaging range of the infrastructure sensor 31. In other words, an abnormal state is a state in which even a part of the target imaging range is not included in the imaging range of the infrastructure sensor 31. Here, the target imaging range refers to the range that should be imaged by the infrastructure sensor 31. When the infrastructure sensor 31 is installed, the performance, installation position, and installation angle of the infrastructure sensor 31 are adjusted so that the target imaging range is included in its entirety. The target imaging range is set to be narrower than the range that can be imaged by the infrastructure sensor 31. The target imaging range includes multiple roads that intersect with each other.

[0021] The following describes in detail the target recognition control using the target recognition system 10. The target recognition control is a control for recognizing a target from an image captured by the infrastructure sensor 31.

[0022] In FIG. 2, when target recognition control is started, the acquisition unit acquires captured images as sensor information from the infrastructure sensor 31 in step S101. The acquisition unit acquires multiple images captured at different times. As a result, the same traveling vehicle will be included in different positions in the multiple captured images. Information regarding the installation position of the infrastructure sensor 31, i.e., the coordinates of the infrastructure sensor 31, is also acquired. After acquiring the sensor information, the process proceeds to step S102.

[0023] In step S102, the coordinate conversion unit 51 converts the local coordinates into global coordinates. By converting the coordinates into global coordinates, the coordinates of the target included in the sensor information can be acquired as global coordinates instead of local coordinates. In the conversion into global coordinates, the values ​​of the yaw angle, roll angle, and pitch angle based on the initial installation angle when the infrastructure sensor 31 was installed are used. However, if the installation angle is corrected after the infrastructure sensor 31 is installed, the values ​​of the yaw angle, roll angle, and pitch angle based on the corrected installation angle are used. After the coordinate conversion into global coordinates is complete, the process proceeds to step S103.

[0024] In step S103, the map acquisition unit 23 acquires map information about the area around the installation position of the infrastructure sensor 31. The map information includes road azimuth angles for multiple roads included in the imaging range of the infrastructure sensor 31. The map information is acquired from the map database 22 by communication via the communication unit 21. However, if the map information stored in the storage unit of the target object recognition system 10 is the same as the latest map information in the map database 22, the map information stored in the storage unit can be used. After acquiring the map information, the process proceeds to step S110.

[0025] In step S110, the calculation unit calculates the sensor installation state, which is the installation state of the infrastructure sensor 31. Details of the method for calculating the sensor installation state will be described below.

[0026] In FIG. 3 , when calculation of the sensor installation state starts, the azimuth angle calculation unit 53 calculates the maximum distance of the traveling vehicle in step S111. The maximum distance is the length at which the distance between the traveling vehicle and the infrastructure sensor 31 is longest when the infrastructure sensor 31 continuously captures images of the same traveling vehicle. Assume that the traveling vehicle is traveling in a direction away from the infrastructure sensor 31. In this case, the maximum distance may be the distance from the position of the traveling vehicle just before the traveling vehicle moves out of the angle of view of the infrastructure sensor 31 to the installation position of the infrastructure sensor 31. Even if the traveling vehicle is within the angle of view of the infrastructure sensor 31, if the coordinates of the traveling vehicle cannot be calculated, that position is not used in calculating the maximum distance. In other words, the maximum distance is the distance from the farthest traveling vehicle within the range where the coordinates can be calculated by the infrastructure sensor 31 to the installation position of the infrastructure sensor 31. After calculating the maximum distance, proceed to step S112.

[0027] In step S112, the abnormality determination unit 55 determines whether the maximum distance is equal to or greater than the distance threshold. The distance threshold can be the distance from the farthest point from the infrastructure sensor 31 in the target imaging range to the installation position of the infrastructure sensor 31. The distance threshold is, for example, 30 m. If the maximum distance is equal to or greater than the distance threshold, the process proceeds to step S113. On the other hand, if the maximum distance is less than the distance threshold, the process determines that the traveling vehicle has not been properly imaged, and proceeds to step S119.

[0028] In step S113, the azimuth angle calculation unit 53 calculates the traveling azimuth angle of the traveling vehicle. The azimuth angle calculation unit 53 calculates the traveling azimuth angle of the traveling vehicle traveling straight on a straight portion of the road. Therefore, the traveling azimuth angle of the vehicle traveling on a curved portion of the road or the vehicle turning right or left is not calculated.

[0029] The traveling azimuth angle is calculated from the coordinates of two points: a nearby point where the traveling vehicle is close to the infrastructure sensor 31, and a distant point where the traveling vehicle is far from the infrastructure sensor 31. Here, the coordinates of traveling vehicles located within the exclusion distance from the installation position of the infrastructure sensor 31 are not used. The exclusion distance is the shortest imaging distance determined by the performance of the infrastructure sensor 31 plus a margin. The exclusion distance is, for example, 2 m. Therefore, as the coordinates of the nearby point, it is possible to use the coordinates of a point that is closer to the infrastructure sensor 31 and located further than the exclusion distance from multiple coordinates acquired for the traveling vehicle. As the coordinates of the distant point, it is possible to use the coordinates of a point that is farthest from the infrastructure sensor 31 from multiple coordinates acquired for the traveling vehicle. A detailed method for calculating the traveling azimuth angle will be described using figures.

[0030] In FIG. 4, a first road 71 is a road that extends linearly in the east-west direction. A first traveling vehicle 81 is traveling on the first road 71. A second road 72 is a road that extends linearly in the north-south direction. A second traveling vehicle 82 is traveling on the second road 72. The first road 71 and the second road 72 intersect with each other. A first road azimuth angle θew, which is the road azimuth angle of the first road 71, and a second road azimuth angle θns, which is the road azimuth angle of the second road 72, are different angles from each other.

[0031] A first guardrail 71G is installed along a first road 71. A second guardrail 72G is installed along a second road 72. The first guardrail 71G and the second guardrail 72G provide an example of road installations.

[0032] The infrastructure sensor 31 is installed on the roadside at the intersection of the first road 71 and the second road 72. The imaging range DA of the infrastructure sensor 31 includes the straight portions of both the first road 71 and the second road 72. The imaging range DA of the infrastructure sensor 31 includes the first guardrail 71G and the second guardrail 72G.

[0033] 5, from time T1 to time T2, a first traveling vehicle 81 travels from west to east on a first road 71. Similarly, from time T1 to time T2, a second traveling vehicle 82 travels from north to south on a second road 72. The first traveling vehicle 81 is a vehicle traveling straight on the first road 71. The second traveling vehicle 82 is a vehicle traveling straight on the second road 72.

[0034] The first detection point q1 is a detection point of the first traveling vehicle 81 as of time T2. Here, the detection point indicates the center position of the traveling vehicle. The second detection point q2 is a detection point of the first traveling vehicle 81 as of time T1. The movement trajectory from the second detection point q2 to the first detection point q1 can be considered as the traveling trajectory of the first traveling vehicle 81. The first detection point q1 corresponds to a distant point of the first traveling vehicle 81, and the second detection point q2 corresponds to a nearby point of the first traveling vehicle 81.

[0035] The third detection point q3 is a detection point of the second traveling vehicle 82 as of time T1. The fourth detection point q4 is a detection point of the second traveling vehicle 82 as of time T2. The movement trajectory from the third detection point q3 to the fourth detection point q4 can be considered to be the traveling trajectory of the second traveling vehicle 82. The third detection point q3 corresponds to a point near the second traveling vehicle 82, and the fourth detection point q4 corresponds to a point far from the second traveling vehicle 82.

[0036] In the following, we will assume that the east-west direction corresponds to the x-axis direction, the north-south direction corresponds to the y-axis direction, and the height direction corresponds to the z-axis direction. The east-west direction, north-south direction, and height direction are perpendicular to each other. The coordinates of the detection point are expressed by three components: x, y, and z.

[0037] Assume that the global coordinates of the first detection point q1 are expressed as (x1g, y1g, z1g), and the global coordinates of the second detection point q2 are expressed as (x2g, y2g, z2g). In this case, the traveling azimuth angle of the first traveling vehicle 81 is expressed as arctan(y2g-y1g) / (x2g-x1g).

[0038] Assume that the global coordinates of the third detection point q3 are expressed as (x3g, y3g, z3g), and the global coordinates of the fourth detection point q4 are expressed as (x4g, y4g, z4g). In this case, the traveling azimuth angle of the second traveling vehicle 82 is expressed as arctan(y4g-y3g) / (x4g-x3g).

[0039] The traveling azimuth angle may be calculated for one traveling vehicle, or may be calculated as an average value calculated for multiple traveling vehicles. For example, if there are 10 vehicles traveling on the first road 71, the 10 vehicles are defined as first traveling vehicles 81, and the traveling azimuth angle of each is calculated. The average value of the traveling azimuth angles of the 10 vehicles may then be calculated and used as the traveling azimuth angle of the first traveling vehicle 81. Furthermore, instead of taking the average value, the traveling azimuth angle of the largest vehicle among the 10 first traveling vehicles 81 may be used. After calculating the traveling azimuth angle, the process proceeds to step S114.

[0040] In step S114 of Fig. 3, the angle calculation unit 54 calculates the azimuth angle difference. The azimuth angle difference can be calculated by taking the difference between the road azimuth angle and the traveling azimuth angle of the traveling vehicle traveling on that road. For example, if the traveling azimuth angle of the first traveling vehicle 81 traveling on the first road 71 is 0° and the road azimuth angle of the first road 71 is 0°, the azimuth angle difference for the first road 71 is 0°. As the azimuth angle difference, a first azimuth angle difference which is the azimuth angle difference of the first road 71 and a second azimuth angle difference which is the azimuth angle difference of the second road 72 are calculated.

[0041] The method for calculating the azimuth angle difference is not limited to calculation in the dimension of angle. For example, a comparison between (y2g-y1g) / (x2g-x1g) using the driving azimuth angle and tan(θew) using the road azimuth angle may be performed. After calculating the azimuth angle difference, the process proceeds to step S115.

[0042] In step S115, the abnormality determination unit 55 determines whether the azimuth angle difference is less than the angle threshold. An angle that includes the entire target imaging range can be used as the angle threshold. For example, assume that the entire target imaging range is included in the angle of view of the infrastructure sensor 31 even if the installation angle of the infrastructure sensor 31 is shifted by 5°. In this case, the traveling azimuth angle when the installation angle is shifted by 5° can be calculated in advance, and the azimuth angle difference between the traveling azimuth angle and the road azimuth angle can be used as the angle threshold. If both the first azimuth angle difference and the second azimuth angle difference are less than the angle threshold, the process proceeds to step S116. On the other hand, if either the first azimuth angle difference or the second azimuth angle difference is equal to or greater than the angle threshold, the process proceeds to step S119.

[0043] In step S116, the abnormality determination unit 55 determines that the installation state of the infrastructure sensor 31 is normal. The normal installation state is stored, calculation of the sensor installation state is terminated, and the process proceeds to step S121.

[0044] In step S119, the abnormality determination unit 55 determines that the installation state of the infrastructure sensor 31 is abnormal. The abnormality of the installation state is stored, calculation of the sensor installation state is terminated, and the process proceeds to step S121.

[0045] 2, the abnormality determination unit 55 determines whether the installation state of the infrastructure sensor 31 is normal or abnormal. If the installation state is normal, it determines that targets within the target image capture range can be recognized, and the process proceeds to step S136. On the other hand, if the installation state is abnormal, it determines that targets within the target image capture range may not be recognized, and the process proceeds to step S149.

[0046] In step S149, the communication unit 21 issues a sensor abnormality notification. The communication unit 21 transmits a signal to the external terminal 90 indicating that the installation state of the infrastructure sensor 31 is abnormal. This allows the user of the user terminal 91 to understand that the infrastructure sensor 31 is in an abnormal state. If the user of the user terminal 91 is a driver, they can recognize that the safety function using target information may not be operating normally. If the user of the user terminal 91 is a pedestrian, they can recognize that the function notifying of approaching vehicles may not be operating normally. The maintenance company operating the maintenance terminal 92 can recognize that the infrastructure sensor 31 is in an abnormal state and can repair the infrastructure sensor 31. After issuing the sensor abnormality notification, the target recognition control is terminated.

[0047] In step S136, the target recognition unit 52 recognizes a target. More specifically, the target is recognized from the image captured by the infrastructure sensor 31, and the coordinates of the target are acquired. Since the coordinate conversion from the local coordinate to the global coordinate has been completed, the coordinates of the target can be acquired in the global coordinate. After the target recognition, the process proceeds to step S139.

[0048] In step S139, the communication unit 21 transmits the recognition result to the user terminal 91. The recognition result includes information on the type, coordinates, and speed of the target included in the imaging range. For example, information on the size, position coordinates, and speed of the moving vehicle is transmitted to each other between moving vehicles that are in a blind spot relationship with each other. This allows the driver to be aware of the presence of another moving vehicle in a blind spot and to drive safely. After transmitting the recognition result, the target recognition control is terminated. However, after the target recognition control is terminated, the target recognition control is started again. As a result, the target recognition control is repeatedly executed while the infrastructure sensor 31 is determined to be in a normal state.

[0049] The effects of the above-described embodiment will be described below. According to the above-described embodiment, whether the installation state of the infrastructure sensor 31 is normal or abnormal is determined based on the azimuth angle difference, which is the difference between the traveling azimuth angle and the road azimuth angle. Therefore, it is possible to determine whether the installation state of the infrastructure sensor 31 is normal or abnormal, regardless of a specific fixed object. Therefore, it is possible to provide a target recognition system that can determine whether there is an abnormality in the infrastructure sensor 31 with a simple configuration.

[0050] The abnormality determination unit 55 determines that the state is normal when the azimuth angle difference is less than the angle threshold, and determines that the state is abnormal when the azimuth angle difference is equal to or greater than the angle threshold. Therefore, by changing the angle threshold, it is possible to easily change whether the state is determined to be normal or abnormal. For example, if the infrastructure sensor 31 can be easily repaired, setting the angle threshold small makes it easier to determine that the state is abnormal, allowing repairs to be made more frequently.

[0051] When the installation state is abnormal, the communication unit 21 transmits information indicating the abnormal state to the external terminal 90. This allows the user of the external terminal 90 to know that the infrastructure sensor 31 is in an abnormal state. This makes it easy to know that an application based on information from the infrastructure sensor 31 is not operating normally or that the infrastructure sensor 31 needs to be repaired.

[0052] The azimuth angle calculation unit 53 does not calculate the traveling azimuth angle of a vehicle located within the exclusion distance from the installation position of the infrastructure sensor 31. This makes it possible to prevent the traveling azimuth angle from being calculated based on coordinates that have not been accurately acquired because the vehicle is too close to the infrastructure sensor 31. Therefore, it is easier to accurately calculate the traveling azimuth angle compared to calculating the traveling azimuth angle using the coordinates of a vehicle located within the exclusion distance.

[0053] The azimuth angle calculation unit 53 does not calculate the traveling azimuth angle of a vehicle that is not traveling straight on the road. Therefore, it does not calculate the traveling azimuth angle of a vehicle that is turning right or left instead of traveling straight on the road, or a vehicle traveling on a curved section of the road. Therefore, it is possible to prevent an azimuth angle that is clearly different from the road azimuth angle from being calculated as the traveling azimuth angle for a vehicle turning right or left or a vehicle traveling on a curved section. Therefore, it is easy to perform abnormality judgment based on accurate information.

[0054] When there are multiple vehicles traveling on the same road, the azimuth angle calculation unit 53 uses the traveling azimuth angle of the largest vehicle. Large vehicles tend to travel straighter than small vehicles, and tend to travel more stably when traveling straight. Therefore, it is easier to accurately calculate the traveling azimuth angle required to calculate the azimuth angle difference compared to when using the traveling azimuth angle of a small vehicle, which tends to travel less stably.

[0055] (Second embodiment) This embodiment is a modification of the preceding embodiment, in which the amount of correction for coordinate transformation is calculated as needed, and target recognition is performed by performing coordinate transformation with the correction applied.

[0056] 6, the target recognition system 10 includes an infrastructure sensor 231 and a correction amount calculation unit 256. The infrastructure sensor 231 includes a first sensor 231A and a second sensor 231B. The first sensor 231A is a camera device, and the second sensor 231B is a LiDAR. The first sensor 231A and the second sensor 231B may be different types of sensor devices, and the combination of the sensor devices can be freely designed. Furthermore, the infrastructure sensor 231 may be configured by combining more than two types of sensor devices, such as by configuring it by using three types of sensor devices.

[0057] The correction amount calculation unit 256 calculates the correction amount to be used when converting coordinates. By appropriately calculating the correction amount, even if the infrastructure sensor 231 is displaced from its initial installation position and installation angle, the coordinate conversion from the local coordinates to the global coordinates can be performed accurately.

[0058] Details of the target recognition control in the second embodiment will be described below. However, the description will focus on parts that are different from the above-described embodiment, and descriptions of parts that are the same as the above-described embodiment may be omitted.

[0059] 7, when target recognition control in the second embodiment is started, in step S201, a captured image is acquired as sensor information from the infrastructure sensor 231. Since the infrastructure sensor 231 is equipped with two types of sensor devices, a first sensor 231A and a second sensor 231B, sensor information is acquired from each of the sensor devices. After acquiring the sensor information, the process proceeds to step S202.

[0060] In step S202, coordinate conversion unit 51 converts the local coordinates into global coordinates. The sensor information acquired by first sensor 231A and the sensor information acquired by second sensor 231B are each subjected to coordinate conversion. After the coordinate conversion, the process proceeds to step S103, where map information is acquired. After the map information is acquired, the process proceeds to step S210, where the sensor installation state is calculated.

[0061] The method for calculating the sensor installation state in the second embodiment will be described in detail below. In Fig. 8, if it is determined in step S112 that the maximum distance is equal to or greater than the threshold, the process proceeds to step S213. In step S213, the traveling azimuth angle is calculated using information on the first guardrail 71G and the second guardrail 72G, which are road installations. The method for calculating the traveling azimuth angle will be described in detail below using the calculation of the traveling azimuth angle of the first traveling vehicle 81 as an example.

[0062] First, similarly to step S113, the traveling azimuth angle of the first traveling vehicle 81 is calculated from the information of the detection points of the first traveling vehicle 81 as arctan(y2g-y1g) / (x2g-x1g).

[0063] Next, of the detection points on the first guardrail 71G, the detection point closest to the infrastructure sensor 231 is set as the near point, and the detection point farthest from the infrastructure sensor 231 is set as the far point. The installed object azimuth angle, which is the azimuth angle of the first guardrail 71G, is calculated from the two coordinates of the near point and the far point.

[0064] Thereafter, the traveling azimuth angle calculated from the detection points of the first traveling vehicle 81 is corrected by the installed object azimuth angle of the first guardrail 71G. For example, a new traveling azimuth angle is calculated as the average value of the traveling azimuth angle calculated from the detection points of the first traveling vehicle 81 and the installed object azimuth angle of the first guardrail 71G. Here, the weighting of the traveling azimuth angle calculated from the detection points of the first traveling vehicle 81 and the installed object azimuth angle of the first guardrail 71G may be changed. For example, when the traveling azimuth angle is calculated using multiple vehicles as the first traveling vehicles 81, the weighting of the traveling azimuth angle calculated from the detection points of the first traveling vehicle 81 is increased compared to when the traveling azimuth angle is calculated based on only one first traveling vehicle 81. In this case, the greater the number of vehicles designated as the first traveling vehicles 81, the heavier the weighting of the traveling azimuth angle calculated from the detection points of the first traveling vehicle 81 should be. Alternatively, when the traveling azimuth angle is calculated using a large vehicle as the first traveling vehicle 81, the weight of the traveling azimuth angle calculated from the detection point of the first traveling vehicle 81 is increased compared to when the traveling azimuth angle is calculated using a small vehicle as the first traveling vehicle 81.

[0065] In cases where the traveling azimuth angle cannot be calculated from the detection point of the traveling vehicle, such as when there is no traveling vehicle, the installed object azimuth angle may be regarded as the traveling azimuth angle. Also, although the case where a guardrail is used to calculate the installed object azimuth angle has been described as an example, the installed object is not limited to a guardrail. For example, an installed object such as a white line along the road may also be used. After calculating the traveling azimuth angle, the process proceeds to step S114.

[0066] If it is determined in step S116 that the infrastructure sensor 231 is normal, the process proceeds to step S217. In step S217, the correction amount calculation unit 256 calculates the correction amount for coordinate conversion. The correction amount can be calculated from information on the coordinates of the traveling vehicle and the road azimuth angle.

[0067] In the multiple equations shown in FIG. 9, (xg, yg, zg) indicate the global coordinates of the detection point, and (x, y, z) indicate the local coordinates of the detection point. Also, (x0, y0, z0) indicate the global coordinates of the installation position of the infrastructure sensor 231. yaw indicates the yaw angle of the infrastructure sensor 231, pitch indicates the pitch angle of the infrastructure sensor 231, and roll indicates the roll angle of the infrastructure sensor 231. sy indicates sin(yaw), and cy indicates cos(yaw). sp indicates sin(pitch), and cp indicates cos(pitch). sr indicates sin(roll), and cr indicates cos(roll).

[0068] Equation (1) is an equation that shows the relationship between the global coordinates of the detection point, the local coordinates of the detection point, and the global coordinates of the installation position of the infrastructure sensor 231. Equation (1) can be said to be an equation that shows that the global coordinates of the detection point can be obtained by rotating the local coordinates of the detection point about the y-axis, the x-axis, and the z-axis, and adding the global coordinates of the installation position of the infrastructure sensor 231 to the resulting coordinates.

[0069] Equation (2A) is obtained by inputting (x1, y1, z1), which are the local coordinates of the first detection point q1, into equation (1), and is an equation that indicates the global coordinates of the first detection point q1. Equation (2B) is obtained by inputting (x2, y2, z2), which are the local coordinates of the second detection point q2, into equation (1), and is an equation that indicates the global coordinates of the second detection point q2. Equation (2C) is obtained by inputting (x3, y3, z3), which are the local coordinates of the third detection point q3, into equation (1), and is an equation that indicates the global coordinates of the third detection point q3. Equation (2D) is obtained by inputting (x4, y4, z4), which are the local coordinates of the fourth detection point q4, into equation (1), and is an equation that indicates the global coordinates of the fourth detection point q4.

[0070] Equation (3A) is a relational expression obtained when it is assumed that the height of the traveling vehicle is the same at the near point and the far point. Equation (3A) may be expressed as z3g = z4g. Equation (3B) is a relational expression obtained when it is assumed that the first road azimuth angle θew and the traveling azimuth angle of the first traveling vehicle 81 traveling on the first road 71 coincide. Equation (3C) is a relational expression obtained when it is assumed that the second road azimuth angle θns and the traveling azimuth angle of the second traveling vehicle 82 traveling on the second road 72 coincide. In equations (3B) and (3C), rmax is the assumed maximum distance assumed as the maximum distance of the infrastructure sensor 231. Any value can be used for rmax, but for example, a value greater than the distance threshold can be used. rmax is, for example, 40 m. Equation (3B) can also be said to be a relational expression obtained when it is assumed that the difference in distance in the y-axis direction between a point rmax away in the direction of the traveling azimuth angle of the first traveling vehicle 81 and a point rmax away in the direction of the first road azimuth angle θew is zero. Similarly, Equation (3C) can also be said to be a relational expression obtained when it is assumed that the difference in distance in the x-axis direction between a point rmax away in the direction of the traveling azimuth angle of the second traveling vehicle 82 and a point rmax away in the direction of the second road azimuth angle θns is zero. The assumed maximum distance provides an example of a specific distance.

[0071] Equation (4A) is a relational expression based on equation (3A), equation (4B) is a relational expression based on equation (3B), and equation (4C) is a relational expression based on equation (3C). Equation (4D) is a relational expression that shows the relationship between the yaw angle, pitch angle, roll angle, e1 in equation (4A), e2 in equation (4B), and e3 in equation (4C). e1, e2, and e3 each represent an error.

[0072] In calculating the correction amount, the yaw angle, pitch angle, and roll angle are calculated based on Equation (4D). More specifically, as shown in Equation (4D), the yaw angle, pitch angle, and roll angle are calculated so that e1^2 + e2^2 + e3^2 is the minimum value. In the calculation, all possible combinations of values ​​for the yaw angle, pitch angle, and roll angle are calculated, and the yaw angle, pitch angle, and roll angle that satisfy Equation (4D) can be determined. Equation (4B) is an equation related to the first azimuth angle difference, and Equation (4C) is an equation related to the second azimuth angle difference. Therefore, calculating the yaw angle, pitch angle, and roll angle based on Equation (4D) can be said to calculate the correction amount based on the first azimuth angle difference and the second azimuth angle difference. After calculating the correction amount, the calculation of the sensor installation state is terminated with the correction amount stored, and the process proceeds to step S121.

[0073] 7, if the infrastructure sensor 231 is in a normal state, the process proceeds to step S232. On the other hand, if the infrastructure sensor 231 is in an abnormal state, the process proceeds to step S149, where a sensor abnormality is notified.

[0074] In step S232, the coordinate conversion unit 51 performs coordinate conversion using the calculated correction amount. As a result, the information converted in step S202 is updated to the newly converted information. After the coordinate conversion using the correction amount is performed, the process proceeds to step S236.

[0075] In step S236, the target recognition unit 52 performs target recognition using the newly coordinate-converted information. This allows the target to be recognized by more accurately calculating the position coordinates and velocity of the target compared to when target recognition is performed based on coordinate-converted information without using the correction amount. After target recognition, the recognition result is notified in step S139, and then the target recognition control is terminated and restarted. When target recognition control is restarted, the correction amount for the coordinate conversion has been stored, so the correction amount is used at the time of the coordinate conversion in step S202.

[0076] The effects of the above-described embodiment will be described below. According to the above-described embodiment, the coordinate conversion unit 51 performs coordinate conversion using the correction amount calculated by the correction amount calculation unit 256. Therefore, it is easier to convert to accurate coordinates than when converting coordinates without using the correction amount. Furthermore, it is possible to continue target recognition control without repairing the infrastructure sensor 231.

[0077] The correction amount calculation unit 256 calculates the correction amount in the coordinate transformation based on the first azimuth angle difference and the second azimuth angle difference. Therefore, the correction amount can be calculated with a simpler configuration than when a dedicated sensor for calculating the correction amount is used separately from the infrastructure sensor 231.

[0078] The correction amount calculation unit 256 calculates the correction amount using the difference in distance between a point that is a specific distance away from the infrastructure sensor 231 in the direction of the driving azimuth angle and a point that is a specific distance away from the infrastructure sensor 231 in the direction of the road azimuth angle. The specific distance can be, for example, the maximum expected distance. Therefore, it is easy to improve the calculation accuracy of the correction amount by appropriately setting the value of the specific distance. Furthermore, since Equation (3A), Equation (3B), and Equation (3C) are all unified into the dimension of length, an effect of facilitating calculation is also obtained.

[0079] (Other embodiments) The disclosure in this specification and drawings, etc. is not limited to the exemplified embodiments. The disclosure encompasses the exemplified embodiments and modifications thereto by those skilled in the art. For example, the disclosure is not limited to the combinations of parts and / or elements shown in the embodiments. The disclosure can be implemented in various combinations. The disclosure can have additional parts that can be added to the embodiments. The disclosure encompasses the omission of parts and / or elements from the embodiments. The disclosure encompasses the substitution or combination of parts and / or elements between one embodiment and another embodiment. The disclosed technical scope is not limited to the description of the embodiments. Some disclosed technical scopes are defined by the claims, and should be interpreted as including all modifications within the meaning and scope equivalent to the claims.

[0080] The disclosure in the specification, drawings, etc. is not limited by the claims. The disclosure in the specification, drawings, etc. encompasses the technical ideas described in the claims, and extends to more diverse and broader technical ideas than the technical ideas described in the claims. Therefore, various technical ideas can be extracted from the disclosure in the specification, drawings, etc. without being bound by the claims.

[0081] The controller and methods described herein may be implemented by a special-purpose computer having a processor programmed to perform one or more functions embodied in a computer program. Alternatively, the apparatus and methods described herein may be implemented by special-purpose hardware logic circuitry. Alternatively, the apparatus and methods described herein may be implemented by one or more special-purpose computers configured by a combination of a processor executing a computer program and one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by a computer on a computer-readable non-transitory storage medium. [Explanation of symbols]

[0082] 10 Target recognition system, 21 Communication unit, 22 Map database, 23 Map acquisition unit, 31 Infrastructure sensor, 51 Coordinate conversion unit, 52 Target recognition unit, 53 Azimuth angle calculation unit, 54 Angle calculation unit, 55 Abnormality determination unit, 71 First road, 71G First guardrail, 72 Second road, 72G Second guardrail, 81 First traveling vehicle, 82 Second traveling vehicle, 90 External terminal, 91 User terminal, 92 Maintenance terminal, 231 Infrastructure sensor, 231A First sensor, 231B Second sensor, 256 Correction amount calculation unit

Claims

1. An infrastructure sensor (31, 231) installed at a position capable of capturing an image of a road; a coordinate conversion unit (51) that converts coordinates of information obtained by the infrastructure sensor; a target recognition unit (52) that recognizes a target in an image captured by the infrastructure sensor; a communication unit (21) that transmits information about the target recognized by the target recognition unit to a user terminal (91); an azimuth angle calculation unit (53) that calculates a traveling azimuth angle of a traveling vehicle (81, 82) traveling on the road among the targets recognized by the target recognition unit; a map acquisition unit (23) that acquires map information including a road azimuth angle, which is an azimuth angle of the road included in the imaging range of the infrastructure sensor; an angle calculation unit (54) that calculates an azimuth angle difference that is the difference between the traveling azimuth angle and the road azimuth angle; and an abnormality determination unit (55) that determines whether the installation state of the infrastructure sensor is normal or abnormal based on the azimuth angle difference calculated by the angle calculation unit.

2. 2. The target recognition system according to claim 1, wherein the abnormality determination unit determines that the state is normal when the azimuth angle difference is less than an angle threshold, and determines that the state is abnormal when the azimuth angle difference is equal to or greater than the angle threshold.

3. The target recognition system according to claim 2, wherein, when the installation state is the abnormal state, the communication unit transmits information indicating the abnormal state to an external terminal (90).

4. The roads include a first road (71) and a second road (72) that intersect with each other, the azimuth angle calculation unit calculates a traveling azimuth angle of a first traveling vehicle (81) traveling on the first road and a traveling azimuth angle of a second traveling vehicle (82) traveling on the second road; the map acquisition unit acquires the map information including the road azimuth angle of the first road and the road azimuth angle of the second road; the angle calculation unit calculates a first azimuth angle difference which is the azimuth angle difference between the traveling azimuth angle of the first traveling vehicle and the road azimuth angle of the first road, and a second azimuth angle difference which is the azimuth angle difference between the traveling azimuth angle of the second traveling vehicle and the road azimuth angle of the second road; a correction amount calculation unit (256) that calculates a correction amount in the coordinate transformation based on the first azimuth angle difference and the second azimuth angle difference when the installation state is the normal state, 4. The target recognition system according to claim 1, wherein the coordinate transformation unit performs coordinate transformation using the correction amount calculated by the correction amount calculation unit.

5. 5. The target recognition system according to claim 4, wherein the correction amount calculation unit calculates the correction amount using a difference in distance between a point that is a specific distance away from the infrastructure sensor in the direction of the traveling azimuth angle and a point that is the specific distance away from the infrastructure sensor in the direction of the road azimuth angle.

6. 4. The target recognition system according to claim 1, wherein the azimuth angle calculation unit does not calculate the traveling azimuth angle of a vehicle located within an exclusion distance from an installation position of the infrastructure sensor.

7. 4. The target recognition system according to claim 1, wherein the azimuth angle calculation unit does not calculate the traveling azimuth angle of a vehicle that is not traveling straight on the road.

8. The target recognition unit recognizes road installations (71G, 72G) installed along the road, 4. The target recognition system according to claim 1, wherein the azimuth angle calculation unit calculates the traveling azimuth angle using information about the road-installed object recognized by the target recognition unit.

Images