Determination Device, Determination Method, and Determination Program

Abstract

An image acquisition unit 20 acquires a group of time-series images captured by a camera mounted in an observation vehicle. A speed difference estimation unit 24 uses time-series changes in a region representing a target object photographed in the group of time-series images to estimate the speed difference between the target object and the observation vehicle. A determination unit 28 determines, on the basis of the speed difference, whether the target object or the observation vehicle is in a danger state.

Description

Technical Field

[0001] The disclosed technology relates to a determination device, a determination method, and a determination program.

Background Art

[0002] There are sensors installed in the city as information that can be collected and analyzed about vehicles. Typical examples are devices that acquire speed information such as loop coils and orbis, and they are mainly used to detect dangerous driving vehicles that exceed the speed limit.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, loop coils, orbis, etc. are fixed types and can only collect information near the installed position, which poses a problem in terms of versatility. Recently, portable types have also emerged, but in addition to being expensive, the problem remains that they can only collect information from the location where they are transported and installed, so it is difficult to cover the information of roads across the country, and it is impossible to determine dangerous driving vehicles in non-installed sections. Also, although it is possible to detect dangerous driving vehicles using the data obtained from the above-mentioned orbis, etc., it is not generally done to notify the running vehicles of the information of the detected dangerous driving vehicles. Also, even if it is shared whether it is dangerous or not, it is impossible to determine whether the danger does not move from the location where it is detected, or if it moves, which vehicles it can be dangerous to.

[0005] The disclosed technology has been made in view of the above points, and an object thereof is to provide a determination device, a determination method, and a determination program capable of determining a dangerous state that can be used not only for the vehicle but also for other vehicles by using a time-series image group captured by a camera mounted on the vehicle.

Means for Solving the Problems

[0006] A first aspect of the present disclosure is a determination device for determining whether an object that can be photographed from an observation vehicle or the observation vehicle is in a dangerous state. The determination device includes an image acquisition unit that acquires a time-series image group captured by a camera mounted on the observation vehicle, a speed difference estimation unit that estimates a speed difference between the object and the observation vehicle using a time-series change of an area representing the object captured in the time-series image group, and a determination unit that determines whether the object or the observation vehicle is in a dangerous state based on the speed difference.

[0007] A second aspect of the present disclosure is a determination method for a determination device that determines whether an object that can be photographed from an observation vehicle or the observation vehicle is in a dangerous state. In the determination method, an image acquisition unit acquires a time-series image group captured by a camera mounted on the observation vehicle, a speed difference estimation unit estimates a speed difference between the object and the observation vehicle using a time-series change of an area representing the object captured in the time-series image group, and a determination unit determines whether the object or the observation vehicle is in a dangerous state based on the speed difference.

[0008] A third aspect of the present disclosure is a determination program for causing a computer to function as the determination device according to the first aspect.

Effects of the Invention

[0009] According to the disclosed technology, it is possible to determine a dangerous state that can be used not only for the vehicle but also for other vehicles by using a time-series image group captured by a camera mounted on the vehicle.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Embodiments for Carrying Out the Invention

[0011] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings. In each drawing, the same or equivalent components and parts are given the same reference numerals. Also, the dimensional ratios in the drawings are exaggerated for convenience of explanation and may be different from the actual ratios.

[0012] <Outline of the Present Embodiment> In this embodiment, the speed difference between the observation vehicle and the target object is estimated from the temporal change of the region representing the target object shown in the image captured by the camera mounted on the observation vehicle. In this embodiment, as shown in FIGS. 1 and 2, the target object is taken as the target vehicle A, and the case where the speed of the target vehicle A is higher than that of the observation vehicle 100 will be described as an example. FIG. 1 shows an example of an image representing the target vehicle A traveling in the adjacent lane. FIG. 2(A) shows an example where the speed of the target vehicle A traveling in the same lane as the observation vehicle 100 is higher than that of the observation vehicle 100. Further, FIG. 2(B) shows an example where the speed of the target vehicle A traveling in a lane away from the traveling lane of the observation vehicle 100 is higher than that of the observation vehicle 100.

[0013] Conventionally, for the enforcement of vehicles violating speed limits, installed sensors such as speed measuring devices including loop coils, orbis, and H-systems are used.

[0014] Recently, portable sensors are also being used, but they are expensive and lack flexibility in terms of transporting and installing them at the locations where data collection is desired, so it is not possible to cover all roads in the country with portable sensors.

[0015] In addition, it takes time and money to install sensors in automobiles and infrastructure so that information on all roads in the country can be collected. Therefore, there is a need for a technology that efficiently collects information on other automobiles and the like using sensors (including cameras) already installed in some automobiles.

[0016] Also, as shown in FIG. 3, there are also problems caused by the target vehicle A that is parked on the road, such as a parked vehicle or an accident vehicle. Even if such a target vehicle A is a connected car, there are cases where the engine is turned off and position information cannot be collected, so it is necessary to be detected and recognized from the outside in order to notify the surrounding area of its presence.

[0017] In addition, in commercial vehicles, a system that evaluates the presence or absence of dangerous driving (such as distracted driving, driving while talking, aggressive driving, speeding, sudden acceleration, and sudden braking) by incorporating a monitoring mechanism using in-vehicle cameras, sensors, and CAN (Controller Area Network) data has also become widespread.

[0018] However, since there is no merit and there is a cost burden in general vehicles (non-commercial vehicles), the possibility of introducing this system is low, and it is still necessary to detect and recognize from the outside.

[0019] Conventionally, as a method for estimating the speed difference between an observation vehicle and an object, there are those that obtain the distance to the object using a millimeter-wave radar mounted on the observation vehicle and measure the speed difference from its time-series change, or those that calculate the feature points and their optical flow from the video taken using a camera mounted on the observation vehicle, and obtain the movement amount and relative speed of surrounding vehicles from the vector quantity. The mainstream application is for approach warnings and collision avoidance.

[0020] However, since the optical flow is also detected for the surrounding background when the observation vehicle is also moving, it is necessary to separately identify the area of the object in the video. Also, there is a possibility that the same part of the same vehicle cannot be tracked. For example, there is a possibility that a vector is drawn from the tire of one vehicle to the tire of another vehicle. In addition, since it is not clear which part of the vehicle the feature points for calculating the optical flow are, it may cause an error. For example, as shown in FIG. 4, even for the optical flow regarding the same vehicle, there is a difference in the vector length depending on which of the two points it is. FIG. 4 shows an example where the vector of the optical flow of the front wheel part of the target vehicle A is shorter than the vector of the optical flow of the rear part of the target vehicle.

[0021] Therefore, in this embodiment, the time-series image group captured by the camera mounted on the observation vehicle is used to accurately estimate the speed difference between the target vehicle and the observation vehicle and the speed of the target vehicle. The estimation results of the speed difference between the target vehicle and the observation vehicle and the speed of the target vehicle may be utilized for information sharing with companies that provide traffic congestion information such as the Japan Road Traffic Information Center, the police, surrounding vehicles, and the like.

[0022] For example, the estimation result of the speed difference between the target vehicle and the observation vehicle is utilized for detecting vehicles that violate the speed limit. When the observation vehicle is traveling at 100 km / h on a highway with a speed limit of 100 km / h, as shown in FIG. 5, if it is estimated that the speed difference between the target vehicle A and the observation vehicle is +50 km / h or more, it is notified to the police with an attached image that the target vehicle A is a vehicle traveling at 150 km / h in violation of the speed limit.

[0023] In addition, the estimation result of the speed difference between the target vehicle and the observation vehicle is utilized for detecting parked vehicles such as broken-down vehicles and vehicles parked on the road. When the observation vehicle is traveling at 30 km / h on a general road, as shown in FIG. 6, if it is estimated that the speed difference between the target vehicle A and the observation vehicle is -30 km / h, and it is confirmed from the position information that there are no ground features such as signals and facilities around that would cause the vehicle to stop, the information that the target vehicle A is a parked vehicle with a traveling speed of 0 km / h is shared with the following vehicles. As the following vehicle to share the information with, it may be configured to notify only the vehicles that will travel after the time when the danger of the parked vehicle or the like is detected in the lane where the parked vehicle exists. For example, it may notify the vehicles existing between the position where the parked vehicle exists and the nearest intersection, or it may notify the vehicles that are scheduled to travel in the lane where the parked vehicle exists by referring to the navigation information. Also, the information may be transmitted to a map or dynamic map that aggregates and distributes road information, or a service provider that provides a navigation service.

[0024] Next, the principle of estimating the speed difference between the target vehicle and the observation vehicle will be described. Here, the case where the target vehicle is faster than the observation vehicle will be described as an example.

[0025] As shown in FIG. 7, as the target vehicle A moves away at time t, time t + a, and time t + b, the size of the area representing the target vehicle A in the image decreases. At this time, the greater the speed difference between the target vehicle A and the observation vehicle, the faster the size of the area representing the target vehicle A decreases. Therefore, the speed difference between the target vehicle A and the observation vehicle can be obtained from the change amount of the size of the area representing the target vehicle A and the speed at which it decreases. When obtaining the absolute speed of the target vehicle A, the speed of the observation vehicle may be added.

[0026] When using a camera (a general monocular camera) mounted on the observation vehicle, objects such as other vehicles and buildings are represented on the image so as to converge to the vanishing point.

[0027] The size of the area representing the object reflected at that time can be approximated as a one-point perspective view and changes according to a law with respect to the distance from the reference point. For example, when the distance from the camera is doubled, the length of the side of the area representing the front part of the object becomes 1 / 2, and when the distance from the camera is tripled, the length of the side of the area representing the front part of the object becomes 1 / 3.

[0028] Also, when the distance from the camera is doubled, the area of the area representing the front part of the object becomes 1 / 4, and when the distance from the camera is tripled, the area of the area representing the front part of the object becomes 1 / 9.

[0029] As shown in FIG. 8, the area representing the target vehicle A detected at time t is set as the area representing the target vehicle A at the reference position in the vertical direction of the image, and the area of the area representing the rear part of the target vehicle A is set to 1.0. The distance between the target vehicle A and the observation vehicle at this time is defined as the reference distance d (m).

[0030] Regarding the target vehicle A detected at time t + a, the area of the area representing the rear part is 1 / 4 times that at time t, and the length of the side of the area representing the rear part is 1 / 2 that at time t.

[0031] At this time, the distance D between the target vehicle A and the observation vehicle is changed to 2d (m).

[0032] In addition, the area of the region representing the rear part of the target vehicle A detected at time t + b is 1 / 9 times that at time t, and the length of the side of the region representing the rear part is 1 / 3 that at time t.

[0033] At this time, the distance D between the target vehicle A and the observation vehicle is changed to 3d (m).

[0034] FIG. 9 shows, as a graph, the relationship between the area of the region representing the rear part of the target vehicle A described above and the distance D to the target vehicle A. When the horizontal direction of the observation vehicle A is the X-axis and the height direction of the observation vehicle A is the Z-axis, the ratio of the area of the region representing the rear part of the target vehicle A, which is horizontal in the X-Z plane, to the area of the region representing the rear part of the target vehicle A at the reference position changes according to the distance between the target vehicle A and the observation vehicle as shown in FIG. 9 above.

[0035] The relational expression of the ratio of the length of the side of the region representing the rear part of the target vehicle to the length of the side of the region representing the rear part of the target vehicle at the reference position, the distance D to the target vehicle, and the reference distance d is represented by the following formula (1).

[0036] JPEG0007683685000001.jpg2474 (1)

[0037] The relational expression of the ratio of the area of the region representing the rear part of the target vehicle to the area of the region representing the rear part of the target vehicle at the reference position, the distance D to the target vehicle, and the reference distance d is represented by the following formula (2).

[0038] JPEG0007683685000002.jpg2574 (2)

[0039] Here, the reference distance d is the distance from the installation position of a camera 60, which will be described later, mounted on the observation vehicle 100, rather than the lower part of the image (see FIG. 10). When shown in a top view, it is as shown in FIG. 11. In FIG. 11, it is assumed that the camera 60 is mounted at the tip of the observation vehicle 100, and a distance D (= reference distance d) at which the ratio of the area of the region representing the rear part of the target vehicle A is 1.0, a distance D (= 2d) at which the ratio of the area is 1 / 4, and a distance D (= 3d) at which the ratio of the area is 1 / 9 are shown.

[0040] Here, as shown in FIG. 9 above, when the horizontal axis is the distance D to the target vehicle A, it becomes the same graph regardless of the speed difference. Also, as in the above relational expression, if the reference distance d is included in the expression, it is essential to calculate the reference distance d in order to calculate the speed difference using the relational expression.

[0041] Therefore, in the present embodiment, as shown in FIG. 12, by setting the horizontal axis as the time axis, the influence of the speed difference is visualized. Similarly, the graph in FIG. 12 is also a graph in which the ratio of the area changes according to time. Also, the greater the speed difference, the faster it converges, and the smaller the speed difference, the more time it takes to converge.

[0042] Thus, it is possible to estimate the speed difference even only from the tendency in the initial stage that changes rapidly.

[0043] In the present embodiment, the speed difference between the target vehicle and the observation vehicle is estimated as described above, and the dangerous state of the target vehicle is determined.

[0044] For example, on a highway where the speed limit is 80 km / h, a rule is determined that a target vehicle traveling at 110 km / h is determined to be in a dangerous state. That is, a speed difference of +30 km / h is set as the determination threshold for whether it is in a dangerous state or not.

[0045] Alternatively, as shown in FIG. 13, the time-series change of the ratio of the area when the speed difference is +30 km / h is obtained as a pattern, and if it seems to decay earlier than this pattern, the target vehicle is determined to be in a dangerous state, and this information is shared with the police and surrounding vehicles.

[0046] Also, as shown in FIG. 14, patterns may be obtained for each speed difference at 10 km / h intervals, and the degree of danger may be finely classified by determining which pattern is the closest. FIG. 14 shows an example in which patterns representing the time-series changes in the area ratio are obtained for each of the speed differences of +10 km / h, +20 km / h, +30 km / h, +40 km / h, and +50 km / h.

[0047] Also, the distance between the position where the camera 60 is installed and the road (hereinafter referred to as height) may be considered. For example, when the lower end of the image is set to Y = 0, the coordinates on the image for vertical unification in the vertical direction of the image may be set to lower values as the height increases. The height of the camera 60 may be obtained using an altitude sensor, may be determined for each vehicle type on which the camera is mounted, or may be input in advance when the camera 60 is installed. When the upper left of the image is the origin, the coordinates on the image for vertical unification in the vertical direction of the image may be set to higher values as the height increases.

[0048] Also, the orientation in which the camera 60 is installed (hereinafter referred to as orientation) may be considered. For example, when the roll angle of the camera 60 is not horizontal with respect to the road, the acquired image may be rotated left and right and used. The orientation may be obtained using an acceleration sensor or a gyro center, or may be obtained from the acquired image.

[0049] Also, when the pitch angle of the camera 60 is not facing forward but upward or downward, the coordinates on the image for vertical unification in the vertical direction of the image may be moved up and down. For example, when the camera 60 is facing downward, it may be shifted to a higher value on the upper side of the vertical axis.

[0050] Also, in this embodiment, the following two are introduced into the processing.

[0051] First, after the entire rear portion of the target vehicle is completely within the image, the calculation of the size of the area representing the rear portion of the target vehicle is started. Specifically, when the area representing the rear portion of the target vehicle detected from the image is determined to be a certain distance away from the edge of the image, the calculation of the size of the area representing the rear portion of the target vehicle is started.

[0052] Second, the timing when the ratio of the area of the region representing the rear portion of the target vehicle is set to 1.0 is unified in the vertical direction of the image. This is to consider the case where the area representing the rear portion of the target vehicle is missing due to the front of the observing vehicle, and to unify the reference distance d.

[0053] Also, in the present embodiment, in order to determine whether the target vehicle is in a dangerous state rather than whether the target vehicle is approaching or moving away, the speed difference between the target vehicle and the observing vehicle is obtained.

[0054] [First Embodiment] <Configuration of the Determination Device According to the First Embodiment> FIGS. 15 and 16 are block diagrams showing the hardware configuration of the determination device 10 of the present embodiment.

[0055] As shown in FIG. 15, a camera 60, a sensor 62, and a communication unit 64 are connected to the determination device 10. The camera 60 is mounted on the observing vehicle 100, captures a time-series image group representing the front of the observing vehicle 100, and outputs it to the determination device 10. The sensor 62 detects CAN data including the speed of the observing vehicle 100 and outputs it to the determination device 10. The communication unit 64 transmits the determination result by the determination device 10 to surrounding vehicles and the server of the company via a network.

[0056] Note that, although the case where the determination device tail 10 is mounted on the observing vehicle 100 is described as an example, it is not limited thereto. An external device capable of communicating with the observing vehicle 100 may be configured as the determination device 10.

[0057] As shown in FIG. 16, the determination device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to be communicable with each other via a bus 19.

[0058] The CPU 11 is a central processing unit that executes various programs and controls each unit. That is, the CPU 11 reads a program from the ROM 12 or the storage 14 and executes the program using the RAM 13 as a work area. The CPU 11 performs control of the above components and various arithmetic processes according to the programs stored in the ROM 12 or the storage 14. In the present embodiment, a determination program for determining the dangerous state of the target vehicle A is stored in the ROM 12 or the storage 14. The determination program may be a single program or a program group composed of a plurality of programs or modules.

[0059] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a work area. The storage 14 is composed of an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including an operating system and various data.

[0060] The input unit 15 is used to perform various inputs including a time-series image group photographed by the camera 60 and CAN data detected by the sensor 62. For example, the input unit 15 receives a time-series image group including the target vehicle A existing in front of the observation vehicle 100 photographed by the camera 60 and CAN data including the speed of the observation vehicle 100 detected by the sensor 62.

[0061] Each image in the time-series image group is an RGB or grayscale image without distortion caused by the camera structure such as a lens or a shutter, or an image in which such distortion is corrected.

[0062] The display unit 16 is, for example, a liquid crystal display, and displays various types of information including the determination result of the dangerous state of the target vehicle A. The display unit 16 may adopt a touch panel method and function as the input unit 15.

[0063] The communication interface 17 is an interface for communicating with other devices via the communication unit 64. For example, standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark) are used.

[0064] Next, the functional configuration of the determination device 10 will be described. FIG. 17 is a block diagram showing an example of the functional configuration of the determination device 10.

[0065] Functionally, as shown in FIG. 17, the determination device 10 includes an image acquisition unit 20, a speed acquisition unit 22, a speed difference estimation unit 24, a speed estimation unit 26, a determination unit 28, and a road database 30.

[0066] The image acquisition unit 20 acquires the time-series image group received by the input unit 15.

[0067] The speed acquisition unit 22 acquires the speed of the observation vehicle 100 when the time-series image group was taken, which was received by the input unit 15.

[0068] The speed difference estimation unit 24 estimates the speed difference between the target vehicle A and the observation vehicle 100 using the time-series change of the area representing the target vehicle A captured in the time-series image group.

[0069] Specifically, for each time, the speed difference estimation unit 24 calculates the ratio between the size of the area representing the target vehicle A at the reference position and the size of the area representing the target vehicle A at that time, and compares the pattern representing the time-series change of the ratio with the pattern representing the time-series change of the ratio obtained in advance for each speed difference, thereby estimating the speed difference between the target vehicle A and the observation vehicle 100.

[0070] As shown in FIG. 18, the speed difference estimation unit 24 includes an object detection unit 40, a tracking unit 42, a region information calculation unit 44, a pattern calculation unit 46, a pattern comparison unit 48, and a pattern database 50.

[0071] The object detection unit 40 detects a region representing a target vehicle A that can be photographed from the observation vehicle 100 from each image of the time-series image group. Specifically, by performing object detection in a classification including cars, trucks, and buses using object detection means such as the object detection algorithm YOLOv3, a region representing the target vehicle A is detected, and as shown in FIG. 19, numbering is performed on the detected region. In FIG. 19, in the Nth frame, regions X and Y representing the target vehicle A are detected, in the N+1th frame, regions X and Y representing the target vehicle A are detected, and in the N+2th frame, an example in which region X representing the target vehicle A is detected is shown.

[0072] The tracking unit 42 tracks the region representing the target vehicle A based on the detection result of the object detection unit 40. Specifically, as shown in FIG. 19 above, by comparing the regions detected in the front and rear frames, among the regions detected in the current frame, the region with a larger number of overlapping pixels with the regions detected in the front and rear frames is estimated as the region representing the same target vehicle, and by repeating this, the region representing each target vehicle A is tracked. On the right side of FIG. 19, for region X in the N+1th frame, an example of calculating a value obtained by dividing the number of pixels in the intersection of the pixels of region X that overlap when compared with the Nth frame and the N+2th frame by the number of pixels in the union of the pixels of region X that overlap when compared with the Nth frame and the N+2th frame is shown. For the region X where this value is the highest, it is determined that it represents the same target vehicle A as the overlapping regions X in the front and rear frames, and the region representing the target vehicle A is tracked.

[0073] The area information calculation unit 44 calculates the size of the area representing the tracked target vehicle A for each time. At this time, the area information calculation unit 44 calculates the size of the area representing the tracked target vehicle A for each time from the timing when the area representing the target vehicle A is away from the edge of the image. For example, as the size of the area representing the tracked target vehicle A, the length of the side or the area is calculated. In the present embodiment, the case of calculating the area of the area representing the target vehicle A will be described as an example.

[0074] The pattern calculation unit 46 calculates a pattern representing the time-series change of the ratio between the size of the area representing the target vehicle A at the reference position and the size of the area representing the target vehicle A at each time for the tracked target vehicle A. Specifically, the pattern calculation unit 46 calculates the size of the area representing the target vehicle A detected at the reference position in the vertical direction of the image as the size of the area representing the target vehicle A at the reference position, and calculates a pattern representing the time-series change of the ratio between the size of the area representing the tracked target vehicle A and the size of the area representing the target vehicle A at the reference position.

[0075] The pattern comparison unit 48 compares the pattern calculated by the pattern calculation unit 46 with the pattern representing the time-series change of the ratio between the size of the area representing the target vehicle A at the reference position and the size of the area representing the target vehicle A stored in the pattern database 50 for each speed difference, and estimates the speed difference corresponding to the most similar pattern as the speed difference between the target vehicle A and the observation vehicle 100.

[0076] In the pattern database 50, for each speed difference, a pattern representing the time-series change of the ratio between the size of the area representing the target vehicle A at the reference position and the size of the area representing the target vehicle A is stored (see FIG. 14 above).

[0077] The speed estimation unit 26 estimates the speed of the target vehicle A from the acquired speed of the observation vehicle 100 and the estimated speed difference.

[0078] The determination unit 28 determines whether the target vehicle A is in a dangerous state using the speed difference from the target vehicle A or the speed of the target vehicle A.

[0079] For example, when the speed of the target vehicle A is equal to or higher than the threshold value, the determination unit 28 determines that the target vehicle A is in a dangerous state. Here, as an example, the threshold value is a speed that is a predetermined speed faster than the speed limit.

[0080] In addition, when the determination unit 28 determines, based on the lane information acquired from the road database 30, that the road has no overtaking lane, and the speed difference between the target vehicle A and the observation vehicle 100 is equal to or higher than the threshold value, the determination unit 28 determines that the target vehicle A is in a dangerous state.

[0081] The road database 30 stores lane information for each point on the road.

[0082] <Operation of the determination device according to the first embodiment> Next, the operation of the determination device 10 will be described.

[0083] FIG. 20 is a flowchart showing the flow of the determination process by the determination device 10. The determination process is performed by the CPU 11 reading a determination program from the ROM 12 or the storage 14, expanding it in the RAM 13, and executing it. In addition, a time-series image group captured by the camera 60 and CAN data detected by the sensor 62 when the time-series image group is captured are input to the determination device 10.

[0084] In step S100, the CPU 11, as the image acquisition unit 20, acquires the time-series image group received by the input unit 15.

[0085] In step S102, the CPU 11, as the speed acquisition unit 22, acquires the speed of the observation vehicle 100 when the time-series image group is captured from the CAN data received by the input unit 15.

[0086] In step S104, the CPU 11, acting as the object detection unit 40, detects the region representing the target vehicle A from each image in the time-series image group. Then, the CPU 11, acting as the tracking unit 42, tracks the region representing the target vehicle A based on the detection result by the object detection unit 40.

[0087] In step S106, the CPU 11, acting as the region information calculation unit 44, calculates the size of the region representing the tracked target vehicle A for each time. Then, the CPU 11, acting as the pattern calculation unit 46, calculates a pattern representing the time-series change of the ratio of the size of the region at the reference position of the tracked target vehicle A to the size of the region representing the target vehicle A.

[0088] In step S108, the CPU 11, acting as the pattern comparison unit 48, compares the pattern calculated by the pattern calculation unit 46 with the pattern representing the time-series change of the ratio of the size of the region representing the target vehicle A at the reference position to the size of the region representing the target vehicle A, which is stored in the pattern database 50 for each speed difference, and estimates the speed difference corresponding to the most similar pattern as the speed difference between the target vehicle A and the observation vehicle 100.

[0089] In step S110, the CPU 11, acting as the speed estimation unit 26, estimates the speed of the target vehicle A from the acquired speed of the observation vehicle 100 and the estimated speed difference.

[0090] In step S112, the CPU 11, acting as the determination unit 28, determines whether the target vehicle A is in a dangerous state using the speed difference between the target vehicle A and the observation vehicle 100 or the speed of the target vehicle A. If it is determined that the target vehicle A is in a dangerous state, the process proceeds to step S114. On the other hand, if it is determined that the target vehicle A is not in a dangerous state, the determination process ends.

[0091] In step S114, the communication unit 64 transmits danger information indicating the determination result by the determination device 10 to the surrounding vehicles and the servers of enterprises via the network. Also, the display unit 16 displays the danger information including the determination result of the dangerous state of the target vehicle A, and the determination process ends.

[0092] As described above, the determination device according to the present embodiment estimates the speed difference between the target vehicle and the observation vehicle using the time-series change of the region representing the target vehicle captured in the time-series image group, and determines whether the target vehicle is in a dangerous state based on the speed difference. Thereby, it is possible to determine a dangerous state that can also be used other than the observation vehicle by using the time-series image group captured by the camera mounted on the observation vehicle.

[0093] Also, compared with Reference 1, the determination device according to the present embodiment can estimate the speed of the target vehicle by considering the speed of the observation vehicle, and can estimate whether the target vehicle is stopped or moving. For example, it is possible to correspond to the difference in the change in the size of the region representing the target vehicle approaching in the oncoming lane when the observation vehicle is stopped and when it is traveling at 80 km / h. Furthermore, it is also possible to correspond to the case where the target vehicle is stopped and the observation vehicle is approaching. [Reference 1]: NTT Communications Corporation, "Success in Automatically Detecting Dangerous Driving Utilizing Artificial Intelligence (AI)", <URL:https: / / www.ntt.com / about-us / press-releases / news / article / 2016 / 20160926_2.html>, September 26, 2016

[0094] Also, in the above Non-Patent Document 1, the movement of the background is not considered. Especially since optical flow is used, when the observation vehicle is moving, the background also moves, so it is difficult to distinguish between the target vehicle and the road surface or street trees where vectors are drawn in the same direction, and a fluttering flag or trees swaying in strong wind may be detected as the target vehicle. Therefore, separate means for specifying the region representing the target vehicle and statistical processing of vectors are required. However, for a target vehicle overtaking the observation vehicle, since the vector is in the opposite direction to the background, it can be extracted neatly using optical flow. On the other hand, in the present embodiment, by tracking the region representing the target vehicle between frames and estimating the speed difference between the target vehicle and the observation vehicle using the time-series change of the region representing the target vehicle, the amount of calculation can be reduced.

[0095] In the above-described embodiment, the case where the speed difference between the target vehicle and the observation vehicle is estimated by comparing patterns representing the time-series changes in the ratio of the size of the region representing the target vehicle at the reference position to the size of the region representing the target vehicle has been described as an example. However, the present invention is not limited to this. An approximate expression approximating the pattern representing the time-series change in the ratio of the size of the region representing the target vehicle to the size of the region representing the target vehicle at the reference position may be calculated, and the approximate expressions may be compared to estimate the speed difference between the target vehicle and the observation vehicle. In this case, first, for each time, the ratio of the size of the region representing the target vehicle to the size of the region representing the target vehicle at the reference position is calculated, and a point group having coordinates with the time on the horizontal axis and the ratio of the sizes on the vertical axis is calculated ([t1, x1], [t2, x2], [t3, x3],...), and an approximate expression corresponding to the point group may be calculated. At this time, the degree of the approximate expression may be determined according to whether the size of the region is the length of a side or the area of the region.

[0096] Also, patterns representing the time-series changes in the ratio of the size of the region representing the target vehicle to the size of the region representing the target vehicle at the reference position may be prepared for each vehicle type such as cars, trucks, buses, and trailers. Further, patterns representing the time-series changes in the ratio may be prepared for each vehicle name. In this case, the object detection unit may detect the region representing the target vehicle A from each image in the time-series image group and identify the vehicle type and vehicle name of the target vehicle A.

[0097] Also, the speed difference between the target vehicle and the observation vehicle may be estimated in consideration of how many lanes away the target vehicle is and which of the front part and the rear part of the target vehicle is shown in the image.

[0098] [Second Embodiment] Next, the determination device according to the second embodiment will be described. Note that the parts having the same configuration as those in the first embodiment are denoted by the same reference numerals and the description thereof will be omitted.

[0099] In the second embodiment, the distance to the target vehicle is estimated using a relational expression represented by the ratio of the size of the region representing the target vehicle at the reference position to the size of the region representing the target vehicle, the reference distance to the target vehicle at the reference position, and the distance to the target vehicle. The difference from the first embodiment is that the speed difference is estimated from the change in the estimated distance.

[0100] <Configuration of the determination device according to the second embodiment> Next, the determination device 10 of the second embodiment includes an image acquisition unit 20, a speed acquisition unit 22, a speed difference estimation unit 224, a speed estimation unit 26, a determination unit 28, and a road database 30.

[0101] For each time, the speed difference estimation unit 224 calculates the distance to the target vehicle A at that time using the reference distance to the target vehicle A at the reference position obtained in advance for the type of the target vehicle A, the size of the region representing the target vehicle A at the reference position obtained in advance for the type of the target vehicle A, and the size of the region representing the target vehicle A at that time, and estimates the speed difference between the target vehicle A and the observation vehicle 100 from the time-series change of the distance.

[0102] Specifically, for each time, the speed difference estimation unit 224 calculates the distance to the target vehicle A at that time using a relational expression represented by the reference distance to the target vehicle A at the reference position, the ratio of the size of the region representing the target vehicle A at that time to the size of the region representing the target vehicle A at the reference position, and the distance to the target vehicle A at that time. Then, the speed difference estimation unit 224 estimates the speed difference between the target vehicle A and the observation vehicle 100 based on the time-series change of the distance to the target vehicle A.

[0103] As shown in FIG. 21, the speed difference estimation unit 224 includes an object detection unit 40, a tracking unit 42, a region information calculation unit 44, a distance calculation unit 246, a speed difference calculation unit 248, and a parameter database 250.

[0104] The object detection unit 40 detects the region representing the target vehicle A from each image in the time-series image group. At this time, the object detection unit 40 further identifies the type of the target vehicle A. Here, the type of the target vehicle A is, for example, the vehicle type.

[0105] The region information calculation unit 44 calculates the size of the region representing the tracked target vehicle A for each time, and calculates the ratio of the size of the region representing the tracked target vehicle A to the size of the region representing the target vehicle A at the reference position for each time.

[0106] The distance calculation unit 246 calculates the distance to the target vehicle A at each time using a relational expression represented by the reference distance corresponding to the type of the target vehicle A, the ratio of the size of the region representing the target vehicle A at that time to the size of the region representing the target vehicle A at the reference position, and the distance to the target vehicle A at that time. Specifically, taking the size of the region representing the target vehicle A as the area, substituting the reference distance obtained in advance for the type of the target vehicle A and the ratio of the size of the region representing the target vehicle A at that time to the size of the region representing the target vehicle A at the reference position obtained in advance for the type of the target vehicle A into the above formula (2), the distance to the target vehicle A is calculated.

[0107] In addition, when the size of the region representing the target vehicle A is the length, substituting the reference distance obtained in advance for the type of the target vehicle A and the ratio of the size of the region representing the target vehicle A at that time to the size of the region representing the target vehicle A at the reference position obtained in advance for the type of the target vehicle A into the above formula (1), the distance to the target vehicle A is calculated.

[0108] The speed difference calculation unit 248 calculates the speed difference between the target vehicle A and the observation vehicle 100 based on the distance to the target vehicle A calculated for each time and the interval of the time step. Specifically, the speed difference between the target vehicle A and the observation vehicle 100 is calculated by dividing the difference in the distance to the target vehicle A between times by the interval of the time step.

[0109] The parameter database 250 stores, for each type of vehicle, a previously determined reference distance and the size of the area representing the target vehicle A at the reference position. For example, for each type of target vehicle A, the size of the area representing the target vehicle A when detected at the reference position in the vertical direction of the image and the distance to the target vehicle A at that time are determined in advance, and the size of the area representing the target vehicle A at the reference position and the reference distance may be stored in the parameter database 250. The reference distance is obtained from the dimensions of the target vehicle A and the angle-of-view information of the camera 60. For example, the width defined for the vehicle type of the target vehicle A may be used as the dimension of the target vehicle A to obtain the reference distance from the width of the target vehicle A and the angle-of-view information of the camera 60. For example, the distance at which the entire width of the target vehicle A can be photographed from the horizontal angle-of-view information of the camera 60 may be obtained and used as the reference distance d. Also, even if the type of the target vehicle A is not specified, if at least a part of the dimensions of the target vehicle A can be obtained, the reference distance may be obtained from the dimensions and the angle-of-view information of the camera 60. That is, among the photographed subjects, those whose size in the real space can be obtained may be used, and the reference distance may be obtained using the relationship with the size of the subject in the image.

[0110] <Operation of the determination device according to the second embodiment> Next, the operation of the determination device 10 will be described.

[0111] FIG. 22 is a flowchart showing the flow of the determination process by the determination device 10. The determination process is performed by the CPU 11 reading the determination program from the ROM 12 or the storage 14, expanding it in the RAM 13, and executing it. Also, a time-series image group photographed by the camera 60 and CAN data detected by the sensor 62 when the time-series image group was photographed are input to the determination device 10.

[0112] In step S100, the CPU 11, as the image acquisition unit 20, acquires the time-series image group received by the input unit 15.

[0113] In step S102, as the speed acquisition unit 22, the CPU 11 acquires the speed of the observation vehicle 100 when the time-series image group was captured from the CAN data received by the input unit 15.

[0114] In step S104, as the object detection unit 40, the CPU 11 detects the region representing the target vehicle A from each image of the time-series image group and identifies the type of the target vehicle A. Then, as the tracking unit 42, the CPU 11 tracks the region representing the target vehicle A based on the detection result by the object detection unit 40.

[0115] In step S200, as the region information calculation unit 44, the CPU 11 calculates the size of the region representing the tracked target vehicle A for each time, and calculates the ratio of the size of the region representing the tracked target vehicle A to the size of the region representing the target vehicle A at the reference position for each time.

[0116] In step S201, as the distance calculation unit 246, for each time, using the relational expression represented by the reference distance, the ratio of the size of the region representing the target vehicle A at the current time to the size of the region representing the target vehicle A at the reference position, and the distance to the target vehicle A at the current time, the CPU 11 calculates the distance to the target vehicle A at the current time from the size of the region representing the target vehicle A at the current time.

[0117] In step S202, as the speed difference calculation unit 248, for each time, based on the distance to the target vehicle A calculated for each time and the interval of the time step, the CPU 11 calculates the speed difference between the target vehicle A and the observation vehicle 100. Specifically, the speed difference between the target vehicle A and the observation vehicle 100 is calculated by dividing the difference in the distance to the target vehicle A between times by the interval of the time step.

[0118] In step S110, the CPU 11, acting as the speed estimation unit 26, estimates the speed of the target vehicle A based on the acquired speed of the observation vehicle 100 and the estimated speed difference. In step S112, the CPU 11, acting as the determination unit 28, determines whether the target vehicle A is in a dangerous state using the speed difference between the target vehicle A and the observation vehicle 100 or the speed of the target vehicle A. If it is determined that the target vehicle A is in a dangerous state, the process proceeds to step S114. On the other hand, if it is determined that the target vehicle A is not in a dangerous state, the determination process ends.

[0119] In step S114, the communication unit 64 transmits danger information including the determination result by the determination device 10 via the network to peripheral vehicles and the servers of enterprises. Also, the display unit 16 displays the danger information including the determination result of the dangerous state of the target vehicle A, and the determination process ends.

[0120] As described above, the determination device according to the present embodiment uses the time-series change of the area representing the target vehicle captured in the time-series image group and the relational expression represented by the size of the area representing the target vehicle and the distance to the target vehicle to estimate the speed difference between the target vehicle and the observation vehicle, and determines whether the target vehicle is in a dangerous state based on the speed difference. Thereby, it is possible to determine a dangerous state that can also be used other than the observation vehicle using the time-series image group captured by the camera mounted on the observation vehicle.

[0121] <Modification Example> Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

[0122] For example, the case where the target vehicle is faster than the speed of the observation vehicle has been described as an example, but it is not limited to this. Even when the target vehicle is slower than the speed of the observation vehicle, the present invention may be applied. In this case, based on the region representing the target vehicle detected from the time-series image group, the size of the region representing the target vehicle at the reference position is obtained, and for each time, the ratio of the size of the region representing the target vehicle to the size of the region representing the target vehicle at the reference position is calculated, a pattern representing the time-series change of the ratio is obtained, the time axis is reversed, and it is made to appear as if the observation vehicle is faster, and it may be compared with the pattern for each speed difference. Also, when the speed of the target vehicle is extremely slow on a highway where the speed limit is 80 km / h, for example, when the target vehicle is a single vehicle that has broken down and stopped on the shoulder, it may be determined that the target vehicle is in a dangerous state, and the detection point of the target vehicle may be notified to the high-speed mobile unit. Further, when the target vehicle is a vehicle in the oncoming lane, if the speed difference from the target vehicle is faster than twice the speed limit or slower than the speed limit, it may be determined that the target vehicle is in a dangerous state.

[0123] Also, as a dangerous state of the target vehicle, it may be determined that there is congestion in each lane. In the determination of congestion in each lane, for example, when the estimated speed of each of a plurality of target vehicles in a predetermined lane is below a threshold value, it may be determined that there is congestion. Also, when the observation vehicle is traveling at high speed, it is assumed that it overtakes a large number of vehicles and the speed difference is large. In such a case, when the speed of the observation vehicle is equal to or higher than a predetermined value, the speed difference for determining congestion or the number of overtaken vehicles may be increased.

[0124] Also, the case where the region representing the rear portion of the target vehicle is detected has been described as an example, but it is not limited to this. For example, the region representing the side portion of the target vehicle may be detected. Here, the time-series change in the size of the region representing the target vehicle is different between the rear portion of the target vehicle and the side portion of the target vehicle.

[0125] Specifically, the relational expression among the ratio of the length of the side of the region representing the side portion of the target vehicle to the length of the side of the region representing the side portion of the target vehicle at the reference position, the distance D to the target vehicle, and the reference distance d is represented by the following formula (3).

[0126] JPEG0007683685000003.jpg2375 (3)

[0127] Further, the relational expression among the ratio of the area of the region representing the side portion of the target vehicle to the area of the region representing the side portion of the target vehicle at the reference position, the distance D to the target vehicle, and the reference distance d is represented by the following formula (4).

[0128] JPEG0007683685000004.jpg2173 (4)

[0129] Also, the region representing both the rear portion and the side portion of the target vehicle may be detected. In this case, a relational expression obtained by adding the relational expression regarding the region representing the rear portion of the target vehicle and the relational expression regarding the region representing the side portion may be used.

[0130] Further, regardless of the type of the target vehicle, for those with a fixed size, for example, the speed difference between the target vehicle and the observation vehicle may be estimated using the time-series change in the size of the region representing the license plate of the target vehicle.

[0131] Also, although the case where the object to be detected is the target vehicle has been described as an example, it is not limited to this. The object to be detected may be other than the target vehicle. For example, it may be a motorcycle or a person walking on the road, or it may be a ground object such as a falling object, a road sign, a billboard, or a utility pole. When the object is a ground object, in the time-series image group, the size of the region representing the object changes so as to expand. Therefore, if it is determined that the object is a ground object from the estimated speed of the object, it may be notified to the following vehicle that there is a ground object, or the position information (longitude and latitude) may be corrected using the distance to the ground object estimated using a relational expression. Also, since the speed of the observation vehicle can be obtained from the speed difference from the estimated object, the dangerous state of the observation vehicle may be determined.

[0132] Also, although the case where the road on which the observation vehicle and the target vehicle are traveling is a straight road has been described as an example, it is not limited to this. The road on which the observation vehicle and the target vehicle are traveling may be a road with curvature. In this case, even if the road has curvature, since the short-time change of the region representing the target vehicle is used, the speed difference from the target vehicle may be estimated in the same way as considering it as a straight road. Alternatively, the speed difference from the target vehicle may be estimated using a relational expression considering the curvature.

[0133] Also, although the case where the image captured by the camera has no distortion or the distortion correction is performed in advance has been described as an example, it is not limited to this. For example, a relational expression adapted to the distortion of the camera lens may be used, or a partial image obtained by cutting out the central portion of an image with less distortion may be used to estimate the speed difference from the target vehicle.

[0134] Also, although the case where the target vehicle is traveling in a lane different from the observation vehicle has been described as an example, it is not limited to this. A vehicle traveling in the same lane as the observation vehicle may be used as the target vehicle.

[0135] In addition, while an example is described in which a reference position is defined in the vertical direction of the image and the same pattern or relational expression is used regardless of the lane position, the present invention is not limited to this. For example, for each lane position, a reference position in the vertical direction of the image may be defined and a pattern or relational expression may be prepared.

[0136] Further, in the period corresponding to the time-series image group, when the target vehicle or the observation vehicle performs a lane change, since the time-series change in the size of the region representing the target region becomes a sudden change, exception processing such as diverting to separate processing may be incorporated.

[0137] In addition, in each of the above embodiments, various processes executed by the CPU by reading software (program) may be executed by various processors other than the CPU. Examples of the processor in this case include a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacture, such as a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array), and a dedicated electric circuit which is a processor having a circuit configuration dedicated to executing specific processing, such as an ASIC (Application Specific Integrated Circuit). Further, the determination process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same type or different types (for example, a plurality of FPGAs, a combination of a CPU and an FPGA, etc.). Further, the hardware structure of these various processors is, more specifically, an electric circuit combining circuit elements such as semiconductor elements.

[0138] Also, in each of the above embodiments, the mode where the determination program is pre-stored (installed) in the storage 14 has been described, but it is not limited to this. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), and a USB (Universal Serial Bus) memory. Further, the program may be in a form downloaded from an external device via a network.

[0139] Regarding the above embodiments, the following additional remarks are disclosed.

[0140] (Supplementary Note 1) A determination device for determining whether an object that can be photographed by an observation vehicle or the observation vehicle is in a dangerous state, a memory, at least one processor connected to the memory, and including the processor acquires a time-series image group photographed by a camera mounted on the observation vehicle, estimates the speed difference between the object and the observation vehicle using the time-series change of the region representing the object photographed in the time-series image group, and determines whether the object or the observation vehicle is in a dangerous state based on the speed difference A determination device configured as described above.

[0141] (Supplementary Note 2) A non-transitory storage medium storing a program executable by a computer so as to execute a determination process for determining whether an object that can be photographed by an observation vehicle or the observation vehicle is in a dangerous state, wherein the determination process acquires a time-series image group photographed by a camera mounted on the observation vehicle, Estimate the speed difference between the object and the observation vehicle by using the time-series change of the region representing the object photographed in the time-series image group, Determine whether the object or the observation vehicle is in a dangerous state based on the speed difference Non-transitory storage medium

Explanation of Signs

[0142] 10 Judgment device 11 CPU 15 Input unit 16 Display unit 17 Communication interface 20 Image acquisition unit 22 Speed acquisition unit 24, 224 Speed difference estimation unit 26 Speed estimation unit 28 Judgment unit 30 Road database 40 Object detection unit 42 Tracking unit 44 Region information calculation unit 46 Pattern calculation unit 48 Pattern comparison unit 50 Pattern database 60 Camera 62 Sensor 64 Communication unit 100 Observation vehicle 246 Distance calculation unit 248 Speed difference calculation unit 250 Parameter database A Target vehicle

Claims

1. A determination device for determining whether an object that can be photographed from an observation vehicle or the observation vehicle is in a dangerous state, comprising: an image acquisition unit that acquires a time-series image group photographed by a camera mounted on the observation vehicle; a speed difference estimation unit that estimates a speed difference between the object and the observation vehicle using a time-series change in an area representing the object photographed in the time-series image group; a determination unit that determines whether the object or the observation vehicle is in a dangerous state based on the speed difference; wherein the speed difference estimation unit calculates the size of the area representing the object at each time from the timing when the area representing the object is at a certain distance from the edge of the image photographed by the camera, and for each time, calculates the ratio between the size of the area representing the object when detected at the reference position in the vertical direction of the image and the size of the area at that time; compares the pattern representing the time-series change of the ratio with the patterns representing the time-series changes of the ratio obtained in advance for each speed difference, and estimates the speed difference corresponding to the most similar pattern as the speed difference between the object and the observation vehicle.

2. The determination device according to claim 1, wherein the speed difference estimation unit calculates the length of a side or the area of the region as the size of the region representing the object.

3. A determination method in a determination device for determining whether an object that can be photographed from an observation vehicle or the observation vehicle is in a dangerous state, comprising: an image acquisition unit acquires a time-series image group photographed by a camera mounted on the observation vehicle; a speed difference estimation unit estimates a speed difference between the object and the observation vehicle using a time-series change in an area representing the object photographed in the time-series image group; a determination unit determines whether the object or the observation vehicle is in a dangerous state based on the speed difference; wherein in estimating the speed difference, the size of the area representing the object is calculated at each time from the timing when the area representing the object is at a certain distance from the edge of the image photographed by the camera, and for each time, the ratio between the size of the area representing the object when detected at the reference position in the vertical direction of the image and the size of the area at that time is calculated; the pattern representing the time-series change of the ratio is compared with the patterns representing the time-series changes of the ratio obtained in advance for each speed difference, and the speed difference corresponding to the most similar pattern is estimated as the speed difference between the object and the observation vehicle.

4. A determination program for causing a computer to function as the determination device according to claim 1 or 2.

Images