Image processing device, vehicle, image processing method, image processing program, and server

Abstract

To provide an image processing device capable of attaining smoothing of traffic.SOLUTION: An image processing device 10 comprises a communication interface 12 and a processor 13. The processor 13 detects an image of another mobile 2 from a periphery video image picking up a periphery of a mobile 1 and performs behavior determination processing for determining a behavior of the mobile 1 based on a state of the detected other mobile 2. The communication interface 12 outputs to the mobile 1 information commanding the determined behavior of the mobile 1. The processor 13 determines a first behavior of the mobile 1 based on a first state of the other mobile 2 detected from the periphery video image as the behavior determination processing and outputs to the mobile 1 information commanding the determined first behavior. The processor 13 then determines a second behavior of the mobile 1 in a case where it is determined that the other mobile 2 is performing dangerous driving based on a second state of the other mobile 2 detected from the periphery video image after the first behavior.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus, an imaging apparatus, a moving body, and an image processing method.

Background Art

[0002] Conventionally, an in-vehicle camera is used to detect the state of a vehicle ahead, the state of a pedestrian, the state of the road ahead, etc., and to notify the driver only when it is possible or impossible to turn right, thereby providing the driver with truly necessary information. There is known a driving support apparatus (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, simply detecting the state of the vehicle ahead as in the prior art has a problem that, for example, it is impossible to accurately determine whether the vehicle ahead is performing aggressive driving with respect to the host vehicle. In the future, in a society where vehicles equipped with an automatic driving function become widespread, there is a concern that misjudgment of aggressive driving may become a factor hindering smooth traffic flow.

[0005] An object of the present disclosure is to provide an image processing apparatus that solves the above-described problems and can achieve smooth traffic flow.

Means for Solving the Problems

[0006] An image processing apparatus according to an embodiment of the present disclosure comprises a processor and an output unit. The processor detects the state of other moving objects from peripheral images captured around a moving object, and performs behavior determination processing to determine the behavior of the moving object based on the detected state of the other moving objects. The output unit outputs information to the moving object indicating the behavior of the moving object determined by the processor. As part of the behavior determination processing, the processor determines a first behavior of the moving object based on a first state of the other moving object detected from the peripheral images, and outputs information to the moving object via the output unit indicating the determined first behavior. Then, if the processor determines that the other moving object is driving dangerously based on a second state of the other moving object detected from the peripheral images after the first behavior by the moving object, it determines a second behavior of the moving object.

[0007] The imaging apparatus according to the embodiment of this disclosure comprises the image processing apparatus described above and an input unit for acquiring the peripheral image.

[0008] The mobile body according to the embodiment of this disclosure is equipped with the imaging device described above.

[0009] An image processing method according to an embodiment of the present disclosure is an image processing method performed by an image processing device, comprising: detecting the state of another moving object detected from a peripheral image captured around a moving object; determining a first behavior of the moving object based on a first state of the other moving object detected; outputting information to the moving object indicating the determined first behavior; and determining a second behavior of the moving object if it is determined that the other moving object is performing dangerous driving based on a second state of the other moving object detected from a peripheral image after the first behavior by the moving object. [Effects of the Invention]

[0010] According to this disclosure, it is possible to facilitate traffic flow. [Brief explanation of the drawing]

[0011] [Figure 1]This figure shows an example of the configuration of an image processing apparatus according to one embodiment of the present disclosure. [Figure 2A] This diagram shows an example of how your vehicle behaves when the other vehicle is engaging in the first type of aggressive driving against your vehicle. [Figure 2B] This diagram shows an example of how your vehicle behaves when the other vehicle is engaging in the first type of aggressive driving against your vehicle. [Figure 2C] This diagram shows an example of how your vehicle behaves when the other vehicle is engaging in the first type of aggressive driving against your vehicle. [Figure 3A] This diagram shows an example of how your vehicle behaves when the other vehicle is engaging in a second form of aggressive driving against your vehicle. [Figure 3B] This diagram shows an example of how your vehicle behaves when the other vehicle is engaging in a second form of aggressive driving against your vehicle. [Figure 3C] This diagram shows an example of how your vehicle behaves when the other vehicle is engaging in a second form of aggressive driving against your vehicle. [Figure 4A] This diagram shows an example of how your vehicle behaves when the other vehicle is engaging in a third type of aggressive driving against your vehicle. [Figure 4B] This diagram shows an example of how your vehicle behaves when the other vehicle is engaging in a third type of aggressive driving against your vehicle. [Figure 4C] This diagram shows an example of how your vehicle behaves when the other vehicle is engaging in a third type of aggressive driving against your vehicle. [Figure 5A] This figure shows an example of how your vehicle behaves when the other vehicle is engaging in a fourth type of aggressive driving against your vehicle. [Figure 5B] This figure shows an example of how your vehicle behaves when the other vehicle is engaging in a fourth type of aggressive driving against your vehicle. [Figure 5C] This figure shows an example of how your vehicle behaves when the other vehicle is engaging in a fourth type of aggressive driving against your vehicle. [Figure 6A]It is a diagram showing an example of the behavior of the host vehicle when the other vehicle performs the fifth provoking driving with respect to the host vehicle. [Figure 6B] It is a diagram showing an example of the behavior of the host vehicle when the other vehicle performs the fifth provoking driving with respect to the host vehicle. [Figure 6C] It is a diagram showing an example of the behavior of the host vehicle when the other vehicle performs the fifth provoking driving with respect to the host vehicle. [Figure 7A] It is a diagram showing an example of the behavior of the host vehicle when the other vehicle performs the sixth provoking driving with respect to the host vehicle. [Figure 7B] It is a diagram showing an example of the behavior of the host vehicle when the other vehicle performs the sixth provoking driving with respect to the host vehicle. [Figure 7C] It is a diagram showing an example of the behavior of the host vehicle when the other vehicle performs the sixth provoking driving with respect to the host vehicle. [Figure 8] It is a flowchart showing an example of an image processing method according to an embodiment of the present disclosure. [Figure 9] It is a diagram showing an example of a state transition diagram. [Figure 10] It is a flowchart showing an example of the first provoking driving determination process in the image processing method according to an embodiment of the present disclosure. [Figure 11] It is a flowchart showing an example of the second provoking driving determination process in the image processing method according to an embodiment of the present disclosure. [Figure 12] It is a flowchart showing an example of the third provoking driving determination process in the image processing method according to an embodiment of the present disclosure. [Figure 13] It is a flowchart showing an example of the fourth provoking driving determination process in the image processing method according to an embodiment of the present disclosure. [Figure 14] It is a flowchart showing an example of the fifth provoking driving determination process in the image processing method according to an embodiment of the present disclosure. [Figure 15] It is a flowchart showing an example of the sixth provoking driving determination process in the image processing method according to an embodiment of the present disclosure. [Modes for carrying out the invention]

[0012] Embodiments of the present disclosure will be described below with reference to the drawings. In each figure, the same reference numerals indicate the same or equivalent components.

[0013] In this specification, "aggressive driving" will be explained by giving six specific examples. "Aggressive driving" is an act in which the driver of another vehicle, for some reason or purpose, provokes the driver of one's own vehicle and obstructs the smooth flow of traffic. The first type of aggressive driving is "driving in a manner that shortens the distance between the vehicle being driven and the vehicle being driven by another vehicle." The second type of aggressive driving is "driving in a manner that threatens one's own vehicle with the horn of another vehicle." The third type of aggressive driving is "driving in a manner that swerves behind one's own vehicle." The fourth type of aggressive driving is "driving in a manner that flashes the headlights of another vehicle behind one's own vehicle." The fifth type of aggressive driving is "driving in a manner that tailgates or chases after one's own vehicle." The sixth type of aggressive driving is "driving in a manner that cuts off one's own vehicle."

[0014] <Configuration of the image processing device> Referring to Figure 1, the configuration of the image processing apparatus 10 according to this embodiment will be described. Figure 1 is a diagram showing an example of the configuration of the image processing apparatus 10 according to this embodiment.

[0015] As shown in Figure 1, the image processing device 10 is mounted on a mobile body 1 (for example, its own vehicle) and detects the state of other mobile bodies (for example, another vehicle) from surrounding images captured around the mobile body 1. Based on the detected state of the other mobile bodies, the image processing device 10 determines the action to be performed by the mobile body 1. The image processing device 10 outputs information to the mobile body 1 instructing it on the action to be performed. In addition to the imaging device 1A, the mobile body 1 further includes, for example, a vehicle speed sensor, a distance measuring sensor, a microphone, radar, sonar, and lidar.

[0016] The mobile device 1 may be, for example, a vehicle equipped with an autonomous driving function. In this embodiment, "autonomous driving" includes automating some or all of the user operations for driving the vehicle. For example, autonomous driving may include levels 1 to 5 as defined by the SAE (Society of Automotive Engineers). Hereafter, the mobile device 1 will be described as having a fully autonomous driving function of level 4 or higher as defined by the SAE.

[0017] The image processing device 10 includes a communication interface 12 as an output unit, a processor 13, and a storage unit 14. The mobile body 1 is equipped with an imaging device 1A which includes the image processing device 10 and an imaging unit 11 as an input unit.

[0018] The imaging unit 11 acquires peripheral video footage of the area around the mobile body 1 and outputs the acquired peripheral video footage to the processor 13. The imaging unit 11 is, for example, an in-vehicle camera mounted on the mobile body 1. Multiple imaging units 11 may be mounted on the mobile body 1. If four in-vehicle cameras are mounted on the mobile body 1, for example, the first in-vehicle camera is positioned to capture the peripheral area in front of the mobile body 1 and at least a portion of the front side of the mobile body 1. For example, the second in-vehicle camera is positioned to capture the peripheral area behind the mobile body 1 and at least a portion of the rear side of the mobile body 1. For example, the third in-vehicle camera is positioned to capture the peripheral area to the left of the mobile body 1 and at least a portion of the left side of the mobile body 1. For example, the fourth in-vehicle camera is positioned to capture the peripheral area to the right of the mobile body 1 and at least a portion of the right side of the mobile body 1. With the imaging unit 11 positioned in this manner, it becomes possible for the imaging unit 11 to image the surrounding area on all four sides of the moving body 1.

[0019] The imaging unit 11 comprises at least an imaging optical system and an image sensor.

[0020] The imaging optical system includes, for example, one or more lenses and optical components such as an aperture. The lenses in the imaging optical system are, for example, wide-angle lenses such as fisheye lenses. The imaging optical system forms an image of the subject on the light-receiving surface of the image sensor.

[0021] The image sensor includes, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide Semiconductor) image sensor. Multiple pixels are arranged on the light-receiving surface of the image sensor. The image sensor captures the image of a subject formed on the light-receiving surface and generates an image. The imaging unit 11 outputs the image generated by the image sensor to the processor 13 via wired or wireless connection. The imaging unit 11 may also output the image to external devices such as an ECU (Electronic Control Unit), display, and navigation device mounted on the mobile body 1. The imaging unit 11 may also have a function to perform predetermined image processing on the image, such as white balance adjustment, exposure adjustment, and gamma correction.

[0022] The communication interface 12 is an interface that communicates with various control systems of the mobile unit 1 via wired or wireless connection. The communication interface 12 communicates with control systems that control the movement of the mobile unit 1, control systems that control functions other than the movement of the mobile unit 1 (for example, turning lights on and off, flashing turn signals, sounding the horn, etc.).

[0023] The storage unit 14 includes, for example, a primary storage device or a secondary storage device. The storage unit 14 stores various information and programs necessary for the operation of the image processing device 10.

[0024] The processor 13 includes, for example, a dedicated processor such as a DSP (Digital Signal Processor) or a general-purpose processor such as a CPU (Central Processing Unit). The processor 13 controls the operation of the entire image processing device 10. For example, the processor 13 detects the state of other moving objects from the surrounding image captured by the imaging unit 11 and performs behavior determination processing to determine the behavior of the moving object 1 based on the detected state of the other moving objects. Examples of the states of other moving objects detected by the processor 13 include a first state, a second state, a third state, and so on.

[0025] The processor 13 detects one of the following as a first state of the other moving object: for example, closing the distance to moving object 1, honking the horn to intimidate moving object 1, swerving behind moving object 1, flashing its headlights behind moving object 1, sticking close to or chasing moving object 1, or cutting off moving object 1. The processor 13 detects one of the following as a second state of the other moving object: for example, a continuation of the first state, or a repetition of the first state. The processor 13 detects one of the following as a third state of the other moving object: for example, a continuation of the first state, or a repetition of the first state.

[0026] The processor 13 accesses various control systems of the mobile body 1 via the communication interface 12 and determines the behavior of the mobile body 1. Examples of the behavior of the mobile body 1 determined by the processor 13 include the first behavior, the second behavior, and the third behavior.

[0027] The processor 13 determines one of the following as the first behavior of the mobile body 1: for example, continuing to drive at the legal speed limit, changing lanes, sounding the horn to warn other mobile bodies, pulling over to the nearest convenience store, or pulling over to the nearest service area or parking area. For example, the processor 13 accesses the control system that controls the driving of the mobile body 1 and determines the first behavior of the mobile body 1, such as continuing to drive at the legal speed limit, changing lanes, pulling over to the nearest convenience store, or pulling over to the nearest service area or parking area. For example, the processor 13 accesses the control system that controls everything except the driving of the mobile body 1 and determines the first behavior of the mobile body 1, such as sounding the horn to warn other mobile bodies.

[0028] The processor 13 determines one of the following as the second behavior of the mobile vehicle 1: for example, reporting to the police, stopping, slowing down, taking refuge in the nearest convenience store, taking refuge in the nearest service area or parking area, recording to the drive recorder, uploading surrounding video to a predetermined network, or displaying that another mobile vehicle is driving dangerously. The second behavior is the behavior that the mobile vehicle 1 takes when the first state continues or is repeated by another mobile vehicle even after the mobile vehicle 1 has performed the first behavior, and is also called a "danger avoidance behavior".

[0029] For example, the processor 13 accesses the control system that controls the movement of the mobile vehicle 1 and determines a second action for the mobile vehicle 1, such as stopping, slowing down, pulling over to the nearest convenience store, or pulling over to the nearest service area or parking area. For example, the processor 13 accesses the control system that controls actions other than the movement of the mobile vehicle 1 and determines a second action for the mobile vehicle 1, such as recording to the drive recorder or displaying a message that another mobile vehicle is driving dangerously.

[0030] The processor 13 determines one of the following as the third behavior of the mobile body 1: for example, calling the police, stopping, taking refuge in the nearest convenience store, taking refuge in the nearest service area or parking area, or locking the doors after stopping. The third behavior is the behavior that mobile body 1 performs when the first state continues or is repeated by another mobile body even after mobile body 1 has performed the second behavior, and is also called an "emergency evacuation behavior".

[0031] For example, the processor 13 accesses the control system that controls the movement of the mobile body 1 and determines a third action for the mobile body 1, such as stopping, pulling over to the nearest convenience store, or pulling over to the nearest service area or parking area. For example, the processor 13 accesses the control system that controls actions other than the movement of the mobile body 1 and determines a third action for the mobile body 1, such as locking the doors after stopping.

[0032] The processor 13 performs the following operations as a behavior determination process. The processor 13 detects the state of other moving objects from the surrounding image captured by the imaging unit 11, and determines the first behavior of the moving object 1 based on the first state of the detected other moving objects. The processor 13 outputs information instructing the determined first behavior to the control system of the moving object 1 via the communication interface 12.

[0033] For example, the processor 13 determines the first behavior of the mobile vehicle 1 and outputs information to the mobile vehicle 1 via the communication interface 12 instructing it to continue driving at the legal speed limit, change lanes, sound the horn to warn other mobile vehicles, pull over to the nearest convenience store, pull over to the nearest service area or parking area, etc., and causes the mobile vehicle 1 to perform these actions.

[0034] Furthermore, after the first action by the mobile body 1, the processor 13 detects the state of other mobile bodies from the surrounding image captured by the imaging unit 11, and, based on the second state of the other mobile bodies detected, determines that the other mobile bodies are driving dangerously, and then determines the second action of the mobile body 1. The processor 13 outputs information instructing the determined second action to the control system of the mobile body 1 via the communication interface 12.

[0035] For example, the processor 13 determines the second behavior of the mobile vehicle 1 and outputs information to the mobile vehicle 1 via the communication interface 12 instructing it to report to the police, stop, slow down, pull over to the nearest convenience store, pull over to the nearest service area or parking area, record to the drive recorder, upload surrounding video to a predetermined network, display that another mobile vehicle is driving dangerously, etc., causing the mobile vehicle 1 to perform these actions.

[0036] In other words, if the processor 13 still determines that another vehicle is driving dangerously even after the first behavior of vehicle 1, it estimates that the other vehicle is driving aggressively. Then, in order to more accurately detect whether or not the other vehicle is driving aggressively, the processor 13 determines a second behavior of vehicle 1.

[0037] Furthermore, after the second action by the mobile body 1, the processor 13 detects the state of other mobile bodies from the surrounding image captured by the imaging unit 11, and if it determines that the other mobile bodies are driving dangerously based on the detected third state of the other mobile bodies, it determines the third action of the mobile body 1. The processor 13 outputs information instructing the determined third action to the control system of the mobile body 1 via the communication interface 12.

[0038] For example, the processor 13 determines a third action for the mobile vehicle 1 and outputs information to the mobile vehicle 1 via the communication interface 12 instructing it to report to the police, stop, take refuge in the nearest convenience store, take refuge in the nearest service area or parking area, lock the doors after stopping, etc., causing the mobile vehicle 1 to perform these actions.

[0039] In other words, if the processor 13 still determines that another vehicle is driving dangerously even after the second behavior of vehicle 1, it estimates that the other vehicle is driving aggressively. Then, in order to more accurately detect whether or not the other vehicle is driving aggressively, the processor 13 determines a third behavior of vehicle 1.

[0040] As described above, after the first action by the moving body 1, the processor 13 determines, based on the second state of other moving bodies detected from the surrounding image captured by the imaging unit 11, that the other moving bodies are engaging in dangerous driving, and determines the second action of the moving body 1. Furthermore, after the second action by the moving body 1, the processor 13 determines, based on the third state of other moving bodies detected from the surrounding image captured by the imaging unit 11, that the other moving bodies are engaging in dangerous driving, and determines the third action of the moving body 1.

[0041] According to the image processing device 10 of this embodiment, a first state of other moving objects is detected from surrounding video footage captured around the moving object 1, and a first behavior of the moving object 1 is determined based on the first state. After the first behavior of the moving object 1, a second state of other moving objects is detected from surrounding video footage captured around the moving object 1, and if it is determined that the other moving object is engaging in dangerous driving based on the second state, a second behavior of the moving object 1 is determined. In other words, by determining whether or not the other moving object is engaging in dangerous driving even after the first behavior of the moving object 1, it is possible to accurately determine whether or not the other moving object is engaging in aggressive driving towards the moving object. That is, compared to the conventional method, which simply detected the state of other moving objects to determine whether or not the other moving object is engaging in aggressive driving towards the moving object, the accuracy of the aggressive driving determination can be improved. As a result, the moving object 1 can take appropriate measures against other moving objects that are actually engaging in aggressive driving, thereby improving traffic flow.

[0042] <Specific examples of the behavior of mobile object 1> Next, with reference to Figures 2 to 7, specific examples of the behavior of mobile body 1 when another mobile body 2 is tailgating mobile body 1 will be described.

[0043] [In the case of the first instance of aggressive driving] As shown in Figure 2A, the image processing device 10 uses an imaging unit 11 positioned to capture the area behind the moving object 1, thereby capturing the area behind the moving object 1. The image processing device 10 also calculates the distance between the moving object 1 and other moving objects 2.

[0044] The image processing device 10 estimates that the other mobile vehicle 2 is engaging in a first type of aggressive driving based on the surrounding video footage behind the mobile vehicle 1 and the distance between the mobile vehicle 1 and the other mobile vehicle 2. In this case, as shown in Figure 2B, the image processing device 10 determines that the first behavior of the mobile vehicle 1 is "lane change" and outputs information instructing the mobile vehicle 1 to perform this first behavior.

[0045] Subsequently, based on the surrounding video footage behind the moving body 1 and the distance between the moving body 1 and the other moving body 2, the image processing device 10 determines that the first aggressive driving by the other moving body 2 is still continuing even after the first action of the moving body 1, and then determines the second action of the moving body 1. In this case, as shown in Figure 2C, the image processing device 10 determines that the second action of the moving body 1 is to "take refuge in the nearest convenience store" and outputs information to the moving body 1 instructing this second action.

[0046] [In the case of the second instance of aggressive driving] As shown in Figure 3A, the image processing device 10 uses an imaging unit 11 positioned to capture the area behind the mobile body 1, thereby capturing the area behind the mobile body 1. The image processing device 10 also uses a microphone mounted on the mobile body 1 to detect the sounding of the horn of another mobile body 2.

[0047] The image processing device 10 estimates that the other mobile vehicle 2 is engaging in a second act of aggressive driving based on the surrounding video footage behind the mobile vehicle 1 and the sounding of the other mobile vehicle 2's horn. In this case, as shown in Figure 3B, the image processing device 10 determines that the first action of the mobile vehicle 1 is to "change lanes" and outputs information to the mobile vehicle 1 instructing it to perform this first action.

[0048] Subsequently, based on the surrounding video footage behind the moving body 1 and the sounding of the horn of the other moving body 2, the image processing device 10 determines that the first aggressive driving by the other moving body 2 is still continuing even after the first action of the moving body 1, and then determines the second action of the moving body 1. In this case, as shown in Figure 3C, the image processing device 10 determines that the second action of the moving body 1 is to "take refuge in the nearest convenience store" and outputs information to the moving body 1 instructing this second action.

[0049] [In the case of the third type of aggressive driving] As shown in Figure 4A, the image processing device 10 uses an imaging unit 11 positioned to capture the area behind the moving object 1, thereby capturing the area behind the moving object 1. The image processing device 10 also calculates the distance between the moving object 1 and other moving objects 2.

[0050] The image processing device 10 estimates that the other mobile vehicle 2 is engaging in a third type of aggressive driving based on the surrounding video footage behind the mobile vehicle 1 and the distance between the mobile vehicle 1 and the other mobile vehicle 2. In this case, as shown in Figure 4B, the image processing device 10 determines that the first behavior of the mobile vehicle 1 is "lane change" and outputs information to the mobile vehicle 1 instructing it to perform this first behavior.

[0051] Subsequently, based on the surrounding video footage behind the moving body 1 and the distance between the moving body 1 and the other moving body 2, the image processing device 10 determines that the first aggressive driving by the other moving body 2 is still continuing even after the first action of the moving body 1, and then determines the second action of the moving body 1. In this case, as shown in Figure 4C, the image processing device 10 determines that the second action of the moving body 1 is to "take refuge in the nearest convenience store" and outputs information to the moving body 1 instructing this second action.

[0052] [In the case of the fourth type of aggressive driving] As shown in Figure 5A, the image processing device 10 uses an imaging unit 11 positioned to capture the area behind the moving body 1, thereby capturing the area behind the moving body 1. The image processing device 10 also calculates the distance between the moving body 1 and the other moving body 2. Furthermore, based on the surrounding image captured by the imaging unit 11, the image processing device 10 detects whether the other moving body 2 is flashing its front lights behind the moving body 1, or whether the other moving body 2 is illuminating its front lights upwards at high brightness behind the moving body 1.

[0053] The image processing device 10 estimates that the other mobile vehicle 2 is engaging in a fourth type of aggressive driving based on the surrounding video footage behind the mobile vehicle 1, the distance between the mobile vehicle 1 and the other mobile vehicle 2, and the appearance of the other mobile vehicle 2's headlights. In this case, as shown in Figure 5B, the image processing device 10 determines that the first behavior of the mobile vehicle 1 is "lane change" and outputs information to the mobile vehicle 1 instructing it to perform this first behavior.

[0054] Subsequently, based on the surrounding video footage behind the moving body 1, the distance between the moving body 1 and the other moving body 2, and the appearance of the other moving body 2's headlights, the image processing device 10 determines that the first aggressive driving by the other moving body 2 is still continuing even after the first action of the moving body 1, and then determines the second action of the moving body 1. In this case, as shown in Figure 5C, the image processing device 10 determines that the second action of the moving body 1 is to "take refuge in the nearest convenience store," and outputs information to the moving body 1 instructing this second action.

[0055] [In the case of the fifth type of aggressive driving] As shown in Figure 6A, the image processing device 10 uses an imaging unit 11 positioned to capture the area behind the moving object 1, thereby capturing the area behind the moving object 1. The image processing device 10 also calculates the distance between the moving object 1 and other moving objects 2.

[0056] The image processing device 10 estimates that the other mobile vehicle 2 is engaging in a fifth type of aggressive driving based on the surrounding video footage behind the mobile vehicle 1 and the distance between the mobile vehicle 1 and the other mobile vehicle 2. In this case, as shown in Figure 6B, the image processing device 10 determines that the first behavior of the mobile vehicle 1 is "lane change" and outputs information to the mobile vehicle 1 instructing it to perform this first behavior.

[0057] Subsequently, based on the surrounding video footage behind the moving body 1 and the distance between the moving body 1 and the other moving body 2, if the image processing device 10 determines that the fifth aggressive driving by the other moving body 2 is continuing even after the first action of the moving body 1, it determines the second action of the moving body 1. In this case, as shown in Figure 6C, the image processing device 10 determines that the second action of the moving body 1 is to "take refuge in the nearest convenience store" and outputs information to the moving body 1 instructing this second action.

[0058] Subsequently, based on the surrounding video of the mobile body 1 and the distance between the mobile body 1 and the other mobile body 2, the image processing device 10 determines that the fifth aggressive driving by the other mobile body 2 is still continuing even after the second action of the mobile body 1, and then determines the third action of the mobile body 1. In this case, as shown in Figure 6C, the image processing device 10 determines that the third action of the mobile body 1 is to "lock the doors and call the police," and outputs information to the mobile body 1 instructing it to perform this third action.

[0059] [In the case of the sixth type of aggressive driving] As shown in Figure 7A, the image processing device 10 uses an imaging unit 11 positioned to capture images of the rear peripheral area of ​​the mobile body 1, at least a portion of the rear side of the mobile body 1, the right peripheral area of ​​the mobile body 1, and at least a portion of the right side of the mobile body 1. The imaging unit 11 captures images of the rear, rear side, right side, and right side peripheral areas of the mobile body 1. The image processing device 10 also calculates the distance between the mobile body 1 and other mobile bodies 2.

[0060] The image processing device 10 estimates that the other mobile vehicle 2 is engaging in a sixth type of aggressive driving based on the surrounding video footage behind the mobile vehicle 1, the surrounding video footage to the right of the mobile vehicle 1, and the distance between the mobile vehicle 1 and the other mobile vehicle 2. In this case, as shown in Figure 7B, the image processing device 10 determines that the first action of the mobile vehicle 1 is to "honk the horn to warn the other mobile vehicle," and outputs information to the mobile vehicle 1 instructing it to perform this first action.

[0061] Subsequently, based on the surrounding video footage behind the moving body 1, the surrounding video footage to the right of the moving body 1, and the distance between the moving body 1 and the other moving body 2, the image processing device 10 determines that the sixth aggressive driving by the other moving body 2 is still continuing even after the first action of the moving body 1, and determines the second action of the moving body 1. In this case, as shown in Figure 7C, the image processing device 10 determines "lane change, deceleration" as the second action of the moving body 1 and outputs information instructing the second action to the moving body 1.

[0062] Subsequently, based on the surrounding video footage behind the moving body 1 and the distance between the moving body 1 and the other moving body 2, the image processing device 10 determines that the sixth aggressive driving by the other moving body 2 is still continuing even after the second action of the moving body 1, and determines the third action of the moving body 1. In this case, the image processing device 10 determines "stopping to avoid a collision" as the third action and outputs information instructing this action to the moving body 1.

[0063] Table 1 is a table summarizing the first state and second state of other mobile bodies 2 detected by the processor 13 based on the first and second behaviors of mobile body 1 determined by the processor 13, as well as surrounding video footage, etc., when driving on a public road.

[0064] [Table 1]

[0065] For example, if the first state of the other moving vehicle 2 is "the other moving vehicle 2 is closing the distance between itself and moving vehicle 1", the processor 13 will decide on the first behavior of moving vehicle 1 to be either "continue driving while adhering to the legal speed limit" or "check the safety of the lane to which it will change lanes and change lanes".

[0066] Furthermore, if the second state of the other mobile object 2 is either "the first state of the other mobile object 2 continues" or "the other mobile object 2 sticks to mobile object 1 and chases it," the processor 13 decides that the second behavior of mobile object 1 is to "stop at the nearest convenience store or similar place and yield the right of way."

[0067] For example, if the first state of the other moving object 2 is "the other moving object 2 honks its horn at the moving object 1", the processor 13 decides on the first behavior of the moving object 1 to be either "continue driving while adhering to the speed limit" or "check the safety of the lane to which it will change lanes and change lanes".

[0068] Furthermore, if the second state of the other mobile object 2 is either "the first state of the other mobile object 2 continues" or "the other mobile object 2 sticks to mobile object 1 and chases it," the processor 13 decides that the second behavior of mobile object 1 is to "stop at the nearest convenience store or similar place and yield the right of way."

[0069] For example, if the first state of the other moving object 2 is "the other moving object 2 is driving erratically behind the moving object 1", the processor 13 will decide on the first behavior of the moving object 1 to be either "continue driving while adhering to the legal speed limit" or "check the safety of the lane to which it will change lanes and change lanes".

[0070] Furthermore, if the second state of the other mobile object 2 is either "the first state of the other mobile object 2 continues" or "the other mobile object 2 sticks to mobile object 1 and chases it," the processor 13 decides that the second behavior of mobile object 1 is to "stop at the nearest convenience store or similar place and yield the right of way."

[0071] For example, if the first state of the other moving object 2 is "the other moving object 2 flashes its headlights at the moving object 1", the processor 13 will decide on the first behavior of the moving object 1 to be either "continue driving while adhering to the speed limit" or "check the safety of the lane to which it will change lanes and change lanes".

[0072] Furthermore, if the second state of the other mobile object 2 is either "the first state of the other mobile object 2 continues" or "the other mobile object 2 sticks to mobile object 1 and chases it," the processor 13 decides that the second behavior of mobile object 1 is to "stop at the nearest convenience store or similar place and yield the right of way."

[0073] For example, if the first state of the other moving object 2 is "the other moving object 2 is sticking to and chasing the moving object 1", the processor 13 decides that the first behavior of the moving object 1 is either "to stop at the nearest convenience store or similar place and yield the right of way" or "to check the safety of the lane to which it will change lanes and change lanes".

[0074] Furthermore, if the second state of the other mobile body 2 is "the first state of the other mobile body 2 continues," the processor 13 decides that the second behavior of mobile body 1 is "to stop at the nearest convenience store or similar place and notify the police."

[0075] For example, if the first state of the other moving object 2 is "the other moving object 2 is moving too close to the moving object 1", the processor 13 decides that the first behavior of the moving object 1 is "to sound the horn to warn the other moving object 2".

[0076] Furthermore, if the second state of the other moving body 2 is either "the other moving body 2 stops the cutting action" or "the other moving body 2 continues the cutting action," the processor 13 decides on the second behavior of moving body 1 to be either "continue driving" or "decelerate to avoid a collision."

[0077] Table 2 is a table summarizing the first state and second state of other mobile bodies 2 detected by the processor 13 based on the first and second behaviors of mobile body 1 determined by the processor 13, surrounding video footage, etc., when driving on a highway.

[0078] [Table 2]

[0079] For example, if the first state of the other moving vehicle 2 is "the other moving vehicle 2 is closing the distance between itself and moving vehicle 1", the processor 13 will decide on the first behavior of moving vehicle 1 to be either "continue driving while adhering to the legal speed limit" or "check the safety of the lane to which it will change lanes and change lanes".

[0080] Furthermore, if the second state of the other mobile body 2 is either "the first state of the other mobile body 2 continues" or "the other mobile body 2 clings to and chases mobile body 1", the processor 13 decides that the second behavior of mobile body 1 is to "take refuge in a safe place such as a service area or parking area".

[0081] For example, if the first state of the other moving object 2 is "the other moving object 2 honks its horn at the moving object 1", the processor 13 decides on the first behavior of the moving object 1 to be either "continue driving while adhering to the speed limit" or "check the safety of the lane to which it will change lanes and change lanes".

[0082] Furthermore, if the second state of the other mobile body 2 is either "the first state of the other mobile body 2 continues" or "the other mobile body 2 clings to and chases mobile body 1", the processor 13 decides that the second behavior of mobile body 1 is to "take refuge in a safe place such as a service area or parking area".

[0083] For example, if the first state of the other moving object 2 is "the other moving object 2 is driving erratically behind the moving object 1", the processor 13 will decide on the first behavior of the moving object 1 to be either "continue driving while adhering to the legal speed limit" or "check the safety of the lane to which it will change lanes and change lanes".

[0084] Furthermore, if the second state of the other mobile body 2 is either "the first state of the other mobile body 2 continues" or "the other mobile body 2 clings to and chases mobile body 1", the processor 13 decides that the second behavior of mobile body 1 is to "take refuge in a safe place such as a service area or parking area".

[0085] For example, if the first state of the other moving object 2 is "the other moving object 2 flashes its headlights at the moving object 1", the processor 13 will decide on the first behavior of the moving object 1 to be either "continue driving while adhering to the speed limit" or "check the safety of the lane to which it will change lanes and change lanes".

[0086] Furthermore, if the second state of the other mobile body 2 is either "the first state of the other mobile body 2 continues" or "the other mobile body 2 clings to and chases mobile body 1", the processor 13 decides that the second behavior of mobile body 1 is to "take refuge in a safe place such as a service area or parking area".

[0087] For example, if the first state of the other moving object 2 is "the other moving object 2 is clinging to and chasing the moving object 1", the processor 13 will decide that the first behavior of the moving object 1 is either "enter the nearest service area or parking area" or "check the safety of the lane to change lanes and change lanes".

[0088] Furthermore, if the second state of the other mobile body 2 is "the first state of the other mobile body 2 continues," the processor 13 decides on the second behavior of mobile body 1 to be either "report to the police" or "enter the nearest service area or parking area and report to the police."

[0089] For example, if the first state of the other moving object 2 is "the other moving object 2 is moving too close to the moving object 1", the processor 13 decides that the first behavior of the moving object 1 is "to sound the horn to warn the other moving object 2".

[0090] Furthermore, if the second state of the other moving body 2 is either "the other moving body 2 stops the cutting action" or "the other moving body 2 continues the cutting action," the processor 13 decides on the second behavior of moving body 1 to be either "continue driving" or "decelerate to avoid a collision."

[0091] As described above, the image processing device 10 detects a first state of other mobile bodies 22 from surrounding video footage captured around the mobile body 1, and determines a first behavior of mobile body 1 based on the first state. Then, after the first behavior of mobile body 1, the image processing device 10 detects a second state of other mobile bodies 22 from surrounding video footage captured around the mobile body 1, and if it determines that the other mobile body 22 is driving dangerously based on the second state, it determines a second behavior of mobile body 1. In other words, by determining whether the other mobile body 22 is still driving dangerously even after the first behavior of mobile body 1, the image processing device 10 can accurately determine whether the other mobile body 2 is driving aggressively towards the mobile body 1. As a result, mobile body 1 can take appropriate measures against the other mobile body 22 that is actually driving aggressively, thereby improving traffic flow.

[0092] <Operation of the image processing device> Next, with reference to Figure 8, an image processing method in the image processing apparatus 10 according to this embodiment will be described. Figure 8 is a flowchart showing an example of an image processing method in the image processing apparatus 10 according to this embodiment.

[0093] Step S0: Processor 13 performs initialization processing.

[0094] Step S1: Processor 13 performs a first aggressive driving determination process. Information used by Processor 13 for the first aggressive driving determination process includes, for example, the vehicle speed of vehicle 1, the distance between vehicle 1 and other vehicle 2 according to the vehicle speed of vehicle 1 (see Table 3), and the duration for which other vehicle 2 remains within a predetermined range of vehicle 1.

[0095] [Table 3]

[0096] For example, if another mobile body 2 remains within a predetermined range of mobile body 1 for 3 seconds or more, the processor 13 determines that the other mobile body 2 is performing the first aggressive driving. For example, if another mobile body 2 remains within a predetermined range of mobile body 1 for less than 3 seconds, the processor 13 determines that the other mobile body 2 is not performing the first aggressive driving.

[0097] Step S2: Processor 13 performs a second aggressive driving determination process. Information used by Processor 13 for the second aggressive driving determination process includes, for example, the vehicle speed of vehicle 1, the distance between vehicle 1 and other vehicle 2 according to the vehicle speed of vehicle 1 (see Table 3), the number of times and volume of the horn sound of other vehicle 2.

[0098] For example, if the other mobile vehicle 2 honks its horn five or more times per minute, the processor 13 determines that the other mobile vehicle 2 is engaging in a second type of aggressive driving. For example, if the other mobile vehicle 2 honks its horn less than five times per minute, the processor 13 determines that the other mobile vehicle 2 is not engaging in a second type of aggressive driving.

[0099] Step S3: Processor 13 performs a third aggressive driving determination process. Information used by Processor 13 for the third aggressive driving determination process includes, for example, the vehicle speed of vehicle 1, the distance between vehicle 1 and other vehicle 2 according to the vehicle speed of vehicle 1 (see Table 3), the time that other vehicle 2 stays within a predetermined range of vehicle 1, the amplitude of the serpentine driving of other vehicle 2, and the period of the serpentine driving of other vehicle 2.

[0100] For example, processor 13 determines that the other mobile body 2 is engaging in a third type of aggressive driving if the amplitude of the serpentine driving of the other mobile body 2 is 900 mm or more, and the period of the serpentine driving of the other mobile body 2 is between 1 second and 5 seconds. For example, processor 13 determines that the other mobile body 2 is not engaging in a third type of aggressive driving if the amplitude of the serpentine driving of the other mobile body 2 is less than 900 mm, or if the period of the serpentine driving of the other mobile body 2 is longer than 5 seconds. Note that 900 mm is approximately half the width of a standard vehicle, which is 1800 mm.

[0101] Step S4: Processor 13 performs a fourth aggressive driving determination process. Information used by Processor 13 for the fourth aggressive driving determination process includes, for example, the vehicle speed of vehicle 1, the distance between vehicle 1 and other vehicle 2 according to the vehicle speed of vehicle 1 (see Table 3), the time that other vehicle 2 stays within a predetermined range of vehicle 1, the illumination of the headlights of other vehicle 2, the frequency of flashing of the headlights of other vehicle 2, and the brightness and darkness cycle of the flashing of the headlights of other vehicle 2.

[0102] For example, if the illuminance of the front lights of the other mobile vehicle 2 is 10,000 lx or more, the processor 13 determines that the other mobile vehicle 2 is engaging in the fourth type of aggressive driving. For example, if the illuminance of the front lights of the other mobile vehicle 2 is less than 10,000 lx, the processor 13 determines that the other mobile vehicle 2 is not engaging in the fourth type of aggressive driving.

[0103] For example, processor 13 determines that other mobile vehicle 2 is engaging in the fourth type of aggressive driving if the flashing cycle of the other mobile vehicle 2 is between 500 milliseconds and 2 seconds. For example, processor 13 determines that other mobile vehicle 2 is not engaging in the fourth type of aggressive driving if the flashing cycle of the other mobile vehicle 2 is longer than 2 seconds.

[0104] Step S5: Processor 13 performs a fifth aggressive driving determination process. Information used by Processor 13 for the fifth aggressive driving determination process includes, for example, the vehicle speed of vehicle 1, the distance between vehicle 1 and other vehicle 2 according to the vehicle speed of vehicle 1 (see Table 3), the duration for which other vehicle 2 stays within a predetermined range of vehicle 1, and the number of times other vehicle 2 engages in tailgating and chasing.

[0105] For example, if another mobile body 2 continues to tailgate and follow another mobile body 1 two or more times after the mobile body 1 has changed lanes, the processor 13 determines that the other mobile body 2 is engaging in a fifth type of aggressive driving. For example, if the other mobile body 2 does not tailgate or follow another mobile body 1 after the mobile body 1 has changed lanes, the processor 13 determines that the other mobile body 2 is not engaging in a fifth type of aggressive driving.

[0106] Step S6: Processor 13 performs a sixth aggressive driving determination process. Information used by Processor 13 for the sixth aggressive driving determination process includes, for example, the vehicle speed of vehicle 1, the distance between vehicle 1 and other vehicle 2 according to the vehicle speed of vehicle 1 (see Table 3), the distance between vehicle 1 and other vehicle 2 along the side, and the number of times other vehicle 2 has cut off vehicle 1.

[0107] For example, if the distance between the side of mobile body 1 and the other mobile body 2 is less than 2m, the processor 13 determines that the other mobile body 2 is engaging in the sixth type of aggressive driving. For example, if the distance between the side of mobile body 1 and the other mobile body 2 is 2m or more, the processor 13 determines that the other mobile body 2 is not engaging in the sixth type of aggressive driving.

[0108] For example, if another mobile body 2 continues to cut off another mobile body 1 two or more times after the first mobile body 1 has changed lanes, the processor 13 determines that the second mobile body 2 is engaging in the sixth type of aggressive driving. For example, if the second mobile body 2 does not cut off another mobile body 1 after the first mobile body 1 has changed lanes, the processor 13 determines that the second mobile body 2 is not engaging in the sixth type of aggressive driving.

[0109] The processor 13 executes the aggressive driving detection process in steps S1 to S6 described above in parallel. The processor 13 may execute the aggressive driving detection process in a single step, or it may execute the aggressive driving detection process in a combination of steps. By executing the aggressive driving detection process in a combination of steps, it becomes possible to more accurately determine whether or not the other mobile body 2 is engaging in aggressive driving.

[0110] Next, we will briefly explain the provocation detection state flag with reference to Figure 9. Figure 9 is a diagram showing an example of a state transition diagram for the provocation detection state flag.

[0111] As shown in Figure 9, the aggressive driving detection status flag has three states, for example, "none," "provisionally confirmed," and "confirmed." The processor 13 manages the aggressive driving detection status flag and executes aggressive driving detection processing as described in steps S1 to S6 above.

[0112] State A is the "none" state, that is, the initial state.

[0113] State B is a "provisionally confirmed" state. In state B, if processor 13 estimates that aggressive driving is occurring based on the first state of the other mobile body 2, it outputs an action instruction to mobile body 1. In state B, if processor 13 estimates that aggressive driving is not occurring based on the first state of the other mobile body 2, it does not output an action instruction to mobile body 1 (state B → state A).

[0114] State C is the "confirmed" state. In state C, if processor 13 estimates that aggressive driving is occurring based on the second state of the other mobile body 2, it outputs a hazard avoidance action instruction to mobile body 1. In state C, if processor 13 estimates that aggressive driving is not occurring based on the second state of the other mobile body 2, it does not output a hazard avoidance action instruction to mobile body 1 (state C → state A).

[0115] According to the image processing method of this embodiment, the aggressive driving determination status flag has three states. For example, in the "provisionally determined" state, the processor 13 outputs an action instruction to the mobile body 1. For example, in the "determined" state, the processor 13 outputs a danger avoidance action instruction to the mobile body 1. That is, according to the image processing method of this embodiment, instead of determining whether the other mobile body 2 is engaging in aggressive driving towards the mobile body 1 based solely on the first state of the other mobile body 2, the second state of the other mobile body 2 after the first behavior of the mobile body 1 is considered to determine whether the other mobile body 2 is engaging in aggressive driving towards the mobile body 1. This suppresses misjudgments of aggressive driving and reduces the likelihood of unnecessary disputes occurring between the mobile body 1 and the other mobile body 2, thereby contributing to smoother traffic flow.

[0116] <Specific examples of aggressive driving detection processes> Next, with reference to Figures 10 to 15, a specific example of the aggressive driving detection process according to this embodiment will be described.

[0117] [First aggressive driving detection process] Figure 10 is a flowchart showing an example of the first aggressive driving detection process. The first aggressive driving detection process is performed by the processor 13 installed in the image processing device 10.

[0118] Step S11: The processor 13 obtains the vehicle speed of the mobile body 1 from the vehicle speed sensor mounted on the mobile body 1.

[0119] Step S12: The processor 13 calculates an appropriate distance between the moving body 1 and the other moving body 2 based on Table 3. That is, the processor 13 determines a safe distance and a dangerous distance depending on the speed of the moving body 1.

[0120] Step S13: The processor 13 determines, based on the surrounding image captured around the mobile body 1, whether or not another mobile body 2 exists within a predetermined range of the mobile body 1. If the processor 13 determines that another mobile body 2 exists within the predetermined range of the mobile body 1 (Step S13: Yes), it proceeds to the process in Step S14. If the processor 13 determines that another mobile body 2 does not exist within the predetermined range of the mobile body 1 (Step S13: No), it proceeds to the process in Step S20.

[0121] Step S14: The processor 13 determines whether the time during which another mobile body 2 remains within a predetermined range of the mobile body 1 is 3 seconds or more. If the processor 13 determines that the vehicle dwell time is 3 seconds or more (Step S14: Yes), it proceeds to the process in Step S15. If the processor 13 determines that the vehicle dwell time is less than 3 seconds (Step S14: No), it proceeds to the process in Step S20.

[0122] Step S15: Processor 13 determines whether the first provocation judgment flag is in the state of "none," "provisionally confirmed," or "confirmed." If Processor 13 determines that the first provocation judgment flag is in the state of "confirmed" (Step S15: "confirmed"), it proceeds to the process of Step S17. If Processor 13 determines that the first provocation judgment flag is in the state of "provisionally confirmed" (Step S15: "provisionally confirmed"), it proceeds to the process of Step S16. If Processor 13 determines that the first provocation judgment flag is in the state of "none" (Step S15: "none"), it proceeds to the process of Step S18.

[0123] Step S16: Processor 13 switches the first taunting judgment flag from the "provisional confirmation" state to the "confirmed" state.

[0124] Step S17: The processor 13 outputs information to the mobile body 1 via the communication interface 12 that instructs the behavior of the mobile body 1 (second behavior), and outputs a danger avoidance action instruction to the mobile body 1.

[0125] Step S18: Processor 13 switches the first taunting judgment flag from the "none" state to the "provisionally confirmed" state.

[0126] Step S19: The processor 13 outputs information instructing the behavior of the mobile body 1 (first behavior) to the mobile body 1 via the communication interface 12, and outputs an action instruction to the mobile body 1.

[0127] Step S20: The processor 13 sets the first aggressive driving detection flag to "none". That is, the processor 13 estimates that the driver of the other mobile vehicle 2 does not intend to engage in aggressive driving and continues the fully automated operation of mobile vehicle 1.

[0128] [Second aggressive driving detection process] Figure 11 is a flowchart showing an example of the first aggressive driving detection process. The first aggressive driving detection process is performed by the processor 13 mounted on the image processing device 10.

[0129] Step S21: The processor 13 obtains the vehicle speed of the mobile body 1 from the vehicle speed sensor mounted on the mobile body 1.

[0130] Step S22: The processor 13 calculates an appropriate distance between the moving body 1 and the other moving body 2 based on Table 3. That is, the processor 13 determines a safe distance and a dangerous distance depending on the speed of the moving body 1.

[0131] Step S23: The processor 13 determines, based on the surrounding image captured around the mobile body 1, whether or not another mobile body 2 exists within a predetermined range of the mobile body 1. If the processor 13 determines that another mobile body 2 exists within the predetermined range of the mobile body 1 (Step S23: Yes), it proceeds to the process in Step S14. If the processor 13 determines that another mobile body 2 does not exist within the predetermined range of the mobile body 1 (Step S23: No), it proceeds to the process in Step S30.

[0132] Step S24: The processor 13 determines whether or not the horn is being continuously detected from the microphone mounted on the mobile body 1. If the processor 13 determines that the horn is being continuously detected (Step S24: Yes), it proceeds to the process in Step S25. If the processor 13 determines that the horn is not being continuously detected (Step S24: No), it proceeds to the process in Step S30.

[0133] Step S25: Processor 13 determines whether the second provocation judgment flag is in the state of "none," "provisionally confirmed," or "confirmed." If Processor 13 determines that the second provocation judgment flag is in the state of "confirmed" (Step S25: "confirmed"), it proceeds to the process of Step S27. If Processor 13 determines that the judgment state of the second provocation judgment flag is "provisionally confirmed" (Step S25: "provisionally confirmed"), it proceeds to the process of Step S26. If Processor 13 determines that the judgment state of the second provocation judgment flag is "none" (Step S25: "none"), it proceeds to the process of Step S28.

[0134] Step S26: Processor 13 switches the second taunting judgment flag from the "provisional confirmation" state to the "confirmed" state.

[0135] Step S27: The processor 13 outputs information to the mobile body 1 via the communication interface 12 that instructs the behavior of the mobile body 1 (second behavior), and outputs a danger avoidance action instruction to the mobile body 1.

[0136] Step S28: Processor 13 switches the second taunting judgment flag from the "none" state to the "provisionally confirmed" state.

[0137] Step S29: The processor 13 outputs information instructing the behavior of the mobile body 1 (second behavior) to the mobile body 1 via the communication interface 12, and outputs an action instruction to the mobile body 1.

[0138] Step S30: The processor 13 sets the second aggressive driving detection flag to the "none" state. That is, the processor 13 estimates that the driver of the other mobile vehicle 2 does not intend to engage in aggressive driving and continues the fully automated operation of mobile vehicle 1.

[0139] [Third Aggressive Driving Detection Process] Figure 12 is a flowchart showing an example of the third aggressive driving detection process. The third aggressive driving detection process is executed by the processor 13 mounted on the image processing device 10.

[0140] Step S31: The processor 13 obtains the vehicle speed of the mobile body 1 from the vehicle speed sensor mounted on the mobile body 1.

[0141] Step S32: The processor 13 calculates an appropriate distance between the moving body 1 and the other moving body 2 based on Table 3. That is, the processor 13 determines a safe distance and a dangerous distance depending on the speed of the moving body 1.

[0142] Step S33: The processor 13 determines, based on the surrounding image captured around the mobile body 1, whether or not another mobile body 2 exists within a predetermined range of the mobile body 1. If the processor 13 determines that another mobile body 2 exists within the predetermined range of the mobile body 1 (Step S33: Yes), it proceeds to the process in Step S34. If the processor 13 determines that another mobile body 2 does not exist within the predetermined range of the mobile body 1 (Step S33: No), it proceeds to the process in Step S40.

[0143] Step S34: The processor 13 determines whether the other moving body 2 remains within a predetermined range of the moving body 1 for 3 seconds or more, and whether the other moving body 2 is performing a meandering motion. If the processor 13 determines that the vehicle dwell time is 3 seconds or more and the other moving body 2 is performing a meandering motion (Step S34: Yes), it proceeds to the process in Step S35. If the processor 13 does not determine that the vehicle dwell time is 3 seconds or more and the other moving body 2 is performing a meandering motion (Step S34: No), it proceeds to the process in Step S40. The processor 13 determines whether the other moving body 2 is performing a meandering motion based, for example, on whether the amplitude of the meandering motion in the other moving body 2 is greater than or equal to a predetermined amplitude, or whether the period of the meandering motion in the other moving body 2 is included in a predetermined period, etc.

[0144] Step S35: Processor 13 determines whether the third provocation judgment flag is in the state of "none," "provisionally confirmed," or "confirmed." If Processor 13 determines that the third provocation judgment flag is in the state of "confirmed" (Step S35: "confirmed"), it proceeds to the process of Step S37. If Processor 13 determines that the third provocation judgment flag is in the state of "provisionally confirmed" (Step S35: "provisionally confirmed"), it proceeds to the process of Step S36. If Processor 13 determines that the third provocation judgment flag is in the state of "none" (Step S35: "none"), it proceeds to the process of Step S38.

[0145] Step S36: Processor 13 switches the third taunting judgment flag from the "provisional confirmation" state to the "confirmed" state.

[0146] Step S37: The processor 13 outputs information to the mobile body 1 via the communication interface 12 that instructs the behavior of the mobile body 1 (second behavior), and outputs a danger avoidance action instruction to the mobile body 1.

[0147] Step S38: Processor 13 switches the third taunting judgment flag from the "none" state to the "provisionally confirmed" state.

[0148] Step S39: The processor 13 outputs information instructing the behavior of the mobile body 1 (third behavior) to the mobile body 1 via the communication interface 12, and outputs an action instruction to the mobile body 1.

[0149] Step S40: The processor 13 sets the third aggressive driving detection flag to "none". That is, the processor 13 estimates that the driver of the other mobile vehicle 2 does not intend to engage in aggressive driving and continues the fully automated operation of mobile vehicle 1.

[0150] [Fourth Aggressive Driving Detection Process] Figure 13 is a flowchart showing an example of the fourth aggressive driving detection process. The fourth aggressive driving detection process is executed by the processor 13 mounted on the image processing device 10.

[0151] Step S41: The processor 13 obtains the vehicle speed of the mobile body 1 from the vehicle speed sensor mounted on the mobile body 1.

[0152] Step S42: The processor 13 calculates an appropriate distance between the moving body 1 and the other moving body 2 based on Table 3. That is, the processor 13 determines a safe distance and a dangerous distance depending on the speed of the moving body 1.

[0153] Step S43: The processor 13 determines, based on the surrounding image captured around the mobile body 1, whether or not another mobile body 2 exists within a predetermined range of the mobile body 1. If the processor 13 determines that another mobile body 2 exists within the predetermined range of the mobile body 1 (Step S43: Yes), it proceeds to the process in Step S44. If the processor 13 determines that another mobile body 2 does not exist within the predetermined range of the mobile body 1 (Step S43: No), it proceeds to the process in Step S50.

[0154] Step S44: The processor 13 determines whether the other mobile body 2 remains within a predetermined range of mobile body 1 for 3 seconds or more, and whether the other mobile body 2 is flashing its headlights. If the processor 13 determines that the vehicle remains for 3 seconds or more and the other mobile body 2 is flashing its headlights (Step S44: Yes), it proceeds to the process in Step S45. If the processor 13 does not determine that the vehicle remains for 3 seconds or more and the other mobile body 2 is flashing its headlights (Step S44: No), it proceeds to the process in Step S50. The processor 13 determines whether the other mobile body 2 is swerving, for example, based on whether the other mobile body 2 is flashing its headlights behind mobile body 1, or whether the other mobile body 2 is illuminating its headlights at high brightness behind mobile body 1.

[0155] Step S45: Processor 13 determines whether the fourth provocation judgment flag is in the state of "none," "provisionally confirmed," or "confirmed." If Processor 13 determines that the fourth provocation judgment flag is in the state of "confirmed" (Step S45: "confirmed"), it proceeds to the process of Step S47. If Processor 13 determines that the fourth provocation judgment flag is in the state of "provisionally confirmed" (Step S45: "provisionally confirmed"), it proceeds to the process of Step S46. If Processor 13 determines that the fourth provocation judgment flag is in the state of "none" (Step S45: "none"), it proceeds to the process of Step S48.

[0156] Step S46: Processor 13 switches the fourth taunting judgment flag from the "provisional confirmation" state to the "confirmed" state.

[0157] Step S47: The processor 13 outputs information to the mobile body 1 via the communication interface 12 that instructs the behavior of the mobile body 1 (second behavior), and outputs a danger avoidance action instruction to the mobile body 1.

[0158] Step S48: Processor 13 switches the fourth taunting judgment flag from the "none" state to the "provisionally confirmed" state.

[0159] Step S49: The processor 13 outputs information instructing the behavior of the mobile body 1 (fourth behavior) to the mobile body 1 via the communication interface 12, and outputs an action instruction to the mobile body 1.

[0160] Step S50: The processor 13 sets the fourth aggressive driving detection flag to "none". That is, the processor 13 estimates that the driver of the other mobile vehicle 2 does not intend to engage in aggressive driving and continues the fully automated operation of mobile vehicle 1.

[0161] [Fifth Aggressive Driving Detection Process] Figure 14 is a flowchart showing an example of the fifth aggressive driving detection process. The fifth aggressive driving detection process is executed by the processor 13 mounted on the image processing device 10.

[0162] Step S51: The processor 13 obtains the vehicle speed of the mobile body 1 from the vehicle speed sensor mounted on the mobile body 1.

[0163] Step S52: The processor 13 calculates an appropriate distance between the moving body 1 and the other moving body 2 based on Table 3. That is, the processor 13 determines a safe distance and a dangerous distance depending on the speed of the moving body 1.

[0164] Step S53: The processor 13 determines, based on the surrounding image captured around the mobile body 1, whether or not another mobile body 2 exists within a predetermined range of the mobile body 1. If the processor 13 determines that another mobile body 2 exists within the predetermined range of the mobile body 1 (Step S53: Yes), it proceeds to the process in Step S54. If the processor 13 determines that another mobile body 2 does not exist within the predetermined range of the mobile body 1 (Step S53: No), it proceeds to the process in Step S60.

[0165] Step S54: The processor 13 determines whether the other moving body 2 remains within a predetermined range of the moving body 1 for 3 seconds or more, and whether the other moving body 2 continues to stick to and follow the moving body 1 after it changes lanes. If the processor 13 determines that the other moving body 2 remains within a predetermined range of the moving body 1 for 3 seconds or more, and that the other moving body 2 continues to stick to and follow the moving body 1 after it changes lanes (Step S54: Yes), the process proceeds to Step S55. If the processor 13 does not determine that the other moving body 2 remains within a predetermined range of the moving body 1 for 3 seconds or more, and that the other moving body 2 continues to stick to and follow the moving body 1 after it changes lanes (Step S54: No), the process proceeds to Step S60.

[0166] Step S55: Processor 13 determines whether the fifth provocation judgment flag is in the state of "none," "provisionally confirmed," or "confirmed." If Processor 13 determines that the fifth provocation judgment flag is in the state of "confirmed" (Step S55: "confirmed"), it proceeds to the process of Step S57. If Processor 13 determines that the fifth provocation judgment flag is in the state of "provisionally confirmed" (Step S55: "provisionally confirmed"), it proceeds to the process of Step S56. If Processor 13 determines that the fifth provocation judgment flag is in the state of "none" (Step S55: "none"), it proceeds to the process of Step S18.

[0167] Step S56: Processor 13 switches the fifth taunting judgment flag from the "provisional confirmation" state to the "confirmed" state.

[0168] Step S57: The processor 13 outputs information to the mobile body 1 via the communication interface 12 that instructs the behavior of the mobile body 1 (second behavior), and outputs a danger avoidance action instruction to the mobile body 1.

[0169] Step S58: Processor 13 switches the fifth taunting judgment flag from the "none" state to the "provisionally confirmed" state.

[0170] Step S59: The processor 13 outputs information instructing the behavior of the mobile body 1 (the fifth behavior) to the mobile body 1 via the communication interface 12, and outputs an action instruction to the mobile body 1.

[0171] Step S60: The processor 13 sets the fifth aggressive driving detection flag to "none". That is, the processor 13 estimates that the driver of the other mobile vehicle 2 does not intend to engage in aggressive driving and continues the fully automated operation of mobile vehicle 1.

[0172] [Sixth Aggressive Driving Detection Process] Figure 15 is a flowchart showing an example of the sixth aggressive driving detection process. The sixth aggressive driving detection process is executed by the processor 13 mounted on the image processing device 10.

[0173] Step S61: The processor 13 obtains the vehicle speed of the mobile body 1 from the vehicle speed sensor mounted on the mobile body 1.

[0174] Step S62: The processor 13 calculates an appropriate distance between the moving body 1 and the other moving body 2 based on Table 3. That is, the processor 13 determines a safe distance and a dangerous distance depending on the speed of the moving body 1.

[0175] Step S63: The processor 13 determines, based on the surrounding image captured around the mobile body 1, whether or not another mobile body 2 exists within a predetermined range of the mobile body 1. If the processor 13 determines that another mobile body 2 exists within the predetermined range of the mobile body 1 (Step S63: Yes), it proceeds to the process in Step S64. If the processor 13 determines that another mobile body 2 does not exist within the predetermined range of the mobile body 1 (Step S63: No), it proceeds to the process in Step S70.

[0176] Step S64: The processor 13 determines whether the time during which another mobile body 2 remains within a predetermined range of the mobile body 1 is 3 seconds or more. If the processor 13 determines that the vehicle dwell time is 3 seconds or more (Step S64: Yes), it proceeds to the process in Step S65. If the processor 13 determines that the vehicle dwell time is less than 3 seconds (Step S64: No), it proceeds to the process in Step S70.

[0177] Step S65: Processor 13 determines whether the sixth provocation judgment flag is in the state of "none," "provisionally confirmed," or "confirmed." If Processor 13 determines that the sixth provocation judgment flag is in the state of "confirmed" (Step S65: "confirmed"), it proceeds to the process of Step S67. If Processor 13 determines that the sixth provocation judgment flag is in the state of "provisionally confirmed" (Step S65: "provisionally confirmed"), it proceeds to the process of Step S66. If Processor 13 determines that the sixth provocation judgment flag is in the state of "none" (Step S65: "none"), it proceeds to the process of Step S68.

[0178] Step S66: Processor 13 switches the sixth taunting judgment flag from the "provisional confirmation" state to the "confirmed" state.

[0179] Step S67: The processor 13 outputs information to the mobile body 1 via the communication interface 12 that instructs the behavior of the mobile body 1 (second behavior), and outputs a danger avoidance action instruction to the mobile body 1.

[0180] Step S68: Processor 13 switches the sixth taunting judgment flag from the "none" state to the "provisionally confirmed" state.

[0181] Step S69: The processor 13 outputs information instructing the behavior of the mobile body 1 (sixth behavior) to the mobile body 1 via the communication interface 12, and outputs an action instruction to the mobile body 1.

[0182] Step S70: The processor 13 sets the sixth aggressive driving detection flag to "none". That is, the processor 13 estimates that the driver of the other mobile vehicle 2 does not intend to engage in aggressive driving and continues the fully automated operation of mobile vehicle 1.

[0183] As described above, according to the image processing method of this embodiment, instead of determining whether or not the other moving body 2 is engaging in aggressive driving towards the moving body 1 based solely on the first state of the other moving body 2, the method considers the second state of the other moving body 2 after the first behavior of the moving body 1 to determine whether or not the other moving body 2 is engaging in aggressive driving towards the moving body 1. This suppresses misjudgments of aggressive driving and reduces the likelihood of unnecessary disputes occurring between the moving body 1 and the other moving body 2, thereby contributing to smoother traffic flow.

[0184] <Variation> In this embodiment, the first behavior of the mobile vehicle 1 has been described as, for example, continuing to drive at the legal speed limit, changing lanes, sounding the horn to warn other mobile vehicles, pulling over to the nearest convenience store, or pulling over to the nearest service area or parking area, but it is not limited to these. The first behavior of the mobile vehicle 1 may be a combination of these actions as appropriate. Furthermore, if the mobile vehicle 1 is equipped with an audio output function, the first behavior of the mobile vehicle 1 may include audio output. Furthermore, if the mobile vehicle 1 is equipped with a projection function onto the ground or the like, the first behavior of the mobile vehicle 1 may include projecting a predetermined image, such as a message, onto a sidewalk or the like.

[0185] Furthermore, in this embodiment, examples of the second behavior of the mobile body 1 include reporting to the police, stopping, slowing down, taking refuge in the nearest convenience store, taking refuge in the nearest service area or parking area, recording to a drive recorder, uploading surrounding video to a predetermined network, and displaying that another mobile body is driving dangerously, but the second behavior is not limited to these. Also, if the mobile body 1 has an audio output function, the second behavior of the mobile body 1 may include audio output. Also, if the mobile body 1 has a projection function onto the ground or the like, the second behavior of the mobile body 1 may include projecting a predetermined image, such as a message, onto a sidewalk or the like.

[0186] Furthermore, in this embodiment, the third behavior of the mobile body 1 has been described with examples such as notifying the police, stopping, taking refuge in the nearest convenience store, taking refuge in the nearest service area or parking area, and locking the doors after stopping, but it is not limited to these. Also, if the mobile body 1 has an audio output function, the third behavior of the mobile body 1 may include audio output. Also, if the mobile body 1 has a projection function onto the ground or the like, the third behavior of the mobile body 1 may include projecting a predetermined image, such as a message, onto a sidewalk or the like.

[0187] For example, the processor 13 may sequentially perform actions to avoid the state of the other mobile body 2, such as slowing down at the edge of the driving lane and continuing to drive, gradually moving out of the driving lane, or stopping on the shoulder of the road, and after each action, it may determine whether the other mobile body 2 is driving dangerously or not. Alternatively, for example, the processor 13 may cancel the fully automatic operation of mobile body 1 if the other mobile body 2 does not stop driving dangerously even after repeating the first, second, and third actions a predetermined number of times.

[0188] Furthermore, in this embodiment, an example was described using an image processing device 10 mounted on the mobile body 1 that detects the first, second, and third states of other mobile bodies 2 from surrounding video footage, determines a first behavior based on the first state of other mobile bodies 2, determines a second behavior based on the second state of other mobile bodies 2, determines a third behavior based on the third state of other mobile bodies 2, and so on, but the embodiment is not limited to this. For example, a server on a network may have functions such as detecting the first, second, and third states of other mobile bodies 2 from surrounding video footage, determining a first behavior based on the first state of other mobile bodies 2, determining a second behavior based on the second state of other mobile bodies 2, and determining a third behavior based on the third state of other mobile bodies 2, which are mainly provided by the processor 13. In this case, the mobile body 1 can communicate with the server via the network, and the mobile body 1 transmits surrounding video footage to the server. The server detects the first, second, and third states of other mobile bodies 2 from the surrounding video footage acquired from mobile body 1, or determines a first behavior based on the first state of other mobile body 2, a second behavior based on the second state of other mobile body 2, and a third behavior based on the third state of other mobile body 2, and transmits information instructing the determined behavior to mobile body 1 via the network.

[0189] Furthermore, in this embodiment, we have described an example where the aggressive driving detection status flag has three states: "none," "provisionally confirmed," and "confirmed." However, the aggressive driving detection status flag is not limited to these states. Also, in this embodiment, we have described an example where aggressive driving detection processing is performed using the aggressive driving detection status flag. However, it is also possible to perform aggressive driving detection processing using a known algorithm.

[0190] Furthermore, in this embodiment, the processor has been described as detecting the state of another mobile body 2 based on the surrounding image captured by the imaging unit 11, but the system is not limited to this example. The processor may detect the state of the other mobile body 2 based on various sensors mounted on the mobile body 1. For example, the processor may detect the distance between the mobile body 1 and the other mobile body 2 using a distance measuring sensor mounted on the mobile body 1. For example, the processor may detect the flashing of the headlights of the other mobile body 2 using an illuminance sensor mounted on the mobile body 1.

[0191] Furthermore, in this embodiment, descriptions such as "First," "Second," "Third," "Fourth," "Fifth," and "Sixth" are identifiers for distinguishing the configurations. Configurations distinguished by descriptions such as "First" and "Second" in this disclosure can have their numbers swapped. For example, the first aggressive driving can swap the identifiers "First" and "Second" with the second aggressive driving. The second aggressive driving can swap the identifiers "Second" and "Third" with the third aggressive driving. The third aggressive driving can swap the identifiers "Third" and "Fourth" with the fourth aggressive driving. The fourth aggressive driving can swap the identifiers "Fourth" and "Fifth" with the fifth aggressive driving. The fifth aggressive driving can swap the identifiers "Fifth" and "Sixth" with the sixth aggressive driving. The swapping of identifiers occurs simultaneously. The configurations remain distinguishable even after the swapping of identifiers. Identifiers may be deleted. Configurations from which identifiers have been removed are distinguished by codes. The names of identifiers such as “1st,” “2nd,” “3rd,” “4th,” “5th,” and “6th” in this disclosure should not be used alone to interpret the order of such configurations or to justify the existence of lower-numbered identifiers.

[0192] While one embodiment of this disclosure has been described based on the drawings and examples, it should be noted that those skilled in the art will find it easy to make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are included within the scope of this disclosure. [Explanation of symbols]

[0193] 1 Mobile Unit 1A Imaging device 2 Other mobile bodies 10 Image Processing Device 11. Imaging Unit (Input Unit) 12. Communication Interface (Output Section) 13 processors 14 Storage section

Claims

1. An image processing apparatus comprising a processor, If the processor makes a first determination that it has detected erratic driving by another moving object in the vicinity of the vehicle to which the image processing device is attached for a first time or longer, it decides to output an audio signal or display an indication that the other moving object is driving dangerously, without yet determining that the other moving object is engaging in aggressive driving. If the processor makes a second determination that it has detected the erratic driving for a second time or longer after the first determination, it decides to record video footage of the other moving object into the drive recorder. Image processing device.

2. If the processor determines that it has not detected the other moving object for a period of time or longer after the first determination, it decides not to record the video of the other moving object to the drive recorder. The image processing apparatus according to claim 1.

3. A vehicle equipped with the image processing device described in claim 1.

4. An image processing method performed by the processor of an image processing device, If the image processing device has detected erratic driving by another moving object in the vicinity of the vehicle to which it is attached for a first period of time or longer, the first step is to decide to output an audio signal or display a message indicating that the other moving object is driving dangerously, without yet determining that the other moving object is engaging in aggressive driving. If it is determined that the serpentine driving has been detected for a second time or longer after the first determination, the step includes deciding to record video footage of the other moving object onto the drive recorder. Image processing methods.

5. The processor of the image processing device is instructed to perform the following actions: if it makes a first determination that it has detected erratic driving by another moving object in the vicinity of the vehicle to which the image processing device is attached for a first time or longer, it will decide to output an audio signal or display a message indicating that the other moving object is engaging in dangerous driving, without yet determining that the other moving object is engaging in aggressive driving; and if it makes a second determination that it has detected the erratic driving for a second time or longer after the first determination, it will decide to record video footage of the other moving object to the drive recorder. Image processing program.

6. A server comprising a processor, If the processor makes a first determination that it has detected erratic driving by another moving object in the vicinity of the vehicle for a first time or longer, it transmits information to the vehicle via the network instructing it to output an audio signal or display an indication that the other moving object is engaging in dangerous driving, even if it has not yet determined that the other moving object is engaging in aggressive driving. If the processor makes a second determination that it has detected the erratic driving for a second time or longer after the first determination, it transmits information to the vehicle via the network instructing it to record video of the other moving object in the drive recorder. server.

Images