Image processing device, image processing method, image processing program, and recording medium

Abstract

To provide an image processing apparatus capable of detecting a flying object flying towards a vehicle or in an area in front of the vehicle.SOLUTION: An image processing apparatus is provided with: an image acquisition unit that acquires a plurality of images sequentially taken around a vehicle; a flying object detection unit that determines, based on the plurality of images, whether or not the vehicle is captured in any of the plurality of images or a flying object is captured in an area in front of the vehicle; and an image specifying unit that specifies the image as a specific image within a predetermined period of time including the time when the flying object was captured in the image if it is determined that the flying object is captured.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, an image processing program, and a recording medium, and particularly to an image processing apparatus, an image processing method, an image processing program, and a recording medium that perform image processing based on a plurality of images.

Background Art

[0002] There has been proposed a device that detects dangerous driving such as aggressive driving by detecting the behavior of other vehicles traveling around a host vehicle while the host vehicle is in motion.

[0003] For example, in Patent Document 1, a peripheral vehicle is detected from an image of the surroundings of a vehicle, the vehicle type of the peripheral vehicle is recognized, and based on the size of the peripheral vehicle in the image and a size criterion for determining dangerous driving defined in advance for each vehicle type, a dangerous driving determination device that detects dangerous driving due to the approach of a peripheral vehicle is disclosed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] As a threatening or obstructive act against a vehicle, which is a dangerous event for a vehicle (hereinafter, also simply referred to as a dangerous event), there is an act of throwing an object from around the vehicle. Further, as a dangerous event, there is also an event in which an object flies due to wind or the like, regardless of the threatening or obstructive act against the vehicle. In any case, if an object flies into the vehicle or the area in front of the vehicle, it will obstruct driving and is dangerous. One of the problems is that depending on the above-described dangerous driving determination device, such a dangerous event in which an object flies into the host vehicle or the area in front of the host vehicle cannot be detected unless there is an approach of another vehicle to the host vehicle.

[0006] The present invention has been made in view of the above-mentioned points, and one of its objectives is to provide an image processing device capable of detecting flying objects that are flying toward a vehicle or into the area in front of a vehicle. [Means for solving the problem]

[0007] The invention described in claim 1 is an image processing apparatus characterized by comprising: an image acquisition unit that acquires a plurality of images taken sequentially around a vehicle; a flying object detection unit that determines, based on the plurality of images, whether or not the vehicle or a flying object in the area in front of the vehicle is captured in any of the plurality of images; and an image identification unit that, if it is determined that a flying object is captured, identifies an image within a predetermined time period including the time when the image containing the flying object was captured as a specific image.

[0008] The invention described in claim 17 is an image processing method performed by an image processing device, comprising: an image acquisition step of acquiring a plurality of images taken sequentially around a vehicle; a flying object detection step of determining, based on the plurality of images, whether or not a flying object toward the vehicle or toward the area in front of the vehicle is captured in any of the plurality of images; and, if it is determined that a flying object is captured, an image identification step of identifying an image within a predetermined time period including the time when the image containing the flying object was captured as a specific image.

[0009] The invention described in claim 18 is an image processing program executed by an image processing device equipped with a computer, which causes the computer to perform an image acquisition step of acquiring a plurality of images taken sequentially around a vehicle; a flying object detection step of determining whether or not a flying object toward the vehicle or toward the area in front of the vehicle is captured in any of the plurality of images based on the plurality of images; and an image identification step of identifying an image within a predetermined time period including the time when the image containing the flying object was captured as a specific image, if it is determined that a flying object is captured.

[0010] The invention described in claim 19 is a computer-readable recording medium that stores an image processing program which causes the computer-equipped image processing device to perform the following steps: an image acquisition step of acquiring a plurality of images taken sequentially around a vehicle; a flying object detection step of determining whether or not a flying object toward the vehicle or toward the area in front of the vehicle is captured in any of the plurality of images based on the plurality of images; and an image identification step of identifying an image within a predetermined time period including the time when the image containing the flying object was captured as a specific image, if it is determined that a flying object is captured. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows an example of an event detected by the image processing according to Example 1. [Figure 2] This diagram shows the configuration of the image processing system according to Example 1. [Figure 3] This figure shows the front seat area of ​​a vehicle equipped with the image processing device according to Example 1. [Figure 4] This is a block diagram showing the configuration of the image processing apparatus according to Example 1. [Figure 5] This diagram shows the server configuration according to Example 1. [Figure 6] This figure shows an example of a trajectory calculated by the image processing device according to Example 1. [Figure 7] This figure shows an example of a trajectory calculated by the image processing device according to Example 1. [Figure 8] This flowchart shows an example of a routine executed by the image processing device according to Example 1. [Figure 9] This flowchart shows a modified example of the routine executed by the image processing device according to Example 1. [Figure 10] This figure shows an example of the optical flow calculated by the image processing device according to Example 2. [Figure 11] This flowchart shows an example of a routine executed by the image processing device according to Example 2. [Figure 12] It is a flowchart showing a modified example of a routine executed by the image processing apparatus according to Example 2.

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present invention will be described in detail. In the following description and the accompanying drawings, the same reference numerals are given to substantially the same or equivalent parts.

Embodiment

[0013] With reference to the accompanying drawings, the configuration of the image processing apparatus 10 according to this embodiment will be described.

[0014] FIG. 1 is a diagram showing an example of an event detected by image processing executed by the image processing apparatus 10 mounted on the host vehicle M. FIG. 1 shows a state where the image processing apparatus 10 is mounted on the host vehicle M1. In FIG. 1, the host vehicle M1 traveling on the road R, other vehicles M2 and pedestrians P1 existing around the host vehicle M1 are shown. Further, in FIG. 1, as an example of an object flying onto the road R, empty cans A1, A2, and A3 are shown.

[0015] In the example shown in FIG. 1, the empty can A1 is an object that has flown onto the road R by being thrown away from the other vehicle M2. In other words, the empty can A1 is a thrown object thrown by a passenger in the other vehicle M2.

[0016] The empty can A2 in FIG. 1 is an object that has flown onto the road R by being thrown by the pedestrian P1. In other words, the empty can A2 is a thrown object thrown by the pedestrian P1.

[0017] The empty can A3 in FIG. 1 is an object existing on the road R by being carried by the wind. In other words, the empty can A3 is a flying object that has flown onto the road R by the wind.

[0018] In the example shown in Figure 1, empty cans A1, A2, and A3 can all be considered flying objects that landed in the area in front of the vehicle M1 on the road R.

[0019] In this embodiment, the flying object is described as not including animals such as birds or actively moving objects such as unmanned aerial vehicles.

[0020] As described above, incidents such as objects like empty cans A1 or A2 flying towards the vehicle due to throwing objects from around the vehicle, or objects like empty can A3 flying towards the vehicle due to wind, etc., are all dangerous events for the vehicle M1. The image processing device 10 is a device that enables the detection of such dangerous events.

[0021] Specifically, the image processing device 10 acquires multiple images taken sequentially around the vehicle M1, detects flying objects such as empty cans A1 to A3 based on these multiple images, and detects dangerous events by identifying the images in which the flying objects are captured. For example, by analyzing the images identified by the image processing device 10 as containing flying objects, it is possible to investigate the cause of the flying object's occurrence, such as verifying which vehicle it was thrown from.

[0022] Figure 2 is a schematic diagram of an image processing system 100 including an image processing device 10. As shown in Figure 2, the image processing system 100 consists of an image processing device 10 mounted on the vehicle M1 and a server device 20.

[0023] The server device 20 is a server device that communicates with the image processing device 10 via a network NW. The image processing device 10 and the server device 20 can send and receive data from each other using a communication protocol such as TCP / IP. The connection between the image processing device 10 and the network NW can be made by mobile communication such as 4G (4th Generation) or 5G (5th Generation), or by wireless communication such as Wi-Fi (registered trademark).

[0024] In this embodiment, the image processing device 10 acquires a plurality of images sequentially captured by a camera attached to the vehicle M1, identifies the image containing the flying object based on the plurality of images, labels it as the identified image, and transmits the identified image to the server device 20.

[0025] The server device 20 permanently stores a specific image received from the image processing device 10.

[0026] Thus, the image processing system 100 identifies images containing flying objects based on multiple images sequentially taken around the vehicle M1, and permanently stores the identified images. For example, a user of the image processing system 100 can use the image data stored in the server device 20 to extract locations with a high incidence of hazardous events and to verify hazardous events.

[0027] Figure 3 shows the front seat area of ​​the vehicle M1. As shown in Figure 3, the image processing device 10 is located on the center console of the vehicle M1.

[0028] The front camera 11 is mounted on the dashboard DB. The front camera 11 is oriented to capture images in the direction of travel of the vehicle M1. When the direction of travel of the vehicle M1 is considered to be forward, the front camera 11 is positioned to capture images in a range that includes the area in front of the vehicle M1, the right front, and the left front.

[0029] The front camera 11 may be installed anywhere on the vehicle M1 as long as it can capture an image of the area in front of the vehicle M1. For example, the front camera 11 may be located behind the rearview mirror RM, that is, on the surface facing the windshield FG, or it may be installed on the outer surface (exterior surface) of the vehicle M1, for example, on the hood or front bumper.

[0030] The front camera 11 is connected to the image processing device 10 in a communicative manner and can transmit the signal of the captured image to the image processing device 10.

[0031] The touch panel display 15 consists of a touch panel and a display. The touch panel display 15 is communicatively connected to the image processing device 10. The display of the touch panel display 15 displays an image supplied by the image processing device 10. For example, the display may show a specific image that the image processing device 10 has identified as containing a flying object when the image processing device 10 detects a dangerous event.

[0032] Furthermore, the touch panel of the touch panel display 15 transmits signals to the image processing device 10 indicating input operations by touching the touch panel. For example, the touch panel display 15 transmits signals to the image processing device 10 indicating input operations for making various settings related to image processing, such as starting or ending image processing.

[0033] For example, the image processing device 10 may receive input operations for setting image processing via voice through a microphone (not shown) provided on its own vehicle M1.

[0034] The GNSS receiver 17 is a device that receives signals (GNSS signals) from GNSS (Global Navigation Satellite System) satellites. The GNSS receiver 17 is located, for example, on the dashboard DB. However, the GNSS receiver 17 can be located anywhere as long as it can receive GNSS signals.

[0035] The GNSS receiver 17 is connected to the image processing device 10 in a communicative manner and can transmit the received GNSS signals to the image processing device 10. The image processing device 10 uses the GNSS signals to acquire the current position information of the vehicle M1. For example, the image processing device 10 may add the current position information of the vehicle M1 at the time the image was taken to an image that has been identified as containing an incoming object.

[0036] The image processing device 10 acquires and stores video footage captured by the front camera 11. In other words, in this embodiment, the image processing device 10 has the same functions as a so-called drive recorder.

[0037] The image processing device 10 detects an incoming object by performing image processing based on a plurality of images sequentially captured by the forward camera 11.

[0038] Referring to Figure 4, the configuration and functions of the image processing device 10 installed in the vehicle M1 will be explained.

[0039] Figure 4 is a block diagram showing the configuration of the image processing device 10. As shown in Figure 4, the image processing device 10 is configured with each part connected via a system bus 21.

[0040] The input unit 23 is an interface that enables communication between the image processing device 10 and the equipment installed on the vehicle M1.

[0041] The input unit 23 is an interface that enables communication between the image processing device 10 and the front camera 11. The image processing device 10 acquires images of the area around the vehicle M1 from the front camera 11 via the input unit 23.

[0042] The input unit 23 is an interface that enables communication between the image processing device 10 and the touch panel of the touch panel display 15. The image processing device 10 receives signals indicating input operations by touching the touch panel via the input unit 23.

[0043] The input unit 23 is an interface that connects the image processing device 10 to the GNSS receiver 17. The image processing device 10 receives GNSS signals from the GNSS receiver 17 via the input unit 23 and acquires information indicating the current position of the vehicle M1.

[0044] The large-capacity storage device 25 is composed of, for example, a hard disk drive, an SSD (solid state drive), flash memory, etc., and stores various programs executed in the image processing device 10. These programs may be acquired, for example, from other server devices via a network, or they may be recorded on a recording medium and read via various drive devices. In other words, the programs stored in the large-capacity storage device 25 can be transmitted via a network, and they can also be recorded on a computer-readable recording medium and transferred to other devices.

[0045] For example, the large-capacity storage device 25 stores an image processing program that the image processing device 10 executes when it identifies an image containing an incoming object based on multiple images taken sequentially around the vehicle M1.

[0046] Furthermore, the large-capacity storage device 25 has an image temporary storage unit 25A. The image temporary storage unit 25A stores moving images captured by the front camera 11, that is, a series of images taken sequentially around the vehicle M1. A series of images refers to images arranged in chronological order.

[0047] The image temporary storage unit 25A stores, for example, video footage for a certain period of time. The image temporary storage unit 25A also stores, for example, video footage up to a predetermined capacity.

[0048] For example, the image temporary storage unit 25A is a memory section where new images are stored moment by moment, like a ring buffer. The image temporary storage unit 25A can store, for example, several hours to tens of hours of video footage. For example, in the image temporary storage unit 25A, images are updated moment by moment by overwriting old images with new images.

[0049] The control unit 27 is composed of a CPU (Central Processing Unit) 27A, ROM (Read Only Memory) 27B, RAM (Random Access Memory) 27C, etc., and functions as a computer. The CPU 27A reads and executes various programs stored in the ROM 27B and the mass storage device 25 to realize various functions.

[0050] The control unit 27 functions as an image acquisition unit that acquires multiple images sequentially taken around the vehicle M1 from the front camera 11 via the input unit 23.

[0051] The control unit 27 functions as a flying object detection unit that determines, based on the acquired plurality of images, whether or not a flying object is captured in any of the plurality of images, or whether a flying object is captured in the area in front of the vehicle M1.

[0052] For example, the control unit 27, acting as a flying object detection unit, identifies the object to be tracked in each of multiple images through image recognition, calculates the trajectory traced by the object to be tracked based on the images in chronological order that include the image in which the object to be tracked was identified, and determines whether or not a flying object is captured in the image by determining whether or not the object to be tracked is a flying object based on the trajectory.

[0053] For example, the control unit 27 identifies objects that are recognized as objects other than people or vehicles through image recognition, or objects that are potentially flying objects such as objects on the road, as objects to be tracked.

[0054] For example, the control unit 27 extracts feature points of the object to be tracked in each of the consecutive images and calculates the trajectory of each feature point based on the coordinates of the feature points in the image.

[0055] For example, when tracking the movement of a flying object in a video, the trajectory traced by the object has characteristics such as a distinctive curve, which can be distinguished from the trajectories traced by fallen objects or surrounding vehicles that were already on the road before the video was taken. The control unit 27 determines whether the calculated trajectory is that of a flying object based on the criteria for determining the characteristics of the trajectory.

[0056] For example, the control unit 27, acting as a flying object detection unit, determines that the object being tracked is a flying object if at least one of the following conditions is met: the maximum curvature of the trajectory traced by the object being tracked exceeds a predetermined threshold for maximum curvature; the direction of the trajectory is downward and approaches the axis indicating the direction of travel of the vehicle M1; or the length of the trajectory exceeds a predetermined threshold for the length of the trajectory.

[0057] Furthermore, the determination using the characteristics of the tracking object's trajectory as described above can also be achieved using a learning model. For example, the control unit 27, acting as a flying object detection unit, may use a pre-trained model that has been machine-learned to output information indicating whether or not the trajectory traced by the tracking object is that of a flying object, when the trajectory traced by the tracking object is input, to determine whether or not the tracking object is a flying object.

[0058] The learning model can be constructed, for example, in a large-capacity storage device 25 using pre-calculated trajectories of numerous incoming objects and information indicating that each of these numerous objects is an incoming object as training data.

[0059] When the control unit 27 determines that an incoming object is captured, it functions as an image identification unit that identifies images within a predetermined time period, including the time when the image containing the incoming object was captured, as a specific image.

[0060] The control unit 27 acquires the current location information of the vehicle M1 from the GNSS receiver 17 via the input unit 23. This current location information is, for example, information that associates information indicating the time with information indicating the position of the vehicle M1 at that time. The information indicating the position of the vehicle M1 may be information indicating its position on a map.

[0061] The communication unit 29 is a network adapter such as a NIC (Network Interface Card) connected to a wireless device (not shown). The communication unit 29 communicates with the server device 20 according to commands from the control unit 27. For example, the communication unit 29 communicates when the image processing device 10 determines that an incoming object is captured in a specific image and sends it to the server device 20.

[0062] The output unit 31 is connected to the display of the touch panel display 15 installed in the vehicle M1. The output unit 31 is an interface for supplying various information to the display in accordance with commands from the control unit 27.

[0063] For example, the output unit 31 supplies information to the display for displaying a screen that accepts input for settings related to image processing by the image processing device 10. For example, the output unit 31 may supply the display with an image that has been determined to contain a flying object.

[0064] Next, the configuration and functions of the server device 20 will be explained with reference to Figure 5.

[0065] Figure 5 is a block diagram showing the configuration of the server device 20. As shown in Figure 5, the server device 20 is a server device in which each part is connected via the system bus 41.

[0066] The large-capacity storage device 43 is a storage device composed of, for example, a hard disk drive, an SSD (solid state drive), flash memory, etc. The large-capacity storage device 43 stores various programs executed on the server device 20. These programs may be obtained, for example, from other server devices via a network, or they may be recorded on a recording medium and read via various drive devices.

[0067] The large-capacity storage device 43 includes a permanent storage unit 43A. The permanent storage unit 43A permanently stores specific images, including images that the image processing device 10 has determined to contain an object flying towards the vehicle M1 or an object flying towards the area in front of the vehicle M1.

[0068] The control unit 45 is composed of a CPU (Central Processing Unit) 45A, a ROM (Read Only Memory) 45B, a RAM (Random Access Memory) 45C, etc., and functions as a computer. The CPU 45A reads and executes various programs stored in the ROM 45B and the mass storage device 43 to realize various functions.

[0069] The control unit 45 reads and executes an image processing program stored in the large-capacity storage device 43, thereby causing the specific image identified by the image processing device 10 to be stored in the permanent storage unit 43A.

[0070] The communication unit 47 is a network adapter such as a NIC (Network Interface Card) connected to a wireless device (not shown). The communication unit 47 communicates with the server device 20 and the image processing device 10 according to commands from the control unit 45.

[0071] For example, the communication unit 47 handles communication when the server device 20 receives a specific image from the image processing device 10.

[0072] The server device 20 receives a specific image from the image processing device 10 via the communication unit 47 and permanently stores the received specific image in the permanent storage unit 43A.

[0073] For example, the server device 20 receives specific images from each of the multiple image processing devices 10 installed in each of the multiple vehicles, and stores the specific images in the permanent storage unit 43A for each image processing device 10 or for each vehicle.

[0074] The image processing performed by the control unit 27 of the image processing device 10 will be described in detail below, with reference to Figures 6-8.

[0075] Figure 6 shows an example of a trajectory calculated by the image processing device 10 when the object being tracked is not a flying object, but a fallen object that was already on the road before being photographed by the front camera 11. Figure 6 shows image F1, which is an example of an image taken in front of the vehicle M1 by the front camera 11. As shown in Figure 6, image F1 shows another vehicle M2 traveling in the adjacent lane and an empty can B1 that is on the road in front of the vehicle M1.

[0076] The control unit 27, for example, sequentially acquires images captured by the front camera 11 and performs image processing. For example, the control unit 27 sequentially acquires images captured by the front camera 11 and stores them in the image temporary storage unit 25A, and uses the images stored in the image temporary storage unit 25A for image processing.

[0077] For example, when the control unit 27 acquires image F1, which is one of the images sequentially captured by the front camera 11, it performs image recognition on image F1 to identify the object to be tracked. For example, the control unit 27 classifies the objects in image F1 into several types and identifies the objects classified into a predetermined type as the object to be tracked.

[0078] For example, the control unit 27 classifies objects in image F1 into three categories: the first category is "vehicle," the second category is "person," and the third category is "other than vehicles and people." It then identifies objects classified as the third category as objects to be tracked.

[0079] For example, the control unit 27 classifies objects in image F1 as either "fallen objects on the road" or "fallen objects," and then tracks objects classified as such.

[0080] For example, in the example shown in Figure 6, empty can B1 is classified as "other than vehicles and people" or "fallen object" and is identified as an object to be tracked.

[0081] When the control unit 27 identifies the object to be tracked, it calculates the trajectory traced by the object based on a series of images in chronological order, including the image in which the object was identified.

[0082] For example, when the control unit 27 identifies the object to be tracked, it retrieves a series of images from the temporary image storage unit 25A in chronological order over a predetermined period of time, starting from the time when the image of the object was captured, and calculates the trajectory using the retrieved series of images.

[0083] For example, when the control unit 27 acquires a series of images, it identifies the object identified as the object to be tracked in each of the frame images included in the series of images by image recognition and calculates the trajectory of the object to be tracked.

[0084] For example, the control unit 27 extracts feature points of the object to be tracked in each of the consecutive images and calculates the trajectory of each feature point based on the coordinates of the feature points in the image. For example, the control unit 27 may use the centroid of the object to be tracked as a feature point to calculate the trajectory of the object to be tracked.

[0085] Figure 6 schematically shows the trajectory TR1 calculated by plotting and connecting the centroids of the empty can B1 in each frame image included in the sequence of images.

[0086] Furthermore, regarding empty can B1 in image F1, empty can B2 in the first frame of a sequence of images preceding the time when image F1 was taken is shown with a dashed line. Therefore, the optical flow of empty can B1 is directed from empty can B2, shown with a dashed line, towards empty can B1, shown with a solid line. Also, in Figure 6, the axis indicating the direction of travel of the vehicle M1 is shown as axis X.

[0087] As shown in Figure 6, the trajectory TR1 is a straight line following the road. Furthermore, the direction of the trajectory TR1 is away from axis X.

[0088] Figure 7 shows an example of a trajectory calculated by the image processing device 10 when the object being tracked is a flying object. Figure 7 shows image F2, which is an example of an image taken in front of the vehicle M1 by the front camera 11. As shown in Figure 7, image F2 shows another vehicle M2 traveling in the adjacent lane and an empty can B1 on the road in front of the vehicle M1. In the example shown in Figure 7, the empty can B1 is a projectile thrown from the other vehicle M2 and is a flying object.

[0089] In the example shown in Figure 7, when the control unit 27 identifies empty can B1 as the object to be tracked, the trajectory TR2 calculated by plotting and connecting the centroids of empty can B1 in each of the consecutive images is schematically shown.

[0090] [Determination based on maximum curvature] For example, when the control unit 27 calculates the trajectory of the object to be tracked, it calculates the maximum curvature of the trajectory, and if the maximum curvature exceeds a threshold, it determines that the trajectory is that of an incoming object.

[0091] The trajectories of flying objects tend to include characteristic curved sections. On the other hand, the trajectories of objects other than flying objects, such as falling objects that existed before the image was taken, vehicles moving in a straight line, or stationary objects, tend to be straight lines. In other words, the trajectories of flying objects tend to have a greater maximum curvature compared to the trajectories of stationary objects. Therefore, by focusing on the difference in the shape of the trajectories between ordinary falling objects and flying objects, a threshold for the curvature of the trajectory can be set, and if the maximum curvature exceeds the threshold, it can be determined that it is a flying object, thereby reliably detecting flying objects and distinguishing them from ordinary falling objects.

[0092] In the example shown in Figure 7, the trajectory TR2, which is the trajectory of an incoming object, contains a characteristic curve near its starting point. In contrast, as shown in Figure 6, the trajectory TR1, which is the trajectory of a falling object, is a straight line along the road. By determining whether the maximum curvature of trajectories TR1 and TR2 exceeds a threshold, it is possible to distinguish and detect trajectory TR2, which is the trajectory of an incoming object, from trajectory TR1, which is the trajectory of a falling object that is not an incoming object.

[0093] [Determination based on orientation] For example, the control unit 27 determines that a trajectory is the trajectory of an incoming object if at least a portion of the calculated trajectory is oriented downwards in the image and approaches the axis X that indicates the direction of travel of the vehicle M1.

[0094] The optical flow of objects flying into the area in front of the vehicle M1 tends to have a downward orientation in the vertical direction of the image and an orientation in the horizontal direction that approaches the axis X, which indicates the direction of travel of the vehicle M1. On the other hand, the trajectory of stationary objects, such as falling objects, tends to be away from axis X. Furthermore, for example, if the tracked objects include vehicles, the trajectory of a vehicle changing lanes or overtaking may include a portion that approaches axis X, but its vertical orientation is upward, which differs from the tendency of flying objects.

[0095] Therefore, by focusing on the difference in the direction of the trajectories of flying objects and stationary objects that are not flying objects, if at least a portion of the trajectory is oriented downwards and approaching axis X, it is determined that the object tracing that trajectory is a flying object, thereby reliably detecting flying objects in distinction from mere falling objects or other vehicles.

[0096] In the example shown in Figure 7, trajectory TR2 includes a portion near the starting point where the trajectory is downward and approaching axis X. In contrast, as shown in Figure 6, the trajectory TR1 of the falling object is moving away from axis X. By determining based on the direction of the trajectory, trajectory TR2, which is the trajectory of the flying object, can be distinguished and detected from trajectory TR1, which is the trajectory of the falling object.

[0097] [Determination by length] Furthermore, the control unit 27 may determine that the calculated trajectory is the trajectory of an incoming object if the length of the trajectory exceeds a predetermined threshold.

[0098] In this embodiment, the length of the trajectory calculated is determined by the relative velocity between the vehicle M1 and the object, and the longer the trajectory, the greater the relative velocity.

[0099] For example, the relative velocity between the vehicle M1 and the incoming object is likely to be greater than the relative velocity between the vehicle M1 and a vehicle traveling in the same direction. Therefore, the length of the trajectory of the incoming object will be longer than the length of the trajectory of a vehicle traveling in the same direction as the vehicle M1. By setting a lower threshold for the trajectory length and determining that a trajectory exceeds this threshold, it is possible to detect the trajectory of the incoming object and distinguish it from the trajectories of vehicles traveling in the same direction.

[0100] For example, the relative velocity between the vehicle M1 and the incoming object is likely to be lower than the relative velocity between the vehicle M1 and an oncoming vehicle. Therefore, the length of the trajectory of the incoming object will be shorter than, for example, the trajectory of an oncoming vehicle. By setting an upper limit on the trajectory length and determining that a trajectory shorter than this limit is that of an incoming object, it is possible to detect the trajectory of an incoming object and distinguish it from the trajectories of oncoming vehicles, etc.

[0101] For example, the image processing device 10 may be able to acquire the speed of the vehicle M1 from an acceleration sensor (not shown) or the like provided on the vehicle M1, and a threshold value for the length of the trajectory may be set for each speed of the vehicle M1.

[0102] Thus, the control unit 27 determines whether the trajectory is that of a flying object, that is, whether a flying object is captured in the continuous images used to calculate the trajectory, by using the maximum curvature of the trajectory of the object being tracked, the direction of the trajectory, or the length of the trajectory. The control unit 27 may determine that a flying object is captured if any of the following conditions are met: the maximum curvature exceeds a threshold, the direction of the trajectory is downward and approaching axis X, or the length of the trajectory exceeds a threshold; or it may determine that a flying object is captured if all of the following conditions are met.

[0103] Furthermore, these determinations can be combined. For example, a determination using maximum curvature can be combined with a determination using the direction of the trajectory, or a determination using maximum curvature can be combined with a determination using the length of the trajectory.

[0104] [Decision based on a pre-trained model] Furthermore, as described above, the control unit 27 may, after calculating the trajectory of the object to be tracked, input the calculated trajectory into a trained model that has been trained in machine learning to output information indicating whether or not the trajectory is that of a flying object when the trajectory drawn by the object to be tracked is input, thereby determining whether or not it is the trajectory of a flying object.

[0105] [Detection of thrown objects] Furthermore, if the control unit 27 determines, in the determination using the maximum curvature or the like, or in the determination using a trained model, that the trajectory is that of a flying object, it may further determine whether a person or vehicle is visible near the starting point of the trajectory in the continuous images, thereby detecting a projectile among the flying objects. In this case, if the control unit 27 determines that a person or vehicle is visible, it determines that a projectile is visible in the continuous images.

[0106] If a person or vehicle is present near the starting point of the trajectory, there is a high probability that the person inside or outside the vehicle threw the object being tracked, and the thrown object can be accurately detected among the incoming objects.

[0107] Referring to Figures 8 and 9, the control routines executed by the control unit 27 of the image processing device 10 in this embodiment will be described.

[0108] Figure 8 is a flowchart showing an example of a flying object detection routine RT1, which is executed by the control unit 27 when it detects a flying object. For example, when the forward camera 11 starts taking pictures of the area around the vehicle M1, the control unit 27 starts the flying object detection routine RT1.

[0109] When the control unit 27 starts the incoming object detection routine RT1, it acquires one frame image from the images sequentially captured by the forward camera 11, for example, from the image temporary storage unit 25A (step S101). In step S101, the control unit 27 may also acquire the frame image by receiving it from the forward camera 11.

[0110] After step S101 is executed, the control unit 27 performs image recognition processing on the frame image acquired in step S101 (step S102). In step S102, for example, the control unit 27 classifies the type of object depicted in the frame image by identifying which of a predetermined number of types it belongs to.

[0111] In step S102, the control unit 27 classifies objects into three categories, for example, by designating the first category as "vehicles," the second category as "people," and the third category as "other than vehicles and people," and identifies the objects classified as the third category as objects to be tracked.

[0112] In step S102, the control unit 27 classifies, for example, objects that are on the road from among the "other than vehicles and people" classified as the third type as fallen objects, and sets the objects classified as fallen objects as objects to be tracked.

[0113] After step S102 is executed, the control unit 27 determines whether or not the object to be tracked is included in the objects classified in step S102 (step S103). In step S103, for example, the control unit 27 determines that the object to be tracked is included if there are objects classified as "other than vehicles and people" or objects classified as "fallen objects".

[0114] If the control unit 27 determines in step S103 that there is no object to track (step S103: NO), it terminates the flying object detection routine RT1 and starts a new flying object detection routine RT1 to acquire the next frame image.

[0115] In step S103, the control unit 27 determines that there is an object to be tracked (step S103: YES), and then retrieves a series of images, including the frame image acquired in step S101, from the image temporary storage unit 25A (step S104). In step S104, for example, if the object to be tracked is a falling object, the control unit 27 retrieves images in which the object classified as a "falling object" is identified, that is, a series of images in chronological order going back from the time when the frame image acquired in step S101 was taken.

[0116] In step S104, for example, the control unit 27 acquires a predetermined number of consecutive images for a time period before or after the time when the frame image acquired in step S101 was captured. In step S104, for example, a number of consecutive images lasting several seconds to more than ten seconds, for example, a number of consecutive images lasting about 10 seconds, is acquired.

[0117] In step S104, the control unit 27 functions as an image acquisition unit that acquires multiple images sequentially taken around the vehicle M1.

[0118] After step 104 is completed, the control unit 27 calculates the trajectory of the object to be tracked based on the continuous images acquired in step S104 (step S105).

[0119] In step S105, for example, the control unit 27 extracts feature points of the object to be tracked in each of the consecutive images, and calculates the trajectory of each feature point by connecting corresponding feature points between the images, thereby calculating the trajectory traced by the object to be tracked.

[0120] In step S105, for example, the control unit 27 calculates the trajectory of the object being tracked by connecting the centers of gravity of the object between images.

[0121] After step S105 is executed, the control unit 27 determines whether the trajectory calculated in step S105 corresponds to the trajectory of an incoming object (step S106). In step S106, for example, the control unit 27 determines that the trajectory is the trajectory of an incoming object if the maximum curvature of the trajectory calculated in step S105 exceeds a threshold, the direction of the trajectory is downward and approaching axis X, or the length of the trajectory exceeds a threshold.

[0122] In step S106, for example, when the control unit 27 receives the trajectory traced by the object to be tracked as input, it uses a trained model that has been machine-learned to output information indicating whether or not the trajectory is that of a flying object. If the output of the trained model indicates that the trajectory is that of a flying object, the control unit 27 determines that the trajectory calculated in step S105 is that of a flying object.

[0123] In step S106, the control unit 27 functions as a flying object detection unit that determines, based on the plurality of images acquired in step S104, whether or not a flying object is captured in any of the plurality of images, either towards the vehicle M1 or in the area in front of the vehicle M1.

[0124] In step S106, if it is determined that the trajectory does not match that of an incoming object (step S106: NO), the control unit 27 terminates the incoming object detection routine RT1 and starts a new incoming object detection routine RT1 to acquire the next frame image.

[0125] In step S106, if it is determined that the image corresponds to the trajectory of an incoming object (step S106: YES), the control unit 27 identifies and labels an image containing the incoming object, for example, an image taken within a predetermined time period including the time the image acquired in step S101 was captured, as a specific image (step S107). In step S107, the control unit 27 functions as an image identification unit.

[0126] In step S107, for example, the control unit 27 labels all frame images included in the sequence of images acquired in step S104 as specific images. Alternatively, in step S107, the control unit 27 may label only a portion of the frame images included in the sequence of images acquired in step S104 as specific images. For example, if the object to be tracked is only visible in a portion of the sequence of images, only the frame images containing the object to be tracked may be labeled as specific images.

[0127] After step S107 is executed, the control unit 27 transmits the specific image to the server device 20 (step S108). In step S108, the control unit 27 functions as an image storage unit that permanently stores the specific image in the server device 20.

[0128] After step S108 is executed, the control unit 27 terminates the flying object detection routine RT1 and starts a new flying object detection routine RT1 to acquire the next frame image. For example, the image acquired in step S101 of the new flying object detection routine RT1 is the next frame image in chronological order of the last frame image of the sequence acquired in the previous flying object detection routine RT1.

[0129] Figure 9 is a modified example of the incoming object detection routine RT1, and is a flowchart showing the projectile detection routine RT2 that the control unit 27 executes when it detects projectiles in particular among incoming objects.

[0130] The projectile detection routine RT2 differs from the flying object detection routine RT1 only in that it includes step S207; for the other steps, it is executed in the same way as the flying object detection routine RT1, so some of the explanation for the overlapping parts will be omitted.

[0131] When the projectile detection routine RT2 is started, the control unit 27, similar to the case of the flying object detection routine RT1, acquires one frame image from the images sequentially captured by the forward camera 11 (step S201), performs image recognition processing (step S202), and if the object to be tracked is included in the object shown in the frame image (step S203: YES), acquires a series of images including the frame image (step S204).

[0132] Subsequently, the control unit 27 calculates the trajectory of the object to be tracked (step S205) and determines whether or not it corresponds to the trajectory of an incoming object (step S206).

[0133] In step S206, if it is determined that the calculated trajectory corresponds to the trajectory of an incoming object (step S206: YES), the control unit 27 determines whether or not a person or vehicle is captured near the starting point of the trajectory (step S207). In step S207, for example, the control unit 27 refers to the image recognition result in step S202 for the image containing the starting point of the trajectory from the sequence of images used to calculate the trajectory, and determines whether or not there is an object classified as a person or vehicle. In step S207, the control unit 27 may also determine whether or not a person or vehicle is captured in an image that is slightly earlier in the timeline than the image containing the starting point of the trajectory.

[0134] For example, in the example shown in Figure 7, another vehicle M2 is located near the starting point of the trajectory traced by the empty can B1, so in step S207, it is determined that a person or vehicle is in the image.

[0135] In step S207, if it is determined that no person or vehicle is captured in the image (step S207: NO), the control unit 27 terminates the projectile detection routine RT2 and starts a new projectile detection routine RT2 to acquire the next frame image.

[0136] In step S207, if it is determined that a person or vehicle is in the image (step S207: YES), the control unit 27 identifies the image containing the flying object, for example, an image taken within a predetermined time period including the time the image acquired in step S201 was captured, as a specific image and labels it (step S208).

[0137] After step S208 is executed, the control unit 27 sends the specific image to the server device 20 (step S209), then terminates the projectile detection routine RT2, and starts a new projectile detection routine RT2 to acquire the next frame image.

[0138] This routine makes it possible to detect projectiles among incoming objects. In step S207 of this routine, if it is determined that a person or vehicle is captured near the starting point of the trajectory, there is a high probability that the object being tracked is a projectile. Therefore, it is possible to accurately detect projectiles and prevent the detection of non-projectiles. For example, the specific images identified by this routine can be used to verify dangerous driving, etc.

[0139] In this embodiment, when detecting projectiles among incoming objects, the control unit 27, acting as the projectile detection unit, may use a pre-trained model that has been trained using machine learning as training data, which includes multiple projectile images taken of a person throwing an object, and information indicating whether a projectile is visible in those multiple projectile images. When the control unit 27 inputs multiple images taken sequentially around the vehicle by the projectile detection unit into the pre-trained model, the pre-trained model outputs information indicating whether or not a projectile is visible in those multiple images. Based on the information output from the pre-trained model, the control unit 27 determines whether or not a projectile is visible in the multiple images.

[0140] For example, the trained model can be built in a large-capacity storage device 25 using as training data images of people throwing various types of objects from inside or outside a vehicle, and information indicating that the thrown objects are visible in the images.

[0141] For example, the determination of whether or not a thrown object is present in an image using the trained model may be performed using only one image from a sequence of images.

[0142] As described above, the image processing system 100, including the image processing apparatus 10 of this embodiment, includes an image acquisition unit that acquires a plurality of images taken sequentially around a vehicle, a flying object detection unit that determines whether or not a flying object towards the vehicle or a flying object towards the area in front of the vehicle is captured in any of the plurality of images based on the plurality of images, and an image identification unit that, if it is determined that a flying object is captured, identifies the images within a predetermined time period including the time when the image containing the flying object was captured as a specific image.

[0143] Therefore, the image processing system 100, including the image processing device 10 of this embodiment, provides an image processing device capable of detecting a vehicle or an incoming object in the area in front of the vehicle.

[0144] For example, in this embodiment, the specific image identified by the image identification unit can be permanently stored on the server. For example, by analyzing the permanently stored specific image, the analysis results can be used for various purposes, such as verifying dangerous events where an object is flying towards or into the area in front of the vehicle.

[0145] For example, in this embodiment, the flying object detection unit identifies the object to be tracked in each of a plurality of images by image recognition, calculates the trajectory traced by the object to be tracked based on the images in chronological order that include the image in which the object to be tracked was identified, and determines whether or not a flying object is captured in the image by determining whether or not the object to be tracked is a flying object based on the trajectory.

[0146] This method of detection allows for efficient detection of incoming objects because, for example, it is not necessary to calculate the trajectories of all objects in an image, but rather to calculate the trajectory of the object being tracked. [Examples]

[0147] Referring to Figures 10-12, the image processing apparatus 10 and image processing system 100 according to Embodiment 2 will be described. The image processing apparatus 10 and image processing system 100 of this embodiment are configured in the same way as in Embodiment 1, with only the function of the control unit 27 as a flying object detection unit differing from Embodiment 1.

[0148] In this embodiment, the control unit 27, acting as a flying object detection unit, acquires a series of images taken sequentially around the vehicle M1, calculates the optical flow for the series of images, and determines whether or not a flying object is captured based on the calculated optical flow.

[0149] Optical flow is a motion vector that represents the apparent movement of each pixel between two frames in a moving image. In this specification, the trajectory drawn by accumulating the inter-image vectors of each pixel between two consecutive images contained in a sequence of images is also referred to as optical flow.

[0150] The control unit 27 calculates the optical flow using a known method, such as the block matching method or the gradient method.

[0151] For example, before calculating the optical flow, the control unit 27 extracts feature points such as edges and corners of the grayscale pattern in the first frame image included in the sequence of images. The control unit 27 then calculates the optical flow of these feature points.

[0152] The control unit 27 may calculate the optical flow for all pixels in each of the continuous images, but calculating the optical flow for all pixels would increase the processing load. Therefore, in this embodiment, the optical flow is calculated for the pixels that have been thinned out using feature points as described above.

[0153] Figure 10 is a schematic diagram showing an example of the optical flow calculated by the image processing device 10 in this embodiment. Figure 10 shows image F3, which is an example of an image taken in front of the vehicle M1 by the front camera 11. As shown in Figure 10, image F3 shows another vehicle M2, an oncoming vehicle M3, an empty can B1, and the scenery around the road, such as buildings and street trees. In the example shown in Figure 10, the empty can B1 is a projectile thrown from the other vehicle M2 and is a flying object.

[0154] Furthermore, as shown in Figure 10, the optical flow calculated using a sequence of images with image F3 as the last frame is superimposed and plotted on image F3.

[0155] Specifically, image F3 displays the optical flow OF1 of the empty can B1, the optical flow OF2 of the oncoming car M3, and the optical flow OF3 of stationary background objects such as buildings and street trees around the road. In Figure 10, the empty can B2 in the first frame of the sequence is shown with a dashed line. Therefore, in image F3, the starting point of optical flow OF1 is on the empty can B2 shown with a dashed line, and the ending point of optical flow OF1 is on the empty can B1 shown with a solid line.

[0156] The optical flow shown in Figure 10 can be calculated, for example, by extracting feature points based on the grayscale values ​​of pixels as described above, and then using a method such as the gradient method. The control unit 27 may also calculate the optical flow for all pixels in a continuous image including image F3.

[0157] For example, when the control unit 27 calculates the optical flow, it determines whether the calculated optical flow is the optical flow of an incoming object, that is, whether an incoming object is captured in the continuous image used to calculate the optical flow, by using a determination based on the maximum curvature of the optical flow, a determination based on the direction of the optical flow, or a determination based on the length of the optical flow, similar to the case in Embodiment 1.

[0158] [Determination based on maximum curvature] For example, when the control unit 27 calculates the optical flow, it calculates the maximum curvature of the optical flow, and if the maximum curvature exceeds a threshold, it determines that the optical flow is the optical flow of an incoming object.

[0159] For example, the control unit 27 calculates the maximum curvature for each of the calculated optical flows and determines whether the maximum curvature for each of the optical flows exceeds a threshold.

[0160] For example, the control unit 27 may divide the frame image included in the continuous image into multiple regions, calculate the average value of the maximum curvature of multiple optical flows in each of the multiple regions, and determine whether or not the maximum curvature exceeds a threshold.

[0161] Similar to the trajectory described above, the optical flow of a flying object tends to include characteristic curved portions. On the other hand, among objects other than flying objects, the optical flow of at least a vehicle moving in a straight line and stationary objects tends to be linear. Therefore, by using the maximum curvature of the optical flow as the criterion, the optical flow of a flying object can be detected.

[0162] In the example shown in Figure 10, the optical flow OF1 of the empty can B1 includes a curved portion, while the optical flow OF2 of the oncoming car M3 and the optical flow OF3 of the background are linear. By using a threshold value for the maximum curvature of the optical flow, the optical flow OF1 of the flying object, the empty can B1, can be detected separately from the linear optical flows of stationary objects such as the background optical flow OF3 and the oncoming car M3 optical flow OF2.

[0163] [Determination based on orientation] For example, the control unit 27 determines that the optical flow is the optical flow of an incoming object if the direction of at least a portion of the calculated optical flow is downward and approaches the axis X that indicates the direction of travel of the vehicle M1.

[0164] Similar to the trajectory described above, the optical flow of an object flying into the area in front of the vehicle M1 tends to have a downward orientation in the vertical direction of the image and an orientation toward the axis X indicating the direction of travel of the vehicle M1 in the horizontal direction.

[0165] For example, the optical flow direction in images of stationary objects such as falling objects or oncoming vehicles tends to be away from axis X. Also, the optical flow of a vehicle traveling in the same direction as the vehicle M1 and changing lanes or overtaking may be towards axis X, but its vertical direction is upward, which differs from the optical flow of a flying object. By using the direction of optical flow as a criterion, the optical flow of a flying object can be detected.

[0166] In the example shown in Figure 10, the orientation of optical flow OF1 in the image is downward and approaching axis X in the curved portion near the starting point. In contrast, the orientations of optical flow OF2 and optical flow OF3 in the images are moving away from axis X. By making a determination based on the orientation of the optical flow, optical flow OF1, which is the optical flow of the incoming object empty can B1, can be detected separately from optical flows such as the background optical flow OF3 and the optical flow OF2 of the oncoming vehicle.

[0167] [Determination by length] Furthermore, the control unit 27 determines that the optical flow is the optical flow of an incoming object if the calculated optical flow length exceeds a threshold.

[0168] In this embodiment, the length of the optical flow is determined by the relative velocity between the vehicle M1 and the object.

[0169] The length of the optical flow of an incoming object is expected to be longer than, for example, the length of a vehicle traveling in the same direction. Therefore, by setting a lower threshold for the length and making a determination, it is possible to detect the optical flow of an incoming object by distinguishing it from the optical flow of a vehicle traveling in the same direction, which is shorter than the threshold.

[0170] The length of the optical flow of an incoming object is expected to be shorter, for example, compared to that of an oncoming vehicle. Therefore, by setting an upper limit on the length and making a determination based on that, it is possible to detect the optical flow of an incoming object in distinction from the optical flow of an oncoming vehicle or other object that is longer than the upper limit.

[0171] In the example shown in Figure 10, the optical flow OF1 of the empty can B1 is shorter than the background optical flow OF3 and the optical flow OF2 of the oncoming vehicle M3. By using the length of the optical flow as a criterion for determination, the optical flow OF1 of the flying object, the empty can B1, can be detected separately from the background optical flow OF3 and the oncoming vehicle's optical flow OF2, etc.

[0172] For example, the image processing device 10 may be able to acquire the speed of the vehicle M1 from an acceleration sensor (not shown) or the like provided on the vehicle M, and a threshold value for the length of the optical flow may be set for each speed of the vehicle M1.

[0173] The control unit 27 may combine the determination using maximum curvature, determination using the direction of optical flow, and determination using the length of optical flow. For example, if a vehicle overtakes the vehicle M1, the optical flow of that vehicle will be in the direction of approaching axis X. In this case, if only determination using the direction of optical flow is performed, the overtaken vehicle may be detected as a flying object. In such cases, by combining determination using maximum curvature and length, false detections can be prevented and flying objects can be detected with high accuracy.

[0174] [Determination by splitting the image] For example, when determining whether or not a flying object is captured in a continuous image, the control unit 27 may divide the frame image into multiple regions and make the determination based on optical flow-related values ​​such as the slope, curvature, or length of the optical flow in each of the multiple regions.

[0175] For example, if there is an optical flow within a region where the value related to the optical flow shows a singular value, it may be determined that an incoming object is captured in each of the consecutive images from which the optical flow was calculated.

[0176] For example, the control unit 27 may divide the frame image into multiple regions (blocks), calculate the average value of optical flow-related values ​​such as the slope, curvature, or length of the optical flow for each block, and determine that an incoming object is captured in one block if there is an optical flow within that block whose difference from the average value exceeds a threshold.

[0177] Alternatively, if the standard deviation of the slope of the optical flow within block 1 exceeds a threshold, it is considered highly likely that the block contains an optical flow exhibiting an outlier in the slope, and if such a block exists, it may be determined that an incoming object is captured in the image. In this case, the threshold may be set according to the average of the standard deviations of the slopes of the optical flow of multiple blocks located around block 1 in the frame image.

[0178] The control unit 27 may, when determining whether or not a flying object is captured in a continuous image, divide the frame image into multiple regions and make the determination based on a comparison of optical flow-related values ​​such as the slope, curvature, or length of the optical flow in each of the multiple regions.

[0179] In this case, for example, the average value of the optical flow within each block may be calculated and compared with adjacent blocks. If the difference between the average values ​​of adjacent blocks exceeds a predetermined threshold, it may be determined that an incoming object is captured in each of the consecutive images.

[0180] [Exclusion of area] Furthermore, for example, the control unit 27 may perform image recognition on each of the multiple frame images included in the continuous image before calculating the optical flow, classify the objects depicted in each of the multiple frame images into multiple types, and exclude the regions containing objects of a predetermined type from the calculation of the optical flow.

[0181] For example, the control unit 27 classifies objects in a frame image into three categories: the first category is "vehicles," the second category is "people," and the third category is "other than vehicles and people." The control unit then excludes areas containing objects classified as the first and second categories from the calculation of optical flow.

[0182] Since people and vehicles are clearly not flying objects, optical flow calculation is unnecessary for pixels in areas containing people or vehicles when detecting flying objects. By excluding such pixels from the calculation beforehand, the processing load can be reduced.

[0183] For example, before calculating the optical flow, if the control unit 27 performs image recognition on each of the multiple frames included in the continuous image and finds an object classified as a "fallen object" that was on the road before being captured by the front camera 11, it may exclude the area other than the area in which the "fallen object" is captured from the calculation of the optical flow. In other words, the control unit 27 will include the area in which the "fallen object" is captured as the target for calculating the optical flow.

[0184] Alternatively, for example, the "falling object" could be identified in the first frame image, a portion of the image containing the area where the "falling object" is visible could be determined, and optical flow calculations could be performed only for the pixels within that portion of the image.

[0185] Referring to Figures 11 and 12, the control routine executed by the control unit 27 of the image processing device 10 in this embodiment will be described.

[0186] Figure 11 is a flowchart showing an example of a flying object detection routine RT3, which is executed by the control unit 27 when it detects a flying object. For example, when the forward camera 11 starts taking pictures of the area around the vehicle M1, the control unit 27 starts the flying object detection routine RT3.

[0187] When the control unit 27 starts the incoming object detection routine RT3, it acquires a series of images from the images sequentially captured by the forward camera 11 (step S301). In step S301, for example, the control unit 27 acquires a series of images for a predetermined time or a predetermined number of frame images. In step S301, for example, a series of images lasting several seconds to more than ten seconds, for example, a series of images lasting about 10 seconds, is acquired.

[0188] In step S301, for example, the control unit 27 acquires a series of images from the image temporary storage unit 25A. Alternatively, in step S301, a series of images may be acquired from the front camera 11.

[0189] After step S301 is executed, the control unit 27 calculates the optical flow by accumulating the inter-image vectors for each pixel between two consecutive images included in the continuous image (step S302). In step S302, for example, the control unit 27 extracts pixels that become feature points based on the grayscale values ​​of pixels in the first frame image of the continuous image, and calculates the optical flow of the extracted pixels in the continuous image using a method such as the gradient method. In step S302, for example, an optical flow like the one shown in Figure 10 is calculated.

[0190] After step S302 is executed, the control unit 27 determines whether or not the optical flow of the incoming object is included in the optical flow calculated in step S302 (step S303).

[0191] In step S303, for example, the control unit 27 determines whether or not the optical flow of the incoming object is included, using the maximum curvature, direction, or length of the optical flow as a determination criterion. For example, in step S303, if the maximum curvature of the optical flow exceeds a threshold, if the direction of the optical flow is downward and approaches the axis indicating the direction of travel of the vehicle M1, or if the length of the trajectory exceeds a predetermined threshold, it is determined that the optical flow of the incoming object is included.

[0192] In step S303, for example, the control unit 27 may divide the frame image into multiple regions as described above, and determine whether or not the optical flow of an incoming object is included based on the presence or absence of singular values ​​within a region for optical flow-related values ​​such as the slope, curvature, or length of the optical flow in each of the multiple regions, or by comparing the multiple regions.

[0193] In step S303, if it is determined that the optical flow of the incoming object is not included (step S303: NO), the control unit 27 terminates the incoming object detection routine RT3 and starts a new incoming object detection routine RT3 to acquire a new series of images.

[0194] In step S303, if it is determined that the optical flow of the flying object is included (step S303: YES), the control unit 27 identifies the images within a predetermined time period including the time when the image containing the flying object was captured as specific images and labels them (step S304).

[0195] In step S304, for example, the control unit 27 identifies all frame images included in the sequence of images acquired in step S301 as specific images and labels each of those frame images. For example, in step S304, a portion of the sequence of images acquired in step S301 may be identified as specific images.

[0196] After step S304 is completed, the control unit 27 transmits the specific image to the server device 20 (step S305). In step S305, the specific image is permanently stored in the server device 20.

[0197] After step S305 is executed, the control unit 27 terminates the flying object detection routine RT3 and starts a new flying object detection routine RT3 to acquire the next sequence of images. For example, the sequence of images acquired in step S301 of the new flying object detection routine RT3 may start from the frame image that follows the last frame image in the chronological order of the sequence of images acquired in the previous flying object detection routine RT3, or it may be a sequence of images that partially overlaps with the sequence of images acquired in the previous flying object detection routine RT3.

[0198] Figure 12 is a flowchart showing a modified version of the flying object detection routine RT3, called the flying object detection routine RT4. The only difference between the flying object detection routine RT4 and the flying object detection routine RT3 is that it includes step S402. Since the other steps are performed in the same way as the flying object detection routine RT3, some of the explanation for the overlapping parts will be omitted.

[0199] When the control unit 27 starts the incoming object detection routine RT4, it acquires a series of images from the images sequentially captured by the forward camera 11, similar to the case of the incoming object detection routine RT3 (step S401).

[0200] After step S401 is executed, the control unit 27 performs an area exclusion process to exclude areas where optical flow calculation is unnecessary from the optical flow calculation (step S402). In step S402, for example, the control unit 27 performs image recognition on each of the multiple frame images included in the continuous image, classifies the objects depicted in each of the multiple frame images into multiple types, and excludes areas containing objects of a predetermined type from the optical flow calculation.

[0201] For example, in step S402, the control unit 27 classifies the objects in each of the multiple frame images into "vehicles," "people," and "other than vehicles and people," and pixels in the areas where objects classified as "vehicles" and "people" are depicted are excluded from the optical flow calculation.

[0202] After step S402 is executed, the optical flow is calculated by accumulating the inter-image vectors for each pixel between two consecutive images included in the continuous image (step S403). In step S403, the optical flow is calculated using the pixel data of the region that was not excluded in step S402.

[0203] After step S403 is executed, the control unit 27 determines whether or not the optical flow of the incoming object is included in the optical flow calculated in step S403 (step S404). In step S404, for example, the control unit 27 determines whether or not the optical flow of the incoming object is included by making a determination based on the maximum curvature, direction, or length of the optical flow, or by determining whether or not there is a region in which the values ​​related to the optical flow show singular values, similar to step S303 of the incoming object detection routine RT3.

[0204] In step S404, if it is determined that the optical flow of an incoming object is included (step S404: YES), then, for example, all frame images included in the continuous image acquired in step S401 are identified as specific images, each of these frame images is labeled (step S405), and the specific images are transmitted to the server device 20 (step S406).

[0205] This routine reduces the processing load associated with optical flow calculations compared to calculating optical flow for the entire region of a continuous image.

[0206] Furthermore, when detecting projectiles among incoming objects, step S207 of the projectile detection routine RT2 of Example 1 may be combined with the incoming object detection routine RT3 or the incoming object detection routine RT4. Specifically, it may be determined whether a person or vehicle is visible near the starting point of the optical flow in the continuous image, and if it is determined that a person or vehicle is visible, it may be determined that a projectile is visible in at least one of the frames included in the continuous image.

[0207] As described in detail above, in the image processing apparatus 10 of this embodiment, the flying object detection unit is configured to calculate an optical flow obtained by accumulating the inter-image vectors for each pixel between two consecutive images included in a series of images in chronological order, and to determine whether or not a flying object is captured in either of the consecutive images based on the optical flow.

[0208] For example, in determining whether or not a flying object is captured in the image, the incoming object detection unit determines whether or not the optical flow of the flying object is included in the calculated optical flow by using the maximum curvature, direction, or length of the optical flow as the determination criteria.

[0209] Furthermore, for example, in determining whether or not a flying object is captured in an image, the object detection unit divides the frame image into multiple regions and determines whether or not a flying object is captured in each of the consecutive images based on whether or not there is an optical flow in each of the multiple regions where the values ​​related to optical flow, such as the slope, curvature, or length, show an outlier value.

[0210] The image processing device 10 of this embodiment provides an image processing device capable of detecting flying objects that are flying toward a vehicle or into the area in front of a vehicle.

[0211] The configurations and routines in the above-described embodiments are merely illustrative examples and can be appropriately selected, combined, and modified depending on the application and other factors.

[0212] For example, in the above embodiment, an example was described in which a specific image is stored in the permanent storage unit of the server device 20, but the embodiment is not limited to this. The permanent storage unit may be provided in the large-capacity storage device 25 of the image processing device 10, and the control unit 27 may configure the unit to permanently store the identified specific image in the storage unit of the large-capacity storage device 25.

[0213] Furthermore, the control unit 45 of the server device 20 may have the functions of an image acquisition unit, a flying object detection unit, and an image identification unit of the control unit 27 of the image processing device 10.

[0214] In the above embodiment, image processing of images captured by a front camera was described as an example, but the invention is not limited to this. For example, the image processing device of the present invention can detect flying objects using images captured by side cameras mounted on the vehicle. For example, the side cameras are mounted above the left and right doors of the vehicle, etc., so that the shooting direction includes a horizontal direction that is substantially perpendicular to the direction of travel of the vehicle M1.

[0215] For example, by using images from the side camera, it is possible to detect objects thrown at the vehicle from a vehicle traveling in an adjacent lane, or objects thrown from the side towards the vehicle from the roadside, as flying objects towards the vehicle.

[0216] Furthermore, in the present invention, a rear camera mounted so that the rear of the vehicle is the shooting direction, or images captured by, for example, a front camera 11, a side camera 13, and a 360-degree camera mounted on the vehicle instead of the rear camera, may be used to detect objects flying towards the vehicle from various directions.

[0217] For example, the image processing device 10 may be a terminal device having a similar configuration to the image processing device 10, with a front camera 11 and a touch panel display 15 integrated into one unit. Specifically, for example, the image processing device 10 may be a terminal device such as a smartphone, tablet, or PC with a camera equipped with an application that performs the same functions as the image processing device 10 described above. In this case, the image processing device 10 may be mounted on the dashboard DB, for example, in a cradle, so that the built-in camera can capture images of the area in front of the vehicle M1 through the windshield of the vehicle M1.

[0218] Furthermore, the image processing device 10 may be configured not to display a screen to the driver of the vehicle M1. For example, the image processing device 10 may be a device with a configuration such as a drive recorder integrated with the front camera 11. Specifically, the image processing device 10 may be a device in which, for example, hardware that performs the function of an image processing device 10 that detects flying objects and identifies specific images, as described above, is built into the housing of the front camera 11. [Explanation of symbols]

[0219] 10 Image Processing Device 20 Server Devices 11. Front camera 23 Input section 25 Mass storage 25A Image Temporary Storage Unit 27 Control Unit 29 Communications Department 43 Mass storage 43A Permanent Memory Unit 45 Control Unit 47 Communications Department

Claims

1. An image acquisition unit that acquires multiple images taken sequentially around the vehicle, A flying object detection unit identifies objects that have fallen onto the road from among the objects captured in each of the aforementioned multiple images as objects to be tracked, calculates the trajectory traced by the object based on images in chronological order going backward from the time the image in which the object to be tracked was captured was taken, and determines whether or not an object flying towards the vehicle or into the area in front of the vehicle is captured in any of the aforementioned multiple images based on the trajectory. An image processing apparatus characterized by having an image identification unit that, when it is determined that the aforementioned flying object is in the image, identifies the images within a predetermined time period including the time when the image containing the flying object was taken as a specific image.

2. An image acquisition unit that acquires multiple images taken sequentially around the vehicle, A flying object detection unit analyzes the aforementioned multiple images to identify the object to be tracked in each image, calculates the trajectory traced by the object based on the time-series images including the image in which the object to be tracked was identified, and determines that an object flying towards the vehicle or into the area in front of the vehicle is captured in any of the aforementioned multiple images if the maximum curvature of the trajectory exceeds a first threshold, the direction of the trajectory is downward and approaches the axis indicating the direction of travel of the vehicle, or the length of the trajectory exceeds a second threshold. If it is determined that the aforementioned flying object is captured in the image, the image identification unit extracts images within a predetermined time period including the time when the image containing the flying object was taken as a specific image. An image processing apparatus characterized by having

3. An image acquisition unit that acquires multiple images by sequentially photographing the area around the vehicle, A flying object detection unit that determines, based on the plurality of images, whether or not a flying object is visible in any of the plurality of images toward the vehicle or in the area in front of the vehicle, When it is determined that the aforementioned flying object is captured in the image, the image identification unit identifies the images within a predetermined time period, including the time when the image containing the flying object was taken, as a specific image. It has, The flying object detection unit analyzes the multiple images to identify the object to be tracked in each image, calculates the trajectory traced by the object based on the images in chronological order that include the image in which the object to be tracked was identified, determines whether a person or vehicle is visible near the starting point of the trajectory in the multiple images, and if it is determined that a person or vehicle is visible, determines that the flying object, which is a projectile, is visible in one of the multiple images. An image processing apparatus characterized by the following:

4. An image acquisition unit that acquires a series of images in chronological order by sequentially photographing the area around the vehicle, A flying object detection unit calculates an optical flow by accumulating the inter-image vectors for each pixel between two consecutive images included in the aforementioned continuous image, and determines that a flying object is captured on the vehicle or in the area in front of the vehicle if the maximum curvature of the optical flow exceeds a first threshold, the direction of the accumulated optical flow is downward and approaches the axis indicating the direction of travel of the vehicle, or the length of the optical flow exceeds a second threshold. When it is determined that the aforementioned flying object is captured in an image, the image identification unit identifies the images within a predetermined time period, including the time when the image containing the flying object was taken, as a specific image. An image processing apparatus characterized by having

5. The image processing apparatus according to claim 4, wherein the incoming object detection unit divides the frame images included in the continuous image into a plurality of regions, and determines whether or not the incoming object is captured in each of the continuous images based on whether or not there is an optical flow in each of the plurality of regions in which the value related to optical flow shows a singular value.

6. The image processing apparatus according to claim 4, wherein the incoming object detection unit divides the frame images included in the continuous image into a plurality of regions, and determines whether or not the incoming object is captured in each of the continuous images based on a comparison of the optical flow values ​​in each of the plurality of regions between the plurality of regions.

7. The image processing apparatus according to claim 4, characterized in that the incoming object detection unit classifies the objects depicted in each of the consecutive images into a plurality of types by image recognition, and excludes the regions in the images in which objects classified into a predetermined type are depicted from the calculation of the optical flow.

8. The image processing apparatus according to claim 4, characterized in that the incoming object detection unit identifies a falling object on the road from among the objects depicted in each of the consecutive images by image recognition, and the region in the image in which the object identified as the falling object is depicted is the target of the optical flow calculation.

9. The aforementioned flying object is a projectile, The image processing apparatus according to claim 4, wherein the incoming object detection unit determines whether a person or vehicle is captured near the starting point of the optical flow in the continuous image, and if it determines that a person or vehicle is captured, it determines that the projectile is captured in any of the continuous images.

10. An image processing method performed by an image processing device, Image acquisition step: Obtain multiple images by sequentially taking pictures around the vehicle, A flying object detection step involves identifying objects that have fallen onto the road from among the objects captured in each of the aforementioned multiple images as objects to be tracked, calculating the trajectory traced by the object based on images in chronological order going backward from the time the image in which the object to be tracked was captured, and determining whether or not an object flying towards the vehicle or into the area in front of the vehicle is captured in any of the aforementioned multiple images based on the trajectory. If it is determined that the aforementioned flying object is captured in the image, the image identification step involves identifying the images taken within a predetermined time period, including the time when the image containing the flying object was taken, as the specific image. An image processing method characterized by including [a certain element].

11. An image processing program executed by an image processing device equipped with a computer, wherein the computer, Image acquisition step: Obtain multiple images by sequentially taking pictures around the vehicle, A flying object detection step involves identifying objects that have fallen onto the road from among the objects captured in each of the aforementioned multiple images as objects to be tracked, calculating the trajectory traced by the object based on images in chronological order going backward from the time the image in which the object to be tracked was captured, and determining whether or not an object flying towards the vehicle or into the area in front of the vehicle is captured in any of the aforementioned multiple images based on the trajectory. If it is determined that the aforementioned flying object is captured in the image, the image identification step involves identifying the images taken within a predetermined time period, including the time when the image containing the flying object was taken, as the specific image. An image processing program to execute this.

12. An image processing method performed by an image processing device, Image acquisition step: Obtain multiple images by sequentially taking pictures around the vehicle, A flying object detection step in which a target object to be tracked is identified in each of the multiple images by analyzing the multiple images, the trajectory traced by the target object is calculated based on the images in chronological order including the image in which the target object to be tracked is identified, and if the maximum curvature of the trajectory exceeds a first threshold, if the direction of the trajectory is downward and approaches the axis indicating the direction of travel of the vehicle, or if the length of the trajectory exceeds a second threshold, it is determined that a flying object towards the vehicle or into the area in front of the vehicle is captured in any of the multiple images. If it is determined that the aforementioned flying object is captured in the image, the image identification step involves extracting images from within a predetermined time period, including the time when the image containing the flying object was taken, as a specific image. An image processing method characterized by including [a certain element].

13. An image processing program executed by an image processing device equipped with a computer, wherein the computer, Image acquisition step: Obtain multiple images by sequentially taking pictures around the vehicle, A flying object detection step in which a target object to be tracked is identified in each of the multiple images by analyzing the multiple images, the trajectory traced by the target object is calculated based on the images in chronological order including the image in which the target object to be tracked is identified, and if the maximum curvature of the trajectory exceeds a first threshold, if the direction of the trajectory is downward and approaches the axis indicating the direction of travel of the vehicle, or if the length of the trajectory exceeds a second threshold, it is determined that a flying object towards the vehicle or into the area in front of the vehicle is captured in any of the multiple images. If it is determined that the aforementioned flying object is captured in the image, the image identification step involves extracting images from within a predetermined time period, including the time when the image containing the flying object was taken, as a specific image. An image processing program to execute this.

14. An image processing method performed by an image processing device, Image acquisition step: Obtain multiple images by sequentially taking pictures around the vehicle, A flying object detection step, based on the plurality of images, determines whether or not a flying object is visible in any of the plurality of images toward the vehicle or in the area in front of the vehicle. If it is determined that the aforementioned flying object is captured in the image, the image identification step involves identifying the images taken within a predetermined time period, including the time when the image containing the flying object was taken, as the specific image. Includes, The aforementioned flying object detection step includes: identifying the object to be tracked in each image by analyzing the plurality of images; calculating the trajectory traced by the object to be tracked based on the images in chronological order, including the image in which the object to be tracked was identified; determining whether a person or vehicle is visible near the starting point of the trajectory in the plurality of images; and, if it is determined that a person or vehicle is visible, determining that the flying object, which is a projectile, is visible in one of the plurality of images. An image processing method characterized by the following:

15. An image processing program executed by an image processing device equipped with a computer, wherein the computer, Image acquisition step: Obtain multiple images by sequentially taking pictures around the vehicle, A flying object detection step, based on the plurality of images, determines whether or not a flying object is visible in any of the plurality of images toward the vehicle or in the area in front of the vehicle. If it is determined that the aforementioned flying object is captured in the image, the image identification step involves identifying the images taken within a predetermined time period, including the time when the image containing the flying object was taken, as the specific image. Make it run, The aforementioned flying object detection step includes: identifying the object to be tracked in each image by analyzing the plurality of images; calculating the trajectory traced by the object to be tracked based on the images in chronological order, including the image in which the object to be tracked was identified; determining whether a person or vehicle is visible near the starting point of the trajectory in the plurality of images; and, if it is determined that a person or vehicle is visible, determining that the flying object, which is a projectile, is visible in one of the plurality of images. An image processing program characterized by the following:

16. An image processing method performed by an image processing device, The image acquisition step involves obtaining a series of images in chronological order by sequentially photographing the area around the vehicle, A flying object detection step in which an optical flow is calculated by accumulating the inter-image vectors for each pixel between two consecutive images included in the continuous image, and if the maximum curvature of the optical flow exceeds a first threshold, if the direction of the accumulated optical flow is downward and approaches the axis indicating the direction of travel of the vehicle, or if the length of the optical flow exceeds a second threshold, it is determined that a flying object is captured on the vehicle or in the area in front of the vehicle. When it is determined that the aforementioned flying object is captured in the image, the image identification step involves identifying the images taken within a predetermined time period, including the time when the image containing the flying object was taken, as the specified image. An image processing method characterized by including [a certain element].

17. An image processing program executed by an image processing device equipped with a computer, wherein the computer, The image acquisition step involves obtaining a series of images in chronological order by sequentially photographing the area around the vehicle, A flying object detection step in which an optical flow is calculated by accumulating the inter-image vectors for each pixel between two consecutive images included in the continuous image, and if the maximum curvature of the optical flow exceeds a first threshold, if the direction of the accumulated optical flow is downward and approaches the axis indicating the direction of travel of the vehicle, or if the length of the optical flow exceeds a second threshold, it is determined that a flying object is captured on the vehicle or in the area in front of the vehicle. When it is determined that the aforementioned flying object is captured in the image, the image identification step involves identifying the images taken within a predetermined time period, including the time when the image containing the flying object was taken, as the specified image. An image processing program to execute this.

18. A computer-readable recording medium storing an image processing program according to claim 11, 13, 15, or claim 17.

Images