Image processing apparatus, image processing method, and image processing program

Abstract

To reduce a processing load even when a high-resolution moving image is photographed by a camera.SOLUTION: The image processing device includes a controller that performs a process of determining whether a vehicle is included in a frame image of a moving image in which a road is captured by a camera, a first detection process of generating a reduced image in which the number of pixels of the frame image is reduced and detecting an object on the road from the reduced image when it is determined that the vehicle is not included in the frame image, and a second detection process of detecting the object on the road from the frame image when it is determined that the vehicle is included in the frame image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and an image processing program. [Background technology]

[0002] A monitoring system has been proposed that uses a camera to monitor fallen objects on the road (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-124986 Summary of the Invention [Problem to be solved by the invention]

[0004] It is preferable to use high-resolution video for detecting falling objects. However, the higher the resolution of the video, the greater the amount of video data, which increases the processing load. Therefore, when high-resolution video is input into a computer with low processing power to detect falling objects, processing delays may occur.

[0005] One aspect of the disclosed technology is to provide an image processing device, an image processing method, and an image processing program that can reduce the processing load even when a high-resolution video is captured by a camera. [Means for solving the problem]

[0006] One aspect of the disclosed technology is exemplified by the following image processing device: The image processing device includes a control unit that executes a process of determining whether or not a vehicle is included in a frame image of a video in which a road is captured by a camera, a first detection process of generating a reduced image by reducing the number of pixels of the frame image and detecting an object on the road from the reduced image if it is determined that the vehicle is not included in the frame image, and a second detection process of detecting an object on the road from the frame image if it is determined that the vehicle is included in the frame image.

[0007] It is believed that objects falling onto the road are often vehicles. Therefore, when it is determined that the vehicle is not included in the frame image, the image processing device generates a reduced image by reducing the number of pixels of the frame image and detects an object on the road from the reduced image. Detecting an object from the reduced image by reducing the number of pixels can reduce the processing load of the image processing device compared to detecting an object from a frame image before the number of pixels is reduced. The vehicle detection may be performed by inputting the frame image into a learning model constructed using training data including image data of the road with the vehicle traveling and image data of the road without the vehicle traveling.

[0008] Here, the process of generating the reduced image may include a process of deleting pixels in even-numbered columns and even-numbered rows from among the pixels in the frame image, and a process of deleting pixels in odd-numbered columns and odd-numbered rows from among the pixels in the frame image.

[0009] The image processing device may further include the following feature: The method includes alternately applying, to each of a series of frame images in the moving image, a first deletion process that deletes pixels in even columns and even rows from among the pixels in the frame images, and a second deletion process that deletes pixels in odd columns and odd rows from among the pixels in the frame images. The deleted image generated by the first deletion process includes pixels that would be deleted by the second deletion process. Furthermore, the deleted image generated by the second deletion process includes pixels that would be deleted by the first deletion process. Therefore, the image processing device can prevent specific pixels from being overlooked.

[0010] The image processing device may further include the following feature. When it is determined that the vehicle is included in the first frame image, the control unit detects an object on the road from a second frame image acquired within a predetermined period after the first frame image, even if the second frame image does not include the vehicle. The possibility of an object falling onto the road is considered to be higher after a vehicle is detected. The image processing device can reduce the possibility of overlooking a fallen object by detecting an object based on frame images in which the number of pixels has not been reduced during the predetermined period after the detection of a vehicle suspected of containing a fallen object.

[0011] The image processing device may further include the following feature: the road includes a plurality of lanes, the determining process includes a process of determining whether or not a vehicle is present in each of the plurality of lanes, and the first detection process and the second detection process omit detecting the object for the lane in which it is determined that the vehicle is not present. It is considered that the possibility of a new fallen object occurring in a lane in which no vehicle is detected is low. By including such a feature, the image processing device can reduce the processing load related to object detection.

[0012] The image processing device may further include the following feature: When the control unit determines in the determining process that the vehicle is included in the frame image, it sets a predetermined timer, and executes the second detection process until the timer expires, even if it is determined that the vehicle is not included in the frame image. By including this feature, the image processing device can reduce the possibility of overlooking a fallen object that occurs after a vehicle has passed.

[0013] The image processing device may further include the following feature: When the vehicle is detected in a predetermined range of pixels in the frame image in the determining process, the control unit sets a predetermined timer for the predetermined range of pixels, and excludes the predetermined range of pixels from targets for reduction from the frame image in generating the reduced image until the timer expires. By including such a feature, the image processing device can reduce the processing load due to the reduction process.

[0014] The disclosed technology can also be understood from the aspects of an image processing method and an image processing program. [Effects of the Invention]

[0015] According to the disclosed technology, the processing load can be reduced even when high-resolution video is captured by a camera. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram illustrating an example of a falling object detection system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of the detection device according to the embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of a processing block of the detection device according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of processing for reducing the number of pixels in a frame image by the reduction unit in the embodiment. [Figure 5]FIG. 5 is a diagram illustrating an example of foreground extraction by the extraction unit. [Figure 6] FIG. 6 is a diagram illustrating an example of a processing flow of the detection device according to the embodiment. [Figure 7] FIG. 7 is a diagram showing an example of a processing flow of processing performed by the extraction unit to reduce the number of pixels in a frame image. [Figure 8] FIG. 8 is a diagram illustrating an example of a processing block of a detection device according to a first modified example. [Figure 9] FIG. 9 is a diagram schematically showing the masking process by the vehicle detection unit in the first modified example. [Figure 10] FIG. 10 is a diagram illustrating an example of a processing block of a detection device according to the second modification. [Figure 11] FIG. 11 is a diagram schematically illustrating a state before the timer is set by the reduction unit 22B in the second modified example. [Figure 12] FIG. 12 is a diagram schematically showing a state in which a timer is set when a vehicle is detected in a frame image in the second modified example. [Figure 13] FIG. 13 is a diagram schematically showing the setting of the timer when the vehicle moves in the second modified example. [Figure 14] FIG. 14 is a diagram schematically showing the setting of the timer when the vehicle moves out of the frame image in the second modified example. [Figure 15] FIG. 15 is a diagram illustrating an example of a processing block of a detection device according to a third modified example. [Figure 16] FIG. 16 is a diagram schematically illustrating a state in which some pixels are excluded from the reduction process by the reduction unit in the third modified example. [Figure 17] FIG. 17 is a diagram schematically showing a state in which a timer is set when a vehicle is detected in a frame image in the third modified example. DETAILED DESCRIPTION OF THE INVENTION

[0017] <Application example> An application example of the present invention will be described below. An application example of the present invention is a falling object detection system 100 illustrated in Fig. 1. The falling object detection system 100 detects a falling object on a road 500 based on a video captured by a camera 1 facing the road 500.

[0018] The detection device 2 acquires frame images of a video in which the road 500 is captured by the camera 1. If the acquired frame image does not include a vehicle, the detection device 2 generates a reduced image by reducing the number of pixels of the acquired frame image. The detection device 2 then detects objects on the road from the reduced image. Furthermore, if the detection device 2 determines that the acquired frame image includes a vehicle, it detects objects on the road 500 from the frame image rather than the reduced image. The objects are, for example, vehicles or fallen objects on the road 500.

[0019] It is believed that objects falling onto the road 500 are often generated by vehicles. Therefore, in this application example, for frame images in which it is believed that there is little possibility of objects falling because no vehicles are detected, a reduced image with a reduced number of pixels is generated, and object detection is performed from the reduced image. By using the reduced image, the detection device 2 can reduce the processing load of the detection device 2.

[0020] <Embodiment> Hereinafter, an embodiment will be described with reference to the drawings. FIG. 1 is a diagram showing an example of a falling object detection system 100 according to an embodiment. The falling object detection system 100 is a system that detects an object that has fallen onto a road 500. The falling object detection system 100 includes a camera 1, a detection device 2, and a display device 3. The camera 1 and the detection device 2 are connected by a computer network N1. The detection device 2 and the display device 3 are connected by a connection cable L1. .

[0021] The road 500 is, for example, a motorway or a national expressway, but may also be a general national road, a prefectural road, or a municipal road.

[0022] Camera 1 is a video camera that captures images of road 500 to be monitored. Camera 1 may be, for example, a network camera that transmits captured images via computer network N1. Camera 1 is, for example, a digital video camera that uses a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) as an imaging element. Video data captured by camera 1 is transmitted to detection device 2 via computer network N1. The frame rate of the video data captured by camera 1 is, for example, 1 fps.

[0023] The detection device 2 is an information processing device that detects a fallen object on the road 500 based on video acquired from the camera 1 via the computer network N1. When the detection device 2 detects a fallen object on the road 500, it outputs the video of the detected fallen object to the display device 3 via the connection cable L1.

[0024] The display device 3 is a display monitored by the traffic wardens M1. The display device 3 is, for example, a Liquid Crystal Display (LCD), a Plasma Display Panel (PDP), an inorganic electroluminescence (EL) panel, or an organic EL panel. The display device 3 displays the information output by the detection device 2.

[0025] 2 is a diagram illustrating an example of the hardware configuration of a detection device 2 according to an embodiment. The detection device 2 includes a central processing unit (CPU) 201, a main memory unit 202, an auxiliary memory unit 203, a communication unit 204, a connection terminal 205, and a bus B1. The CPU 201, the main memory unit 202, the auxiliary memory unit 203, the communication unit 204, and the connection terminal 205 are interconnected by the bus B1.

[0026] The CPU 201 is also referred to as a microprocessor unit (MPU) or processor. The CPU 201 is not limited to a single processor and may have a multiprocessor configuration. Furthermore, a single CPU 201 connected via a single socket may have a multi-core configuration. At least a portion of the processing performed by the CPU 201 may be performed by a processor other than the CPU 201, such as a dedicated processor such as a digital signal processor (DSP), a graphics processing unit (GPU), a numerical calculation processor, a vector processor, or an image processing processor. Furthermore, at least a portion of the processing performed by the CPU 201 may be performed by an integrated circuit (IC) or other digital circuit. Furthermore, at least a portion of the CPU 201 may include an analog circuit. Integrated circuits include large-scale integrated circuits (LSIs), application-specific integrated circuits (ASICs), and programmable logic devices (PLDs). PLDs include, for example, field-programmable gate arrays (FPGAs). The CPU 201 may be a combination of a processor and an integrated circuit. The combination is called, for example, a microcontroller unit (MCU), a system-on-a-chip (SoC), a system LSI, a chipset, etc. In the detection device 2, the CPU 201 expands the program stored in the auxiliary memory 203 into the work area of ​​the main memory 202, and controls the peripheral devices through the execution of the program. This allows the detection device 2 to execute processing that meets a predetermined purpose. The main memory 202 The auxiliary storage unit 203 is a recording medium that can be read by the detection device 2.

[0027] The main storage unit 202 is exemplified as a storage unit that is directly accessed by the CPU 201. The main storage unit 202 includes a random access memory (RAM) and a read only memory (ROM).

[0028] The auxiliary storage unit 203 stores various programs and various data on a readable and writable recording medium. The auxiliary storage unit 203 is also called an external storage device. The auxiliary storage unit 203 stores an operating system (OS), various programs, various tables, etc. The OS includes a communication interface program that exchanges data with external devices connected via the communication unit 204. The external devices include, for example, other information processing devices and external storage devices connected via a computer network, etc. The auxiliary storage unit 203 may be, for example, part of a cloud system, which is a group of computers on a network.

[0029] The auxiliary storage unit 203 is, for example, an erasable programmable ROM (EPROM), a solid state drive (SSD), a hard disk drive (HDD), etc. Furthermore, the auxiliary storage unit 203 is, for example, a compact disc (CD) drive device, a digital versatile disc (DVD) drive device, a Blu-ray (registered trademark) disc (BD) drive device, etc. Furthermore, the auxiliary storage unit 203 may be provided by a network attached storage (NAS) or a storage area network (SAN).

[0030] The communication unit 204 is, for example, an interface with the computer network N1, and communicates with external devices such as the camera 1 via the computer network N1.

[0031] The connection terminal 205 is a connection terminal to which the connection cable L1 is connected. When the detection device 2 detects a falling object, it outputs an alarm to the display device 3 via the connection cable L1.

[0032] <Detection device 2 processing block> 3 is a diagram showing an example of a processing block of the detection device 2 according to the embodiment. The detection device 2 includes a vehicle detection unit 21, a reduction unit 22, an extraction unit 23, a tracking unit 24, and a reporting unit 25. The detection device 2 executes the processes of each unit of the detection device 2, such as the vehicle detection unit 21, the reduction unit 22, the extraction unit 23, the tracking unit 24, and the reporting unit 25, by the CPU 201 executing a computer program deployed in an executable manner in the main memory unit 202.

[0033] The vehicle detection unit 21 determines whether or not a vehicle is included in a frame image in a video received from the camera 1. For vehicle detection by the vehicle detection unit 21, for example, a learning model constructed by machine learning using images of roads on which vehicles are traveling and images of roads on which vehicles are not traveling as training data can be used. Such a learning model is constructed, for example, by a convolutional neural network (CNN), which is an example of machine learning.

[0034] When the vehicle detection unit 21 does not detect a vehicle in a frame image, the reduction unit 22 reduces the data amount of the frame image. One method for reducing the data amount of a frame image is to reduce the number of pixels in the frame image. Note that the reduction unit 22 refrains from executing the process of reducing the number of pixels in the frame image for a predetermined period of time after the vehicle detection unit 21 detects a vehicle.

[0035] FIG. 4 is a diagram schematically illustrating an example of processing for reducing the number of pixels in a frame image by the reduction unit 22 in an embodiment. FIG. 4 illustrates processing for reducing the number of pixels in a frame image. In FIG. 4, the reduction unit 22 performs processing for reducing the number of pixels on an original frame image F100 of 20 × 20 pixels. The pixel reduction processing by the reduction unit 22 uses at least one of a first method for reducing the number of pixels by deleting pixels in even-numbered columns and even-numbered rows, starting from pixel P1 in the upper right corner of the original frame image F100 (the first pixel), and a second method for reducing the number of pixels by deleting pixels in odd-numbered columns and odd-numbered rows. Note that the pixel reduction processing by the reduction unit 22 is not limited to the first method or the second method, and other methods may also be used. In the example of FIG. 4, frame image F100A is an example of an image in which the number of pixels has been reduced by the first method, and frame image F100B is an example of an image in which the number of pixels has been reduced by the second method.

[0036] For example, the reduction unit 22 may alternately apply the first method and the second method to frame images continuously input from the camera 1. Furthermore, for example, the reduction unit 22 may apply either the first method or the second method to frame images continuously input from the camera 1.

[0037] Returning to FIG. 3 , the extraction unit 23 generates an extracted image by extracting the foreground from a frame image in the video received from the camera 1 or a frame image whose data amount has been reduced by the reduction unit 22. The extraction of the foreground is performed, for example, by background subtraction. FIG. 5 is a diagram showing an example of foreground extraction by the extraction unit 23. In the example of FIG. 5 , a frame image I1 at time T and a frame image I2 at time T+1, which is the time immediately following time T, are input to the extraction unit 23. Frame image I1 shows only a road 500, while frame image I2 shows a box O1 and a passenger car O2 on the road 500. The extraction unit 23 extracts the box O1 as the foreground based on the difference between frame image I1 and frame image I2, and generates an extracted image D1 including the box O1 and the passenger car O2. The extracted image D1 is, for example, an image in which the box O1 and the passenger car O2 extracted as the foreground are shown in white and the road 500 in the background has been removed.

[0038] Returning to FIG. 3, the tracking unit 24 tracks the foreground (e.g., a box O1 and an automobile O2) extracted by the extraction unit 23. For example, the tracking unit 24 sets a tracking frame surrounding the foreground and tracks the foreground using the tracking frame. If the tracked foreground is a fallen object onto the road 500, the tracking unit 24 causes the reporting unit 25 to report it. For example, the tracking unit 24 determines that a foreground whose tracking duration is equal to or longer than a predetermined time is a fallen object that is not moving on the road 500. Furthermore, the tracking unit 24 determines that a foreground whose tracking duration is less than the predetermined time is not a fallen object. In the example of FIG. 5, for example, the automobile O2 passes by in less than the predetermined time, and is therefore determined not to be a fallen object. The predetermined time may be determined based on, for example, the time required for the automobile O2 to pass by.

[0039] When the tracking unit 24 detects a fallen object on the road 500, the reporting unit 25 notifies the traffic warden M1 of the detection of the fallen object by displaying a screen of the detection on the display device 3. The reporting unit 25 may notify the traffic warden M1 of the detection of the fallen object on the road 500 by, for example, outputting a sound from a speaker.

[0040] <Processing flow of detection device 2> Fig. 6 is a diagram showing an example of a processing flow of the detection device 2 according to the embodiment. Fig. 6 illustrates processing up to extraction of the foreground by the extraction unit 23 of the detection device 2. Hereinafter, an example of the processing flow of the detection device 2 will be described with reference to Fig. 6.

[0041] In step S1, the vehicle detection unit 21 detects a vehicle in a frame image input from the camera 1. If a vehicle is detected (YES in step S1), the process proceeds to step S2. If a vehicle is not detected (NO in step S1), the process proceeds to step S5.

[0042] In step S2, the reduction unit 22 sets a timer indicating a predetermined period during which the reduction process by the reduction unit 22 is suppressed. The timer is set to, for example, "30." The processes from step S3 to step S5 are repeatedly executed until the timer set in step S2 expires (until the timer reaches "0").

[0043] In step S3, extraction unit 23 generates an extracted image by extracting the foreground from the frame image input from camera 1. In step S4, tracking unit 24 tracks the foreground in the extracted image extracted in step S3, and if the tracked foreground is a falling object, reporting unit 25 reports the detection of the falling object.

[0044] In step S5, the extraction unit 23 acquires the next frame image from the camera 1. In step S6, the reduction unit 22 decrements the timer by "1".

[0045] In step S7, the reduction unit 22 performs processing to reduce the number of pixels in the frame image. Details of the processing in step S7 will be described later with reference to FIG.

[0046] In step S8, extraction unit 23 generates an extracted image by extracting the foreground from the frame image whose pixel count has been reduced in step S6. In step S9, tracking unit 24 tracks the foreground in the extracted image extracted in step S8, and if the tracked foreground is a falling object, reporting unit 25 reports the detection of the falling object.

[0047] 7 is a diagram showing an example of a processing flow of processing to reduce the number of pixels in a frame image by the extraction unit 23. Hereinafter, an example of a processing flow of processing to reduce the number of pixels in a frame image by the extraction unit 23 will be described with reference to FIG.

[0048] In step S71, the extraction unit 23 switches the flag. That is, if the flag is ON, the extraction unit 23 turns the flag OFF. Also, if the flag is OFF, the extraction unit 23 turns the flag ON. The state indicating the ON / OFF state of the flag is stored in, for example, the auxiliary storage unit 203.

[0049] In step S72, the extraction unit 23 determines whether the flag is ON. If the flag is ON (YES in step S72), the process proceeds to step S73. If the flag is OFF (NO in step S72), the process proceeds to step S74.

[0050] In step S73, the extraction unit 23 reduces the number of pixels in the frame image using the first method, which has been described with reference to FIG.

[0051] In step S74, the extraction unit 23 reduces the number of pixels in the frame image using the second method, which has been described with reference to FIG.

[0052] <Effects of the embodiment> In this embodiment, if the vehicle detection unit 21 does not detect a vehicle in a frame image of the road 500, the reduction unit 22 executes a process of reducing the number of pixels in the frame image. By reducing the number of pixels in the frame image, the processing load on the extraction unit 23 and tracking unit 24 is reduced. In other words, according to this embodiment, the processing load on the detection device 2 can be reduced. Note that, since it is considered that objects falling on the road 500 fall from vehicles, Therefore, even if the number of pixels in frame images in which no vehicle is detected is reduced, the decrease in accuracy in detecting fallen objects is suppressed.

[0053] Furthermore, in this embodiment, even if the number of pixels of the frame image is reduced, tracking is performed by the tracking unit 24, and if a fallen object is detected by the tracking, a report is issued by the reporting unit 25. Therefore, according to this embodiment, even in a situation where the occurrence frequency of fallen objects is considered low because the vehicle is not traveling on the road 500, detection of fallen objects continues.

[0054] In this embodiment, when a vehicle is detected by the vehicle detection unit 21 in a frame image of the road 500, the reduction unit 22 does not perform the process of reducing the number of pixels of the frame image. Therefore, for a frame image in which there is a risk of an object falling from a vehicle, the fallen object can be detected in detail using the frame image in which the number of pixels is not reduced.

[0055] In this embodiment, the processing for reducing the number of pixels in a frame image can be performed using the first method and the second method. Both methods reduce the number of pixels in a frame image, thereby reducing the number of pixels in the frame image.

[0056] In this embodiment, when reducing the number of pixels in a frame image, the first method and the second method are applied alternately. In the first method, pixels in even-numbered columns and even-numbered rows are deleted to reduce the number of pixels. In addition, in the second method, pixels in odd-numbered columns and odd-numbered rows are deleted to reduce the number of pixels. That is, an image whose number of pixels has been reduced by the first method includes pixels that would be deleted by the second method. In addition, an image whose number of pixels has been reduced by the second method includes pixels that would be deleted by the first method. By applying the first method and the second method alternately in this manner, overlooking certain pixels is suppressed. In addition, in this embodiment, a timer is set to indicate a predetermined period during which the reduction process by the reduction unit 22 is suppressed, thereby reducing the possibility of overlooking a fallen object that occurs after a vehicle has passed.

[0057] In this embodiment, the reduction unit 22 does not perform processing to reduce the number of pixels in the frame images within a predetermined period after the vehicle detection unit 21 detects a vehicle. In other words, within a predetermined period after the vehicle detection unit 21 detects a vehicle, the tracking unit 24 performs tracking based on frame images whose number of pixels has not been reduced. It is considered that the possibility of an object falling onto the road 500 increases after a vehicle is detected. According to this embodiment, the tracking unit 24 performs tracking based on frame images whose number of pixels has not been reduced within a predetermined period after the detection of a vehicle suspected of having a fallen object, thereby reducing the possibility of the fallen object being overlooked.

[0058] In this embodiment, the vehicle detection unit 21 detects vehicles using a learning model constructed by machine learning using images of roads on which vehicles are traveling and images of roads on which vehicles are not traveling as training data. Therefore, according to this embodiment, vehicles can be detected by the simple process of inputting frame images into the learning model.

[0059] <First Modification> In the embodiment described above, even if a lane in which no vehicle is detected is included in a frame image of a video captured by camera 1, the lane is not excluded from the detection targets for fallen objects. In the first modified example, a lane in which no vehicle is detected in a frame image is excluded from the detection targets for fallen objects. The same components as those in the embodiment are given the same reference numerals, and their description will be omitted. The first modified example will be described below with reference to the drawings.

[0060] 8 is a diagram showing an example of a processing block of a detection device 2A according to a first modification. The detection device 2A includes a vehicle detection unit 21A instead of the vehicle detection unit 21, and an extraction unit 23 instead of the extraction unit 23. 23A is provided, which is different from the detection device 2 according to the embodiment.

[0061] When a frame image of a road 500 is input from the camera 1, the vehicle detection unit 21A detects the presence or absence of a vehicle for each lane of the road 500. Then, the vehicle detection unit 21A performs a masking process on lanes where no vehicles are detected.

[0062] Fig. 9 is a diagram schematically showing mask processing by vehicle detection unit 21A in the first modified example. The upper part of Fig. 9 schematically shows frame image I3 captured by camera 1 and input to vehicle detection unit 21A. The lower part of Fig. 9 exemplifies frame image I4 obtained by performing mask processing on frame image I3 by vehicle detection unit 21A.

[0063] When a frame image I3 is input from the camera 1, the vehicle detection unit 21A determines whether or not a vehicle is present in each of the lanes 501 and 502. To determine whether or not a vehicle is present, a learning model constructed for vehicle detection using, for example, machine learning or a convolutional neural network (CNN) can be used. An example of such a learning model is YOLO.

[0064] In frame image I3, vehicle detection unit 21A detects vehicle O3 in lane 502, but does not detect any vehicle in lane 501. Therefore, vehicle detection unit 21A performs mask processing on lane 501, as illustrated in the lower part of FIG. 9. For example, vehicle detection unit 21A performs mask processing by overlaying pattern image 504, which has a predetermined pattern, on lane 501 in which no vehicle has been detected. Note that, although the lower part of FIG. 9 illustrates an image with diagonal lines as pattern image 504, it may also be an image with a pattern other than diagonal lines (such as a dot pattern with multiple dots, or a solid pattern filled with a predetermined color).

[0065] Extraction unit 23A generates an extracted image by extracting the foreground from frame image I4 that has been masked by vehicle detection unit 21A. Extraction unit 23A excludes from the foreground extraction the area (lane) of frame image I4 to which pattern image 504 has been applied by vehicle detection unit 21A. In other words, extraction unit 23A extracts the foreground from the area (lane) of frame image I4 to which pattern image 504 has been applied.

[0066] Extraction unit 23A does not extract the foreground from the area of ​​frame image I4 where pattern image 504 is applied. According to the first modification, the area to be subjected to foreground extraction can be reduced, thereby reducing the processing load related to foreground extraction. Note that it is considered that there is little possibility of new fallen objects occurring in lanes where no vehicles are detected. Therefore, even if foreground extraction from the area where pattern image 504 is applied is omitted as in the first modification, a decrease in the accuracy of fallen object detection is suppressed.

[0067] <Second Modification> In the embodiment described above, a timer indicating a predetermined period for suppressing the reduction process is set for each frame image. In the second modified example, a timer is set for each pixel in a frame image. The same components as in the embodiment are given the same reference numerals, and their description will be omitted. The second modified example will be described below with reference to the drawings.

[0068] 10 is a diagram showing an example of a processing block of a detection device 2B according to Modification 2. The detection device 2B differs from the detection device 2 according to the embodiment in that it includes a reduction unit 22B instead of the reduction unit 22.

[0069] The reduction unit 22B sets a timer for each pixel in the frame image. 11 is a diagram illustrating a state before the timer is set by the reduction unit 22B in the example. Fig. 11 illustrates a state in which the timer is not set for each pixel of the frame image I5 (the timer is set to "0").

[0070] FIG. 12 is a diagram schematically illustrating a state in which a timer is set when a vehicle O4 is detected in a frame image I6 in the second modified example. Frame image I6 is, for example, the frame image following frame image I5. The reduction unit 22B sets a new timer for pixels of frame image I6 that overlap with vehicle O4 and for pixels within a predetermined range around vehicle O4. Here, the reduction unit 22B sets a timer of "30" for pixels that overlap with vehicle O4 and for pixels within a predetermined range around vehicle O4.

[0071] FIG. 13 is a diagram showing the setting of a timer when vehicle O4 moves in the second modified example. Frame image I7 is the frame image following frame image I6. The reduction unit 22B sets new timers for the pixels of frame image I7 that overlap with vehicle O4 and for pixels within a predetermined range around vehicle O4. Furthermore, for pixels for which a timer was set in frame image I6, the frame image preceding frame image I7, but for which a timer is not to be set anew in frame image I7, the timer is advanced (decremented by "1") to set the timer value to "29."

[0072] 14 is a diagram showing the timer settings when vehicle O4 moves out of the frame image in the second modified example. Frame image I8 is the frame image next to frame image I7. The reduction unit 22B reduces the timers set for each pixel in frame image I7 by "1," thereby setting the timers for pixels that were set to "30" in frame image I7 to "29," and setting the timers for pixels that were set to "29" in frame image I7 to "28."

[0073] According to the second modified example, a timer can be set for each pixel, so that it is possible to determine for each pixel whether or not to execute the reduction process.

[0074] <Third Modification> In the second modified example, pixels in a frame image for which the timer is set to "0" are subject to the reduction process, but some of the pixels for which the timer is set to "0" may be excluded from the reduction process. The same components as those in the embodiment are given the same reference numerals, and their description will be omitted. The third modified example will be described below with reference to the drawings.

[0075] 15 is a diagram showing an example of a processing block of a detection device 2C according to Modification 3. The detection device 2C differs from the detection device 2 according to the embodiment in that it includes a reduction unit 22C instead of the reduction unit 22.

[0076] The reduction unit 22C sets a timer for each pixel in the frame image. Here, the reduction unit 22C excludes some of the pixels for which the timer is set to "0" from the reduction process. FIG. 16 is a diagram schematically showing a state in which some pixels are excluded from the reduction process by the reduction unit 22C in the third modified example. In the frame image I9 illustrated in FIG. 16, pixels in odd-numbered rows and odd-numbered columns (pixels with dots drawn in the background) are excluded from the reduction process by the reduction unit 22B.

[0077] FIG. 17 is a diagram showing a state in which a timer is set when a vehicle O5 is detected in a frame image I10 in the third modified example. The frame image I10 is a frame image following the frame image I9. The reduction unit 22B reduces the number of pixels in the frame image I10 by The reduction unit 22B sets new timers for the pixels that overlap with the vehicle O5 and for the pixels in a predetermined range around the vehicle O5. At this time, the reduction unit 22B also sets timers for the pixels that are not subject to the reduction process.

[0078] According to the third modified example, by excluding some pixels from the reduction process by the reduction section 22C, it is possible to reduce the processing load of the reduction process by the reduction section 22C.

[0079] <Other variations> In the embodiment described above, one camera 1 is connected to the detection device 2, but multiple cameras 1 may be connected to the detection device 2. When multiple cameras 1 are connected, it is thought that the processing load on the detection device 2 due to processing of frame images will become higher. In this embodiment, the number of pixels is reduced by the reduction unit 22 for frame images in which a vehicle is detected by the vehicle detection unit 21, so that the processing load on the detection device 2 can be reduced even when multiple cameras 1 are connected.

[0080] In this embodiment, the vehicle detection unit 21 detects vehicles using a learning model, but the detection of vehicles by the vehicle detection unit 21 is not limited to a method using a learning model. The vehicle detection unit 21 can employ various known methods for detecting vehicles.

[0081] The frame image input to the detection device 2 may be the image (original image) output from the camera 1 itself, or may be an image obtained by performing preprocessing (color conversion, brightness conversion, resolution conversion, etc.) on the original image. In other words, it is sufficient if the image used for falling object detection when it is determined that no vehicle is present has fewer pixels than the image used for falling object detection when it is determined that a vehicle is present.

[0082] The embodiments and modifications disclosed above can be combined with each other.

[0083] <Computer-readable recording medium> An information processing program that causes a computer or other machine or device (hereinafter referred to as a computer, etc.) to realize any of the above functions can be recorded on a computer-readable recording medium. Then, by having the computer, etc. read and execute the program from this recording medium, the function can be provided.

[0084] Here, a computer-readable recording medium refers to a recording medium that stores information such as data and programs electrically, magnetically, optically, mechanically, or chemically and that can be read by a computer. Among such recording media, those that are removable from a computer include, for example, flexible disks, magneto-optical disks, Compact Disc Read Only Memory (CD-ROM), Compact Disc-Recordable (CD-R), Compact Disc-Rewriteable (CD-RW), Digital Versatile Disc (DVD), Blu-ray Disc (BD), Digital Audio Tape (DAT), 8mm tape, flash memory, external hard disk drives, and solid state drives (SSDs). Furthermore, recording media that are fixed to a computer include built-in hard disk drives, SSDs, ROMs, etc.

[0085] <Appendix 1> A process of acquiring frame images of a video of a road (500) captured by a camera (1); a process of determining whether or not a vehicle (O2) is included in the frame image; If it is determined that the vehicle (O2) is not included in the frame image, a first detection process for generating reduced images (F100A, F100B) by reducing the number of pixels of the image, and detecting an object on the road (500) from the reduced images (F100A, F100B); a control unit (201) that executes a second detection process to detect an object on the road (500) from the frame image when it is determined that the vehicle (O2) is included in the frame image. Image processing device (2). <Appendix 2> the process of generating the reduced image includes a process of deleting pixels in even-numbered columns and even-numbered rows from pixels in the frame image; The image processing device (2) according to appendix 1. <Appendix 3> the process of generating the reduced image (F100A) includes a process of deleting pixels in odd-numbered columns and odd-numbered rows from pixels in the frame image; The image processing device (2) according to appendix 1. <Appendix 4> the process of generating the reduced image (F100B) includes a process of alternately applying, to each of a series of frame images in the moving image, a first deletion process of deleting pixels in even-numbered columns and even-numbered rows from among the pixels in the frame image, and a second deletion process of deleting pixels in odd-numbered columns and odd-numbered rows from among the pixels in the frame image; The image processing device (2) according to appendix 1. <Appendix 5> When it is determined that the vehicle is included in the first frame image, the control unit (201) detects an object on the road from a second frame image acquired within a predetermined period after the first frame image, even if the second frame image does not include the vehicle. An image processing device (2) according to any one of appendices 1 to 4. <Appendix 6> The process of determining whether the vehicle (O2) is included includes a process of inputting the frame image into a learning model constructed using image data of the road (500) in a state where the vehicle (O2) is traveling and image data of the road in a state where the vehicle (O2) is not traveling as training data, and determining whether the vehicle (O2) is included in the frame image. The image processing device (2) according to appendix 1. <Appendix 7> The road (500) includes a plurality of lanes (501, 502), The process of determining includes a process of determining whether or not a vehicle exists in each of the plurality of lanes (501, 502), In the first detection process and the second detection process, detection of the object is omitted for the lane (501) in which it is determined that the vehicle does not exist. An image processing device (2) according to any one of appendices 1 to 6. <Appendix 8> The control unit (201) In the determination process, if it is determined that the vehicle is included in the frame image, a predetermined timer is set; The second detection process is executed until the timer expires, even if it is determined that the vehicle is not included in the frame image. The image processing device (2) according to appendix 1. <Appendix 9> The control unit (201) In the determination process, when the vehicle is detected in a predetermined range of pixels in the frame image, a predetermined timer is set for the predetermined range of pixels; until the timer expires, the pixels in the predetermined range are excluded from the frame image to be reduced in generating the reduced image. 9. An image processing device according to any one of claims 1 to 8. <Appendix 10> Computer (2) A process of acquiring frame images of a video of a road (500) captured by a camera (1); a process of determining whether or not a vehicle (O2) is included in the frame image; a first detection process of generating reduced images (F100A, F100B) by reducing the number of pixels of the frame images when it is determined that the vehicle (O2) is not included in the frame images, and detecting an object on the road (500) from the reduced images (F100A, F100B); If it is determined that the vehicle (O2) is included in the frame image, a second detection process is executed to detect an object on the road from the frame image. Image processing methods. <Appendix 11> On the computer (2), A process of acquiring frame images of a video of a road (500) captured by a camera (1); a process of determining whether or not a vehicle (O2) is included in the frame image; a first detection process of generating reduced images (F100A, F100B) by reducing the number of pixels of the frame images when it is determined that the vehicle (O2) is not included in the frame images, and detecting an object on the road (500) from the reduced images (F100A, F100B); If it is determined that the vehicle (O2) is included in the frame image, a second detection process is executed to detect an object on the road (500) from the frame image. Image processing program. [Explanation of symbols]

[0086] 1. Camera 2. Detection device 3...Display device 21 Vehicle detection unit 21A Vehicle detection unit 22·Reduction section 23...Extraction part 23A·Extraction part 24 Tracking section 25. Reporting Department 100··Falling Object Detection System 201 CPU 202...Main memory 203...Auxiliary storage section 204··Communications Department 205 Connection terminal 500...road B1 Bus D1 Extracted image L1 Connection cable M1...Traffic warden N1: Computer Network O1··Box O2··Passenger car O4··Vehicle O5 Vehicle

Claims

1. A process of acquiring frame images of a video of a road captured by a camera; a process of determining whether a vehicle is included in the frame image; a first detection process for generating a reduced image by reducing the number of pixels of the frame image when it is determined that the vehicle is not included in the frame image, and detecting an object on the road from the reduced image; a control unit that executes a second detection process of detecting an object on the road from the frame image when it is determined that the vehicle is included in the frame image. Image processing device.

2. the process of generating the reduced image includes a process of deleting pixels in even-numbered columns and even-numbered rows from pixels in the frame image; The image processing device according to claim 1 .

3. the process of generating the reduced image includes a process of deleting pixels in odd-numbered columns and odd-numbered rows from pixels in the frame image; The image processing device according to claim 1 .

4. the process of generating the reduced image includes a process of alternately applying, to each of a series of frame images in the moving image, a first deletion process of deleting pixels in even-numbered columns and even-numbered rows from among the pixels in the frame image, and a second deletion process of deleting pixels in odd-numbered columns and odd-numbered rows from among the pixels in the frame image. The image processing device according to claim 1 .

5. When it is determined that the vehicle is included in the first frame image, the control unit detects an object on the road from a second frame image acquired within a predetermined period after the first frame image, even if the second frame image does not include the vehicle. The image processing device according to claim 1 .

6. The process of determining whether or not the vehicle is included includes a process of inputting the frame image into a learning model constructed using image data of the road in a state where the vehicle is traveling and image data of the road in a state where the vehicle is not traveling as training data, and determining whether or not the vehicle is included in the frame image. The image processing device according to claim 1 .

7. the road includes a plurality of lanes; the determining process includes a process of determining whether or not a vehicle is present in each of the plurality of lanes, In the first detection process and the second detection process, detection of the object is omitted for the lane in which it is determined that the vehicle does not exist. The image processing device according to claim 1 .

8. The control unit In the determination process, if it is determined that the vehicle is included in the frame image, a predetermined timer is set; The second detection process is executed until the timer expires, even if it is determined that the vehicle is not included in the frame image. The image processing device according to claim 1 .

9. The control unit In the determination process, when the vehicle is detected in a predetermined range of pixels in the frame image, a predetermined timer is set for the predetermined range of pixels; until the timer expires, the pixels in the predetermined range are excluded from the frame image to be reduced in generating the reduced image. The image processing device according to claim 1 .

10. The computer A process of acquiring frame images of a video of a road captured by a camera; a process of determining whether a vehicle is included in the frame image; a first detection process for generating a reduced image by reducing the number of pixels of the frame image when it is determined that the vehicle is not included in the frame image, and detecting an object on the road from the reduced image; If it is determined that the vehicle is included in the frame image, a second detection process is executed to detect an object on the road from the frame image. Image processing methods.

11. On the computer, A process of acquiring frame images of a video of a road captured by a camera; a process of determining whether a vehicle is included in the frame image; a first detection process for generating a reduced image by reducing the number of pixels of the frame image when it is determined that the vehicle is not included in the frame image, and detecting an object on the road from the reduced image; a second detection process for detecting an object on the road from the frame image when it is determined that the vehicle is included in the frame image; Image processing program.

Images