Moving object detection device, method and system, and storage medium

Abstract

A motion object detection device, a motion object detection method, a motion object detection system, and a storage medium are provided, capable of easily and quickly detecting moving objects present around a vehicle. A motion object detection device comprises: an image acquisition unit that acquires image data including a plurality of frames representing the surrounding conditions of a moving body, captured sequentially by a camera mounted on a moving body; a difference calculation unit that calculates the differences between the plurality of frames and binarizes the differences into a first value and a second value, thereby calculating a difference image between the plurality of frames; a mesh extraction unit that extracts meshes from a plurality of meshes defined in the difference image where the density of pixels of the first value is greater than or equal to a first threshold; and a motion object detection unit that detects the extracted meshes as moving objects, wherein the plurality of meshes are configured to have smaller pixel sizes the farther away from the camera.

Description

Technical Field

[0001] This invention relates to a moving object detection device, a moving object detection method, a moving object detection system, and a storage medium. Background Technology

[0002] Previously, techniques for detecting moving objects in the vicinity of a vehicle based on image data of the area in front of the vehicle captured by an onboard camera were known. For example, Japanese Patent Application Publication No. 2021-144689 discloses a technique that performs signal processing based on pre-learned results on image data of the vehicle's surrounding environment, thereby outputting a recognition result of moving objects present in the image data.

[0003] The technology described in Patent Document 1 uses deep neural networks (DNNs) such as convolutional neural networks to detect moving objects around a vehicle. However, such machine learning methods require a large amount of training data in advance and tend to have a high processing load during execution. As a result, it is sometimes unable to detect moving objects around a vehicle at high speed. Summary of the Invention

[0004] The present invention was made in consideration of such circumstances, and one of its objectives is to provide a moving object detection device, a moving object detection method, a moving object detection system, and a storage medium capable of easily and quickly detecting moving objects present around a vehicle.

[0005] The moving object detection device, moving object detection method, moving object detection system, and storage medium of the present invention adopt the following structure.

[0006] (1): One aspect of the present invention relates to a moving object detection device, wherein the moving object detection device comprises: an image acquisition unit that acquires image data, the image data including a plurality of frames representing the surrounding conditions of the moving object captured by a camera mounted on a moving object in a time sequence; a difference calculation unit that calculates the difference between the plurality of frames and binarizes the difference into a first value and a second value, thereby calculating a difference image between the plurality of frames; a mesh extraction unit that extracts a mesh in a plurality of meshes set in the difference image where the density of the pixels of the first value is above a first threshold; and a moving object detection unit that detects the extracted meshes as moving objects, wherein the plurality of meshes are set to have smaller pixel sizes the farther away from the camera.

[0007] (2): Based on the above (1) scheme, the differential calculation unit magnifies the frame captured at the previous time point based on the speed of the moving body in the shooting interval of the multiple frames captured, and calculates the differential image between the magnified frame captured at the previous time point and the frame captured at the current time point.

[0008] (3): Based on the above (2) scheme, the differential calculation unit magnifies the frame captured at the previous time point with the vanishing point of the frame captured at the previous time point as the center.

[0009] (4): Based on any of the above schemes (1) to (3), the differential calculation unit performs a parallel movement of the frame captured at the previous time point based on the yaw rate of the moving body during the shooting interval of the multiple frames, and calculates the differential image between the frame captured at the previous time point and the frame captured at the current time point after the parallel movement.

[0010] (5): Based on any of the above schemes (1) to (4), the mesh extraction unit changes the threshold according to the distance of each of the plurality of meshes from the camera.

[0011] (6): Based on any of the above schemes (1) to (5), when the distance from the camera is less than a first distance, the mesh extraction unit sets the size of the plurality of meshes to a first size; when the distance from the camera is greater than the first distance but less than a second distance, the mesh extraction unit sets the size of the plurality of meshes to a second size smaller than the first size; and when the distance from the camera is greater than the second distance, the mesh extraction unit sets the size of the plurality of meshes to a third size smaller than the second size.

[0012] (7): A moving object detection system, wherein the moving object detection system comprises: a moving object detection device according to any one of the embodiments (1) to (6); and a driving support device that provides driving support for the moving object based on the detection result obtained by the moving object detection device.

[0013] (8): Another aspect of the present invention relates to a moving object detection method, wherein the moving object detection method causes a computer to perform the following processing: acquiring image data, the image data comprising multiple frames representing the surrounding conditions of the moving object captured in a time sequence by a camera mounted on the moving object; calculating the difference between the multiple frames, binarizing the difference into a first value and a second value, thereby calculating a difference image between the multiple frames; extracting a grid in a plurality of grids set in the difference image where the density of the pixels of the first value is above a first threshold; and detecting the extracted grid as a moving object, wherein the plurality of grids are set to have smaller pixel sizes the farther away from the camera.

[0014] (9): Another aspect of the present invention relates to a storage medium storing a program, wherein the program causes a computer to perform the following processing: acquiring image data comprising a plurality of frames representing the surrounding conditions of the moving body, captured in a time sequence by a camera mounted on the moving body; calculating the difference between the plurality of frames, binarizing the difference into a first value and a second value, thereby calculating a difference image between the plurality of frames; extracting a grid in a plurality of grids set in the difference image where the density of the first value of pixels is above a first threshold; and detecting the extracted grids as moving objects, wherein the plurality of grids are set to have smaller pixel sizes the farther away from the camera.

[0015] According to the schemes (1) to (9), it is possible to easily and quickly detect moving objects around the vehicle.

[0016] According to scheme (2) or (3), the difference image between the frame captured at the previous time point and the frame captured at the current time point can be accurately calculated.

[0017] According to the scheme in (4), the difference image can be accurately calculated by taking into account the yaw rate of the moving body.

[0018] According to scheme (5) or (6), moving objects can be accurately detected based on their distance from the camera.

[0019] According to the scheme in (7), the detection results detected by the moving object detection device can be appropriately used for driving support. Attached Figure Description

[0020] Figure 1 This is a diagram illustrating an example of the structure and peripheral equipment of the moving object detection device according to the first embodiment.

[0021] Figure 2 This is a diagram illustrating an example of the surrounding conditions of a vehicle M equipped with a moving object detection device.

[0022] Figure 3 It means through Figure 2 An example of an image obtained by a camera taking a picture of the front of a vehicle in the surrounding environment shown.

[0023] Figure 4 This is a diagram illustrating an example of a vanishing point used as a reference point for magnifying a frame.

[0024] Figure 5 This diagram illustrates the method of correcting the previous frame in order to calculate the difference image.

[0025] Figure 6 This is a diagram showing an example of a difference image calculated by the difference calculation unit.

[0026] Figure 7 This is a diagram illustrating an example of the structure of the mesh G set by the mesh extraction unit.

[0027] Figure 8 This is a diagram illustrating an example of a method for extracting mesh G by the mesh extraction unit.

[0028] Figure 9 This is a diagram showing an example of a mesh image GI calculated by the mesh extraction unit.

[0029] Figure 10 This is a diagram illustrating an example of an action performed by a driving control device.

[0030] Figure 11 This is a diagram showing another example of the structure and peripheral equipment of a moving object detection device.

[0031] Figure 12 This is a diagram illustrating an example of an action performed by a reporting device.

[0032] Figure 13 This is a diagram illustrating an example of the processing flow performed by a moving object detection device.

[0033] Figure 14 This diagram illustrates the actions performed by the moving object detection unit of the second embodiment.

[0034] Figure 15 This is a diagram illustrating an example of the processing flow performed by the moving object detection device 100 of the second embodiment.

[0035] Figure 16 This is a diagram illustrating an example of the structure and peripheral equipment of the moving object detection device 100 according to the third embodiment.

[0036] Figure 17 This is a diagram illustrating an example of a mesh G search method performed by the bounding box setting unit 132.

[0037] Figure 18 This is a diagram illustrating an example of the search range of the grid G ​​searched by the bounding box setting unit 132.

[0038] Figure 19 This is a diagram illustrating an example of the processing flow performed by the moving object detection device 100 according to the third embodiment. Detailed Implementation

[0039] [First Implementation Method]

[0040] Hereinafter, embodiments of the moving object detection device, moving object detection method, moving object detection system, and storage medium of the present invention will be described with reference to the accompanying drawings. The moving object detection device is, for example, mounted on a mobile body. The mobile body is, for example, a self-moving mobile body such as a four-wheeled vehicle, a two-wheeled vehicle, a micro-mobile body, or a robot, or a mobile device such as a smartphone that is mounted on a self-moving mobile body or moved by a person. In the following description, the mobile body is a four-wheeled vehicle, and will be referred to as a "vehicle". The moving object detection device is not limited to being mounted on a mobile body; the following description may also be based on images captured by a fixed-point observation camera or a smartphone camera.

[0041] Figure 1 This diagram illustrates an example of the structure and peripheral equipment of the moving object detection device 100. The moving object detection device 100 communicates with the camera 10, the driving control device 200, and the like.

[0042] Camera 10 is mounted on the back of the windshield of vehicle M, etc., and takes pictures of at least the road along the direction of travel of vehicle M in a time sequence, and outputs the captured images to moving object detection device 100. It should be noted that a sensor fusion device or the like may exist between camera 10 and moving object detection device 100, but this will not be described in detail. Driving control device 200 is, for example, an automatic driving control device that enables vehicle M to drive autonomously, or a driving support device that performs inter-vehicle distance control, automatic braking control, automatic lane change control, etc.

[0043] The moving object detection device 100 includes, for example, an image acquisition unit 110, a difference calculation unit 120, a mesh extraction unit 130, and a moving object detection unit 140. These components are implemented by executing a program (software) using a hardware processor such as a CPU (Central Processing Unit). Some or all of these components can be implemented by hardware (including the circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit), or through the coordinated use of software and hardware. The program can be pre-saved in a storage device such as an HDD (Hard Disk Drive) or flash memory (a storage device with a non-transitory storage medium), or it can be saved in a removable storage medium such as a DVD or CD-ROM (a non-transitory storage medium), and installed by mounting the storage medium onto a drive device.

[0044] Figure 2 This is a diagram illustrating an example of the surrounding conditions of a vehicle M equipped with a moving object detection device 100. Figure 2 The following example illustrates a scenario where a vehicle M equipped with a moving object detection device 100 is traveling on a road, and a bicycle B is traveling in front of the vehicle M. In the following description, the scenario in which the moving object detection device 100 detects the bicycle B as a moving object will be used as an example, but the present invention is not limited to such scenarios.

[0045] Figure 3 It means through Figure 2 The image shown is an example of an image obtained by camera 10 capturing the front of vehicle M in the surrounding environment. Image acquisition unit 110 acquires image data comprising multiple frames representing the surrounding environment of vehicle M captured sequentially by camera 10 mounted on vehicle M. More specifically, for example, image acquisition unit 110 acquires image data from camera 10 at a frame rate of 30Hz.

[0046] The difference calculation unit 120 calculates the difference in pixel values ​​for the multiple frames acquired by the image acquisition unit 110, and binarizes the calculated difference into a first value (e.g., 1) and a second value (e.g., 0), thereby calculating the difference image DI between the multiple frames. More specifically, firstly, the difference calculation unit 120 performs grayscale transformation on the multiple frames acquired by the image acquisition unit 110, transforming the RGB image into a grayscale image.

[0047] Next, the differential calculation unit 120 calculates the speed of the vehicle M during the shooting interval of multiple frames based on the speed of the vehicle M. The frame captured at the last time point (hereinafter sometimes referred to as the "last frame") is magnified with the vanishing point of the frame as the center, thereby aligning it with the frame captured at the current time point (hereinafter sometimes referred to as the "current frame").

[0048] Figure 4 This is a diagram illustrating an example of a vanishing point used as a reference point for frame magnification. In Figure 4 In this context, VP represents the vanishing point of the frame. The vanishing point VP is defined, for example, as the intersection point formed by extending both sides of the lane through which vehicle M travels. The differential calculation unit 120 estimates the distance traveled by vehicle M based on the speed (average speed) of vehicle M measured between the previous time point and the current time point, and amplifies the previous frame by a magnification corresponding to that travel distance, centered on the vanishing point VP.

[0049] Figure 5 This diagram illustrates the method of correcting the previous frame to calculate the difference image DI. Figure 5 In this context, h represents the frame height, and w represents the frame width. For example... Figure 5 As shown, the differential calculation unit 120 magnifies the previous frame by the magnification corresponding to the distance the vehicle M has traveled between the previous time point and the current time point. At this time, the size of the magnified previous frame is larger than that before magnification, so the differential calculation unit 120 trims the ends of the magnified previous frame to bring the size of the magnified previous frame back to its original size.

[0050] It should be noted that the differential calculation unit 120 may also consider the yaw rate of the vehicle M during the shooting interval between the previous frame and the current frame, in addition to the vehicle M's speed during the shooting interval, to correct the previous frame. More specifically, the differential calculation unit 120 may also calculate the difference between the yaw rate of the vehicle M at the acquisition time of the previous frame and the yaw rate of the vehicle M at the acquisition time of the current frame based on the yaw rate during the shooting interval, and shift the previous frame along the yaw direction by an angle corresponding to the difference, thereby aligning the previous frame with the current frame.

[0051] Figure 6 This diagram illustrates an example of the difference image DI calculated by the difference calculation unit 120. After aligning the previous frame with the current frame, the difference calculation unit 120 calculates the difference between the pixel values ​​of the previous frame and the current frame. If the difference calculated for each pixel is greater than or equal to a predetermined value, the difference calculation unit 120 assigns a first value representing a candidate moving object to that pixel. Conversely, if the calculated difference value is less than a predetermined value, the difference calculation unit 120 assigns a second value representing a non-moving object to that pixel. Figure 6As shown, in the difference image DI calculated by the difference calculation unit 120, bicycle B is detected as a moving object.

[0052] The grid extraction unit 130 sets up a grid in units of multiple pixels in the difference image DI calculated by the difference calculation unit 120, and extracts the grid when the density (ratio) of the pixels of the first value of each set grid is above a threshold.

[0053] Figure 7 This is a diagram illustrating an example of the structure of the mesh set by the mesh extraction unit 130. Figure 7 In this context, G represents a set of pixels in the difference image DI that are defined as a grid. For example... Figure 7 As shown, the mesh extraction unit 130 sets the size of the mesh G to 10×10 pixels (an example of the "first size") for regions in the differential image DI that are at or below a first distance (e.g., 10m) from the camera 10; for regions that are at or below a second distance (e.g., 20m) from the camera 10 but are at a distance greater than the first distance; for regions that are at or below a second distance (e.g., 20m); and for regions that are at or above the second distance from the camera 10; and for regions that are at or above the second distance. The size of the mesh G is set to 5×5 pixels (an example of the "third size"). This is because the farther the distance from the camera 10, the smaller the change in the area captured by the camera 10, and a finer mesh G size needs to be set to detect moving objects. By setting the mesh G size according to the distance of the differential image DI from the camera 10, moving objects can be detected more accurately.

[0054] Figure 8 This diagram illustrates an example of a method for extracting a grid G ​​performed by the grid extraction unit 130. The grid extraction unit 130 determines, for each of the multiple grids G, whether the density of pixels with a first value is above a threshold (e.g., 85%). For grids G where the density of pixels with the first value is determined to be above the threshold, ... Figure 8 As shown in the upper part, the entire pixel array constituting the grid G ​​is extracted (set as a first value). On the other hand, the grid extraction unit 130 extracts grids G whose pixel density, determined to be the first value, is less than a threshold, such as... Figure 8 As shown in the lower part, the entire pixel array constituting the grid G ​​is discarded (set to the second value).

[0055] It should be noted that in the above description, the grid extraction unit 130 determines whether the density of the pixels of the first value is above a single threshold for each of the multiple grids G. However, the present invention is not limited to that structure, and the grid extraction unit 130 may also change the threshold according to the distance of the differential image DI from the camera 10. For example, generally, the closer the distance to the camera 10, the greater the variation in the area captured by the camera 10, and the easier it is to generate errors. Therefore, the closer the distance to the camera 10, the higher the threshold set by the grid extraction unit 130 may be. Moreover, the grid extraction unit 130 may not be limited to the pixel density of the first value, and may use any statistical value obtained based on the pixels of the first value for determination.

[0056] The mesh extraction unit 130 performs a process on the differential image DI to set all pixels of the mesh with a density of first value above a threshold as the first value (mesh replacement processing), thereby calculating the mesh image GI. Figure 9 This is a diagram showing an example of a grid image GI calculated by the grid extraction unit 130. Figure 8 The components of the differential image DI shown are pixels; on the other hand, Figure 9 The grid image GI shown is composed of grids. Figure 9 As shown, by performing mesh permutation processing on the differential image DI, the mesh representing bicycle B is detected in the mesh image GI. That is, the moving object detection unit 140 detects the mesh represented in the mesh image GI as a moving object.

[0057] The detection results of the moving object detected by the moving object detection unit 140 are sent to the driving control device 200, and the driving control device 200 controls the driving of the vehicle M based on the received detection results. Figure 10 This is a diagram illustrating an example of an action performed by the driving control device 200. Figure 10 As an example, a situation is shown where the moving object detection unit 140 detects bicycle B as a moving object. In this case, the driving control device 200 generates a track for vehicle M to avoid the moving object detected by the moving object detection unit 140, and causes vehicle M to travel along the generated track. Figure 10 In this case, the driving control device 200 controls the vehicle M to stop at a point SP just before the pedestrian crossing where the bicycle B is traveling. Thus, the detection results detected by the moving object detection device 100 can be appropriately applied to the automatic driving or driving support of the vehicle M. The driving control device 200 is an example of a "driving support device".

[0058] It should be noted that the above description illustrates the use of the detection results detected by the moving object detection device 100 for autonomous driving. However, the present invention is not limited to that structure, and the detection results detected by the moving object detection device 100 can also be used, for example, as driving support information provided to occupants performing manual driving.

[0059] Figure 11 This is a diagram showing another example of the structure and peripheral equipment of the moving object detection device 100. Figure 11 In this configuration, the moving object detection device 100 communicates with the reporting device 210 instead of communicating with the driving control device 200. Other structures are similar. Figure 1 The structure shown is the same.

[0060] The reporting device 210 includes a display device, speaker, vibrator, light-emitting device, etc., for outputting information to the occupants of vehicle M, and reports to the occupants of vehicle M the information that there is a moving object in front of vehicle M. The reporting device 210 is an example of a "driving support device".

[0061] Figure 12 This is a diagram illustrating an example of an action performed by the reporting device 210. Figure 12 As an example, the reporting device 210 is shown functioning as a navigation device for vehicle M. For example... Figure 12 As shown, when the grid extraction unit 130 detects a moving object, the reporting device 210 surrounds the detected moving object with a bounding box BX, thereby displaying the moving object on the screen.

[0062] At this time, the reporting device 210 can also display an attention-grabbing message W indicating the presence of a moving object in front of the vehicle M, or it can notify the driver of the presence of a moving object in front of the vehicle M via sound. Through this processing, useful driving support information can be provided to the occupants of the manually driven vehicle M.

[0063] Next, refer to Figure 13 This will explain the processing flow performed by the moving object detection device 100. Figure 13 This is a diagram illustrating an example of the processing flow performed by the moving object detection device 100.

[0064] First, the image acquisition unit 110 acquires an image frame showing the surrounding conditions of the vehicle M captured by the camera 10 as the current image frame (step S100). Next, the differential calculation unit 120 takes the previous image frame acquired just before the current image frame, and based on the speed of the vehicle M between the previous image frame and the current image frame, enlarges the previous image frame with the vanishing point VP as the center, and trims the ends of the enlarged previous image frame, thereby making its size consistent with the current image frame (step S102).

[0065] Next, the difference calculation unit 120 calculates the difference image between the previous image frame and the current image frame (step S104). More specifically, the difference calculation unit 120 calculates the difference value of each pixel for each pixel of the previous image frame and the current image frame. If the calculated difference value is greater than or equal to a predetermined value, a first value is assigned to the pixel. On the other hand, if the calculated difference value is less than or equal to a predetermined value, a second value is assigned to the pixel.

[0066] Next, the mesh extraction unit 130 sets a mesh G in units of multiple pixels in the calculated difference image, and extracts mesh G from the pixels within mesh G whose density of pixels with a first value is above a threshold, thereby calculating the mesh image GI (step S106). Next, the moving object detection unit 140 detects the mesh G represented in the mesh image GI as a moving object (step S108). Next, the driving control device 200 controls the driving of the vehicle M to avoid the moving object detected by the moving object detection unit 140 (step S110). Thus, the processing of this flowchart ends.

[0067] According to the first embodiment described above, a difference image is calculated based on time-series image frames captured by a camera. A grid of different sizes is set up, using multiple pixels in the calculated difference image as units, and the presence or absence of a moving object is detected for each set grid. Therefore, moving objects existing around a vehicle can be detected easily and at high speed.

[0068] [Second Implementation]

[0069] The first embodiment detects grids G mapped to a grid image GI calculated based on the difference image DI as moving objects. However, not all grids G mapped to the grid image GI are moving objects; sometimes they include stationary objects such as pedestrian crossings. The moving object detection device 100 of the second embodiment improves the detection accuracy of moving objects by comparing multiple calculated grid images GI. The functional structure of the moving object detection device 100 of the second embodiment is the same as that of the first embodiment, so its description is omitted.

[0070] In the second embodiment, the moving object detection unit 140 detects moving objects more accurately by comparing multiple grid images obtained at different time points. Figure 14 This diagram illustrates the actions performed by the moving object detection unit 140 according to the second embodiment. Figure 14 The upper part shows the grid image GI1 calculated based on the difference image between time point t0 and time point t1 after time point t0. Figure 14 The lower part shows a grid image GI2 calculated based on the difference image between time point t1 and a subsequent time point t2 (e.g., the current time point). The reference numerals G1 and G2 indicate grids where the density of pixels determined to have a first value is above a threshold. Time point t0 is an example of a "first time point," time point t1 is an example of a "second time point," and time point t3 is an example of a "third time point."

[0071] The moving object detection unit 140 detects moving objects around the vehicle M by comparing grids G1 (G2) in grid image GI1 with grids G1 (G2) in grid image GI2. More specifically, the moving object detection unit 140 first acquires information related to the speed and yaw rate of the vehicle M during the period between time point t1 and time point t2. Next, based on the acquired information related to speed and yaw rate, the moving object detection unit 140 determines the position of grid image GI2 in grid image GI1 corresponding to grid G. Then, the moving object detection unit 140 compares grid G ​​in grid image GI1 with grid G ​​present at the determined position in grid image GI2, and if their shapes and pixel densities are consistent (or similar), determines that these grids G represent the same object. Figure 14 As an example, this illustrates that grid G1 in grid image GI1 corresponds to grid G1 in grid image GI2, and grid G2 in grid image GI1 corresponds to grid G2 in grid image GI2.

[0072] Next, the moving object detection unit 140 determines whether the grid G ​​in the grid image GI2 has moved towards the image center relative to the corresponding grid G ​​in the grid image GI1. If the moving object detection unit 140 determines that the grid G ​​in the grid image GI2 has moved towards the image center, it detects that grid G ​​as a moving object. Figure 14 In the example, the moving object detection unit 140 determines that grid G1 in grid image GI2 has moved towards the image center (i.e., the lower right direction) based on grid G1 in grid image GI1, and therefore detects grid G1 as a moving object. On the other hand, as Figure 14As shown, the grid G2 located in the center of the image does not move towards the center of the image, so the moving object detection unit 140 does not detect the grid G2 as a moving object. However, generally, objects located in the center of the image tend to have their apparent movement amount evaluated as too small before and after frames, so the moving object detection unit 140 may also detect the grid G2 as a monitored object that may collide with the vehicle M.

[0073] Next, refer to Figure 15 This will illustrate the process performed by the moving object detection device 100. Figure 15 This diagram illustrates an example of the processing flow performed by the moving object detection device 100 according to the second embodiment. The processing up to step S106 is... Figure 13 The flowchart of the first embodiment shown is the same, so the description is omitted.

[0074] When the mesh extraction unit 130 calculates the current mesh image GI in step S106, the moving object detection unit 140 obtains the previous mesh image GI calculated one cycle ago (step S200). Next, the moving object detection unit 140 determines the mesh G to be compared based on the speed and yaw rate of the vehicle M between the time the previous mesh image GI was calculated and the time the current mesh image GI was calculated (step S202).

[0075] Next, the moving object detection unit 140 determines whether the grid G ​​in the current grid image GI has moved towards the image center based on the corresponding grid G ​​in the previous grid image GI (step S204). If the moving object detection unit 140 determines that the grid G ​​in the current grid image GI has moved towards the image center based on the corresponding grid G ​​in the previous grid image GI, it detects the grid G ​​as a moving object (step S206).

[0076] On the other hand, if the moving object detection unit 140 determines that a grid G ​​in the current grid image GI has not moved towards the image center based on the corresponding grid G ​​in the previous grid image GI, it detects the grid G ​​as a non-moving object (step S208). Next, the driving control device 200 controls the movement of the vehicle M to avoid the moving object detected by the moving object detection unit 140 (step S210). Thus, the processing of this flowchart ends.

[0077] According to the second embodiment described above, the corresponding grid in the grid image calculated at the previous time point and the grid image calculated at the current time point are determined based on the speed and yaw rate of the vehicle M. If the determined grid has moved towards the center of the vehicle based on the previous time point, the grid is detected as a moving object. Thus, moving objects can be detected more accurately based on the grid image.

[0078] [Third Implementation Method]

[0079] The first embodiment detects moving objects as grids G mapped onto a grid image GI calculated based on the difference image DI. However, not all grids G mapped onto the grid image GI are moving objects; sometimes they include stationary objects such as pedestrian crossings. The moving object detection apparatus 100 of the third embodiment improves the detection accuracy of moving objects by comparing the specified size of the object to be detected (pedestrian, bicycle, vehicle, etc.) with the grids G mapped onto the grid image GI.

[0080] Figure 16 This diagram illustrates an example of the structure and peripheral equipment of the moving object detection device 100 according to the third embodiment. Figure 16 As shown, the moving object detection device 100, in addition to the components of the moving object detection device 100 of the first embodiment, also includes a bounding box setting unit 132. The bounding box setting unit 132 searches for a set of meshes G extracted by the mesh extraction unit 130 that meet a predetermined criterion, and sets a bounding box for the set of meshes G searched.

[0081] Figure 17 This diagram illustrates an example of a mesh G search method performed by the bounding box setting unit 132. The bounding box setting unit 132 first searches for a set of meshes G whose lower end is a certain length L1 or more, based on the mesh image GI calculated by the mesh extraction unit 130. At this time, as... Figure 17 As shown on the left, in order to determine that a set of grids G has a lower end of a certain length L1 or more, it is not necessarily limited to the case that the set contains grids G without any defects. It can also be determined that the lower end has a lower end of a certain length L1 or more, provided that the density of the grids G contained in the lower end is above the reference value.

[0082] Next, the bounding box setting unit 132, upon determining a set of grids G with a lower end having a certain length L1 or more, determines whether the set of grids G has a height having a certain length L2 or more. That is, by determining whether the set of grids G has a lower end with a certain length L1 or more and a height with a certain length L2 or more, it can be determined whether the set of grids G belongs to an object such as a pedestrian, bicycle, or vehicle. In this case, the combination of the lower end's certain length L1 and the height's certain length L2 is set to a fixed value for each of the objects such as pedestrians, bicycles, and vehicles.

[0083] Next, the boundary frame setting unit 132, upon determining a set of grids G with a lower end of a certain length L1 or more and a height of a certain length L2 or more, sets a boundary frame for the set of grids G. Then, the boundary frame setting unit 132 determines whether the density of the grids G contained within the set boundary frame is above a threshold. If the boundary frame setting unit 132 determines that the density of the grids G contained within the set boundary frame is above the threshold, it detects the boundary frame as a moving object. By performing the boundary frame setting and density-related determination, it can be confirmed that the determined set of grids G is a real object (in other words, not an object that happens to meet the specified lower end and height, even though the set of grids G is not an object).

[0084] Thus, the bounding box setting unit 132 searches for a set of grids G that satisfy a predetermined criterion from the grid image GI, but the processing load of searching for grids G can sometimes become heavy. Therefore, the bounding box setting unit 132 can also prioritize searching for areas that are important for the driving of the vehicle M in order to reduce the processing load involved in searching for grids G.

[0085] Figure 18 This is a diagram illustrating an example of the search range of the mesh G searched by the bounding box setting unit 132. Figure 18 In the attached diagram, reference numeral R1 indicates the range of the search grid G ​​at a close distance from vehicle M, reference numeral R2 indicates the range of the search grid G ​​at a medium distance from vehicle M, and reference numeral R3 indicates the range of the search grid G ​​at a distant distance from vehicle M. For example... Figure 18 As shown, the bounding box setting unit 132 can search the entire area of ​​the grid image GI within a range R1 close to the vehicle M, while searching only the central portion of the grid image GI within ranges R1 at a medium distance from the vehicle M and R2 at a distant distance. This allows for priority searching of areas important to the movement of the vehicle M, reducing the processing load associated with searching the grid G.

[0086] It should be noted that, in Figure 18As an example, the case where the search range of the grid G ​​is divided into three stages is described, but the present invention is not limited to such a structure. For example, the bounding box setting unit 132 may search the grid G ​​with a search range of two or more stages, or it may search the grid G ​​in a manner in which the search width of a single search range is continuously focused on the central part.

[0087] Next, refer to Figure 19 This will explain the processing flow performed by the moving object detection device 100. Figure 19 This diagram illustrates an example of the processing flow performed by the moving object detection device 100 according to the third embodiment. The processing up to step S106 is... Figure 13 The flowchart of the first embodiment shown is the same, so the description is omitted.

[0088] When the mesh extraction unit 130 calculates the mesh image GI in step S106, the bounding box setting unit 132 searches for a set of meshes G with a lower end having a certain length L1 or more in the mesh image GI (step S300). If no set of meshes G with a lower end having a certain length L1 or more is found in the mesh image GI, the bounding box setting unit 132 ends the processing of this flowchart.

[0089] On the other hand, if a set of grids G with a lower end of a certain length L1 or more is found in the grid image GI, the bounding box setting unit 132 determines whether the set of grids G has a height of a certain length L2 or more based on the searched lower end (step S304). If it is not determined that the set of searched grids G has a height of a certain length L2 or more, the bounding box setting unit 132 ends the processing of this flowchart.

[0090] On the other hand, if it is determined that the set of searched meshes G has a height of a certain length L2 or more, the bounding box setting unit 132 sets a bounding box that surrounds the set of meshes G (step S306). Next, the bounding box setting unit 132 determines whether the density of meshes G within the set bounding box is above a threshold (step S308). If it is not determined that the density of meshes G within the set bounding box is above the threshold, the bounding box setting unit 132 ends the processing of this flowchart.

[0091] On the other hand, if the density of the grid G ​​within the set bounding box is determined to be above a threshold, the moving object detection unit 140 detects the bounding box as a moving object (step S310). Next, the driving control device 200 controls the driving of the vehicle M to avoid the moving object detected by the moving object detection unit 140 (step S210). Thus, the processing of this flowchart ends.

[0092] According to the third embodiment described above, a set of grids satisfying a predetermined criterion is searched from the grid image, a bounding box is set for the searched set of grids, and moving objects are detected based on whether the density of the set bounding box is above a threshold. Thus, moving objects can be detected more accurately from the grid image.

[0093] The implementation methods described above can be performed as follows.

[0094] A moving object detection device, comprising:

[0095] A storage device containing a program; and

[0096] Hardware processor,

[0097] The hardware processor executes the program stored in the storage device to perform the following processing:

[0098] Image data is acquired, which includes multiple frames representing the surrounding conditions of the moving body, captured in a time sequence by a camera mounted on the moving body.

[0099] The differences between the multiple frames are calculated, and the differences are binarized into a first value and a second value, thereby calculating the difference image between the multiple frames;

[0100] Extract the grids from the plurality of grids defined in the difference image where the density of the pixels of the first value is above a first threshold.

[0101] The extracted mesh is detected as a moving object; and

[0102] The multiple grids are configured to have smaller pixel sizes the farther away from the camera.

[0103] The above description illustrates specific embodiments of the present invention, but the present invention is not limited to such embodiments in any way, and various modifications and substitutions can be made without departing from the spirit of the present invention.

Claims

1. An object motion detection device, wherein the object motion detection device includes: an image acquisition unit that acquires image data including a plurality of frames representing a surrounding situation of a moving body, the frames being captured by a camera mounted on the moving body in a time series; a difference calculation unit that calculates a difference between the plurality of frames and binarizes the difference into a first value and a second value, thereby calculating a difference image between the plurality of frames; a grid extraction unit that extracts a grid in which a density of pixels of the first value in a plurality of grids set in the difference image is equal to or greater than a first threshold value; and an object motion detection unit that detects the extracted grid as an object motion, the plurality of grids are grids in which a distance from the camera is greater, the grids are set to have smaller pixel sizes, the grid extraction unit changes the first threshold value in accordance with the distance from the camera of each of the plurality of grids. 2.The object motion detection device according to claim 1, wherein the difference calculation unit enlarges a frame captured at a previous time point based on a speed of the moving body in a capturing interval in which the plurality of frames are captured, and calculates a difference image between the frame captured at the previous time point after the enlargement and a frame captured at a current time point. 3.The object motion detection device according to claim 2, wherein the difference calculation unit enlarges the frame captured at the previous time point with a vanishing point of the frame captured at the previous time point as a center. 4.The object motion detection device according to any one of claims 1 to 3, wherein the difference calculation unit corrects a frame captured at a previous time point based on a yaw rate of the moving body in a capturing interval in which the plurality of frames are captured, and calculates a difference image between the frame captured at the previous time point after the correction and a frame captured at a current time point. 5.The object motion detection device according to any one of claims 1 to 3, wherein the grid extraction unit sets a size of the plurality of grids to a first size in a case where a distance from the camera is equal to or smaller than a first distance, sets the size of the plurality of grids to a second size smaller than the first size in a case where the distance from the camera is greater than the first distance and equal to or smaller than a second distance, and sets the size of the plurality of grids to a third size smaller than the second size in a case where the distance from the camera is greater than the second distance. 6.An object motion detection system, wherein the object motion detection system includes: the object motion detection device according to any one of claims 1 to 5; and a driving support device that performs driving support of the moving body based on a detection result obtained by the object motion detection device. 7.An object motion detection method, wherein the object motion detection method causes a computer to perform the following processing: acquire image data including a plurality of frames representing a surrounding situation of a moving body, the frames being captured by a camera mounted on the moving body in a time series; calculate a difference between the plurality of frames and binarize the difference into a first value and a second value, thereby calculating a difference image between the plurality of frames; ​ ​ extracting a grid in which a density of pixels of the first value in a plurality of grids set in the difference image is equal to or greater than a first threshold value; and detecting the extracted grid as a moving object, the plurality of grids are grids in which a pixel size is set to be smaller as a distance from the camera is farther, the first threshold value is varied in accordance with a distance from the camera for each of the plurality of grids.

8. A storage medium storing a program, wherein the program causes a computer to perform the following processing: acquiring image data including a plurality of frames representing a surrounding situation of a moving body, the frames being captured by a camera mounted on the moving body in time series; calculating a difference between the plurality of frames, binarizing the difference into a first value and a second value, and thereby calculating a difference image between the plurality of frames; extracting a grid in which a density of pixels of the first value in a plurality of grids set in the difference image is equal to or greater than a first threshold value; and detecting the extracted grid as a moving object, the plurality of grids are grids in which a pixel size is set to be smaller as a distance from the camera is farther, the first threshold value is varied in accordance with a distance from the camera for each of the plurality of grids.

Images