Moving object detection device, moving object detection method, system, and program

Abstract

To simply and quickly detect a moving body existing around a vehicle.SOLUTION: A moving body detection apparatus includes: an image acquisition unit which acquires image data including a plurality of frames representing surroundings of a mobile body, the image data being captured in time series by a camera mounted on the mobile body; a difference calculation unit which calculates a difference between the frames, and binarizes the difference into a first value and a second value, to calculate a difference image between the frames; a grid extraction unit which extracts, out of multiple grids set in the difference image, a grid in which density of pixels of the first value is equal to or higher than a first threshold; and a moving body detection unit which detects, as a moving body, the extracted grid. The multiple grids are set so that the grids located far from the camera have small pixel size.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a moving object detection device, a moving object detection method, a system, and a program. [Background technology]

[0002] Conventionally, there is known a technology for detecting a moving object existing around a vehicle based on image data of the area in front of the vehicle captured by an on-board camera. For example, Patent Document 1 discloses a technology for outputting an identification result of a moving object existing in image data by performing signal processing based on a pre-learned result on image data of the vehicle's surrounding environment. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-144689 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 uses a deep neural network (DNN) such as a convolutional neural network to detect moving objects around a vehicle. However, such machine learning techniques require the preparation of a large amount of training data in advance, and also tend to impose a high processing load during execution. As a result, there are cases where moving objects around the vehicle cannot be detected quickly.

[0005] The present invention has been made in consideration of these circumstances, and one of its objectives is to provide a moving object detection device, a moving object detection method, a system, and a program that can easily and quickly detect moving objects present around a vehicle. [Means for solving the problem]

[0006] The moving object detection device, moving object detection method, system, and program according to the present invention employ the following configuration. (1): A moving object detection device according to one embodiment of the present invention includes an image acquisition unit that acquires image data including a plurality of frames that represent the surrounding conditions of the moving object, captured in chronological order by a camera mounted on the moving object; a difference calculation unit that calculates a difference between the plurality of frames and binarizes the difference into a first value and a second value, thereby calculating a difference image between the plurality of frames; a grid extraction unit that extracts, from a plurality of grids set in the difference image, grids whose pixel density of the first value is equal to or greater than a first threshold; and a moving object detection unit that detects the extracted grids as moving objects, wherein the plurality of grids are set to have smaller pixel sizes the farther they are from the camera.

[0007] (2): In the above aspect (1), the difference calculation unit enlarges the frame captured at the previous time point based on the speed of the moving body during the shooting interval in which the multiple frames were captured, and calculates a difference image between the enlarged frame captured at the previous time point and the frame captured at the current time point.

[0008] (3) In the above aspect (2), the difference calculation unit enlarges the frame captured at the previous time point, with the vanishing point of the frame captured at the previous time point as the center.

[0009] (4): In any of the above aspects (1) to (3), the difference calculation unit translates the frame captured at the previous time point based on the yaw rate of the moving body during the shooting interval in which the multiple frames were captured, and calculates a difference image between the translated frame captured at the previous time point and the frame captured at the current time point.

[0010] (5): In any of the above aspects (1) to (4), the grid extraction unit changes the threshold value depending on the distance from the camera to each of the plurality of grids.

[0011] (6): In any of the above aspects (1) to (5), the grid extraction unit sets the size of the plurality of grids to a first size when the distance from the camera is equal to or less than a first distance, sets the size of the plurality of grids to a second size smaller than the first size when the distance from the camera is greater than the first distance and equal to or less than a second distance, and sets the size of the plurality of grids to a third size smaller than the second size when the distance from the camera is greater than the second distance.

[0012] (7): A system comprising a moving object detection device according to any one of aspects (1) to (6) above, and a driving assistance device that provides driving assistance to the moving object based on the detection results obtained by the moving object detection device.

[0013] (8): Another aspect of the moving object detection method of the present invention is a method in which a computer acquires image data including a plurality of frames representing the surrounding conditions of a moving object, which are captured in chronological order by a camera mounted on the moving object, calculates the difference between the plurality of frames, and binarizes the difference into a first value and a second value to calculate a difference image between the plurality of frames, extracts grids from a plurality of grids set in the difference image, the density of pixels of the first value being equal to or greater than a first threshold, and detects the extracted grids as moving objects, and the plurality of grids are set to have smaller pixel sizes the farther they are from the camera.

[0014] (9): Another aspect of the present invention provides a program that causes a computer to acquire image data including a plurality of frames representing the surrounding conditions of a moving object, captured in chronological order by a camera mounted on the moving object, calculate the difference between the plurality of frames, and calculate a difference image between the plurality of frames by binarizing the difference into a first value and a second value, extract grids from a plurality of grids set in the difference image that have a pixel density of the first value equal to or greater than a first threshold, and detect the extracted grids as a moving object, with the plurality of grids being set to have smaller pixel sizes the farther they are from the camera. [Effects of the Invention]

[0015] According to aspects (1) to (9), moving objects existing around the vehicle can be detected easily and quickly.

[0016] According to the aspect (2) or (3), it is possible to accurately calculate the difference image between the frame captured at the previous time point and the frame captured at the current time point.

[0017] According to the aspect (4), the difference image can be calculated accurately, taking into account the yaw rate of the moving object.

[0018] According to the aspect (5) or (6), a moving object can be accurately detected according to the distance from the camera.

[0019] According to the seventh aspect, the detection result by the moving object detection device can be suitably utilized for driving assistance. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a diagram illustrating an example of the configuration of a moving object detection device 100 according to a first embodiment and peripheral devices. [Figure 2] 1 is a diagram showing an example of the surroundings of a vehicle M equipped with a moving object detection device 100. FIG. [Figure 3]3 is a diagram showing an example of an image obtained by the camera 10 capturing an image of the area ahead of the vehicle M in the surrounding situation shown in FIG. 2. FIG. [Figure 4] FIG. 10 is a diagram showing an example of a vanishing point used as a reference point for enlarging a frame. [Figure 5] FIG. 10 is a diagram for explaining a method of correcting a previous frame in order to calculate a difference image DI. [Figure 6] 10 is a diagram showing an example of a difference image DI calculated by a difference calculation unit 120. FIG. [Figure 7] 10 is a diagram showing an example of the configuration of a grid G ​​set by a grid extraction unit 130. FIG. [Figure 8] 10 is a diagram showing an example of a method for extracting a grid G ​​by a grid extraction unit 130. FIG. [Figure 9] 10 is a diagram showing an example of a grid image GI calculated by a grid extraction unit 130. FIG. [Figure 10] 3 is a diagram illustrating an example of an operation executed by the cruise control device 200. FIG. [Figure 11] FIG. 10 is a diagram showing another example of the configuration of the moving object detection device 100 and peripheral devices. [Figure 12] 10 is a diagram illustrating an example of an operation executed by a notification device 210. FIG. [Figure 13] 1 is a diagram showing an example of the flow of processing executed by the moving object detection device 100. FIG. [Figure 14] FIG. 10 is a diagram for explaining the operation executed by the moving object detection unit 140 according to the second embodiment. [Figure 15] FIG. 10 is a diagram showing an example of the flow of processing executed by the moving object detection device 100 according to the second embodiment. [Figure 16] FIG. 10 is a diagram illustrating an example of the configuration of a moving object detection device 100 according to a third embodiment and peripheral devices. [Figure 17] 10 is a diagram showing an example of a method for searching for a grid G ​​executed by a bounding box setting unit 132. FIG. [Figure 18] 10 is a diagram showing an example of a search range for a grid G ​​by a bounding box setting unit 132. FIG. [Figure 19] FIG. 11 is a diagram showing an example of the flow of processing executed by the moving object detection device 100 according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0021] [First embodiment] Hereinafter, with reference to the drawings, embodiments of a moving object detection device, a moving object detection method, a system, and a program of the present invention will be described. The moving object detection device is mounted on, for example, a moving object. The moving object may be, for example, a four-wheeled vehicle, a two-wheeled vehicle, micromobility, a robot that moves on its own, or a portable device such as a smartphone that is mounted on a moving object or carried by a person. In the following description, the moving object is assumed to be a four-wheeled vehicle, and the moving object will be referred to as a "vehicle." The moving object detection device is not limited to a device mounted on a moving object, but may also be a device that performs the processing described below based on images captured by a fixed-point observation camera or a smartphone camera.

[0022] 1 is a diagram showing an example of the configuration of a moving object detection device 100 and peripheral devices. The moving object detection device 100 communicates with a camera 10, a driving control device 200, and the like.

[0023] The camera 10 is attached to the rear surface of the windshield of the vehicle M, captures images of at least the road in the traveling direction of the vehicle M in time series, and outputs the captured images to the moving object detection device 100. Note that a sensor fusion device or the like may be interposed between the camera 10 and the moving object detection device 100, but a description thereof will be omitted. The driving control device 200 is, for example, an automatic driving control device that causes the vehicle M to drive autonomously, or a driving assistance device that performs inter-vehicle distance control, automatic braking control, automatic lane change control, etc.

[0024] The moving object detection device 100 includes, for example, an image acquisition unit 110, a difference calculation unit 120, a grid extraction unit 130, and a moving object detection unit 140. These components are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (a storage device with a non-transitory storage medium) such as an HDD (Hard Disk Drive) or flash memory, or may be stored in a removable storage medium (a non-transitory storage medium) such as a DVD or CD-ROM, and installed by inserting the storage medium into a drive device.

[0025] Fig. 2 is a diagram showing an example of the surrounding situation of a vehicle M equipped with a moving object detection device 100. Fig. 2 shows, as an example, a scene in which a motorcycle B is traveling in front of the vehicle M equipped with the moving object detection device 100 while the vehicle M is traveling on a road. In the following explanation, a scene in which the moving object detection device 100 detects the motorcycle B as a moving object will be described as an example, but the present invention is not limited to application to such a scene.

[0026] Fig. 3 is a diagram showing an example of an image obtained by the camera 10 capturing an image of the area ahead of the vehicle M in the surrounding situation shown in Fig. 2. The image acquisition unit 110 acquires image data including a plurality of frames that represent the surrounding situation of the vehicle M, captured in time series by the camera 10 mounted on the vehicle M. More specifically, for example, the image acquisition unit 110 acquires image data from the camera 10 at a frame rate of 30 Hz.

[0027] The difference calculation unit 120 calculates differences in pixel values ​​for multiple frames acquired by the image acquisition unit 110, and calculates a difference image DI between the multiple frames by binarizing the calculated differences into a first value (e.g., 1) and a second value (e.g., 0). More specifically, first, the difference calculation unit 120 performs gray conversion on the multiple frames acquired by the image acquisition unit 110, converting the RGB image into a grayscale image.

[0028] Next, the difference calculation unit 120 aligns the frame captured at the previous time point (hereinafter sometimes referred to as the "previous frame") with the frame captured at the current time point (hereinafter sometimes referred to as the "current frame") by enlarging the frame captured at the previous time point (hereinafter sometimes referred to as the "previous frame") around the vanishing point of that frame, based on the speed of the vehicle M during the shooting interval when multiple frames were captured.

[0029] Fig. 4 is a diagram showing an example of a vanishing point used as a reference point for enlarging a frame. In Fig. 4, VP indicates the vanishing point of the frame. The vanishing point VP is defined, for example, as the intersection point formed by extending both sides of the lane on which vehicle M is traveling. The difference calculation unit 120 estimates the travel distance of vehicle M from, for example, the speed (average speed) of vehicle M measured between the previous time point and the current time point, and enlarges the previous frame by an enlargement rate corresponding to the travel distance, with vanishing point VP as the center.

[0030] FIG. 5 is a diagram illustrating a method for correcting the previous frame to calculate the difference image DI. In FIG. 5, h represents the height of the frame, and w represents the width of the frame. As shown in FIG. 5, the difference calculation unit 120 enlarges the previous frame by an enlargement ratio corresponding to the travel distance of the vehicle M measured between the previous time point and the current time point. At this time, the size of the enlarged previous frame is larger than before enlargement, so the difference calculation unit 120 returns the size of the enlarged previous frame to its original size by trimming the edges of the enlarged previous frame.

[0031] The difference calculation unit 120 may correct the previous frame by taking into consideration the yaw rate of the vehicle M during the shooting interval between the previous frame and the current frame, in addition to the speed of the vehicle M during the shooting interval between the previous frame and the current frame. More specifically, the difference calculation unit 120 may calculate the difference between the yaw angle of the vehicle M at the time of acquisition of the previous frame and the yaw angle of the vehicle M at the time of acquisition of the current frame based on the yaw rate during the shooting interval, and align the previous frame with the current frame by shifting the previous frame in the yaw direction by an angle corresponding to the difference.

[0032] FIG. 6 is a diagram showing an example of a difference image DI calculated by the difference calculation unit 120. After aligning the previous frame with the current frame, the difference calculation unit 120 calculates the difference in pixel values ​​between the previous frame and the current frame. If the difference value calculated for each pixel is equal to or greater than a specified value, the difference calculation unit 120 assigns to the pixel a first value indicating that the pixel is a candidate for a moving object. On the other hand, if the calculated difference value is less than the specified value, the difference calculation unit 120 assigns to the pixel a second value indicating that the pixel is not a candidate for a moving object. As shown in FIG. 6, in the difference image DI calculated by the difference calculation unit 120, it can be seen that motorcycle B has been detected as a moving object.

[0033] The grid extraction unit 130 sets a grid for each of multiple pixels in the difference image DI calculated by the difference calculation unit 120, and extracts the grid when the density (proportion) of pixels with the first value in each of the set grids is equal to or greater than a threshold value.

[0034] FIG. 7 is a diagram illustrating an example of the configuration of a grid set by the grid extraction unit 130. In FIG. 7, G represents a set of multiple pixels defined as a grid in the difference image DI. As shown in FIG. 7, the grid extraction unit 130 sets the size of the grid G ​​to 10×10 pixels (an example of a “first size”) for areas in the difference image DI where the distance from the camera 10 is equal to or less than a first distance (e.g., 10 m), sets the size of the grid G ​​to 8×8 pixels (an example of a “second size”) for areas where the distance from the camera 10 is greater than the first distance but equal to or less than a second distance (e.g., 20 m), and sets the size of the grid G ​​to 5×5 pixels (an example of a “third size”) for areas where the distance from the camera 10 is greater than the second distance. This is because the greater the distance from the camera 10, the smaller the change in the area captured by the camera 10. Therefore, in order to detect a moving object, the size of the grid G ​​needs to be set more finely. Setting the size of the grid G ​​according to the distance from the camera 10 in the difference image DI allows for more accurate detection of a moving object.

[0035] Fig. 8 is a diagram showing an example of a method for extracting a grid G ​​by the grid extraction unit 130. For each of a plurality of grids G, the grid extraction unit 130 determines whether the density of pixels having a first value is equal to or greater than a threshold value (e.g., 85%), and for a grid G ​​for which it is determined that the density of pixels having a first value is equal to or greater than the threshold value, the grid extraction unit 130 extracts all of the pixels constituting the grid G ​​(sets them to a first value), as shown in the upper part of Fig. 8. On the other hand, for a grid G ​​for which it is determined that the density of pixels having a first value is less than the threshold value, the grid extraction unit 130 discards all of the pixels constituting the grid G ​​(sets them to a second value), as shown in the lower part of Fig. 8.

[0036] In the above description, the grid extraction unit 130 determines whether the density of pixels having the first value is equal to or greater than a single threshold for each of the plurality of grids G. However, the present invention is not limited to such a configuration, and the grid extraction unit 130 may change the threshold depending on the distance from the camera 10 in the difference image DI. For example, the closer the distance from the camera 10, the greater the change in the area captured by the camera 10 and the greater the likelihood of errors. Therefore, the grid extraction unit 130 may set a higher threshold the closer the distance from the camera 10. Furthermore, the grid extraction unit 130 may make its determination using any statistical value based on the pixels having the first value, rather than being limited to the density of pixels having the first value.

[0037] The grid extraction unit 130 calculates a grid image GI by performing a process (grid replacement process) on the difference image DI, in which all pixels of a grid where the density of pixels having a first value is equal to or greater than a threshold value are set to the first value. FIG. 9 is a diagram showing an example of the grid image GI calculated by the grid extraction unit 130. The components of the difference image DI shown in FIG. 8 are pixels, while the components of the grid image GI shown in FIG. 9 are grids. As shown in FIG. 9, it can be seen that by performing the grid replacement process on the difference image DI, a grid representing motorbike B has been detected in the grid image GI. In other words, the moving object detection unit 140 detects the grid represented in the grid image GI as a moving object.

[0038] The detection result of the moving object by the moving object detection unit 140 is transmitted to the cruise control device 200, and the cruise control device 200 controls the driving of the vehicle M based on the received detection result. FIG. 10 is a diagram showing an example of the operation executed by the cruise control device 200. As an example, FIG. 10 shows a situation in which the moving object detection unit 140 detects a motorcycle B as a moving object. At this time, the cruise control device 200 generates a trajectory for the vehicle M so as to avoid the moving object detected by the moving object detection unit 140, and causes the vehicle M to drive along the generated trajectory. In the case of FIG. 10, the cruise control device 200 controls the vehicle M so that the vehicle M stops at a point SP just before the pedestrian crossing where the motorcycle B is traveling. This makes it possible to suitably utilize the detection result by the moving object detection device 100 for automatic driving or driving assistance of the vehicle M. The cruise control device 200 is an example of a "driving assistance device."

[0039] In the above description, the detection results by the moving object detection device 100 are used for automatic driving. However, the present invention is not limited to such a configuration, and the detection results by the moving object detection device 100 can also be used, for example, as driving assistance information to be provided to a driver who is manually driving the vehicle.

[0040] 11 is a diagram showing another example of the configuration of the moving object detection device 100 and peripheral devices. In Fig. 11, the moving object detection device 100 is configured to communicate with a notification device 210 instead of the driving control device 200. The other configuration is the same as the configuration shown in Fig. 1.

[0041] The notification device 210 is a display device, speaker, vibrator, light-emitting device, etc. for outputting information to the occupants of the vehicle M, and notifies the occupants of the vehicle M of information that a moving object is present in front of the vehicle M. The notification device 210 is an example of a "driving assistance device."

[0042] Fig. 12 is a diagram showing an example of an operation executed by the notification device 210. Fig. 12 shows, as an example, a case where the notification device 210 functions as a navigation device for the vehicle M. As shown in Fig. 12, when the grid extraction unit 130 detects a moving object, the notification device 210 displays the moving object on the screen, for example, by surrounding the detected moving object with a bounding box BX.

[0043] At this time, the notification device 210 may further display a warning message W indicating that a moving object is present ahead of the vehicle M, or may notify by voice that a moving object is present ahead of the vehicle M. Through such processing, useful driving assistance information can be provided to the occupant who is manually driving the vehicle M.

[0044] Next, the flow of processing executed by the moving object detection device 100 will be described with reference to Fig. 13. Fig. 13 is a diagram showing an example of the flow of processing executed by the moving object detection device 100.

[0045] First, the image acquisition unit 110 acquires an image frame captured by the camera 10 and representing the surrounding conditions of the vehicle M as the current image frame (step S100). Next, the difference calculation unit 120 enlarges the previous image frame acquired immediately before the current image frame, centered on the vanishing point VP, based on the speed of the vehicle M between the previous image frame and the current image frame, and trims the edges of the enlarged previous image frame to match its size with the current image frame (step S102).

[0046] Next, the difference calculation unit 120 calculates a difference image between the previous image frame and the current image frame (step S104). More specifically, the difference calculation unit 120 calculates a pixel difference value for each pixel between the previous image frame and the current image frame, and if the calculated difference value is equal to or greater than a specified value, assigns a first value to the pixel, whereas if the calculated difference value is less than the specified value, assigns a second value to the pixel.

[0047] Next, the grid extraction unit 130 sets a grid G ​​for each of a plurality of pixels in the calculated difference image, and calculates a grid image GI by extracting grids G in which the density of pixels having a first value is equal to or greater than a threshold value from among the pixels in the grid G ​​(step S106). Next, the moving object detection unit 140 detects the grid G ​​represented in the grid image GI as a moving object (step S108). Next, the driving control device 200 controls the driving of the vehicle M so as to avoid the moving object detected by the moving object detection unit 140 (step S110). This ends the processing of this flowchart.

[0048] According to the first embodiment described above, a difference image is calculated from a time series of image frames captured by a camera, grids of different sizes are set for each of a plurality of pixels in the calculated difference image, and the presence or absence of a moving object is detected for each of the set grids. This makes it possible to simply and quickly detect moving objects present around a vehicle.

[0049] [Second embodiment] In the first embodiment, grids G captured in a grid image GI calculated from a difference image DI are detected as moving objects. However, not all grids G captured in a grid image GI are necessarily moving objects, and stationary objects such as crosswalks may also be included. The moving object detection device 100 according to the second embodiment improves the accuracy of detecting moving objects by comparing a plurality of calculated grid images GI. The functional configuration of the moving object detection device 100 according to the second embodiment is the same as that of the first embodiment, and therefore description thereof will be omitted.

[0050] In the second embodiment, the moving object detection unit 140 detects a moving object by comparing multiple grid images obtained at different times in order to more accurately detect a moving object from a grid image. FIG. 14 is a diagram for explaining the operation performed by the moving object detection unit 140 according to the second embodiment. The upper part of FIG. 14 shows a grid image GI1 calculated from a difference image between time t0 and time t1, which is later than time t0, and the lower part of FIG. 14 shows a grid image GI2 calculated from a difference image between time t1 and time t2 (e.g., the current time) which is later than time t1. The symbols G1 and G2 each indicate grids in which the density of pixels having a first value is determined to be equal to or greater than a threshold. Time t0 is an example of a "first time point," time t1 is an example of a "second time point," and time t3 is an example of a "third time point."

[0051] The moving object detection unit 140 detects a moving object present around the vehicle M by comparing a grid G1 (G2) in the grid image GI1 with a grid G1 (G2) in the grid image GI2. More specifically, the moving object detection unit 140 first acquires information regarding the speed and yaw rate of the vehicle M during the period between time t1 and time t2. Next, the moving object detection unit 140 identifies the position of the grid image GI2 corresponding to the grid G ​​in the grid image GI1 based on the acquired information regarding the speed and yaw rate. Next, the moving object detection unit 140 compares the grid G ​​in the grid image GI1 with the grid G ​​present at the identified position of the grid image GI2, and determines that these grid Gs represent the same object if their shapes and densities of first-value pixels match (or are similar). FIG. 14 illustrates an example in which the grid G1 in the grid image GI1 corresponds to the grid G1 in the grid image GI2, and the grid G2 in the grid image GI1 corresponds to the grid G2 in the grid image GI2.

[0052] Next, the moving object detection unit 140 determines whether a grid G ​​in the grid image GI2 has moved toward the center of the image, using the corresponding grid G ​​in the grid image GI1 as a reference. If the moving object detection unit 140 determines that the grid G ​​in the grid image GI2 has moved toward the center of the image, the moving object detection unit 140 detects the grid G ​​as a moving object. In the example of FIG. 14 , the moving object detection unit 140 determines that the grid G1 in the grid image GI2 has moved toward the center of the image (i.e., toward the lower right) using the grid G1 in the grid image GI1 as a reference, and therefore detects the grid G1 as a moving object. On the other hand, as shown in FIG. 14 , the grid G2 located in the center of the image has not moved toward the center of the image, and therefore the moving object detection unit 140 does not detect the grid G2 as a moving object. However, in general, the amount of movement of an object located in the center of the image tends to be underestimated in appearance between frames. Therefore, the moving object detection unit 140 may detect the grid G2 as a monitored object that may collide with the vehicle M.

[0053] Next, the flow of processing executed by the moving object detection device 100 will be described with reference to Fig. 15. Fig. 15 is a diagram showing an example of the flow of processing executed by the moving object detection device 100 according to the second embodiment. The processing up to step S106 is the same as that in the flowchart of the first embodiment shown in Fig. 13, and therefore description thereof will be omitted.

[0054] In step S106, when the grid extraction unit 130 calculates the current grid image GI, the moving object detection unit 140 acquires the previous grid image GI calculated one cycle before (step S200). Next, the moving object detection unit 140 identifies a grid G ​​to be compared based on the speed and yaw rate of the vehicle M between the time when the previous grid image GI was calculated and the time when the current grid image GI was calculated (step S202).

[0055] Next, the moving object detection unit 140 determines whether a grid G ​​in the current grid image GI has moved toward the center of the image using the corresponding grid G ​​in the previous grid image GI as a reference (step S204). If the moving object detection unit 140 determines that a grid G ​​in the current grid image GI has moved toward the center of the image using the corresponding grid G ​​in the previous grid image GI as a reference, it detects the grid G ​​as a moving object (step S206).

[0056] On the other hand, if the moving object detection unit 140 determines that a grid G ​​in the current grid image GI has not moved toward the center of the image with respect to the corresponding grid G ​​in the previous grid image GI, the moving object detection unit 140 detects the grid G ​​as a non-moving object (step S208). Next, the driving control device 200 controls the driving of the vehicle M so as to avoid the moving object detected by the moving object detection unit 140 (step S210). This ends the processing of this flowchart.

[0057] According to the second embodiment described above, corresponding grids in the grid image calculated at the previous time point and the grid image calculated at the current time point are identified based on the speed and yaw rate of the vehicle M, and if the identified grid has moved toward the center of the vehicle with respect to the previous time point as a reference, the grid is detected as a moving object. This makes it possible to more accurately detect a moving object from the grid image.

[0058] [Third embodiment] In the first embodiment, grids G captured in a grid image GI calculated from a difference image DI are detected as moving objects. However, not all grids G captured in the grid image GI are necessarily moving objects, and stationary objects such as crosswalks may also be included. The moving object detection device 100 according to the third embodiment improves the accuracy of detecting moving objects by comparing the grids G captured in the grid image GI with the specified size of the object to be detected (pedestrians, motorcycles, vehicles, etc.).

[0059] 16 is a diagram showing an example of the configuration of a moving object detection device 100 according to the third embodiment and peripheral devices. As shown in Fig. 16, the moving object detection device 100 includes a bounding box setting unit 132 in addition to the components of the moving object detection device 100 according to the first embodiment. The bounding box setting unit 132 searches for a set of grids G that are extracted by the grid extraction unit 130 and that satisfy a predetermined criterion, and sets a bounding box for the set of grids G that has been found.

[0060] 17 is a diagram showing an example of a method for searching for grid G ​​executed by the bounding box setting unit 132. The bounding box setting unit 132 first searches for a set of grid G ​​whose bottom ends have a certain length L1 or more from the grid image GI calculated by the grid extraction unit 130. At this time, as shown in the left part of FIG. 17, in order for a set of grid G ​​to be determined to have a bottom end of the certain length L1 or more, the set does not necessarily have to include grid G ​​without any missing pieces of grid G. It may also be determined that the set has a bottom end of the certain length L1 or more on the prerequisite that the density of grid G ​​included in the bottom end is equal to or greater than a reference value.

[0061] Next, when the bounding box setting unit 132 identifies a set of grids G having a bottom end that is equal to or greater than a certain length L1, it determines whether the set of grids G has a height that is equal to or greater than a certain length L2. That is, by determining whether the set of grids G has a bottom end that is equal to or greater than a certain length L1 and a height that is equal to or greater than a certain length L2, it is possible to identify whether the set of grids G corresponds to an object such as a pedestrian, a motorcycle, or a vehicle. In this case, the combination of the bottom end constant length L1 and the height constant length L2 is set as a unique value for each object such as a pedestrian, a motorcycle, or a vehicle.

[0062] Next, when the bounding box setting unit 132 identifies a set of grids G having a bottom end that is equal to or greater than a certain length L1 and a height that is equal to or greater than a certain length L2, it sets a bounding box for the set of grids G. Next, the bounding box setting unit 132 determines whether the density of the grids G included in the set bounding box is equal to or greater than a threshold. If the bounding box setting unit 132 determines that the density of the grids G included in the set bounding box is equal to or greater than a threshold, it detects the bounding box as a moving object. By setting the bounding box and determining the density, it is possible to confirm that the identified set of grids G is an actual object (in other words, that the set of grids G is not an object but just happens to have a specified bottom end and height).

[0063] In this way, the bounding box setting unit 132 searches for a set of grids G that satisfy a predetermined criterion from the grid image GI, but the search for the grids G may impose a heavy processing load. Therefore, in order to reduce the processing load associated with the search for the grids G, the bounding box setting unit 132 may preferentially search areas that are important for the traveling of the vehicle M.

[0064] FIG. 18 is a diagram showing an example of a search range for grid G ​​by the bounding box setting unit 132. In FIG. 18, reference symbol R1 indicates a range for searching for grid G ​​at a short distance from vehicle M, reference symbol R2 indicates a range for searching for grid G ​​at a medium distance from vehicle M, and reference symbol R3 indicates a range for searching for grid G ​​at a long distance from vehicle M. As shown in FIG. 18, the bounding box setting unit 132 may search the entire area of ​​the grid image GI in the range R1 for a short distance from vehicle M, but may search only the central portion of the grid image GI in the range R1 for a medium distance from vehicle M and the range R2 for a long distance from vehicle M. This allows for preferential searching of areas important for the traveling of vehicle M, thereby reducing the processing load associated with searching for grid G.

[0065] 18 illustrates an example in which the search range for grid G ​​is divided into three stages, but the present invention is not limited to such a configuration. For example, the bounding box setting unit 132 may search for grid G ​​using a search range of two stages or four or more stages, or may search for grid G ​​by continuously narrowing the search width of a single search range to the center.

[0066] Next, the flow of processing executed by the moving object detection device 100 will be described with reference to Fig. 19. Fig. 19 is a diagram showing an example of the flow of processing executed by the moving object detection device 100 according to the third embodiment. The processing up to step S106 is the same as that in the flowchart of the first embodiment shown in Fig. 13, and therefore description thereof will be omitted.

[0067] In step S106, the grid extraction unit 130 calculates the grid image GI, and then the bounding box setting unit 132 searches for a set of grids G in the grid image GI whose lower ends are equal to or longer than a certain length L1 (step S300). If a set of grids G in the grid image GI whose lower ends are equal to or longer than the certain length L1 is not found, the bounding box setting unit 132 ends the processing of this flowchart.

[0068] On the other hand, if a set of grids G having a bottom end of equal to or greater than the fixed length L1 is found in the grid image GI, the bounding box setting unit 132 determines whether the set of grids G has a height of equal to or greater than the fixed length L2, based on the found bottom end (step S304). If it is not determined that the found set of grids G has a height of equal to or greater than the fixed length L2, the bounding box setting unit 132 ends the processing of this flowchart.

[0069] On the other hand, if it is determined that the set of found grids G has a height equal to or greater than the certain length L2, the bounding box setting unit 132 sets a bounding box that encloses the set of grids G (step S306). Next, the bounding box setting unit 132 determines whether the density of grids G within the set bounding box is equal to or greater than a threshold (step S308). If it is not determined that the density of grids G within the set bounding box is equal to or greater than the threshold, the bounding box setting unit 132 ends the processing of this flowchart.

[0070] On the other hand, if it is determined that the density of the grid G ​​within the set bounding box is equal to or greater than the threshold, the moving object detection unit 140 detects the bounding box as a moving object (step S310). Next, the driving control device 200 controls the driving of the vehicle M so as to avoid the moving object detected by the moving object detection unit 140 (step S210). This ends the processing of this flowchart.

[0071] According to the third embodiment described above, a grid image is searched for a set of grids that meets a predetermined criterion, a bounding box is set for the searched set of grids, and a moving object is detected based on whether the density of the set bounding box is equal to or greater than a threshold value. This makes it possible to more accurately detect moving objects from a grid image.

[0072] The above-described embodiment can be expressed as follows. a storage device storing a program; a hardware processor; The hardware processor executes the program stored in the storage device, acquiring image data including a plurality of frames that are captured in time series by a camera mounted on a moving object and represent the surrounding situation of the moving object; calculating a difference between the plurality of frames and binarizing the difference into a first value and a second value to calculate a difference image between the plurality of frames; extracting grids in which the density of pixels having the first value is equal to or greater than a first threshold value from among a plurality of grids set in the difference image; Detecting the extracted grid as a moving object; The plurality of grids are set to have smaller pixel sizes as the distance from the camera increases. Motion detection device.

[0073] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0074] 10 Camera 100 Motion detection device 110 Image acquisition unit 120 Difference calculation section 130 Grid Extraction Unit 132 Bounding box setting section 140 Motion detection unit 200 Driving control device 210 Alarm device

Claims

1. an image acquisition unit that acquires image data including a plurality of frames that are captured in time series by a camera mounted on the moving body and represent the surrounding situation of the moving body; a difference calculation unit that calculates a difference between the plurality of frames and binarizes the difference into a first value and a second value, thereby calculating a difference image between the plurality of frames; a grid extraction unit that extracts grids in which the density of pixels having the first value is equal to or greater than a first threshold value from among a plurality of grids set in the difference image; a moving object detection unit that detects the extracted grid as a moving object, The plurality of grids are set to have smaller pixel sizes as the distance from the camera increases. Motion detection device.

2. the difference calculation unit enlarges the frame captured at a previous time point based on the speed of the moving object during an imaging interval in which the plurality of frames are captured, and calculates a difference image between the enlarged frame captured at a previous time point and the frame captured at a current time point. The moving object detection device according to claim 1 .

3. the difference calculation unit enlarges the frame captured at the previous time point with a vanishing point of the frame captured at the previous time point as a center; The moving object detection device according to claim 2 .

4. the difference calculation unit corrects the frame captured at a previous time point based on a yaw rate of the moving object during an imaging interval in which the plurality of frames are captured, and calculates a difference image between the corrected frame captured at the previous time point and the frame captured at a current time point. The moving object detection device according to claim 1 .

5. the grid extraction unit varies the first threshold in accordance with a distance from the camera to each of the plurality of grids; The moving object detection device according to claim 1 .

6. the grid extraction unit sets the sizes of the plurality of grids to a first size when the distance from the camera is equal to or less than a first distance, sets the sizes of the plurality of grids to a second size smaller than the first size when the distance from the camera is greater than the first distance and equal to or less than a second distance, and sets the sizes of the plurality of grids to a third size smaller than the second size when the distance from the camera is greater than the second distance. The moving object detection device according to claim 1 .

7. The moving object detection device according to any one of claims 1 to 6, a driving assistance device that provides driving assistance to the moving object based on the detection result obtained by the moving object detection device, system.

8. The computer acquiring image data including a plurality of frames that are captured in time series by a camera mounted on a moving object and represent the surrounding situation of the moving object; calculating a difference between the plurality of frames and binarizing the difference into a first value and a second value, thereby calculating a difference image between the plurality of frames; extracting grids in which the density of pixels having the first value is equal to or greater than a first threshold value from among a plurality of grids set in the difference image; Detecting the extracted grid as a moving object; The plurality of grids are set to have smaller pixel sizes as the distance from the camera increases. Motion detection method.

9. On the computer, acquiring image data including a plurality of frames that represent the surrounding conditions of the moving object, the frames being captured in time series by a camera mounted on the moving object; calculating a difference between the plurality of frames and binarizing the difference into a first value and a second value, thereby calculating a difference image between the plurality of frames; extracting grids in which the density of pixels having the first value is equal to or greater than a first threshold value from among a plurality of grids set in the difference image; Detecting the extracted grid as a moving object; The plurality of grids are set to have smaller pixel sizes as the distance from the camera increases. program.

Images