Mobile object detection device and mobile object detection program

Abstract

This invention provides a mobile object detection device and a mobile object detection program that efficiently detect moving objects using the background subtraction method. [Solution] The moving object detection device 200 comprises an object detection unit 250 and a background update unit 290. The object detection unit 250 detects an object estimation region represented in the input data 610 based on difference data 620 representing the difference between background data 600 and input data 610. The background update unit 290 updates the background data 600 based on fixed object data representing the object when background conditions are met, including the fact that the velocity of the object represented in the object estimation region is below a threshold. The difference data 620 is used to detect the moving object 20 represented in the input data 610.

Description

Technical Field

[0004] , ,

[0005] , ,

[0001] The present disclosure relates to a moving object detection device and a moving object detection program that detect a moving object using a background difference method.

Background Art

[0002] Techniques for detecting the area of a moving object from an acquired image using the background difference method have been studied. For example, Patent Document 1 discloses a technique for discriminating an area where the degree of texture change is small (such as the shadow of a cloud) in an area detected by the background difference method as a disturbance, and replacing the background image in the disturbance area. Here, the degree of texture change represents the correlation value between the background image and the input image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

[0004] Here, in the technique described in Patent Document 1, an area where the degree of texture change is small due to a cloud shadow or the like can be excluded, but it cannot handle the case where an object that should not be detected as a moving object itself changes. For example, when a tree in the background is cut down, the image representing the area where the tree was cut down is different before and after the cutting in the background difference method, and there is a high possibility that the degree of texture change is large. Therefore, the tree may be detected as a moving object. Similarly, when a signboard or the like is installed, the installed signboard may be detected as a moving object.

[0005] In view of the above situation, one of the objectives of the present disclosure is to provide a moving object detection device and a moving object detection program that efficiently detect a moving object using the background difference method by updating the background image for an area where the background object has changed. Other objectives can be understood from the following description and the explanation of the embodiments.

[0006] A moving object detection device (200) according to one embodiment for achieving the above objective comprises an object detection unit (250) and a background update unit (290). The object detection unit (250) detects object estimation regions (15, 25, 35) represented in the input data (510, 610) based on difference data (520, 620) representing the difference between background data (600) and input data (510, 610). The background update unit (290) updates the background data (600) based on solid object data representing the objects (11, 12, 20, 30) when background conditions are met, including the fact that the velocity of the objects (11, 12, 20, 30) represented in the object estimation regions (15, 25, 35) is below a threshold. The difference data (520, 620) is used to detect moving objects (20) represented in the input data (510, 610).

[0007] A moving object detection program (400) according to one embodiment for achieving the above objective causes the computing unit (220) to detect object estimation regions (15, 25, 35) represented in the input data (510, 610) based on difference data (520, 620) representing the difference between background data (600) and input data (510, 610). Furthermore, when background conditions are met, including the fact that the velocity of the objects (11, 12, 20, 30) represented in the object estimation regions (15, 25, 35) is below a threshold, the moving object detection program (400) causes the computing unit (220) to update the background data (600) based on solid object data representing the objects (11, 12, 20, 30). The difference data (520, 620) is used to detect the moving object (20) represented in the input data (510, 610).

[0008] According to the above configuration, moving objects can be efficiently detected using the background subtraction method. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram illustrating a moving object detection system in one embodiment. [Figure 2]This is a diagram illustrating a moving object detection system in one embodiment. [Figure 3] This is a diagram illustrating the configuration of a moving object detection device in one embodiment. [Figure 4] This diagram illustrates the functional block of a moving object detection system in one embodiment. [Figure 5] This is a flowchart illustrating the process by which a moving object detection system in one embodiment initializes background data. [Figure 6] This figure shows the initial input data that is acquired first in one embodiment. [Figure 7] This figure shows the second initial input data to be acquired in one embodiment. [Figure 8] This is a flowchart illustrating the process by which a moving object detection system in one embodiment identifies an object. [Figure 9] In one embodiment, this figure shows the initial difference data, which represents the difference between the initial input data in Figure 6 and the initial input data in Figure 7. [Figure 10] This figure shows the initial input data acquired third in one embodiment. [Figure 11] In one embodiment, this figure shows the initial difference data, which represents the difference between the initial input data in Figure 6 and the initial input data in Figure 10. [Figure 12] This flowchart illustrates the process by which a moving object detection system in one embodiment removes the area of ​​a moving object from background data. [Figure 13] This is a flowchart illustrating the process by which a motion detection system in one embodiment updates background data. [Figure 14] This figure shows the background data generated in one embodiment. [Figure 15] This flowchart illustrates the process by which a mobile object detection system in one embodiment detects a mobile object and updates background data. [Figure 16] This figure shows the input data that is initially acquired in one embodiment. [Figure 17] In one embodiment, it is a diagram showing difference data indicating the difference between the background data shown in FIG. 14 and the input data shown in FIG. 16. [Figure 18] In one embodiment, it is a diagram showing the input data acquired second. [Figure 19] In one embodiment, it is a diagram showing difference data indicating the difference between the background data shown in FIG. 14 and the input data shown in FIG. 18. [Figure 20] It is a diagram showing the process in which the mobile object detection system in one embodiment updates background data. [Figure 21] In one embodiment, it is a diagram showing the updated background data. [Figure 22] It is a diagram for explaining an object estimation area in other embodiments.

Embodiments for Carrying Out the Invention

[0010] (Embodiment 1) The mobile object detection system 1000 according to the present embodiment of the present invention will be described with reference to the drawings. In the present embodiment, as shown in FIG. 1, the mobile object detection system 1000 includes a sensor 100, a mobile object detection device 200, and an automatic control device 300. The mobile object detection device 200 is connected to the sensor 100 by wire or wirelessly so as to be communicable. Further, the mobile object detection device 200 is connected to the automatic control device 300 wirelessly so as to be communicable.

[0011] Sensor 100 images the surrounding situation, for example, the moving body 20 traveling on road 5. Sensor 100 outputs, as a plurality of input data, a plurality of image information representing a plurality of captured images obtained by imaging at different times to the moving body detection device 200. The moving body detection device 200 detects the moving body 20 (for example, the first moving body 20-1, the second moving body 20-2) represented by the image information by using the background difference method based on the plurality of image information. The moving body detection device 200 outputs the target information of the detected moving body 20, such as the type, position, moving direction, speed, etc. of the moving body 20, to the automatic control device 300. The automatic control device 300 controls the third moving body 20-3 that moves toward the intersection in front of the first moving body 20-1 based on the target information of the moving body 20 (for example, the first moving body 20-1). For example, the automatic control device 300 controls the third moving body 20-3 so as not to contact the first moving body 20-1.

[0012] Also, the moving body detection system 1000 may have a plurality of sensors 100. For example, the moving body detection system 1000 may have a sensor 100 that can image the first moving body 20-1 and a sensor 100 that can image the third moving body 20-3. The moving body detection system 1000 may output the information regarding the third moving body 20-3 as target information to the automatic control device 300 mounted on the second moving body 20-2. The automatic control device 300 of the second moving body 20-2 controls the second moving body 20-2 so as not to contact the third moving body 20-3 based on the target information regarding the third moving body 20-3.

[0013] In this way, the moving body detection system 1000 suppresses the contact between the moving bodies 20 by detecting the moving bodies 20 and outputting the target information of the moving bodies 20 (for example, the first moving body 20-1, the second moving body 20-2, etc.). Also, the target information may be output to and used by any system such as a system for monitoring the traffic volume on road 5.

[0014] The area captured by sensor 100 contains multiple stationary objects such as trees 10. The moving object detection system 1000 detects the moving object 20 and acquires target information of the moving object 20 by utilizing the difference between background data representing a background image of stationary objects and input data representing an image of the moving object 20.

[0015] However, objects that do not move may change. For example, the first tree 10-1 may be cut down and transformed into a stump 11, as shown in Figure 2. Also, construction work may result in the installation of signs 30 on the road 5, such as signs notifying traffic of alternating one-way traffic.

[0016] Therefore, the moving object detection system 1000 extracts stationary objects from the objects detected by the background subtraction method (e.g., the moving object 20, the tree stump 11, the sign 30, etc.) and updates the background data with information representing the extracted objects. By updating the background data, the moving object detection system 1000 can suppress the detection of stationary objects even when using the background subtraction method, and efficiently detect the moving object 20.

[0017] (Configuration of the mobile object detection system) The configuration of the moving object detection system 1000 will now be described. The sensor 100 included in the moving object detection system 1000 captures images of the surrounding environment and outputs image information representing the captured images to the moving object detection device 200. For example, the sensor 100 captures images of the surrounding environment at predetermined intervals, for example, every 100 milliseconds. The sensor 100 outputs image information representing the captured images to the moving object detection device 200. For example, the sensor 100 outputs image information to the moving object detection device 200 each time it captures images of the surrounding environment. Alternatively, the sensor 100 may output multiple pieces of image information together to the moving object detection device 200 at predetermined intervals (for example, every 1 second). The sensor 100 may also output the captured images as a video, for example, a video at 10 FPS (Frames Per Second).

[0018] As shown in Figure 3, the mobile object detection device 200 comprises an input / output device 210, an arithmetic unit 220, a communication device 230, and a storage device 240. The mobile object detection device 200 is, for example, a computer. Information for the arithmetic unit 220 to perform processing is input to the input / output device 210. The input / output device 210 also outputs the results of the processing performed by the arithmetic unit 220. The input / output device 210 may include various input and output devices, such as a keyboard, mouse, microphone, display, speaker, touch panel, etc. The input / output device 210 may be detachably connected or may be omitted.

[0019] The communication device 230 communicates with each device, such as the sensor 100 and the automatic control device 300. For example, the communication device 230 transfers image information acquired from the sensor 100 to the processing unit 220. The communication device 230 also transfers signals generated by the processing unit 220 to the automatic control device 300. The communication device 230 may output information to other devices connected via a network, such as the Internet, and may acquire information from other devices. The communication device 230 may also acquire information from other devices and output information to other devices via any storage medium, such as a memory card or USB (Universal Serial Bus) memory. The communication device 230 may include various interfaces, such as transceivers used for short-range wireless communication such as wireless LAN (Local Area Network), transceivers used for long-range wireless communication such as cellular networks, NICs (Network Interface Cards), and USB terminals.

[0020] The storage device 240 stores various data for detecting the moving object 20 based on image information acquired from the sensor 100, such as a moving object detection program 400. The storage device 240 is used as a non-transitory tangible storage medium for storing the moving object detection program 400. The moving object detection program 400 may be provided as a computer program product recorded on a computer-readable storage medium 1, or as a computer program product downloadable from a server.

[0021] The arithmetic unit 220 reads and executes the moving object detection program 400 from the storage device 240 and performs various data processing to detect the moving object 20. For example, the arithmetic unit 220 may include a central processing unit (CPU).

[0022] The arithmetic unit 220 reads and executes the moving object detection program 400, and in cooperation with the storage device 240, as shown in Figure 4, realizes the object detection unit 250, the speed determination unit 260, the classification unit 270, the output unit 280, and the background update unit 290. The object detection unit 250 detects objects, including the moving object 20, using the background subtraction method based on image information acquired from the sensor 100. Objects include, for example, tree stumps 11 and signs 30. The speed determination unit 260 determines the speed of the detected objects. The classification unit 270 classifies the detected objects into categories that include target categories, which are the types of objects to be detected as moving objects 20 (for example, large vehicles, passenger cars, bicycles, pedestrians, etc.), and non-target categories, which are other types. The output unit 280 outputs the target information of the detected moving object 20 to the automatic control device 300. The background update unit 290 updates the background data based on image information acquired from the sensor 100.

[0023] The automatic control device 300 shown in Figure 1 controls the mobile body 20 on which the automatic control device 300 is mounted, for example, the third mobile body 20-3, based on target information from the mobile body detection device 200. For example, the automatic control device 300 controls the third mobile body 20-3 so as not to come into contact with the mobile body 20 represented in the target information, for example, the first mobile body 20-1. For example, the automatic control device 300 controls the third mobile body 20-3 so as not to come into contact with the first mobile body 20-1.

[0024] (Operation of the mobile object detection system) The operation of the mobile object detection system 1000 will be explained. First, the initial operation when the mobile object detection device 200 is started will be explained. The initial operation is generally performed in a situation where it is unlikely that the mobile object 20 will be imaged by the sensor 100. For example, the initial operation is performed in a situation where the mobile object 20 is restricted from entering the road 5. When the mobile object detection device 200 is started, it generates background data to be used in the background difference method. For example, when the mobile object detection device 200 is started, the arithmetic unit 220 shown in Figure 3 reads and executes the mobile object detection program 400 from the storage device 240. Once the mobile object detection program 400 is executed, the arithmetic unit 220 starts the process shown in Figure 5, which is part of the mobile object detection method.

[0025] In step S110, the object detection unit 250, implemented by the arithmetic unit 220, acquires image information from the sensor 100 for a predetermined period. The object detection unit 250 acquires image information from the sensor 100 each time, and when it has acquired a quantity of image information corresponding to a predetermined period (for example, 1 minute), for example, two or more predetermined numbers of image information, it may execute the process of the next step S120.

[0026] In step S120, the object detection unit 250 initializes background data based on multiple image information. For example, the object detection unit 250 determines the image information representing the first image captured as background data. Here, to distinguish the image information acquired in the initial operation from the image information acquired after the initial operation, it is sometimes called initial input data 510. For example, when the first image captured by the sensor 100 is the first initial input data 510-1 shown in Figure 6, the object detection unit 250 determines the first initial input data 510-1 as background data. In the example shown in Figure 6, the first initial input data 510-1 represents an image of multiple trees 10 planted along the side of the road 5 and a moving object 20 traveling on the road 5.

[0027] Next, in step S130 shown in Figure 5, the object detection unit 250 selects the next image information from a plurality of image information. For example, the sensor 100 acquires the image represented by the second initial input data 510-2 shown in Figure 7, following the first initial input data 510-1. In this case, the object detection unit 250 selects the image information of the second initial input data 510-2.

[0028] Next, in step S140 shown in Figure 5, the moving object detection device 200 identifies an object represented in the selected image information, for example, a moving object 20. For example, the moving object detection device 200 uses background subtraction to detect a moving object from the selected image information and acquires target information of the detected object, such as the object's category, position, and velocity. For example, in order to acquire target information of a moving object from the image information, the moving object detection device 200 performs the process shown in Figure 8.

[0029] In step S210, the object detection unit 250 detects an object estimation region, which represents the area where a moving object is estimated to exist, based on the initial input data 510 of the selected image information, using the background subtraction method. For example, the object detection unit 250 detects an object estimation region based on the second initial input data 510-2.

[0030] For example, the position of the moving object 20 shown in the first initial input data 510-1 in Figure 6 differs from the position of the moving object 20 shown in the second initial input data 510-2 in Figure 7 due to movement. Therefore, the position where the moving object 20 existed in the first initial input data 510-1 is not the position where the moving object 20 exists in the second initial input data 510-2. Also, the position where the moving object 20 existed in the second initial input data 510-2 is not the position where the moving object 20 existed in the first initial input data 510-1.

[0031] As a result, the object detection unit 250 generates first initial difference data 520-1, which represents the difference between the first initial input data 510-1 and the second initial input data 510-2, as shown in Figure 9. For example, in the first initial difference data 520-1, the fourth moving object 20-4 is represented at the position where the moving object 20 was located in the first initial input data 510-1. Also, in the first initial difference data 520-1, the fifth moving object 20-5 is represented at the position where the moving object 20 was located in the second initial input data 510-2.

[0032] Therefore, the object detection unit 250 detects a fourth mobile object region 25-4 corresponding to the fourth mobile object 20-4 at the location where the mobile object 20 was located in the first initial input data 510-1, using the background subtraction method. Similarly, in the first initial difference data 520-1, a fifth mobile object region 25-5 corresponding to the fifth mobile object 20-5 is detected at the location where the mobile object 20 was located in the second initial input data 510-2. Note that any known method for the background subtraction method can be used, so a detailed explanation is omitted here.

[0033] Here, the mobile object region 25 represents the region in which the mobile object 20 was detected in the first initial difference data 520-1. The mobile object region 25 is represented by any shape that encloses the region in which the mobile object 20 was detected. For example, the mobile object region 25 may be represented by a rectangle. For example, when the first initial difference data 520-1 represents a rectangular image enclosed by edges extending in a first direction and a second direction, the mobile object region 25 is represented by a rectangle having edges extending in a first direction and a second direction. For example, the mobile object region 25 is represented by the smallest rectangle that encloses the region of the detected mobile object 20 and has edges extending in a first direction and a second direction. The mobile object region 25 is determined for each detected mobile object 20.

[0034] Furthermore, the object detection unit 250 determines an arbitrary position within the object estimation region as an object estimation position representing the position of the object represented in the object estimation region. For example, the object detection unit 250 determines an arbitrary position within the moving body region 25 as a moving body position 26 representing the position of the moving body 20 represented in the detected moving body region 25. For example, as shown in Figure 9, the moving body position 26 represents the center of the moving body region 25 in the first direction and the endpoint in the opposite direction of the second direction (for example, downwards in the image). The moving body position 26 is also determined for each detected moving body region 25. For example, the object detection unit 250 determines the fourth moving body position 26-4 for the fourth moving body region 25-4 and the fifth moving body position 26-5 for the fifth moving body region 25-5.

[0035] Next, in step S220 shown in Figure 8, the object detection unit 250 tracks the object represented in the object estimation region detected in the initial input data 510. For example, the object detection unit 250 identifies a region in the second initial input data 510-2 that represents the same range as the object estimation region detected in the first initial difference data 520-1. For example, the object detection unit 250 identifies the fourth moving object region 25-4 shown in Figure 9 and the fourth moving object region 25-4 that represents the same range in the second initial input data 510-2 shown in Figure 7. Similarly, the object detection unit 250 identifies the fifth moving object region 25-5 shown in Figure 9 and the fifth moving object region 25-5 that represents the same range in the second initial input data 510-2 shown in Figure 7.

[0036] Similarly, the estimated object positions are determined based on their positions in the initial input data 510. For example, the positions of the fourth moving object 26-4 and the fifth moving object 26-5 in the first initial difference data 520-1 are identified as the positions of the fourth moving object 26-4 and the fifth moving object 26-5 in the second initial input data 510-2.

[0037] The object detection unit 250 identifies the object estimation region where the object represented in the selected initial input data 510 exists, based on the object estimation region detected in the immediately preceding initial input data 510. For example, the object detection unit 250 identifies the object estimation region where the object (e.g., moving object 20) represented in the object estimation region of the second initial input data 510-2 (e.g., the fifth moving object region 25-5) exists, based on the object estimation region detected in the first initial input data 510-1. Here, since the first initial input data 510-1 is treated as background data, no object estimation region is detected in the first initial input data 510-1. Therefore, the object detection unit 250 determines that the object represented in the object estimation region of the second initial input data 510-2 is the first object detected.

[0038] Next, in step S230 shown in Figure 8, the velocity determination unit 260 determines the velocity of an object based on the position of the object represented in the object estimation region when the selected initial input data 510 was imaged. For example, the velocity determination unit 260 determines the velocity of an object based on the distance between the position of the object estimation region of the selected initial input data 510 and the position of the corresponding object estimation region in the immediately preceding initial input data 510, and the time interval when the initial input data 510 was imaged. Here, since no object estimation region was detected in the first initial input data 510-1, the velocity of the object represented in the object estimation region of the second initial input data 510-2 is not calculated.

[0039] Next, in step S240, the classification unit 270 classifies the objects (e.g., moving objects 20) represented in the object estimation area (e.g., moving object area 25) into categories. For example, the categories include large vehicles, regular cars, bicycles, pedestrians, and others. For example, the classification unit 270 determines whether the object represented in the object estimation area belongs to any of the target categories (e.g., large vehicles, regular cars, bicycles, and pedestrians). If the object represented in the object estimation area does not belong to any of the target categories, the classification unit 270 classifies the object in the object estimation area into the non-target category, "other".

[0040] The classification unit 270 may determine, by any means, whether or not an object represented in the object estimation region belongs to a target category. For example, the classification unit 270 may classify using a learned model that has been trained to output the category to which an object represented in the object estimation region belongs, based on an image in the object estimation region, for example, the mobile object region 25, in the initial input data 510. For example, the classification unit 270 determines the category to which an object represented in the object estimation region belongs based on an image in the object estimation region. For example, based on an image in the fourth mobile object region 25-4 shown in Figure 7, the classification unit 270 determines that the object represented in the fourth mobile object region 25-4 does not belong to a target category and therefore belongs to "Other". Also, based on an image in the fifth mobile object region 25-5, the classification unit 270 determines that it belongs to "Ordinary Car".

[0041] When the process of step S240 shown in Figure 8 is completed, the object detection unit 250 executes the process of step S150 shown in Figure 5.

[0042] In step S150, the object detection unit 250 determines whether all image information has been processed. If there is still image information that does not identify an object, the object detection unit 250 returns to step S130 and repeats the process. For example, suppose the sensor 100 captures the second initial input data 510-2 shown in Figure 7, followed by the third initial input data 510-3 shown in Figure 10. In this case, the object detection unit 250 returns to step S130 shown in Figure 5 and selects the image information of the third initial input data 510-3.

[0043] Next, in step S140, processing is performed on the third initial input data 510-3. For example, in step S210 shown in Figure 8, the object detection unit 250 detects an object estimation region in the third initial input data 510-3 shown in Figure 10.

[0044] For example, as shown in Figure 11, the object detection unit 250 generates second initial difference data 520-2, which represents the difference between the first initial input data 510-1, which is background data, and the third initial input data 510-3. In the second initial difference data 520-2, the sixth moving object 20-6 is represented at the position of the moving object 20 in the first initial input data 510-1, and the seventh moving object 20-7 is represented at the position of the moving object 20 in the third initial input data 510-3. Also, the first tree 10-1 in the first initial input data 510-1 shown in Figure 6 has been cut down in the third initial input data 510-3 shown in Figure 10 and has been transformed into a stump 11. Therefore, in the second initial difference data 520-2 shown in Figure 11, a difference tree 12 representing the difference between the first tree 10-1 and the stump 11 is represented at the position of the first tree 10-1 in the first initial input data 510-1.

[0045] Therefore, the object detection unit 250 detects the sixth mobile object region 25-6 corresponding to the sixth mobile object 20-6, the seventh mobile object region 25-7 corresponding to the seventh mobile object 20-7, and the difference tree region 15 corresponding to the difference tree 12 using the background subtraction method. The object detection unit 250 also determines the sixth mobile object position 26-6, the seventh mobile object position 26-7, and the difference tree position 16 corresponding to each object estimation region.

[0046] Next, in step S220 shown in Figure 8, the object detection unit 250 tracks the objects represented in the object estimation regions detected in the initial input data 510. For example, the object detection unit 250 identifies the regions in the third initial input data 510-3 that represent the same range as each object estimation region detected in the second initial difference data 520-2. For example, the object detection unit 250 identifies the sixth mobile region 25-6, the seventh mobile region 25-7, and the difference tree region 15 as regions in the third initial input data 510-3 of Figure 10 that represent the same range as the sixth mobile region 25-6, the seventh mobile region 25-7, and the difference tree region 15 shown in Figure 11. Furthermore, the sixth mobile object position 26-6, the seventh mobile object position 26-7, and the difference tree position 16 shown in Figure 11 are identified as the positions in the third initial input data 510-3 in Figure 10, respectively.

[0047] The object detection unit 250 uses any known tracking method to identify the object estimation region where the object represented in the selected initial input data 510 exists, based on the object estimation region detected in the immediately preceding initial input data 510. For example, the object detection unit 250 identifies the object estimation region where the moving object 20 represented in the seventh moving object region 25-7 of the third initial input data 510-3 exists, based on the object estimation region in the second initial input data 510-2. For example, the object detection unit 250 determines whether the moving object 20 represented in the seventh moving object region 25-7 shown in Figure 10 and the moving object 20 represented in the fifth moving object region 25-5 shown in Figure 7 represent the same object. For example, the object detection unit 250 determines that the moving object 20 represented in the seventh moving object region 25-7 and the moving object 20 represented in the fifth moving object region 25-5 represent the same object when the image in the seventh moving object region 25-7 in Figure 10 is similar to the image in the fifth moving object region 25-5 in Figure 7. Alternatively, the object detection unit 250 may determine whether the objects represented in both moving object regions 25 are the same using the distance between the seventh moving object position 26-7, which represents the position of the seventh moving object region 25-7, and the fifth moving object position 26-5, which represents the position of the fifth moving object region 25-5. Alternatively, the object detection unit 250 may determine whether the objects represented in both moving object regions 25 are the same using the velocity, direction of movement, or both of the moving object 20 represented in the fifth moving object region 25-5.

[0048] In this way, the object detection unit 250 groups together object estimation regions from each initial input data 510 that represent the region of the same object, and identifies the position of a single object at each time point. For example, the object detection unit 250 determines that an object (e.g., moving object 20) represented in a grouped object estimation region was located at the object position (e.g., moving object position 26) of the corresponding initial input data 510 at the time the corresponding initial input data 510 was imaged. For example, the object detection unit 250 determines that the moving object 20 represented in the fifth moving object region 25-5 in Figure 7 was located at the fifth moving object position 26-5 at the imaging time of the second initial input data 510-2, and was located at the seventh moving object position 26-7 at the imaging time of the third initial input data 510-3 in Figure 10. The object detection unit 250 performs this process for all object estimation regions.

[0049] For example, the object detection unit 250 also performs this process for the sixth mobile area 25-6 shown in Figure 10. Although the mobile area 20 is not shown in the sixth mobile area 25-6 in Figure 10, the object detection unit 250 identifies the object estimation area of ​​the second initial input data 510-2 in which an object represented in the sixth mobile area 25-6 exists, for example, the road 5 in that area. For example, the object detection unit 250 determines that the object represented in the sixth mobile area 25-6 in Figure 10 (for example, the road 5 in that area) is the same as the object represented in the fourth mobile area 25-4 in Figure 7 (for example, the road 5 in that area).

[0050] The difference tree region 15 in Figure 10 is processed similarly, and the object detection unit 250 determines that there is no object estimation region corresponding to the difference tree region 15 in the object estimation region of the immediately preceding second initial input data 510-2. Once tracking has been performed for all object estimation regions in this manner, the process moves to step S230 shown in Figure 8.

[0051] In step S230, the velocity determination unit 260 determines the velocity of an object based on the position of the object represented in the object estimation region when the selected initial input data 510 was captured. For example, the velocity determination unit 260 determines the velocity of an object based on the distance between the position of the object estimation region of the selected initial input data 510 and the position of the corresponding object estimation region in the previous initial input data 510, and the time interval between the captures of the initial input data 510.

[0052] For example, the velocity determination unit 260 determines the distance from the position of the object represented in the object estimation region of the immediately preceding initial input data 510 to the position of the object represented in the object estimation region of the selected initial input data 510. For example, the velocity determination unit 260 determines the distance from the position of the moving body 20 in the fifth moving body region 25-5 of the second initial input data 510-2 in Figure 7 to the position of the moving body 20 in the seventh moving body region 25-7 of the third initial input data 510-3 in Figure 10. The velocity determination unit 260 determines the distance from the position represented in the fifth moving body position 26-5 in Figure 7 to the position represented in the seventh moving body position 26-7 in Figure 10.

[0053] The actual positions represented at each position in the initial input data 510 are determined by any known method. For example, the speed determination unit 260 determines the relative direction of the object's position to the sensor 100's position based on the direction the sensor 100 is facing (e.g., the optical axis direction) and the position of the fifth moving object position 26-5 represented in the second initial input data 510-2. Based on this relative direction and the position of the sensor 100, the speed determination unit 260 determines the distance from the sensor 100's position to the object's position according to the slope of the road 5. Thus, the speed determination unit 260 determines the relative direction and distance of the object's position to the sensor 100's position. Based on this relative direction and distance and the position of the sensor 100, e.g., latitude and longitude, the object's position, e.g., latitude and longitude, is determined. The position of the sensor 100 and the slope of the road 5 are pre-inputted by the user. The speed determination unit 260 may also process the data assuming that the road 5 is horizontal. Similarly, the position of the seventh moving object position 26-7, represented in the third initial input data 510-3, is also determined.

[0054] The velocity determination unit 260 determines the interval between the time when the second initial input data 510-2 was captured and the time when the third initial input data 510-3 was captured. The velocity determination unit 260 determines the velocity of the object by dividing the distance from the position represented by the fifth moving object position 26-5 to the position represented by the seventh moving object position 26-7 by the determined time interval.

[0055] Furthermore, the velocity determination unit 260 also determines the direction of movement of the object represented in the object estimation region. The velocity determination unit 260 determines the direction from the position represented by the object estimation position of the immediately preceding initial input data 510 to the position represented by the object estimation position of the selected initial input data 510 as the direction of movement of the object.

[0056] The velocity determination unit 260 similarly determines the velocity of all objects represented in the detected object estimation regions. The velocity determination unit 260 also determines the velocity of the object represented in the sixth moving object region 25-6 in Figure 10, for example. The velocity of the object represented in the difference tree region 15 is not determined because there is no object estimation region corresponding to the preceding initial input data 510.

[0057] Next, in step S240 shown in Figure 8, the classification unit 270 classifies the objects represented in the object estimation region into categories. The classification unit 270 classifies the categories using the categories of objects represented in the corresponding object specification region of the immediately preceding initial input data 510 through tracking processing. For example, the classification unit 270 may determine the category of the object represented in the object specification region of the immediately preceding initial input data 510 as the category of the object represented in the object specification region of the immediately preceding initial input data 510 that belongs to the same group as the object specification region of the immediately preceding initial input data 510. For example, the classification unit 270 may determine that the category of the object represented in the 7th mobile region 25-7 in Figure 10 is the category of the object represented in the corresponding 5th mobile region 25-5 in Figure 7 (e.g., "ordinary car"). Alternatively, the classification unit 270 may determine that the category of the object represented in the 6th mobile region 25-6 in Figure 10 is the category of the object represented in the corresponding 4th mobile region 25-4 in Figure 7 (e.g., "other").

[0058] The classification unit 270 may also classify the objects represented in all object estimation regions of the selected initial input data 510 into categories. In this case, the classification unit 270 may determine the category with the highest quantity among the categories classified for each object specification region belonging to the same group by the tracking process as the category of the objects represented in the grouped object specification regions. For example, the tracking process classifies the fifth mobile area 25-5 shown in Figure 7 and the seventh mobile area 25-7 shown in Figure 10 into the same group. Here, the classification unit 270 classifies the objects in the fifth mobile area 25-5 as "ordinary cars" based on the image in the fifth mobile area 25-5 shown in Figure 7, and classifies the objects in the seventh mobile area 25-7 as "ordinary cars" based on the image in the seventh mobile area 25-7 shown in Figure 10. The classification unit 270 calculates the quantity classified as "ordinary cars" in the group including the seventh mobile area 25-7 as "2". Similarly, the quantity classified as anything other than "ordinary cars" is calculated as "0". Therefore, the classification unit 270 classifies the objects represented in the object estimation region as "ordinary cars" in the group that includes the seventh mobile region 25-7.

[0059] Furthermore, the classification unit 270 may determine the likelihood of a category being classified. In this case, the classification unit 270 may determine the category with the highest likelihood among the categories classified for each object designation area belonging to the same group as the category to which the objects in the grouped object estimation area belong. Alternatively, the classification unit 270 may determine the category determined in any one of the grouped object estimation areas as the category to which the objects in the grouped object estimation area belong.

[0060] When the process of step S240 shown in Figure 8 is completed, the object detection unit 250 executes the process of step S150 shown in Figure 5.

[0061] In step S150, the object detection unit 250 determines whether or not all image information has been processed. Similarly, the processing in steps S130 and S140 is performed for the initial input data 510 captured after the third initial input data 510-3. Once all initial input data 510 has been processed, the processing in step S160 is performed.

[0062] In step S160, the background update unit 290 removes the area of ​​the moving object 20 from the background data (e.g., the first initial input data 510-1). As shown in Figure 6, the initial input data 510 (e.g., the first initial input data 510-1) determined as background data may contain an area representing the moving object 20. For this reason, the background update unit 290 removes the area representing the moving object 20 from the initial input data 510 determined as background data. For example, in order to remove the area of ​​the moving object 20 from the background data, the background update unit 290 executes the process shown in Figure 12.

[0063] In step S310, the background update unit 290 selects one of the groups based on the object estimation region. For example, the background update unit 290 selects a group from among one or more groups grouped by the tracking process that includes the fourth moving object region 25-4 shown in Figure 7.

[0064] Next, in step S320, the background update unit 290 determines whether the selected group satisfies the exclusion conditions. The exclusion conditions include, for example, that the object represented in the object estimation region of the selected group is stationary. In this case, the background update unit 290 determines whether the object represented in the object estimation region of the selected group is stationary. The background update unit 290 determines that the object represented in the object estimation region is stationary when the velocity of the object represented in the object estimation region is below a threshold. For example, the background update unit 290 determines that the object is stationary when the velocity of the object determined by the velocity determination unit 260 is below a threshold. For example, the background update unit 290 determines that the object represented in the sixth moving object region 25-6 shown in Figure 10 (for example, the road 5 in that region) is stationary when the velocity of the object represented in the sixth moving object region 25-6 is below a threshold. In this way, the background update unit 290 determines whether the object represented in the object estimation region of the selected group is stationary. When the background update unit 290 determines that an object is stationary, it determines that the selected group satisfies this exclusion condition. Here, since the velocity of the object represented in the sixth moving object region 25-6 shown in Figure 10 is below the threshold, the background update unit 290 determines that the group including the sixth moving object region 25-6 satisfies this exclusion condition.

[0065] The exclusion condition includes the ability to track an object represented in the region corresponding to the object estimation region of the selected group in the initial input data 510 captured immediately before the first time the initial input data 510 of the selected group was captured. In this case, the background update unit 290 determines whether or not an object represented in the region corresponding to the object estimation region of the selected group can be tracked in the initial input data 510 captured immediately before the first time the initial input data 510 of the selected group was captured. For example, the background update unit 290 identifies the first initial input data 510 captured from the initial input data 510 of the selected group. The background update unit 290 determines the initial input data 510 captured immediately before the identified initial input data 510. In the determined initial input data 510, the background update unit 290 determines the region corresponding to the object estimation region of the selected group. For example, the background update unit 290 determines a region in the determined initial input data 510 that represents the same range as the object estimation region of the selected group. The background update unit 290 determines whether or not an object represented in a determined region in the determined initial input data 510 can be tracked after the time the initial input data 510 was captured.

[0066] For example, the background update unit 290 identifies the second initial input data 510-2 as the first initial input data 510 captured from the group of initial input data 510 including the sixth mobile object region 25-6. Furthermore, the first initial input data 510-1 is determined as the initial input data 510 captured immediately before the second initial input data 510-2. As shown in Figure 6, the background update unit 290 determines the initial mobile object region 27 in the first initial input data 510-1 as a region representing the same range as the sixth mobile object region 25-6. The background update unit 290 determines whether an object represented in the initial mobile object region 27, for example, a mobile object 20, can be tracked after this time in the first initial input data 510-1. The background update unit 290 may determine whether an object in the initial mobile object region 27 can be tracked using any known tracking method, similar to the process shown in step S140 of Figure 5.

[0067] For example, the background update unit 290 determines whether the object represented in the initial mobile area 27 is the same as the mobile area 20 represented in the fifth mobile area 25-5 shown in Figure 7. When it is determined that the object represented in the initial mobile area 27 is the same as the mobile area 20 represented in the fifth mobile area 25-5, the background update unit 290 determines that it is trackable.

[0068] Furthermore, when the background update unit 290 selects a group that includes the difference tree region 15 shown in Figure 10, the difference tree region 15 does not move in the initial input data 510 captured after the time when the third initial input data 510-3 was captured, in order to represent the felled tree 10. Therefore, the group that includes the difference tree region 15 also satisfies the exclusion condition that the object represented in the object estimation region of the selected group is stationary.

[0069] On the other hand, in the group including the difference tree region 15, the first initial input data 510 captured is the third initial input data 510-3. Therefore, as shown in Figure 7, the background update unit 290 determines a difference tree region 15 in the second initial input data 510-2 that shows the same range as the difference tree region 15 of the third initial input data 510-3. The first tree 10-1 represented in the difference tree region 15 is not included in the region detected as the object estimation region, as shown in Figure 11. Therefore, the background update unit 290 determines that it cannot track the object represented in the difference tree region 15.

[0070] When all exclusion conditions are met, the background update unit 290 executes the process of step S330 shown in Figure 12. When any of the exclusion conditions are not met, the background update unit 290 executes the process of step S340. For this reason, the process of step S330 is executed in the group that includes the sixth mobile area 25-6, but the process of step S330 is not executed in the group that includes the difference tree area 15.

[0071] In step S330, the background update unit 290 sets an exclusion flag for the selected group. The exclusion flag indicates that the background group includes an area representing the moving object 20, and that the object estimation area of ​​the group has been detected due to the movement of the moving object 20.

[0072] In step S340, the background update unit 290 determines whether the processes from steps S320 to S330 have been executed for all groups. If there are any groups for which this process has not been executed, the background update unit 290 returns to the process in step S310, selects one of the groups for which the process has not been executed, and repeats the process. If there are no groups for which this process has not been executed, the background update unit 290 executes the process in step S350.

[0073] In step S350, the background update unit 290 generates representative data representing an image for updating the background data, based on the images represented in the object estimation regions of the corresponding multiple initial input data 510 in the group where the exclusion flag is set. The background update unit 290 generates representative data in any way based on the images represented in the object estimation regions of the initial input data 510. For example, the background update unit 290 may select any one of the multiple initial input data 510, for example, the one with the latest acquisition time, and determine the image represented in the object estimation region of the selected initial input data 510 as representative data. For example, the background update unit 290 may determine the image represented in the object estimation region of the last acquired initial input data 510 in the group where the exclusion flag is set as representative data.

[0074] Alternatively, for example, the background update unit 290 may calculate the average value of the brightness values ​​in multiple initial input data 510 for each pixel included in the object estimation region, and generate an image of the object estimation region having the calculated average value of the brightness value of that pixel as representative data. For example, the group including the fourth moving object region 25-4 shown in Figure 7 includes the corresponding regions of each initial input data 510 captured after the time when the second initial input data 510-2 was captured. In this case, for example, the background update unit 290 calculates the average value of the brightness values ​​in each initial input data 510 captured after the time when the second initial input data 510-2 was captured for each pixel included in the fourth moving object region 25-4. For example, the background update unit 290 selects one pixel included in the fourth moving object region 25-4, extracts the brightness value of that pixel in each initial input data 510, and calculates the average value of the extracted brightness values. The background update unit 290 determines the calculated average value as the brightness value of the pixel corresponding to the position of that pixel in the representative data. The background update unit 290 calculates an average value for all pixels included in the fourth moving object region 25-4, and determines the calculated average value as the luminance value of the corresponding pixel in the representative data. If the image is a color image, the luminance value of each color (e.g., red, blue, yellow) of each pixel in the representative data represents the average value of the luminance values ​​for each color from the multiple initial input data 510.

[0075] Next, in step S360 shown in Figure 12, the background update unit 290 replaces a portion of the background data with representative data. For example, the background update unit 290 replaces the mobile initial region 27, which is the corresponding region of the background data (the first initial input data 510-1 shown in Figure 6), with representative data in the group including the fourth mobile region 25-4 shown in Figure 7. For example, the background update unit 290 replaces the position in the background data that corresponds to the actual position where the representative data is represented with the representative data. As a result, the mobile body 20 represented in the background data is excluded.

[0076] In this way, the area of ​​the moving object 20 is excluded from the background data. Once this process is complete, the process returns to Figure 5, and the process of step S170 is performed.

[0077] In step S170, the background update unit 290 updates the background data for the region where the background object has changed. For example, to update the background data, the background update unit 290 performs the process shown in Figure 13.

[0078] In step S410, the background update unit 290 selects one of the groups based on the object estimation region, excluding the group for which the exclusion flag has been set. For example, from one or more groups grouped by the tracking process, it selects a group other than the group for which the exclusion flag has been set (for example, the group including the fourth moving object region 25-4 shown in Figure 7), for example, the group including the difference tree region 15 shown in Figure 10.

[0079] In step S420 shown in Figure 13, the background update unit 290 determines whether the selected group satisfies the background conditions. The background conditions include that the objects represented in the object estimation region (e.g., the moving object region 25, the difference tree region 15, etc.) are stationary. For example, the background update unit 290 determines that the objects represented in the object estimation region are stationary when the velocity of the objects represented in the object estimation region is below a threshold. The background conditions may also include that the period during which the objects are stationary is continuous for a certain period of time or longer. For example, the background update unit 290 determines that the objects represented in the object estimation region are stationary when the velocity of the objects represented in the object estimation region is below a threshold for all initial input data 510 included in a continuous period of time. The background update unit 290 may also determine that the objects represented in the object estimation region are stationary when the statistical values ​​of the velocity of the objects represented in the object estimation region, such as the maximum value, mean value, median value, etc., for all initial input data 510 included in a continuous period of time are below a threshold. The initial input data 510 included in a continuous period of time represents all initial input data 510 captured from the time the initial input data 510 was captured until a certain period of time has elapsed.

[0080] For example, the difference tree region 15 shown in Figure 10 does not move after the time the third initial input data 510-3 was captured. Therefore, the background update unit 290 determines that the group including the difference tree region 15 shown in Figure 10 satisfies this background condition. For example, when the group including the fifth mobile body region 25-5 shown in Figure 7 is selected, the mobile body 20 represented in the fifth mobile body region 25-5 in the second initial input data 510-2 is not stopped. Therefore, the background update unit 290 determines that the group including the fifth mobile body region 25-5 shown in Figure 7 does not satisfy this background condition.

[0081] Furthermore, the background conditions may include the category to which the object represented in the object estimation region belongs, which is a non-target category, such as "other". For example, the background update unit 290 determines that the group of objects in the group including the difference tree region 15 shown in Figure 10 belongs to "other", and therefore determines that this group also satisfies this background condition. Also, since the group of objects in the group including the fifth mobile object region 25-5 shown in Figure 7 has been determined to belong to "ordinary car", the unit determines that this group does not satisfy this background condition.

[0082] The background update unit 290 executes the process in step S440 shown in Figure 13 if the selected group does not satisfy any of the background conditions. If the selected group satisfies all of the background conditions, the background update unit 290 executes the process in step S430.

[0083] In step S430, the background update unit 290 sets a background flag for the selected group. The background flag indicates that the object estimation region for that group has been detected because the background objects have changed.

[0084] In step S440, the background update unit 290 determines whether the processes from steps S420 to S430 have been executed for all groups except those for which an exclusion flag has been set. If there are any groups for which this process has not been executed, the background update unit 290 returns to the process in step S410, selects one of the groups for which the process has not been executed, and repeats the process. If there are no groups for which this process has not been executed, the background update unit 290 executes the process in step S450.

[0085] In step S450, the background update unit 290 generates representative data representing an image for updating the background data, based on the images represented in the object estimation regions of the corresponding multiple initial input data 510 in the group where the background flag is set. The process in step S450 is equivalent to the process in step S350 shown in Figure 12, but with the "exclusion flag" replaced by the "background flag". For this reason, a detailed explanation of the process in step S450 is omitted.

[0086] In step S460 shown in Figure 13, the background update unit 290 replaces a portion of the background data with representative data. For example, the background update unit 290 replaces the difference tree region 15, which is the corresponding region of the background data (first initial input data 510-1 shown in Figure 6), with representative data from the group including the difference tree region 15 shown in Figure 10. For example, the background update unit 290 replaces the image position in the background data that corresponds to the actual position represented in the difference tree region 15 with representative data.

[0087] As a result, the background update unit 290 generates background data 600 in which the moving object 20 is excluded and the first tree 10-1 is replaced with the stump 11, as shown in Figure 14. In this way, the moving object detection system 1000 performs the process shown in Figure 5 and generates background data 600.

[0088] Next, the operation of the mobile object detection system 1000 in which it detects a mobile object 20 and updates background data 600 will be described. For example, the user inputs an operation to the input / output device 210 of the mobile object detection device 200 to start detecting the mobile object 20. When the input / output device 210 receives an operation to start detecting the mobile object 20, the arithmetic unit 220 of the mobile object detection device 200 reads and executes the mobile object detection program 400 from the storage device 240. Alternatively, if a period for performing initial operations is set in advance, the arithmetic unit 220 may read and execute the mobile object detection program 400 after that period has elapsed. Once the mobile object detection program 400 is executed, the arithmetic unit 220 periodically executes the process shown in Figure 15, which is part of the mobile object detection method. For example, each time the sensor 100 captures an image of the surrounding situation, the arithmetic unit 220 may start the process shown in Figure 15. Alternatively, when a predetermined amount of image information is output from the sensor 100, the arithmetic unit 220 may start the process shown in Figure 15.

[0089] In step S510, the object detection unit 250, implemented by the arithmetic unit 220, acquires image information from the sensor 100 as input data. For example, the object detection unit 250 acquires an image from the sensor 100 represented by the first input data 610-1 shown in Figure 16.

[0090] Next, in step S520 shown in Figure 15, the moving object detection device 200 identifies the object represented in the input data 610. For example, the moving object detection device 200 uses the background subtraction method to detect moving objects from the input data 610 and acquires target information of the detected object. For example, in order to acquire target information of a moving object from the input data 610, the moving object detection device 200 executes the process shown in Figure 8. Since the process shown in Figure 8 is the same as the process in the initial operation, the embodiments shown in Figures 16 and 17 will be described.

[0091] In step S210, the object detection unit 250 generates the first difference data 620-1 shown in Figure 17 as the difference between the background data 600 in Figure 14 and the first input data 610-1 in Figure 16. In the first input data 610-1 shown in Figure 16, a moving object 20 is represented on the road 5. Also, in the first input data 610-1, a signboard 30 is represented on the road 5. Therefore, as shown in Figure 17, the object detection unit 250 detects the eighth moving object region 25-8 corresponding to the eighth moving object 20-8 and the signboard region 35 corresponding to the signboard 30 in the first difference data 620-1 using the background difference method. The object detection unit 250 also identifies the eighth moving object position 26-8, which represents the position of the eighth moving object 20-8 represented in the eighth moving object region 25-8. Furthermore, the object detection unit 250 identifies the signboard position 36, which represents the position of the signboard 30 represented in the signboard region 35.

[0092] Next, in step S220 shown in Figure 8, the object detection unit 250 tracks the moving object 20 and the signboard 30, which are represented in the eighth moving object region 25-8 and the signboard region 35 detected in the first input data 610-1. For example, the object detection unit 250 identifies the eighth moving object region 25-8, which is the region in the first input data 610-1 in Figure 16 and represents the same range as the eighth moving object region 25-8 in Figure 17. The object detection unit 250 also identifies the signboard region 35, which is the region in the first input data 610-1 in Figure 16 and represents the same range as the signboard region 35 in Figure 17. Furthermore, the object detection unit 250 identifies the positions of the eighth moving object position 26-8 and the signboard position 36 in the first input data 610-1 in Figure 16.

[0093] The object detection unit 250 determines that the moving object 20 represented in the eighth moving object region 25-8 and the signboard 30 represented in the signboard region 35 of the first input data 610-1 are objects that have been detected for the first time, because there is no input data 610 that was captured immediately before the first input data 610-1.

[0094] Next, in step S230 shown in Figure 8, the speed determination unit 260 determines the speeds of the moving object 20 in the eighth moving object area 25-8 and the signboard 30 in the signboard area 35 at the time the first input data 610-1 was captured. However, since there is no input data captured immediately before the first input data 610-1, the speeds of the moving object 20 in the eighth moving object area 25-8 and the signboard 30 in the signboard area 35 are not calculated.

[0095] Next, in step S240, the classification unit 270 classifies the mobile object 20 represented in the eighth mobile object area 25-8 of the first input data 610-1 and the signboard 30 in the signboard area 35 into categories. For example, the classification unit 270 classifies the mobile object 20 represented in the eighth mobile object area 25-8 as "ordinary car". The classification unit 270 also classifies the signboard 30 represented in the signboard area 35 as "other".

[0096] Once the process in step S240 is completed, the process in step S530 shown in Figure 15 is executed.

[0097] In step S530, the output unit 280 outputs target information to the automatic control device 300. For example, the output unit 280 outputs target information to the automatic control device 300 that represents the type, position, direction of movement, and speed of the detected moving object 20. The type of moving object 20 represents the category determined by the classification unit 270. The position of the moving object 20 represents the actual position represented by the corresponding moving object position 26. The speed of the moving object 20 represents the speed determined by the speed determination unit 260. The direction of movement of the moving object 20 represents the direction of movement determined by the speed determination unit 260.

[0098] The automatic control device 300 shown in Figure 1 controls the third mobile body 20-3 on which the automatic control device 300 is mounted, based on target information acquired from the mobile body detection device 200.

[0099] In step S540 shown in Figure 15, the background update unit 290 determines whether or not to update the background data 600. For example, the background update unit 290 determines to update the background data 600 when the update conditions are met. For example, the update conditions include reaching a predetermined time. The update conditions may also include a predetermined time having elapsed since the last time the background data 600 was updated. The update conditions may also include the user inputting an operation to update the background data 600. When the background update unit 290 determines to update the background data 600, it executes the process in step S550. If it determines not to update the background data 600, the process ends once, waits until the time to acquire the input data 610 again, and when that time is reached, it repeats from step S510.

[0100] Since the background data 600 is updated after acquiring multiple input data 610, the process of acquiring the next input data 610 will be described first, followed by the process of step S550. Here, when it is time to acquire the next input data 610, the process shown in Figure 15 begins. The processing of the second input data 610 will mainly be described in the embodiments shown in Figures 18 and 19.

[0101] In step S510 shown in Figure 15, the object detection unit 250 acquires the image represented by the second input data 610-2 shown in Figure 18.

[0102] In step S520, the process shown in Figure 8 is performed to identify the object of the second input data 610-2.

[0103] In step S210 shown in Figure 8, the object detection unit 250 generates the second difference data 620-2 shown in Figure 19. The object detection unit 250 detects the ninth moving object region 25-9 and the sign region 35 in the second difference data 620-2 using the background subtraction method. The object detection unit 250 also identifies the ninth moving object position 26-9, which represents the position of the ninth moving object 20-9 represented in the ninth moving object region 25-9. Furthermore, the object detection unit 250 identifies the sign position 36, which represents the position of the sign 30 represented in the sign region 35.

[0104] Next, in step S220 shown in Figure 8, the object detection unit 250 tracks the moving object 20 and the signboard 30, which are represented in the ninth moving object region 25-9 and the signboard region 35 detected in the second input data 610-2. For example, in the second input data 610-2 shown in Figure 18, the object detection unit 250 identifies the ninth moving object region 25-9 and the signboard region 35 shown in Figure 18, which correspond to the ninth moving object region 25-9 and the signboard region 35 shown in the second difference data 620-2 shown in Figure 19. The object detection unit 250 also identifies the positions of the ninth moving object position 26-9 and the signboard position 36 in the second input data 610-2.

[0105] The object detection unit 250 uses any known tracking method to identify the object estimation regions of the first input data 610-1, which were captured immediately before, that correspond to the ninth moving object region 25-9 and the signboard region 35 of the second input data 610-2. For example, the object detection unit 250 identifies the eighth moving object region 25-8 and the signboard region 35 in Figure 16, respectively, for the ninth moving object region 25-9 and the signboard region 35 in Figure 18.

[0106] In step S230 shown in Figure 8, the velocity determination unit 260 determines the velocity of the object represented in the object estimation region when the second input data 610-2 was captured. For example, the velocity determination unit 260 determines the distance from the position represented by the eighth moving object position 26-8 and the position represented by the sign position 36 in Figure 16 to the position represented by the ninth moving object position 26-9 and the position represented by the sign position 36 in Figure 18. The velocity determination unit 260 calculates the velocity of the moving object 20 represented in the ninth moving object region 25-9 and the sign 30 represented in the sign region 35 by dividing the determined distance by the time interval between when the first input data 610-1 and the second input data 610-2 were captured.

[0107] In step S240 shown in Figure 8, the classification unit 270 categorizes the mobile vehicle 20 represented in the ninth mobile vehicle area 25-9 of Figure 18 and the signboard 30 represented in the signboard area 35. For example, the classification unit 270 categorizes the mobile vehicle 20 represented in the ninth mobile vehicle area 25-9 as "ordinary vehicle" and the signboard 30 represented in the signboard area 35 as "other".

[0108] Once the process in step S240 is completed, the process in step S530 shown in Figure 15 is executed.

[0109] In step S530, the output unit 280 outputs the target information to the automatic control device 300.

[0110] In step S540, the background update unit 290 determines whether or not to update the background data 600. Here, let's assume that the background update unit 290 determines to update the background data 600. In this case, the background update unit 290 executes the process in step S550.

[0111] In step S550, the background update unit 290 updates the background data for the region where the background object has changed. For example, to update the background data, the background update unit 290 executes the process shown in Figure 20. The process shown in Figure 20 is similar to the process shown in Figure 13.

[0112] In step S610, the background update unit 290 selects one group from the group of object estimation regions that have been grouped by the tracking process. For example, the background update unit 290 selects the group of sign regions 35 shown in Figure 16.

[0113] In step S620 shown in Figure 20, the background update unit 290 determines whether the selected group satisfies the background conditions. The background conditions include the sign 30 represented in the sign area 35 being stationary. For example, the background update unit 290 determines that the sign 30 represented in the sign area 35 is stationary when its speed is below a threshold. For example, the background update unit 290 may determine that the sign 30 represented in the sign area 35 is stationary when its speed at the time the second input data 610-2 was captured is below a threshold. Here, since the speed of the sign 30 determined by the speed determination unit 260 is below a threshold, the background update unit 290 determines that the group including the sign area 35 satisfies this background condition.

[0114] Furthermore, the background conditions may include the condition that the object is stationary for a certain period of time or longer. In this case, the background update unit 290 selects input data 610 that was captured a certain period before the time when the latest second input data 610-2 was captured, within the selected group. For example, the background update unit 290 selects the input data 610 captured at the latest time among the input data 610 captured before a certain period before the time when the latest second input data 610-2 was captured, within the selected group. Alternatively, the background update unit 290 may select the input data 610 captured at the earliest time among the input data 610 captured after a certain period before the time when the latest second input data 610-2 was captured, within the selected group.

[0115] For all input data 610 captured from the time the selected input data 610 was captured until the time the latest second input data 610-2 was captured, the background update unit 290 determines whether the sign 30 represented in the sign area 35 is stationary. For example, if the speed of the sign 30 represented in the sign area 35 is below a threshold for all corresponding input data 610, the background update unit 290 determines that the sign 30 represented in the sign area 35 is stationary. Alternatively, the background update unit 290 may determine that the object represented in the object estimation area is stationary if the statistical values ​​of the speed of the object represented in the object estimation area for all input data 610 included in a continuous period, such as the maximum value, mean value, median value, etc., are below a threshold. Since the input data 610 corresponding to the group including the sign area 35 are only the first input data 610-1 and the second input data 610-2, this background condition may not be met. However, for the sake of clarity, we will assume here that the group including signboard area 35 satisfies this background condition.

[0116] Furthermore, the background condition may include the fact that the category to which the object represented in the object estimation region belongs is a non-target category, such as "other". For example, since the sign 30 represented in the sign region 35 belongs to "other", the background update unit 290 determines that the group including the sign region 35 satisfies this background condition.

[0117] The background update unit 290 executes the process in step S630 because the group including the selected sign area 35 satisfies all background conditions.

[0118] In step S630, the background update unit 290 sets a background flag for the group that includes the selected sign area 35.

[0119] Next, in step S640, since the group including the ninth mobile region 25-9 shown in Figure 18 has not been processed, the background update unit 290 returns to step S610 shown in Figure 20 and selects the group including the unselected ninth mobile region 25-9.

[0120] In step S620, the background update unit 290 determines whether the group including the ninth mobile area 25-9 satisfies the background conditions. Here, the speed of the mobile object 20 represented in the ninth mobile area 25-9 is greater than or equal to the threshold at the time the second input data 610-2 was captured. For this reason, the background update unit 290 determines that the group including the ninth mobile area 25-9 does not satisfy the background conditions and executes the process in step S640 without setting a background flag for the group. Furthermore, since the category of the mobile object 20 represented in the ninth mobile area 25-9 represents "ordinary car," the group including the ninth mobile area 25-9 does not satisfy any other background conditions either.

[0121] In step S640, the background update unit 290 has processed all groups, and therefore executes the process in step S650.

[0122] In step S650, the background update unit 290 generates representative data for the group in which the background flag is set. For example, the background update unit 290 determines the image represented in the sign area 35 in the latest second input data 610-2 shown in Figure 18 as representative data from the group that includes the sign area 35 in which the background flag is set. The background update unit 290 also calculates the average value of the brightness value of the first input data 610-1 and the brightness value of the second input data 610-2 for each pixel of the sign area 35. An image in which the brightness value of the corresponding pixel is set to the calculated average value may be determined as representative data.

[0123] Note that for the group including the 9th mobile area 25-9, no background flag is set, so representative data will not be generated.

[0124] Next, in step S660, the background update unit 290 replaces a portion of the background data with representative data. For example, the background update unit 290 replaces the corresponding area of ​​the background data 600 shown in Figure 14 with the image represented in the sign area 35 shown in Figure 18. Therefore, the corresponding area of ​​the background data 600 is overwritten with the image represented in the sign area 35 shown in Figure 18, for example, the image representing the sign 30. As a result, as shown in Figure 21, background data 600 is generated in which the sign 30 is represented in the corresponding area.

[0125] When the process in step S660 is completed, the process shown in Figure 15 also ends. Subsequently, when the moving object detection device 200 acquires the next input data 610, it starts the process shown in Figure 15 again.

[0126] In this way, the moving object detection system 1000 can efficiently detect the moving object 20 by incorporating changes in background objects into background data 600.

[0127] (Embodiment 2) The sensor 100 shown in Figure 1 may be a LiDAR (Light Detection and Ranging) that measures the distance and direction from the position of the sensor 100 to an object contained in a corresponding area. For example, the sensor 100 may be a ToF (Time Of Flight) camera. The sensor 100 may output an image representing the distance and direction to the object as input data to the moving object detection device 200. For example, the image represents the relative direction from the position of the sensor 100 by the position of the pixels, and the relative distance by the brightness value of the pixels. Alternatively, the sensor 100 may output point cloud data representing the distance and direction to the object in three dimensions as input data to the moving object detection device 200. For example, each point in the point cloud data represents the reflection point of the light wave output from the sensor 100, for example, the position on the surface of the object.

[0128] When the input data is represented as point cloud data, the background data 600 and difference data 620 are also represented as point cloud data. Even when the input data is represented as point cloud data, the moving object detection system 1000 can detect the moving object 20 and update the background data 600 in the same way as in Embodiment 1, simply by replacing the data it handles, such as the input data 610, background data 600, and difference data 620, with point cloud data.

[0129] For example, in step S210 shown in Figure 8, the object detection unit 250 of the moving object detection device 200 excludes background data, such as points that are located at a predetermined distance from each point of the first initial input data 510-1, and determines the point cloud data represented by the remaining points as the initial difference data 520. The object detection unit 250 extracts a point cloud that is estimated to represent a single object from each point in the determined initial difference data 520 in any way. The object detection unit 250 detects the three-dimensional region containing the extracted point cloud as an object estimation region (e.g., moving object region 25, difference tree region 15, signboard region 35, etc.). The three-dimensional region may represent the shape of a rectangular parallelepiped, for example, as shown in Figure 22. The three-dimensional region may have four vertical planes 710 on its surface that include the vertical direction in the in-plane direction.

[0130] In this case, the object estimation position (e.g., moving object position 26, difference tree position 16, signboard position 36, etc.) that represents the position of an object represented in the object estimation region represents any position within the object estimation region. For example, the object estimation position is included in the first vertical plane 710-1 closest to the sensor 100, among the vertical planes 710 that include the vertical direction in the in-plane direction. Furthermore, the object estimation position may also be the position 720 that indicates the center of the edge forming the lower end in the vertical direction on the plane closest to the sensor 100.

[0131] In step S220 shown in Figure 8, the point cloud data represents the surface of the object. Therefore, the object detection unit 250 determines, by any known method, whether the surface of the object represented by the point cloud data is the same as the surface of the object represented in the object region detected in the other initial input data 510. As a result, the object detection unit 250 tracks the object represented in the object estimation region.

[0132] In step S230, the velocity determination unit 260 determines the velocity of the object represented in the object estimation region based on the determined object estimation position.

[0133] In step S240, the classification unit 270 classifies the objects represented at the estimated object locations into categories based on the surface of the objects represented by the point cloud data, using any known method.

[0134] Similarly, in other processes shown in Figures 12 and 13, the moving object detection system 1000 processes the data as point cloud data. As a result, the moving object detection system 1000 detects the moving object 20 and updates the background data 600.

[0135] In this case, when point cloud data is used, the difference tree region 15 shown in Figure 10 does not contain any points representing the surface of an object. Therefore, the object detection unit 250 of the moving object detection device 200 may process the data assuming that there is no object represented in the difference tree region 15, or in other words, that there are no points representing the surface of an object. For example, the representative data generated in step S350 in Figure 12 and step S450 in Figure 13 indicates that there are no points within the object estimation region. Also, in step S360 in Figure 12 and step S460 in Figure 13, the background update unit 290 of the moving object detection device 200 may delete points from the region of the background data 600 that corresponds to the object estimation region.

[0136] Furthermore, in step S210 shown in Figure 8, the object detection unit 250 may determine object data representing the changed object based on the object estimation region, for example, the difference tree region 15. For example, as shown in Figure 10, if the difference tree region 15 is detected in the third initial input data 510-3, the object detection unit 250 detects the region of the object that includes at least a part of the difference tree region 15. Here, since there are no points in the difference tree region 15, the object detection unit 250 detects the difference tree region 15 in the third initial input data 510-3 as a temporary region. Furthermore, in the second initial input data 510-2 shown in Figure 7, which was captured immediately before, the object detection unit 250 detects the region of the object that includes at least a part of the corresponding difference tree region 15. In the second initial input data 510-2, since the difference tree region 15 includes a part of the first tree 10-1, the region including the entire first tree 10-1 is detected as a temporary region. Furthermore, the object detection unit 250 determines the temporary region representing the larger area between the temporary region in the third initial input data 510-3 and the temporary region in the second initial input data 510-2 as the object estimation region. For this reason, the object detection unit 250 determines the region including the entire first tree 10-1 as the object estimation region. In the third initial input data 510-3 of Figure 10, the object detection unit 250 determines the region representing the same range as the determined object estimation region as the object estimation region in the third initial input data 510-3. As a result, the object represented in the object estimation region will represent the tree stump 11. Also, the object data of the object represented in the object estimation region will be point cloud data representing the tree stump 11. Similarly, for the moving object region 25 and the signboard region 35 shown in Figure 16, the object detection unit 250 may modify the detected object estimation region to the region of the object that includes at least a part of it.

[0137] Furthermore, although the object data was described as referring to data representing a changed object, it may also refer to data representing an object represented in the difference tree region 15, for example, data indicating that the object does not exist. Additionally, the object data may refer to data representing a moving object 20 included in the moving object region 25. The object data may represent not only point cloud data, but also images within the object estimation region.

[0138] Furthermore, the position 720 of the object estimation region shown in Figure 22 may be included in the vertical plane 710 in the direction of motion of the object represented in the object estimation region, instead of the first vertical plane 710-1 closest to the sensor 100. In this case, the position 720 of the object estimation region may be determined after the process of step S230 in the process shown in Figure 8.

[0139] (modified version) The embodiments and their modifications described above are examples, and the configurations described in each embodiment and modification may be arbitrarily changed and / or combined as long as they do not impede the function. Furthermore, some of the functions described in the embodiments and modifications may be omitted if the necessary functions can be realized. For example, the process shown in Figure 5 may be performed in any way. For example, the process shown in Figure 5 may be performed when the moving object detection device 200 is first started up. Alternatively, the process shown in Figure 5 may be performed when an operation indicating that the user is performing an initial operation is input to the input / output device 210.

[0140] Furthermore, the process shown in Figure 5 may be executed each time image information is acquired from the sensor 100. For example, the object detection unit 250 of the moving object detection device 200 may execute the process in step S120 when it first acquires image information from the sensor 100, and thereafter execute the processes in steps S130 and S140 each time image information is acquired. In this case, for example, the processes from step S160 onward are executed when a predetermined period (e.g., 1 minute) has elapsed since the start of processing.

[0141] The exclusion conditions in step S320 shown in Figure 12 may include the period during which the object represented in the object estimation region is stationary for a certain period of time or longer. For example, the background update unit 290 determines that the object represented in the object estimation region is stationary when the velocity of the object represented in the object estimation region is below a threshold for all initial input data 510 included in a continuous period of time. The background update unit 290 may also determine that the object represented in the object estimation region is stationary when the statistical values ​​of the velocity of the object represented in the object estimation region, such as the maximum value, mean value, median value, etc., for all initial input data 510 included in a continuous period of time are below a threshold.

[0142] Furthermore, the exclusion conditions may include cases where the category to which an object represented in the object estimation region belongs is a non-target category, such as "other". For example, the region in which the moving object 20 exists in the first initial input data 510-1 that was initially captured is detected because the moving object 20 moves, and therefore the detected initial input data 510 is highly likely to represent a non-target category (e.g., road 5). For this reason, the exclusion conditions may include cases where an object belonging to a non-target category is represented in the movement estimation region detected because the moving object 20 has moved.

[0143] Furthermore, the exclusion condition may include the case where one of the initial input data 510 corresponding to a group represents the last initial input data 510 captured from among the multiple initial input data 510 acquired from the sensor 100. For example, if the first initial input data 510-1 captured initially contains the region of the moving object 20, it is highly likely that the same region will also be detected in the last initial input data 510 captured. Therefore, an object detection region representing the region of the moving object 20 included in the background data is highly likely to be detected in the last initial input data 510 captured. For this reason, the exclusion condition may include the case where an object detection region is detected in the last initial input data 510 captured. Also, in step S310 shown in Figure 12, the background update unit 290 may select one group from the group of object estimation regions detected in the last initial input data 510 captured, thereby limiting the groups for which the exclusion condition is determined.

[0144] In step S320 shown in Figure 12, the background update unit 290 of the moving object detection device 200 may determine that the selected group satisfies the exclusion conditions if the combination of exclusion conditions that is satisfied among the multiple exclusion conditions is a preset combination.

[0145] In step S350 shown in Figure 12, the background update unit 290 of the moving object detection device 200 may generate representative data for an area that is a predetermined length larger than the object estimation area. For example, as shown in Figure 7, the object estimation area may represent only a part of the object if part of the object is missing. Therefore, the background update unit 290 may update the area representing the entire object by updating the background data 600 for an area that is a predetermined length larger than the object estimation area.

[0146] In step S360 shown in Figure 12, the background update unit 290 of the moving object detection device 200 may update the background data using the representative data in any order when multiple representative data are generated. For example, the background update unit 290 may update the background data in order from the earliest time the initial input data 510 corresponding to the representative data was captured the latest.

[0147] The process in step S160 shown in Figure 5 may be omitted. For example, in the process in step S170, the area of ​​the moving body 20 represented in the initial input data 510 may be excluded. For this reason, the process in step S160 shown in Figure 5 may be omitted, and the area of ​​the moving body 20 may be excluded in step S170.

[0148] The background conditions in step S420 shown in Figure 13 may include, similar to the exclusion conditions in step S320 shown in Figure 12, that one of the initial input data 510 corresponding to a group represents the last initial input data 510 captured from among the multiple initial input data 510 acquired from the sensor 100. Furthermore, in step S410 shown in Figure 13, the background update unit 290 may select one group from the groups of object estimation regions detected in the last captured initial input data 510.

[0149] In step S420 shown in Figure 13, similar to step S320 shown in Figure 12, the background update unit 290 of the moving object detection device 200 may determine that the selected group satisfies the background conditions when the combination of satisfying background conditions among the multiple background conditions is a preset combination. Also, in step S450 shown in Figure 13, similar to step S350 shown in Figure 12, the background update unit 290 of the moving object detection device 200 may generate representative data for an area that is a predetermined length larger than the object estimation area. Furthermore, in step S460 shown in Figure 13, similar to step S360 shown in Figure 12, when multiple representative data are generated, the background update unit 290 of the moving object detection device 200 may update the background data using the representative data in any order.

[0150] The background conditions in step S620 shown in Figure 20 may include, similar to the exclusion conditions in step S320 shown in Figure 12, that one of the input data 610 corresponding to the selected group represents the last captured input data 610. Furthermore, in step S610 shown in Figure 20, the background update unit 290 may select one group from the groups of object estimation regions detected in the most recent input data 610.

[0151] The example shown in Figure 15 illustrates how background data is updated using previously acquired input data when the background conditions are met, but the method is not limited to this. For example, when the background conditions are met, the background data may be updated using input data acquired afterward.

[0152] The target category may include any type. For example, the target category may include "motorcycles". If the target category includes "motorcycles", the classification unit 270 of the moving object detection device 200 may distinguish between "motorcycles" and "bicycles" based on the velocity of the object represented in the object estimation domain. For example, the classification unit 270 uses a learning model to determine whether the object represented in the object estimation domain belongs to either a "motorcycle" or a "bicycle", and determines that it belongs to a "motorcycle" if the velocity of the object is greater than a threshold.

[0153] Furthermore, the categories may include types other than "other" as non-target categories. For example, the non-target categories may include "signboards" installed on road 5 and "other". In this case, the classification unit 270 of the moving object detection device 200 may determine whether the object represented in the object estimation area belongs to the target category or "signboards", and if it is determined that it does not belong to either, it may determine that it belongs to "other".

[0154] The background data 600 and the input data 610 represent the same real-world range in at least part of their respective regions. For example, a portion of the background data 600 may represent the entire real-world range represented in the input data 610. Alternatively, a portion of the input data 610 may represent the entire real-world range represented in the background data 600. A portion of the real-world range represented in the background data 600 may include a portion of the real-world range represented in the input data 610, while the remaining portion of the real-world range represented in the input data 610 may not be included in the real-world range represented in the background data 600. The moving object detection device 200 detects the moving object 20 in the overlapping regions of the ranges represented by the background data 600 and the input data 610. [Explanation of Symbols]

[0155] 1: Storage medium, 5: Road, 10: Tree, 11: Stump, 12: Difference tree, 15: Difference tree area, 16: Difference tree position, 20: Moving object, 25: Moving object area, 26: Moving object position, 27: Initial moving object area, 30: Signboard, 35: Signboard area, 36: Signboard position, 100: Sensor, 200: Moving object detection device, 210: Input / output device, 220: Processing unit, 230: Communication device, 240: Memory device, 250: Object detection unit, 260: Speed ​​determination unit, 270: Classification unit, 280: Output unit, 290: Background update unit, 300: Automatic control device, 400: Moving object detection program, 510: Initial input data, 520: Initial difference data, 600: Background data, 610: Input data 620: Differential data, 710: Vertical plane, 720: Position, 1000: Moving object detection system

Claims

1. An object detection unit detects an object estimation region represented in the input data based on difference data that represents the difference between background data and input data. A background update unit updates the background data based on fixed object data representing the object when background conditions are met, including the fact that the velocity of the object represented in the object estimation region is below a threshold. Equipped with, The difference data is used to detect the moving object represented in the input data. Mobile object detection device.

2. The aforementioned background update unit is: Of the background data, the position of the background data corresponding to the position where the fixed object data is represented in the input data is replaced with the fixed object data. The moving object detection device according to claim 1.

3. The aforementioned input data consists of multiple input data acquired at different times. The object detection unit, Based on the difference data representing the difference between each of the plurality of input data and the background data, the object estimation region represented in each of the plurality of input data is detected. The aforementioned moving object detection device is A velocity determination unit determines the velocity of the object represented in the object estimation region based on the position of the object represented in the object estimation region in the plurality of input data. The moving object detection device according to claim 1, further comprising:

4. The object detection unit, Based on the first object data representing the first object represented in the first object estimation region detected in the first input data among the plurality of input data, and the second object data representing the second object represented in the second object estimation region detected in the second input data which is different from the first input data among the plurality of input data, it is determined whether the first object and the second object represent the same object. The speed determination unit, When it is determined that the first object and the second object represent the same object, the velocity of the object represented in the object estimation region is determined based on the position of the first object data in the first input data and the position of the second object data in the second input data. The moving object detection device according to claim 3.

5. The aforementioned background update unit is: Based on the multiple fixed object data representing the object shown in the object estimation region detected from each of the multiple input data, a representative data representing the object is determined. Based on the aforementioned representative data, update the aforementioned background data. The moving object detection device according to claim 3.

6. The system further includes a classification unit that classifies the object into a plurality of categories, including a target category representing the type of object to be detected and a non-target category representing types other than the type of object to be detected. The aforementioned background condition includes the fact that the object is classified into the non-applicable category. A moving object detection device according to any one of claims 1 to 5.

7. The object detection unit determines the size of the object, The aforementioned background condition includes the fact that the size of the object does not fall within the specified range. A moving object detection device according to any one of claims 1 to 5.

8. Based on difference data representing the difference between background data and input data, the object estimation region represented in the input data is detected. When the background conditions are met, including the fact that the velocity of the object represented in the object estimation region is below a threshold, the background data is updated based on the fixed object data representing the object. The computing unit is made to execute this, The difference data is used to detect the moving object represented in the input data. A program for detecting moving objects.

Images