OBJECT DETECTION DEVICE, OBJECT DETECTION PROGRAM, OBJECT DETECTION METHOD, AND OBJECT DETECTION SYSTEM

Abstract

To highly precisely verify the result of a process of detecting an object present in an image.SOLUTION: An object detecting device includes: a background difference executing unit that executes a background difference to generate a difference image using an observed image, which has a possibility of an object being present therein and on which an object detecting process of detecting the object has been executed, and a background image that has no object present therein; and a verifying unit that executes a verifying process of verifying whether or not the object present in the difference image should be taken into consideration when a moving object moves.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an object detection device, an object detection program, an object detection method, and an object detection system. [Background technology]

[0002] Conventionally, there are known systems that analyze images captured by a camera of vehicles traveling on a road, measure the vehicle's position, speed, etc., and contribute to the realization of smooth traffic. Such systems need to accurately detect objects such as people and vehicles on the road in some way.

[0003] For example, the road object detection device disclosed in Patent Document 1 calculates the difference between a background image and an input image, and detects the position of an object based on the difference value. Non-Patent Document 1 also discloses a technology that uses deep learning to train a machine learning model to learn the type and coordinates of objects in an image, and determines the appearance and disappearance of objects in an unknown image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 04-315299 [Non-Patent Document 1] Mingxing Tan, Ruoming Pang, Quoc V. Le, Google Research, Brain Team ”EfficientDet: Scalable and Efficient Object Detection” arXiv:1911.09070v7 [cs.CV] 27 Jul 2020 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the road object detection device disclosed in Patent Document 1 may detect objects other than those that should be considered when a moving body such as a vehicle moves. In addition, the technology disclosed in Non-Patent Document 1 uses deep learning, and therefore may detect objects other than those that should be considered when a moving body such as a vehicle moves.

[0006] Therefore, an object of the present invention is to provide an object detection device, an object detection program, an object detection method, and an object detection system that can verify the results of a process for detecting objects appearing in an image with high accuracy. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, an object detection device according to the present invention comprises: The image is generated by photographing a moving object or a place where the moving object moves, An observation image that may contain an object and has been subjected to an object detection process to detect the object, and a background image that does not contain the object. 、 a background subtraction execution unit that generates a difference image by performing background subtraction using a The above The device further includes a verification unit that executes a verification process for verifying whether or not the movement of the moving body should be taken into consideration. Effect of the Invention

[0008] According to the present invention, the results of processing for detecting an object captured in an image can be verified with high accuracy. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram illustrating an example of an object detection device and an integration device according to an embodiment. [Diagram 2] 2 is a diagram illustrating an example of a hardware configuration of an object detection device and a hardware configuration of an integration device according to an embodiment. FIG. [Diagram 3] FIG. 2 is a diagram illustrating an example of a software configuration of the object detection device according to the embodiment. [Figure 4]3A to 3C are diagrams illustrating examples of an observation image in which the position of an object is defined by camera coordinates and an observation image in which the position of the object is defined by world coordinates according to an embodiment. [Diagram 5] 3A to 3C are diagrams illustrating examples of an observation image in which the position of an object is defined by camera coordinates and an observation image in which the position of the object is defined by world coordinates according to an embodiment. [Figure 6] 5 is a flowchart illustrating an example of a process executed by the object detection device according to the embodiment. [Figure 7] 5 is a flowchart illustrating an example of a process executed by the integration device according to the embodiment. [Figure 8] 7 is a flowchart showing an example of detailed processing of step S62 in FIG. 6. [Figure 9] 9 is a flowchart showing an example of detailed processes in steps S84 and S85 in FIG. 8. [Figure 10] 9 is a flowchart showing an example of detailed processes in steps S84 and S85 in FIG. 8. [Figure 11] 9 is a flowchart showing an example of detailed processing of step S87 in FIG. 8. [Figure 12] 9 is a flowchart showing an example of detailed processing of step S87 in FIG. 8. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] <Embodiment> Hereinafter, an embodiment of the present invention will be described with reference to FIG. 1 to FIG. 12. FIG. 1 is a diagram showing an example of an object detection device and an integrating device according to an embodiment. FIG. 1 shows object detection devices 1-1, ..., and object detection device 1-k (k: an integer equal to or greater than 1), and an integrating device 10. For example, as shown in FIG. 1, the object detection devices 1-1, ..., and object detection device 1-k capture still images or moving images of moving objects such as vehicle B1 and vehicle B2 and places where these moving objects move, and predict the trajectories of these moving objects. In the following description, these still images and still images constituting this moving image are referred to as observed images. The observed images may include objects that should be considered when the moving object moves. Details of the processing executed by the object detection devices 1-1, ..., or object detection device 1-k will be described later.

[0011] The integrating device 10 executes a time prediction process for predicting the time when the support vehicle V will arrive at an attention area A in an intersection based on the result of predicting the trajectories of moving bodies such as vehicle B1 and vehicle B2, for example, as shown in Fig. 1. Then, the integrating device 10 generates data indicating the contents to be notified to the support vehicle V based on the result of the time prediction process, for example, as shown in Fig. 1, and notifies the support vehicle V of the contents. Details of these processes will be described later.

[0012] Furthermore, when the support vehicle V is performing autonomous driving, the above-mentioned contents are reflected in the acceleration, deceleration, stopping, direction changes, etc., that are automatically performed by the support vehicle V. For example, in the case shown in FIG. 1, the support vehicle V stops based on the above-mentioned contents before reaching the attention area A. On the other hand, when the support vehicle V is not performing autonomous driving or the support vehicle V does not have a function for performing autonomous driving, the above-mentioned contents are used for judgments regarding acceleration, deceleration, stopping, direction changes, etc., that are performed by the driver of the support vehicle V.

[0013] Next, the hardware configuration of the object detection device according to the embodiment and the hardware configuration of the integration device will be described with reference to FIG. 2. FIG. 2 is a diagram showing an example of the hardware configuration of the object detection device according to the embodiment and the hardware configuration of the integration device. FIG. 2 shows the object detection device 1-1, ..., the object detection device 1-k, and the integration device 10. As shown in FIG. 2, the object detection device 1-1 includes an information processing unit 2-1, an image capture unit 3-1, and a communication unit 4-1. Moreover, the information processing unit 2-1, the image capture unit 3-1, and the communication unit 4-1 are connected to a LAN (Local Area Network) through a switching hub, and can communicate with each other. Similarly, the object detection device 1-k includes an information processing unit 2-k, an image capture unit 3-k, and a communication unit 4-k as shown in FIG. 2. Moreover, the information processing unit 2-k, the image capture unit 3-k, and the communication unit 4-k are connected to a LAN (Local Area Network) through a switching hub, and can communicate with each other.

[0014] The information processing unit 2-1 includes a CPU (Central Processing Unit) 21-1, a RAM (Random Access Memory) 22-1, and a ROM (Read Only Memory) 23-1. The CPU 21-1, the RAM 22-1, and the ROM 23-1 are connected by a bus and can communicate with each other. Similarly, the information processing unit 2-k includes a CPU 21-k, a RAM 22-k, and a ROM 23-k. The CPU 21-k, the RAM 22-k, and the ROM 23-k are connected by a bus and can communicate with each other.

[0015] The CPU 21-1 realizes each function of the object detection device 1-1 by reading and executing a program. Similarly, the CPU 21-k realizes each function of the object detection device 1-k by reading and executing a program. The RAM 22-1 is a storage medium in which a program read and executed by the CPU 21-1 is temporarily deployed. Similarly, the RAM 22-k is a storage medium in which a program read and executed by the CPU 21-k is temporarily deployed. The ROM 23-1 is a storage medium in which a program read and executed by the CPU 21-1 is stored. Similarly, the ROM 23-k is a storage medium in which a program read and executed by the CPU 21-k is stored. Details of the functions of the object detection devices 1-1, ... and the object detection device 1-k will be described later with reference to the drawings.

[0016] Note that object detection device 1-1, ..., and object detection device 1-k execute the same process. Therefore, in the following description, object detection device 1-1 will be taken as an example, and a description of object detection device 1-2, ..., and object detection device 1-k will be omitted.

[0017] The photographing units 3-1, ... and 3-k are, for example, network cameras. The photographing units 3-1, ... and 3-k each generate an observation image by photographing moving objects such as vehicle B1 and vehicle B2 shown in FIG. 1 and the places where these moving objects move. The communication units 4-1, ... and 4-k are, for example, mobile routers. The communication units 4-1, ... and 4-k communicate with the integrated device 10.

[0018] As shown in FIG. 2, the integrated device 10 includes an information processing unit 20, a notification unit 30, and a communication unit 40.

[0019] The information processing unit 20 includes a CPU 201, a RAM 202, and a ROM 203. The CPU 201 realizes each function of the integrated device 10 by reading and executing a program. The RAM 202 is a storage medium in which the program read and executed by the CPU 201 is temporarily expanded. The ROM 203 is a storage medium in which the program read and executed by the CPU 201 is saved. Details of the functions of the integrated device 10 will be described later with reference to the drawings.

[0020] The notification unit 30 is, for example, a display or a speaker. The notification unit 30 notifies the support vehicle V of the content to be notified. The communication unit 40 executes communication with at least one of the object detection devices 1-1, . . . and the object detection devices 1-k.

[0021] Next, a software configuration of the object detection device according to the embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the software configuration of the object detection device according to the embodiment. As shown in Fig. 3, the object detection device 1-1 includes an image acquisition unit 211-1, an object detection unit 212-1, a background difference execution unit 213-1, a verification unit 214-1, a background image generation unit 215-1, a coordinate conversion unit 216-1, and a trajectory prediction unit 217-1. As described above, these are realized by the CPU 21-1 reading and executing a program.

[0022] The image acquisition unit 211-1 acquires an observation image captured by the image capture unit 3-1. When acquiring observation images constituting a moving image captured by the image capture unit 3-1, the image acquisition unit 211-1 may acquire all of the observation images constituting the moving image. Alternatively, when acquiring observation images constituting a moving image captured by the image capture unit 3-1, the image acquisition unit 211-1 may acquire an observation image for every predetermined number of frames. When acquiring observation images constituting a moving image captured by the image capture unit 3-1, the image acquisition unit 211-1 may acquire only a portion corresponding to an object region, which will be described later, as an observation image.

[0023] The object detection unit 212-1 executes an object detection process to detect an object that should be considered when the moving body moves. For example, the object detection unit 212-1 inputs an observation image to a machine learning model generated by executing deep learning, and detects an object that is reflected in the observation image and should be considered when the moving body moves. The object detection unit 212-1 may also reduce the detection of an object that does not need to be considered when the moving body moves by appropriately using different machine learning models or adjusting various parameters of the machine learning models. The object detection unit 212-1 also assigns a bounding box to each object detected by the object detection process.

[0024] The background difference executing unit 213-1 generates a difference image by performing background difference using an observation image that may contain an object and has been subjected to object detection processing to detect the object, and a background image in which no object is shown. The background difference executing unit 213-1 may also generate a difference image by performing background difference using an object region including an object detected by the object detection processing and a background region in the background image that corresponds to the object region.

[0025] The verification unit 214-1 executes a verification process to verify whether or not an object reflected in the difference image should be taken into consideration when a moving body moves. For example, the verification unit 214-1 executes a verification process to generate a binary image by applying binarization to a background image, and to verify whether or not an object is reflected in the difference image using the binary image.

[0026] The background image generating unit 215-1 generates a background image. For example, the background image generating unit 215-1 sets at least one of an observed image in which no object is detected by an object detection process and an observed image in which it is verified that no object is captured by a verification process as a background image.

[0027] The coordinate conversion unit 216-1 applies coordinate conversion to convert the position of an object in an observation image from camera coordinates to world coordinates. FIG. 4 is a diagram showing an example of an observation image in which the position of an object is defined by camera coordinates and an observation image in which the position of the object is defined by world coordinates according to the embodiment. FIG. 4(A) shows an observation image in which the position of an object is defined by camera coordinates. FIG. 4(B) shows an observation image in which the position of an object is defined by world coordinates. FIG. 4(A) and FIG. 4(B) show a vehicle B4, an object area C4, a road marking R4, and a feature point P. The object area C4 is a rectangular area defined by a bounding box indicating the vehicle B4 detected by the object detection process. This bounding box may circumscribe the vehicle B4. The feature point P is a point located at the midpoint of the lower side of the object area C4, and represents the position of the vehicle B4.

[0028] The coordinate conversion unit 216-1 may execute the above-mentioned coordinate conversion using a projection transformation matrix generated based on the position of the image capture unit 3-1, the three-dimensional information of the vehicle B4 and its surroundings, and the like. The three-dimensional information is generated, for example, by scanning the vehicle B4 and its surroundings with a three-dimensional scanner. The projection transformation matrix is ​​generated, for example, by associating an orthoimage including map information such as latitude and longitude with the feature point P set on the observation image shown in FIG. 4(A) and FIG. 4(B). The projection transformation matrix includes information regarding the depth from the viewpoint of the image capture unit 3-1. The data indicating the projection transformation matrix is ​​stored in the ROM 23-1 and used by the CPU 21-1.

[0029] Fig. 5 is a diagram showing an example of an observation image in which the position of an object is defined by camera coordinates and an observation image in which the position of the object is defined by world coordinates according to the embodiment. Fig. 5(A) shows an observation image in which the position of an object is defined by camera coordinates. Fig. 5(B) shows an observation image in which the position of an object is defined by world coordinates. Fig. 5(A) and Fig. 5(B) show a vehicle B5, an object region C51, a road marking R52, an object region C52, a road marking R53, and an object region C53.

[0030] The object region C51 is a rectangular region defined by a bounding box representing the vehicle B5 detected by the object detection process. This bounding box may circumscribe the vehicle B5. The vehicle B5 is an example of an object that should be taken into consideration when the support vehicle V moves.

[0031] The object region C52 is a rectangular region defined by a bounding box indicating the road marking R52 detected by the object detection process. This bounding box may circumscribe the road marking R52. Furthermore, the road marking R52 is an example of an object that does not need to be taken into consideration when the support vehicle V moves, and should not be detected by the object detection process.

[0032] The object region C53 is a rectangular region defined by a bounding box indicating the road marking R53 detected by the object detection process. This bounding box may circumscribe the road marking R53. The road marking R53 is an example of an object that does not need to be taken into consideration when the support vehicle V moves, and should not be detected by the object detection process.

[0033] The trajectory prediction unit 217-1 executes a trajectory prediction process to predict the trajectory of the support vehicle V. For example, the trajectory prediction unit 217-1 predicts the future position of the support vehicle V using a plurality of observation images captured at different times. Then, the trajectory prediction unit 217-1 transmits data indicating the result of the trajectory prediction process to the integration device 10.

[0034] The integrating device 10 receives data indicating the result of the trajectory prediction process from at least one of the object detection devices 1-1, ... and the object detection device 1-k. The integrating device 10 may synchronize the time associated with each result of the trajectory prediction process. The integrating device 10 executes a time prediction process to predict the time when the support vehicle V will arrive at the attention area based on at least one of the results of the trajectory prediction process. Next, the integrating device 10 generates data indicating the content to be notified to the support vehicle V based on the result of the time prediction process. The integrating device 10 may include, for example, whether the support vehicle V can go straight, go backward, turn right, turn left, etc., and the type of vehicle other than the support vehicle V in the content. In addition, the type of vehicle referred to here is, for example, a large vehicle, a small / medium-sized vehicle, and a two-wheeled vehicle. Then, the integrating device 10 notifies the support vehicle V of the content indicated by this data.

[0035] In addition, when there are multiple support vehicles, it is preferable that the integration device 10 predicts the time when the support vehicle that arrives first in the attention area will arrive in the attention area. In addition, the integration device 10 may calculate the probability that the support vehicle V will collide with another vehicle while taking into account the characteristics of the support vehicle V, and generate data indicating the contents to be notified to the support vehicle V. The characteristics of the support vehicle V referred to here are, for example, the acceleration force, weight, dimensions, and vehicle type of the support vehicle V. In addition, the vehicle type is, for example, a large vehicle, a small or medium-sized vehicle, or a two-wheeled vehicle. In addition, the integration device 10 may generate data indicating the contents to be notified to the support vehicle V based on the characteristics of the road on which the support vehicle V is traveling, the weather, and the like.

[0036] Next, an example of processing executed by the object detection device 1-1, ..., and the object detection device 1-k will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of processing executed by the object detection device according to the embodiment.

[0037] In step S61, the object detection device 1-1 acquires an observation image and executes an object detection process to detect an object.

[0038] In step S62, the object detection device 1-1 executes a verification process for verifying the result of the object detection process and an update process for updating the background image.

[0039] In step S63, object detection device 1-1 performs coordinate conversion of the position of the object in the observation image acquired in step S61 from the camera image to world coordinates.

[0040] In step S64, the object detection device 1-1 executes a trajectory prediction process for predicting the trajectory of the object detected in step S61.

[0041] In step S65, the object detection device 1-1 transmits data indicating the result of the prediction process executed in step S64 to the integrating device.

[0042] Next, an example of processing executed by the integrating device 10 will be described with reference to Fig. 7. Fig. 7 is a flowchart showing an example of processing executed by the integrating device according to the embodiment.

[0043] In step S71, the integrating device 10 receives data indicating the result of the prediction process from the object detection device.

[0044] In step S72, the integrating device 10 executes a time prediction process for predicting the time when the object will arrive in the attention area using the data acquired in step S71.

[0045] In step S73, the integrating device 10 generates data indicating the contents to be notified using the result of the time prediction process executed in step S72.

[0046] In step S74, the integrated device 10 notifies the content indicated by the data generated in step S73.

[0047] Next, an example of detailed processing of step S62 in Fig. 6 will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of detailed processing of step S62 in Fig. 6.

[0048] In step S81, the object detection device 1-1 selects one object that has been detected by the object detection process and is to be verified as to whether or not it should be taken into consideration when the moving body moves.

[0049] In step S82, the object detection device 1-1 obtains data indicating the result of the object detection process for the object selected in step S81.

[0050] In step S83, the object detection device 1-1 acquires an observation image and a background image of the object selected in step S81.

[0051] In step S84, the background subtraction execution unit 213-1 generates a subtraction image by performing background subtraction using the observation image and the background image acquired in step S83.

[0052] In step S85, verification unit 214-1 determines whether or not the object shown in the difference image is an object that should be taken into consideration when the moving body moves. If verification unit 214-1 determines that the object shown in the difference image is an object that should be taken into consideration when the moving body moves (step S85: Yes), it proceeds to step S86. On the other hand, if verification unit 214-1 determines that the object shown in the difference image is not an object that should be taken into consideration when the moving body moves (step S85: No), it returns the process to step S81.

[0053] In step S86, the object detection device 1-1 outputs the data acquired in step S82.

[0054] In step S87, background image generation unit 215-1 updates the background image.

[0055] In step S88, the verification unit 214-1 determines whether or not there is an object that has been detected by the object detection process and has not been selected in step S81. If the verification unit 214-1 determines that there is an object that has been detected by the object detection process and has not been selected in step S81 (step S88: YES), the process returns to step S81. On the other hand, if the verification unit 214-1 determines that there is no object that has been detected by the object detection process and has not been selected in step S81 (step S88: NO), the process proceeds to step S63 in FIG.

[0056] The object detection device 1-1 may execute the verification process in order from the object closest to the support vehicle V in the observation image, background image, or difference image. For example, when the support vehicle V is located at the bottom of the observation image, background image, and difference image, the object detection device 1-1 executes the verification process in order from the object located at the bottom in these images. This order is determined, for example, based on the coordinates of each object in these images. In this case, the verification unit 214-1 executes the verification process in order from the object closest to the moving object in the difference image.

[0057] Furthermore, in the above-mentioned step S85, the verification unit 214-1 may use a mask image. The mask image is used for the purpose of clarifying an area where a moving body such as the support vehicle V may move. For example, in the mask image, white is assigned to an area, such as a road, in the observation image, background image, and difference image, where a moving body such as the support vehicle V may move. In addition, in the mask image, black is assigned to an area, such as a median strip, in the observation image, background image, and difference image, where a moving body such as the support vehicle V may not move. Note that the mask image may not be an image in which white or black is assigned to the above-mentioned areas, but may be an image in which feature points such as vertices of the boundaries of the above-mentioned two types of areas are depicted.

[0058] In this case, the verification unit 214-1 superimposes a mask image on the difference image and determines whether or not the feature points, center point, etc. of the object area of ​​each object shown in the difference image are located in the area assigned to white in the mask image. If these points are located in the area assigned to white in the mask image, the verification unit 214-1 determines that the object corresponding to the points is an object that should be considered when the moving body moves. On the other hand, if these points are not located in the area assigned to white in the mask image, the verification unit 214-1 determines that the object corresponding to the points is not an object that should be considered when the moving body moves.

[0059] Next, an example of detailed processing of steps S84 and S85 in Fig. 8 will be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of detailed processing of steps S84 and S85 in Fig. 8.

[0060] In step S8411, the background subtraction execution unit 213-1 converts the observed image into a grayscale image.

[0061] In step S8412, the background subtraction execution unit 213-1 converts the background image into a grayscale image.

[0062] In step S8413, background subtraction execution unit 213-1 calculates the difference between the grayscale observation image generated in step S8411 and the grayscale background image generated in step S8412, and generates a difference image.

[0063] In step S8511, the verification unit 214-1 applies binarization to the difference image generated in step S8413 to generate a binarized image. Specifically, for example, when a value of brightness or the like assigned to a pixel is less than a predetermined threshold, the verification unit 214-1 regards the difference between the observed image and the background image as not being significant and sets the pixel to black corresponding to "0". On the other hand, for example, when a value of brightness or the like assigned to a pixel is equal to or greater than a predetermined threshold, the verification unit 214-1 regards the difference between the observed image and the background image as being significant and sets the pixel to white corresponding to "1". Note that the verification unit 214-1 may generate a binarized image by applying not only binarization using black and white, binarization using "0" and "1", etc., but also binarization using correct and incorrect values.

[0064] In step S8512, the verification unit 214-1 calculates the ratio of white areas in the binarized image generated in step S8511.

[0065] In step S8513, verification unit 214-1 determines whether the ratio calculated in step S8512 is equal to or greater than a predetermined threshold. If verification unit 214-1 determines that the ratio calculated in step S8512 is equal to or greater than a predetermined threshold (step S8513: YES), the process proceeds to step S86 in Fig. 8. On the other hand, if verification unit 214-1 determines that the ratio calculated in step S8512 is less than the predetermined threshold (step S8513: NO), the process proceeds to step S81 in Fig. 8.

[0066] In the above description, an example is given in which in step S85, the verification unit 214-1 generates a binary image and uses the binary image to verify whether or not an object is reflected in the difference image, but this is not limited to the example.

[0067] For example, the verification unit 214-1 may execute the verification process in step S85 by using statistics related to the difference image. The statistics referred to here include, for example, variance, standard deviation, mean value, and median value.

[0068] In this case, in step S8511, verification unit 214-1 calculates the statistics of the difference image and the statistics of the background image, and in step S8512, calculates the difference between the statistics of the difference image and the statistics of the background image. Then, in step S8513, verification unit 214-1 determines whether this difference is equal to or greater than a predetermined threshold. If verification unit 214-1 determines that this difference is equal to or greater than the predetermined threshold (step S8513: YES), the process proceeds to step S86 in Fig. 8. On the other hand, if verification unit 214-1 determines that this difference is less than the predetermined threshold (step S8513: NO), the process proceeds to step S81 in Fig. 8.

[0069] Alternatively, the verification unit 214-1 may execute the verification process in step S85 by using a color distance related to the difference image. Also, the verification unit 214-1 may execute the verification process in step S85 by applying statistical processing to the difference image. The statistical processing here is, for example, a verification process that is executed after calculating a histogram from the difference image and analyzing the histogram.

[0070] Next, another example of detailed processing of steps S84 and S85 in Fig. 8 will be described with reference to Fig. 10. Fig. 10 is a flowchart showing an example of detailed processing of steps S84 and S85 in Fig. 8.

[0071] In step S8421, background subtraction execution unit 213-1 calculates the difference between the observed image and the background image, and generates a difference image.

[0072] In step S8521, the verification unit 214-1 calculates statistics regarding the background image.

[0073] In step S8522, the verification unit 214-1 calculates statistics regarding the difference image.

[0074] In step S8523, the verification unit 214-1 calculates the difference between the statistics regarding the background image calculated in step S8521 and the statistics regarding the difference image calculated in step S8522.

[0075] In step S8524, verification unit 214-1 determines whether the difference calculated in step S8523 is equal to or greater than a predetermined threshold. If verification unit 214-1 determines that the difference calculated in step S8523 is equal to or greater than the predetermined threshold (step S8523: YES), the process proceeds to step S86 in Fig. 8. On the other hand, if verification unit 214-1 determines that the difference calculated in step S8523 is less than the predetermined threshold (step S8523: NO), the process proceeds to step S81 in Fig. 8.

[0076] Next, an example of detailed processing of step S87 in Fig. 8 will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of detailed processing of step S87 in Fig. 8.

[0077] In step S8711, the background image generation unit 215-1 sets the observed image used for background subtraction as the observed image for each T frame.

[0078] In step S8712, the background image generation unit 215-1 sets the background image used for background subtraction as the background image for every T-1 frames.

[0079] In step S8713, the background image generation unit 215-1 calculates statistics of values ​​assigned to pixels between the observed image set in step S8711 and the background image set in step S8712, and sets the image in which these statistics are assigned to each pixel as a new background image.

[0080] In step S8714, background image generation unit 215-1 saves the new background image related to step S8713 in a storage medium.

[0081] In step S8715, the background image generation unit 215-1 adds "1" to T related to steps S8711 and S8712.

[0082] In step S8716, the background image generation unit 215-1 determines whether or not T exceeded a predetermined threshold in step S8715. If the background image generation unit 215-1 determines in step S8715 that T exceeded the predetermined threshold (step S8716: YES), the process proceeds to step S8717. On the other hand, if the background image generation unit 215-1 determines in step S8715 that T is equal to or less than the predetermined threshold (step S8716: NO), the process proceeds to step S88 in FIG.

[0083] In step S8717, the background image generation unit 215-1 sets T related to steps S8711 and S8712 to "1".

[0084] The background image generating unit 215-1 may execute the process shown in Fig. 11 for all observed images constituting the moving image. The background image generating unit 215-1 may also execute the process shown in Fig. 11 for observed images in which no object is detected by the object detection process among the observed images constituting the moving image.

[0085] Next, an example of detailed processing of step S87 in Fig. 8 will be described with reference to Fig. 12. Fig. 12 is a flowchart showing an example of detailed processing of step S87 in Fig. 8.

[0086] In step S8721, background image generation unit 215-1 creates a synthesis flag list.

[0087] In step S8722, background image generation unit 215-1 fills the composition flag list created in step S8721 with values ​​for which no flags are set.

[0088] In step S8723, background image generation unit 215-1 determines one object to be subjected to synthesis processing for synthesizing an area in the observed image in which no object is captured with an area in the background image excluding the area in question.

[0089] In step S8724, the background image generation unit 215-1 fills in an area of ​​the synthesis flag list that corresponds to the object area including the object that was the subject of synthesis processing in step S8723 with a value for which the flag is set.

[0090] In step S8725, background image generating unit 215-1 executes synthesis processing for the area filled with flagged values, and generates a background image.

[0091] In step S8726, the background image generation unit 215-1 determines whether or not the synthesis process has been performed for all objects detected by the object detection process. If the background image generation unit 215-1 determines that the synthesis process has been performed for all objects detected by the object detection process (step S8726: YES), the process proceeds to step S8727. On the other hand, if the background image generation unit 215-1 determines that the synthesis process has not been performed for all objects detected by the object detection process (step S8726: NO), the process returns to step S8723.

[0092] In step S8727, background image generation unit 215-1 saves the background image generated in step S8726 in a storage medium.

[0093] In the above-described embodiment, an example was given of a case in which the background image generation unit 215-1 uses at least one of an observation image in which no object was detected by the object detection process and an observation image in which it was verified by the verification process that no object is present as the background image, but this is not limited to the above.

[0094] For example, the background image generating unit 215-1 may calculate statistics of values ​​assigned to corresponding pixels between a predetermined number of observed images, and use an image in which the statistics are assigned to each pixel as the background image.

[0095] Alternatively, the background image generating unit 215-1 may calculate statistics of values ​​assigned to corresponding pixels among a plurality of observed images constituting an observed video on which an object detection process has been performed and which may include an object that should be considered when a moving body moves. In this case, the background image generating unit 215-1 sets an image in which these statistics are assigned to each pixel as the background image.

[0096] Alternatively, the background image generating unit 215-1 may calculate statistics of values ​​assigned to corresponding pixels between a plurality of observed images in which no object was detected by the object detection process or between a plurality of observed images in which it was verified by the verification process that no object is captured. In this case, the background image generating unit 215-1 sets the image in which these statistics are assigned to each pixel as the background image.

[0097] Alternatively, the background image generating unit 215-1 may use, as the background image, an image in which the weighting related to the contribution to the statistics assigned to each pixel becomes smaller as the date and time when the observed image was captured becomes older.

[0098] Alternatively, the background image generation unit 215-1 calculates statistics of values ​​assigned to corresponding pixels between a predetermined number of observed images and a predetermined number of background images, and sets the image in which the statistics are assigned to each pixel as the background image.

[0099] Alternatively, the background image generation unit 215-1 calculates statistics of values ​​assigned to corresponding pixels between at least one of an observed image in which no object was detected by the object detection process and an observed image verified by the verification process as not showing an object, and the background image. In this case, the background image generation unit 215-1 sets the image in which these statistics are assigned to each pixel as the background image.

[0100] Alternatively, background image generating section 215-1 may select an observed image, on which background subtraction has been performed with respect to a background image, whose difference from the background image falls within a predetermined range as a new background image.

[0101] Alternatively, the background image generating unit 215-1 may generate a new background image by combining an area of ​​an observation image in which an object has been detected by an object detection process, in which the object is not included, with an area of ​​the background image excluding the area.

[0102] The present invention includes the following inventions that appropriately combine the above-mentioned features.

[0103] (Configuration 1) An object detection device comprising: a background difference execution unit that generates a difference image by performing background difference using an observation image in which an object may be captured and in which an object detection process has been performed to detect the object, and a background image in which the object is not captured; and a verification unit that executes a verification process to verify whether the object captured in the difference image should be taken into consideration when a moving body moves.

[0104] (Configuration 2) 2. The object detection device according to configuration 1, further comprising a background image generation unit that generates the background image.

[0105] (Configuration 3) The object detection device according to configuration 2, wherein the background image generation unit sets at least one of the observation image in which the object is not detected by the object detection processing and the observation image in which it is verified by the verification processing that the object is not included as the background image.

[0106] (Configuration 4) The object detection device according to configuration 3, wherein the background image generation unit calculates statistics of values ​​assigned to corresponding pixels among a predetermined number of the observed images, and sets an image in which the statistics are assigned to each pixel as the background image.

[0107] (Configuration 5) The object detection device according to configuration 2, wherein the background image generation unit calculates statistics of values ​​assigned to corresponding pixels among a plurality of the observation images constituting an observational video in which the object detection process has been executed and which have a possibility of showing the object, and sets an image in which the statistics are assigned to each pixel as the background image.

[0108] (Configuration 6) the background image generation unit calculates statistics of values ​​assigned to corresponding pixels between a plurality of the observed images in which the object was not detected by the object detection processing or between a plurality of the observed images in which the object is verified not to appear by the verification processing, and sets an image in which the statistics are assigned to each pixel as the background image.

[0109] (Configuration 7) The object detection device according to any one of configurations 4 to 6, wherein the background image generation unit uses an image as the background image in which a weight regarding a contribution to the statistics assigned to each pixel is smaller as the date and time when the observed image was captured is older.

[0110] (Configuration 8) The object detection device according to configuration 2, wherein the background image generation unit calculates statistics of values ​​assigned to corresponding pixels between a predetermined number of the observed images and a predetermined number of the background images, and sets an image in which the statistics are assigned to each pixel as the background image.

[0111] (Configuration 9) The object detection device according to configuration 2, wherein the background image generation unit calculates statistics of values ​​assigned to corresponding pixels between at least one of the observation image in which the object was not detected by the object detection processing and the observation image in which it was verified by the verification processing that the object is not included, and the background image, and sets an image in which the statistics are assigned to each pixel as the background image.

[0112] (Configuration 10) The object detection device according to configuration 2, wherein the background image generation unit sets, as a new background image, an observed image on which the background subtraction has been performed with respect to the background image, the observed image having a difference from the background image that falls within a predetermined range.

[0113] (Configuration 11) The object detection device according to configuration 2, wherein the background image generation unit generates a new background image by combining an area of ​​the observation image in which the object is detected by the object detection process, in which the object is not included, with an area of ​​the background image excluding the area.

[0114] (Configuration 12) The object detection device according to any one of configurations 1 to 11, wherein the background subtraction execution unit generates the difference image by performing the background subtraction using an object region including the object detected by the object detection process and a background region in the background image that corresponds to the object region.

[0115] (Configuration 13) The object detection device according to any one of configurations 1 to 12, wherein the verification unit generates a binary image by applying binarization to the background image, and verifies whether the object is reflected in the difference image using the binary image.

[0116] (Configuration 14) 13. The object detection device according to any one of configurations 1 to 12, wherein the verification unit performs the verification process using statistics related to the difference image.

[0117] (Configuration 15) 13. The object detection device according to any one of configurations 1 to 12, wherein the verification unit performs the verification process using a color distance related to the difference image.

[0118] (Configuration 16) 13. The object detection device according to any one of configurations 1 to 12, wherein the verification unit performs the verification process by applying statistical processing to the difference image.

[0119] (Configuration 17) The object detection device according to any one of configurations 1 to 16, wherein the verification unit performs the verification process in order from the object closest to the moving object in the difference image.

[0120] (Program 1) An object detection program that generates a difference image by performing background difference using an observation image in which an object may be captured and in which an object detection process for detecting the object has been performed, and a background image in which the object is not captured, and then performs a verification process to verify whether the object captured in the difference image should be taken into consideration when a moving body moves.

[0121] (Method 1) An object detection method comprising: performing background difference between an observed image in which an object may be captured and in which an object detection process for detecting the object has been performed, and a background image in which the object is not captured, to generate a difference image; and then performing a verification process for verifying whether or not the object captured in the difference image should be taken into consideration when a moving body moves.

[0122] (System 1) An object detection system comprising: a background difference execution unit that generates a difference image by performing background difference using an observation image in which an object may be captured and in which an object detection process has been performed to detect the object, and a background image in which the object is not captured; and a verification unit that executes a verification process to verify whether the object captured in the difference image should be taken into consideration when a moving body moves.

[0123] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit for implementing one or more of the functions, such as an ASIC (Application Specific Integrated Circuit).

[0124] The above describes preferred embodiments of the present invention. However, the present invention is not limited to the above-described embodiments. In other words, the present invention includes embodiments in which various modifications have been made based on the gist of the present invention, and these embodiments are not excluded from the scope of the present invention. [Explanation of symbols]

[0125] 213-1, ..., 213-k: Background subtraction execution unit 214-1, ..., 214-k: Verification section 215-1, ..., 215-k: Background image generation unit

Claims

1. A background difference execution unit that generates a difference image by performing background difference using an observation image that is generated by photographing a moving body or a place where the moving body moves, and that may contain an object and that has been subjected to an object detection process to detect the object, and a background image that does not contain the object; a verification unit that executes a verification process to verify whether or not the object captured in the difference image should be taken into consideration when the moving body moves; An object detection device comprising:

2. A background image generating unit that generates the background image is further provided. The object detection device according to claim 1 .

3. the background image generation unit sets at least one of the observation image in which the object is not detected by the object detection process and the observation image in which it is verified by the verification process that the object is not included as the background image. The object detection device according to claim 2 .

4. the background image generating unit calculates statistics of values ​​assigned to corresponding pixels among a predetermined number of the observed images, and defines an image in which the statistics are assigned to each pixel as the background image. The object detection device according to claim 3 .

5. the background image generation unit calculates statistics of values ​​assigned to corresponding pixels among a plurality of the observed images constituting the observed video on which the object detection process has been executed, and determines an image in which the statistics are assigned to each pixel as the background image, the background image being a plurality of observed images that may include the object. The object detection device according to claim 2 .

6. the background image generation unit calculates statistics of values ​​assigned to corresponding pixels between a plurality of the observation images in which the object is not detected by the object detection process or between a plurality of the observation images in which it is verified by the verification process that the object is not included, and defines an image in which the statistics are assigned to each pixel as the background image. The object detection device according to claim 2 .

7. the background image generating unit uses an image as the background image in which a weight related to a contribution to the statistics assigned to each pixel is smaller as the date and time when the observed image was captured is older. The object detection device according to any one of claims 4 to 6.

8. the background image generation unit calculates statistics of values ​​assigned to corresponding pixels between a predetermined number of the observation images and a predetermined number of the background images, and defines an image in which the statistics are assigned to each pixel as the background image. The object detection device according to claim 2 .

9. the background image generation unit calculates statistics of values ​​assigned to corresponding pixels between at least one of the observation image in which the object is not detected by the object detection process and the observation image in which it is verified by the verification process that the object is not included, and the background image, and defines an image in which the statistics are assigned to each pixel as the background image. The object detection device according to claim 2 .

10. the background image generating unit sets, as a new background image, an observation image on which the background subtraction has been performed with respect to the background image, the observation image having a difference from the background image within a predetermined range. The object detection device according to claim 2 .

11. the background image generating unit generates a new background image by combining a region of the observation image in which the object is detected by the object detection process, in which the object is not included, with a region of the background image excluding the region of the observation image. The object detection device according to claim 2 .

12. the background subtraction execution unit performs the background subtraction using an object region including the object detected by the object detection process and a background region in the background image that corresponds to the object region, thereby generating the difference image. The object detection device according to any one of claims 1 to 6 and claims 8 to 11.

13. the verification unit generates a binary image by applying binarization to the background image, and verifies whether the object is reflected in the difference image by using the binary image; The object detection device according to claim 12.

14. the verification unit performs the verification process using statistics related to the difference image. The object detection device according to any one of claims 1 to 6 and claims 8 to 11.

15. The verification unit performs the verification process using a color distance related to the difference image. The object detection device according to any one of claims 1 to 6 and claims 8 to 11.

16. The verification unit performs the verification process by applying a statistical process to the difference image. The object detection device according to any one of claims 1 to 6 and claims 8 to 11.

17. the verification unit performs the verification process in order from the object closest to the moving object in the difference image; The object detection device according to any one of claims 1 to 6 and claims 8 to 11.

18. A difference image is generated by performing background subtraction using an observation image, which is generated by photographing a moving body or a place where the moving body moves, and which may contain an object and in which an object detection process is performed to detect the object, and a background image in which the object is not captured, performing a verification process for verifying whether or not the object shown in the difference image should be taken into consideration when the moving body moves; Object detection program.

19. A difference image is generated by performing background subtraction using an observation image, which is generated by photographing a moving body or a place where the moving body moves, and which may contain an object and in which an object detection process is performed to detect the object, and a background image in which the object is not captured, performing a verification process for verifying whether or not the object shown in the difference image should be taken into consideration when the moving body moves; Object detection methods.

20. An object detection unit that detects an object in an observation image generated by photographing a moving object or a place where the moving object moves; an object region setting unit that sets a region including the object detected by the object detection unit from the observation image as an object region; a difference image generating unit that performs background subtraction processing on the object region using the observed image and a background image not showing an object, to generate a difference image; An object detection device comprising:

Images