Image processing apparatus, image processing method, and image processing program

Abstract

To provide an image processing apparatus, a method, and a program for accurately detecting a region of a target object from an image.SOLUTION: In an image processing apparatus 1, a processing unit 2 is configured to: detect a detection region 11 of a predetermined shape including a predetermined kind of target object 21 from a captured image 10; detect closed regions from the detection region 11 by performing contour detection on the detection region 11; and identify a closed region having the maximum area from among the detected closed regions, as a region in which the target object 21 appears. Accordingly, even if another object 22 of a kind which is different from that of the target object 21 appears in a position closed to the target object 21, the region of the target object 21 can be accurately detected.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and an image processing program. [Background technology]

[0002] 2. Description of the Related Art In recent years, a technology has become widespread in which a camera is used to capture an image of a monitoring target area, and the captured image is processed to detect a desired target. For example, there is a technology for detecting moving objects such as vehicles and people.

[0003] As for detection of moving objects, for example, a moving object monitoring device has been proposed that extracts a changed area in an image based on the difference between input image information and a background image in a state where a moving object is not present, and judges the presence or absence of a moving object based on the size, brightness, shape, etc. of the changed area. As for creation of a road surface background image, for example, a moving object removal method has been proposed that creates a moving object mask image based on moving object mask information created from a road surface image, and thereafter uses the initial moving object mask image as a base and repeats image synthesis by updating only the portion of the moving object mask image other than the mask area. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2004-208209 A [Patent Document 2] JP 2003-296709 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, there is a technique for detecting an object from a captured image and outputting an area of ​​a predetermined shape including the detected object from the captured image as the detection result of the object. However, this technique has a problem in that when another object, which is not the detection target and is different from the object, is present in a position close to the object and is included in the area of ​​the predetermined shape, the area of ​​the object cannot be distinguished from the area of ​​the other object.

[0006] In one aspect, the present invention aims to provide an image processing device, an image processing method, and an image processing program capable of accurately detecting an object region from an image. [Means for solving the problem]

[0007] In one proposal, an image processing device having the following processing unit is provided: In this image processing device, the processing unit detects a detection area of ​​a predetermined shape including a predetermined type of object from a captured image, detects one or more closed areas from the detection area by performing contour detection on the detection area, and identifies a first closed area having the largest area among the one or more closed areas as the area in which the object is captured.

[0008] Also, one proposal provides an image processing method in which a computer executes the same processing as the above-mentioned image processing device. Furthermore, in one proposal, an image processing program is provided that causes a computer to execute the same processing as the above-mentioned image processing device. Effect of the Invention

[0009] In one aspect, it is possible to accurately detect the region of an object from an image. [Brief description of the drawings]

[0010] [Figure 1] 1 illustrates a configuration example and a processing example of an image processing device according to a first embodiment; [Diagram 2] FIG. 13 illustrates an example of a configuration of an image processing system according to a second embodiment. [Diagram 3]FIG. 2 is a diagram illustrating an example of a hardware configuration of an image processing apparatus. [Figure 4] FIG. 2 is a diagram illustrating an example of the configuration of processing functions included in an image processing apparatus. [Diagram 5] FIG. 11 is a diagram showing a first comparative example of a moving object erasing process using image interpolation. [Figure 6] FIG. 11 is a diagram showing a second comparative example of the moving object erasing process using image interpolation. [Figure 7] FIG. 13 is a diagram illustrating an example of a moving object erasure process in the second embodiment. [Figure 8] 1 is a first example of a flowchart illustrating a procedure for detecting an abnormal object. [Figure 9] 13 is a second example of a flowchart illustrating a procedure for detecting an abnormal object. [Figure 10] 13 is a third example of a flowchart illustrating a procedure for detecting an abnormal object. [Figure 11] 13A and 13B are diagrams illustrating an example of a process for determining whether or not a bounding box overlaps; [Figure 12] 13 is a flowchart illustrating an example of a moving object erasing process using image interpolation. [Figure 13] FIG. 13 is a diagram illustrating an example of an abnormal object detection process using erased frames. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. First Embodiment Fig. 1 is a diagram showing a configuration example and a processing example of an image processing device according to a first embodiment. The image processing device 1 shown in Fig. 1 is a device that detects an area of ​​a predetermined type of object from an input captured image. For example, a moving object such as a vehicle or a person is detected as the object.

[0012] The image processing device 1 includes a processing unit 2. The processing unit 2 is, for example, a processor. In this case, the processing of the processing unit 2 is realized, for example, by the processor executing a predetermined program. The processing unit 2 executes the following processes.

[0013] The processing unit 2 detects a detection area of ​​a predetermined shape including the above-mentioned object from the input captured image. Fig. 1 illustrates an example in which a detection area 11 including an object 21 is detected from a captured image 10. The detection area 11 in Fig. 1 has a rectangular shape. In this case, the detection area 11 may be called a "bounding box."

[0014] As a method for detecting the object 21, for example, an image recognition technology for recognizing a vehicle, a person, etc. can be used. For example, the object 21 can be detected using a trained model for image recognition generated by machine learning.

[0015] 1, another object 22, which is not a detection target and is of a different type from the object 21, is present in a position close to the object 21. Since the object 22 and the object 21 are close to each other, the object 22 is included in the detection area 11. In this case, not only the object 21 but also the area of ​​the object 22 is detected as the area of ​​the object 21 in the detection area 11, and the area of ​​the object 21 cannot be detected separately from the area of ​​the object 22.

[0016] In order to solve such a problem, the processing unit 2 detects one or more closed regions from the detected detection area 11 by performing contour detection on the detected detection area 11. When multiple closed regions are detected, the processing unit 2 identifies the closed region with the largest area among those closed regions as the region in which the object 21 is captured.

[0017] In the image 10a shown in Fig. 1, a contour line detected in the detection area 11 is shown. In this example, a closed area 21a corresponding to the object 21 and a closed area 22a corresponding to the object 22 are detected in the detection area 11 by contour detection. The area of ​​the closed area 21a is larger than that of the closed area 22a. Therefore, the processing unit 2 specifies the closed area 21a as the area in which the object 21 is captured. In the image 10b shown in Fig. 1, the specified area 21b of the object 21 is shown in black.

[0018] The detection area 11 is generated as a result of detecting a desired object 21, and indicates an area in which the object 21 exists. Therefore, the object 21 to be detected is captured large in the detection area 11, and it is unlikely that other objects that are not the object of detection are captured larger than the object 21. By specifying the closed area 21a with the maximum area as the area in which the object 21 is captured as described above, even if the other objects are close to the object 21, it is possible to accurately detect the area in which the object 21 is captured by distinguishing it from the other objects.

[0019] Incidentally, by detecting only the area of ​​the object 21 from the captured image 10 as described above, it becomes possible to generate an image in which the object 21 has been accurately erased from the captured image 10. Since an image in which only the object 21 has been erased from the captured image 10 contains only other objects that are not the detection target (for example, an abnormal object that should not be there), this image can be used as an image for detecting the presence or absence of other objects.

[0020] In the second embodiment described below, an image processing system that can detect other objects using the above technology is illustrated. In the image processing system described below, a predetermined type of moving object is detected as the target object, and an abnormal object other than the predetermined type of moving object is detected as the other object.

[0021] Second Embodiment Fig. 2 is a diagram showing an example of the configuration of an image processing system according to a second embodiment. The image processing system shown in Fig. 2 includes a camera 50 and an image processing device 100. Note that the image processing device 100 is an example of the image processing device 1 shown in Fig. 1.

[0022] The camera 50 is a surveillance camera that monitors an area in which a moving object moves. For example, if the moving object is a vehicle, the camera 50 monitors a road on which the vehicle travels. The camera 50 transmits captured video to the image processing device 100.

[0023] The image processing device 100 receives video from the camera 50 and detects a moving object from the received video. The image processing device 100 further generates an image in which the moving object is not captured but other abnormal objects may be captured by deleting the image of the moving object from the image frame in which the moving object is detected, and detects the abnormal object from the generated image. For example, an object that has fallen from a vehicle is detected as an abnormal object.

[0024] Fig. 3 is a diagram showing an example of the hardware configuration of an image processing device. The image processing device 100 is realized, for example, as a computer as shown in Fig. 3. The image processing device 100 shown in Fig. 3 includes a processor 101, a random access memory (RAM) 102, a hard disk drive (HDD) 103, a graphics processing unit (GPU) 104, an input interface (I / F) 105, a reader 106, and a communication interface (I / F) 107.

[0025] The processor 101 performs overall control of the entire image processing device 100. The processor 101 is, for example, a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), a GPU, an application specific integrated circuit (ASIC), or a programmable logic device (PLD). The processor 101 may also be a combination of two or more elements of a CPU, an MPU, a DSP, a GPU, an ASIC, or a PLD.

[0026] The RAM 102 is used as a main storage device of the image processing device 100. The RAM 102 temporarily stores at least a part of an OS (Operating System) program and application programs to be executed by the processor 101. The RAM 102 also stores various data necessary for processing by the processor 101.

[0027] The HDD 103 is used as an auxiliary storage device for the image processing device 100. An OS program, an application program, and various data are stored in the HDD 103. Note that other types of non-volatile storage devices such as an SSD (Solid State Drive) can also be used as the auxiliary storage device.

[0028] A display device 104a is connected to the GPU 104. The GPU 104 displays an image on the display device 104a in accordance with an instruction from the processor 101. As the display device 104a, a liquid crystal display, an organic EL (ElectroLuminescence) display, or the like can be used.

[0029] An input device 105a is connected to the input interface 105. The input interface 105 transmits a signal output from the input device 105a to the processor 101. A keyboard, a pointing device, or the like can be used as the input device 105a. A mouse, a touch panel, a tablet, a touch pad, a trackball, or the like can be used as the pointing device.

[0030] A portable recording medium 106a is detachably attached to the reading device 106. The reading device 106 reads data recorded on the portable recording medium 106a and transmits it to the processor 101. The portable recording medium 106a may be an optical disk, a semiconductor memory, or the like.

[0031] The communication interface 107 transmits and receives data to and from other devices via the network 107a. For example, when the camera 50 is connected to the image processing device 100 via the network 107a, the communication interface 107 receives video data transmitted from the camera 50.

[0032] The processing functions of the image processing device 100 can be realized by the above hardware configuration. Fig. 4 is a diagram showing a configuration example of processing functions of the image processing device. As shown in Fig. 4, the image processing device 100 includes an image storage unit 111, a BB (bounding box) data storage unit 112, an erased image storage unit 113, an image acquisition unit 121, a moving object detection unit 122, a moving object erasure unit 123, and an abnormal object detection unit 124.

[0033] The image storage unit 111, the BB data storage unit 112, and the erased image storage unit 113 are storage areas secured in a storage device included in the image processing device 100, such as the RAM 102, the HDD 103, or the like.

[0034] In the image storage unit 111, data of moving images acquired by the camera 50 (that is, data of each image frame) is stored. The BB data storage unit 112 stores BB data related to a bounding box detected from an image frame. The bounding box is a rectangular area that contains a moving object, and is detected by the moving object detection unit 122. The BB data includes frame identification information (e.g., image capture time or frame number) indicating the image frame in which the bounding box was detected, and position information of the detected bounding box (e.g., the top left coordinate, top right coordinate, and bottom left coordinate of the rectangular area).

[0035] The erased image storage unit 113 stores data of moving images from which the moving object erasure process has been performed by the moving object erasure unit 123 . The processes of the image acquisition unit 121, the moving body detection unit 122, the moving body elimination unit 123, and the abnormal object detection unit 124 are realized, for example, by the processor 101 executing a predetermined application program.

[0036] The image acquisition unit 121 acquires data of moving images captured by the camera 50. The image acquisition unit 121 sequentially passes data of image frames included in the moving images to the moving object detection unit 122 and stores the data in the image storage unit 111.

[0037] The moving object detection unit 122 detects a moving object from an image frame. When the moving object detection unit 122 detects a moving object, the moving object detection unit 122 generates BB data related to a bounding box that contains the moving object, and stores the BB data in the BB data storage unit 112.

[0038] The moving body detection unit 122 detects a moving body using, for example, a trained model for detecting an object. Such a trained model is generated by, for example, machine learning (for example, deep learning) using a large number of images containing objects as training data. The training data includes position information of a bounding box corresponding to the position of the object, and the position information is used as ground truth data during machine learning. As such a moving body detection technology, for example, YOLO (You Only Look Once), SSD (Single Shot Detector), R-CNN (Region-based Convolutional Neural Network), etc. can be used.

[0039] The moving object erasing unit 123 erases the moving object from the image frame based on the BB data. The moving object erasing unit 123 stores data of each image frame from which the moving object erasing process has been performed in the erased image storage unit 113. In the following description, an image frame from which the moving object erasing process has been performed may be referred to as an "erased frame."

[0040] The abnormal object detector 124 detects abnormal objects from the image frames by performing a difference calculation between the erased frames. Next, the process of erasing a moving object will be described. The image processing device 100 of this embodiment detects a moving object in order to detect an abnormal object. That is, when a moving object is detected from an image frame, the image processing device 100 erases the moving object from the image frame. In each image frame in which such a process of erasing a moving object has been performed, the moving object is not captured, and only abnormal objects other than the moving object may be captured. Therefore, by calculating the difference between erased frames in which the process of erasing a moving object has been performed, it becomes possible to detect an abnormal object.

[0041] Fig. 5 is a diagram showing a first comparative example of a moving object erasing process using image interpolation. A moving object 201 is captured in an image frame 200 shown in Fig. 5. In this example, the moving object 201 is a vehicle traveling on a road. The moving object detection unit 122 executes a moving object detection process for the image frame 200. As a result, a bounding box 202 including the moving object 201 is detected. An image 200a shown in Fig. 5 is an image in which the bounding box 202 is superimposed on the image frame 200.

[0042] In the first comparative example, the moving object is erased from the image frame 200 by image interpolation. Specifically, the pixel values ​​of each pixel in the bounding box 202 are calculated by an interpolation operation using pixel values ​​of pixels around the bounding box 202. Then, the pixel values ​​of each pixel in the bounding box 202 are replaced by the calculated pixel values. For example, the nearest neighbor method, bilinear interpolation, or the like can be used as the interpolation operation method. By such processing, an erased frame 200b in which the moving object 201 is erased from the image frame 200 is generated.

[0043] Such a moving object erasure process has an advantage that when a moving object is detected from an image frame, the moving object can be erased using only that image frame. For example, as described later, there is a method of erasing a moving object by replacing an image in a bounding box in an image frame with an image in a bounding box in an image frame at a past time when no moving object was detected. This method cannot be applied when there is no image frame at a past time when no moving object was detected. For example, when a road where many vehicles frequently pass as moving objects is photographed, it may be difficult to find an image frame in which no vehicles are detected. The moving object erasure process using image interpolation as shown in FIG. 5 can erase the moving object even in such a case.

[0044] However, the moving object erasure process using image interpolation has a problem in that, if an abnormal object is present in a position close to the moving object, not only the moving object but also the abnormal object may be erased. Fig. 6 is a diagram showing a second comparative example of the moving object erasing process using image interpolation. In an image frame 210 shown in Fig. 6, a moving object 211 is captured, and an abnormal object 212 is captured in a position close to the moving object 211. It is assumed that the moving object detection unit 122 detects a bounding box 213 including the moving object 211 from such an image frame 210. An image 210a shown in Fig. 6 is an image in which the bounding box 213 is superimposed on the image frame 210.

[0045] In this example, since the abnormal object 212 is close to the moving object 211, not only the moving object 211 but also the abnormal object 212 is included in the bounding box 213. In such a case, when the moving object erasing unit 123 performs an erasing process of the moving object 211 using image interpolation, not only the moving object 211 but also the abnormal object 212 is erased as shown in the erased frame 210b. For this reason, the abnormal object detection unit 124 cannot detect the abnormal object 212 using the erased frame 210b.

[0046] 7 is a diagram showing an example of moving object erasing processing in the second embodiment. To address the above problem, the moving object erasing unit 123 in this embodiment executes the moving object erasing processing by using contour detection.

[0047] 7, it is assumed that a bounding box 213 including a moving object 211 and an abnormal object 212 is detected from an image frame 210, similarly to FIG. 6. An image 210a is an image in which the bounding box 213 is superimposed on the image frame 210. In this case, the moving object erasing unit 123 executes a contour detection process on the image within the bounding box 213. The moving object erasing unit 123 performs edge detection from the image within the bounding box 213 using, for example, a Sobel filter or a Canny method, and extracts a contour line based on the obtained edge information.

[0048] As a result of such contour detection processing, as shown in an image 210c, a closed region 211a corresponding to the moving object 211 and a closed region 212a corresponding to the abnormal object 212 are detected from a bounding box 213. The closed regions 211a and 212a are regions surrounded by contour lines.

[0049] Next, the moving object erasing unit 123 compares the sizes of the detected closed regions 211a and 212a. Specifically, the area of ​​each of the closed regions 211a and 212a or the area occupancy rate of the bounding box 213 is calculated, and the areas or the area occupancy rates are compared. The areas can be calculated as the number of pixels included in the closed region, for example.

[0050] 7, the area of ​​the bounding box 213 is calculated to be 300. The area of ​​the closed region 211a is calculated to be 230, and the area occupancy rate of the closed region 211a to the bounding box 213 is calculated to be 77%. On the other hand, the area of ​​the closed region 212a is calculated to be 40, and the area occupancy rate of the closed region 212a to the bounding box 213 is calculated to be 13%.

[0051] The moving object erasing unit 123 identifies the area with the largest area among the closed areas detected from the bounding box 213 as the area of ​​the moving object 211. In the example of FIG. 7, the area of ​​the closed area 211a is larger than that of the closed area 212a, so the closed area 211a is identified as the area of ​​the moving object 211. Then, the moving object erasing unit 123 erases the closed area 211a from the image frame 210 by image interpolation. Specifically, the moving object erasing unit 123 calculates the pixel value of each pixel in the closed area 211a by an interpolation operation using pixels surrounding the closed area 211a, and replaces the pixel value of each pixel in the closed area 211a with the calculated pixel value.

[0052] 7 is an image frame in which the pixel values ​​of the pixels in the closed region 211a of the original image frame 210 have been replaced with pixel values ​​generated by an interpolation calculation. In this erased frame 210d, the region of the moving object 211 has been accurately erased, while the abnormal object 212 close to the moving object 211 remains without being erased.

[0053] A bounding box is generated as a result of detecting a certain type of moving object, and indicates the area in which the moving object exists. Therefore, the moving object to be detected is captured large in the bounding box, and it is unlikely that an abnormal object that is not the target of detection will be captured larger than the moving object. By identifying the closed area with the largest area as the area in which the moving object is captured as described above, and erasing only the image information of this area by image interpolation, it is possible to erase only the moving object without erasing the abnormal object, even if the abnormal object is close to the moving object.

[0054] 8 to 10 are examples of flowcharts showing the procedure of the abnormal object detection process. [Step S11] The image acquisition unit 121 acquires one image frame in a moving image captured by the camera 50, and passes it to the moving object detection unit 122 and stores it in the image storage unit 111. Hereinafter, the acquired image frame may be referred to as the "current frame."

[0055] [Step S12] The moving object detection unit 122 detects moving objects in the current frame. [Step S13] The moving object detection unit 122 determines whether one or more moving objects have been detected in the current frame. If one or more moving objects have been detected, the moving object detection unit 122 generates BB data relating to a bounding box including each detected moving object, and stores the BB data in the BB data storage unit 112. Thereafter, the process proceeds to step S21 in Fig. 9. On the other hand, if no moving object has been detected, the process proceeds to step S14.

[0056] [Step S14] The moving object erasure unit 123 stores the current frame as an erased frame in the erased image storage unit 113. After that, the process proceeds to step S35 in FIG.

[0057] The explanation will be continued below with reference to FIG. [Step S21] The moving object elimination processing loop up to step S29 is repeated the number of times equal to the number of bounding boxes detected. In the moving object elimination processing loop, the bounding box of the processing target detected from the current frame may be referred to as the "target bounding box."

[0058] [Step S22] The moving object removal unit 123 selects one image frame (past frame) that is temporally earlier than the current frame. Every time step S22 is executed in the moving object removal processing loop, the image frame immediately preceding the current frame is selected one frame earlier.

[0059] [Step S23] The moving object removal unit 123 determines whether one or more bounding boxes have been detected from the past frame selected in step S22. If one or more pieces of BB data corresponding to the past frame are stored in the BB data storage unit 112, it is determined that one or more bounding boxes have been detected. If one or more bounding boxes have been detected, the process proceeds to step S24; if no bounding boxes have been detected, the process proceeds to step S26.

[0060] [Step S24] The moving object remover 123 determines whether at least a portion of the bounding box detected from the previous frame overlaps with the target bounding box. If the former area overlaps with the latter area, the process proceeds to step S25. If they do not overlap, the process proceeds to step S26.

[0061] An example of the overlap determination process in step S24 will now be described with reference to Fig. 11. Fig. 11 is a diagram showing an example of the overlap determination process for bounding boxes. In Fig. 11, for the area of ​​the target bounding box detected from the current frame, the upper left is designated as point A, the upper right as point B, and the lower left as point C. For the area of ​​the bounding box detected from the past frame (past bounding box), the upper left is designated as point D, the upper right as point E, and the lower left as point F. Note that the coordinate axis extending from the left to the right in Fig. 11 is the X axis, the coordinate axis extending from the top to the bottom in Fig. 11 is the Y axis, and the top left of the image frame is the origin of the coordinates.

[0062] 11, five past bounding boxes 231-235 are detected from the past frames, and the moving object removal unit 123 determines whether at least a portion of the area of ​​the past bounding boxes 231-235 overlaps with the target bounding box 221 that is the determination target. In this determination, of the past bounding boxes 231-235, any past bounding box that satisfies all of the following conditions C1-C4 is determined to overlap with the target bounding box 221. On the other hand, any past bounding box that does not satisfy at least one of the conditions C1-C4 is determined to not overlap with the target bounding box 221.

[0063] Condition C1 is "X coordinate of point A of the target bounding box≦X coordinate of point E of the past bounding box." A past bounding box that does not satisfy condition C1 is a past bounding box that is located to the left of the target bounding box 221 and does not overlap with the target bounding box 221. In FIG. 11, past bounding box 231 that does not satisfy condition C1 is determined to not overlap, and past bounding box 231 is excluded from the determination targets.

[0064] Condition C2 is "X coordinate of point B of the target bounding box≧X coordinate of point D of the past bounding box." A past bounding box that does not satisfy condition C2 is a past bounding box that is located to the right of the target bounding box 221 and does not overlap with the target bounding box 221. In FIG. 11, of the past bounding boxes 232 to 225 that remain as determination targets, past bounding box 235 that does not satisfy condition C2 is determined to not overlap, and past bounding box 235 is excluded from determination targets.

[0065] Condition C3 is “Y coordinate of point A of the target bounding box≦Y coordinate of point F of the past bounding box.” A past bounding box that does not satisfy condition C3 is a past bounding box that is located above the target bounding box 221 and does not overlap with the target bounding box 221.

[0066] Condition C4 is “Y coordinate of point C of the target bounding box≧Y coordinate of point D of the past bounding box.” A past bounding box that does not satisfy condition C4 is a past bounding box that is located below the target bounding box 221 and does not overlap with the target bounding box 221.

[0067] 11, there is no past bounding box that does not satisfy conditions C3 and C4 among the past bounding boxes 232 to 234 remaining as determination targets. For this reason, it is determined that the remaining past bounding boxes 232 to 234 overlap with the target bounding box 221.

[0068] The explanation will be continued below with reference to FIG. [Step S25] The moving object erasure unit 123 determines whether there are any past frames that have not been selected in step S22. If there are any unselected past frames, the process proceeds to step S22, where the latest (latest) past frame among the undetermined past frames is selected. On the other hand, if all past frames have been selected, the process proceeds to step S28.

[0069] [Step S26] The moving object remover 123 extracts an image within the same area as the target bounding box from the previous frame most recently selected in step S22. [Step S27] The moving object remover 123 replaces the image in the target bounding box with the image extracted in step S26.

[0070] [Step S28] The moving object remover 123 performs a moving object removal process using image interpolation on the target bounding box. [Step S29] If the moving object removal processing loop has been executed the number of times equal to the number of detected bounding boxes, the process proceeds to step S31 in FIG.

[0071] 12 is a flowchart showing an example of a moving object erasing process by image interpolation. The process in FIG. 12 corresponds to the process in step S28 in FIG. [Step S41] The moving object remover 123 performs a contour detection process on the image within the target bounding box detected from the current frame. This process detects one or more closed regions surrounded by a contour line.

[0072] [Step S42] The moving object erasing unit 123 determines whether two or more closed regions are detected by the process of step S41. If two or more closed regions are detected, the process proceeds to step S43. On the other hand, if only one closed region is detected, the detected closed region is identified as the region of the moving object, and the process proceeds to step S45.

[0073] [Step S43] The moving object remover 123 calculates the area of ​​each detected closed region. [Step S44] The moving object erasing unit 123 identifies, among the detected closed regions, the closed region with the largest calculated area as the region of the moving object.

[0074] [Step S45] The moving object erasing unit 123 erases the moving object by image interpolation processing. Specifically, the moving object erasing unit 123 calculates the pixel value of each pixel in the closed region by interpolation calculation using the pixels of the current frame surrounding the closed region identified as the region of the moving object. The moving object erasing unit 123 replaces the pixel value of each pixel in the closed region with the calculated pixel value of each pixel.

[0075] 12, the moving object erasing unit 123 can specify only the moving object area, distinguishing it from the area of ​​the abnormal object, even if an abnormal object exists in a position close to the moving object. Therefore, the moving object erasing unit 123 can erase the moving object while leaving the abnormal object in the vicinity of the moving object.

[0076] 9, if there is a past frame in which no bounding box has been detected from an area overlapping with the target bounding box, the image in the target bounding box is replaced with the image in the same area as the target bounding box in the past frame (step S27).On the other hand, if there is no such past frame, the moving object erasure process is performed by image interpolation using only the image data of the current frame (step S28).

[0077] In the moving object erasure process in step S28, the image in the moving object region is replaced with an image generated by an interpolation calculation. Since the replaced image is generated by an interpolation calculation, it does not completely match the actual background image captured by the camera 50 when the moving object is removed in the real space. If there is a large difference between the replaced image and the actual background image, the detection accuracy of the abnormal object using the erased frame after the moving object is erased may deteriorate.

[0078] On the other hand, in the moving object erasure process in step S27, the image after replacement in the target bounding box is likely to match the actual background image when the moving object is removed. Therefore, the detection accuracy of abnormal objects using the erased frame after the moving object is erased is considered to be relatively high.

[0079] For this reason, in the above-mentioned moving object erasure processing loop, the moving object erasure processing by image interpolation is executed only when there is no past frame in which no bounding box has been detected from an area overlapping with the target bounding box. This makes it possible to suppress a decrease in the accuracy of detecting abnormal objects using the erased frames after the moving object has been erased.

[0080] The explanation will be continued below with reference to FIG. [Step S31] The moving object erasing unit 123 stores in the erased image storage unit 113 the erased frame in which the moving object has been erased from the current frame by the processing in step S27 or step S .

[0081] [Step S32] The abnormal object detection unit 124 acquires an erased frame corresponding to a past image frame (a past erased frame) from the erased image storage unit 113. As the past erased frame, for example, an erased frame corresponding to an image frame captured a predetermined time before the current frame (for example, an image frame one second before) is acquired. The abnormal object detection unit 124 calculates the difference in pixel value at the same pixel between the current erased frame (the erased frame stored in step S31) and the acquired past erased frame.

[0082] [Step S33] The abnormal object detection unit 124 determines whether or not there was a difference in the calculation in step S32. If there was a difference, the process proceeds to step S34. If there was no difference, the process proceeds to step S35.

[0083] [Step S34] The abnormal object detection unit 124 outputs alert information indicating that an abnormal object has been detected. For example, the abnormal object detection unit 124 records position information indicating the position where the difference occurred in step S32 as alert information in the storage device of the image processing device 100 together with an identification number indicating the image frame in which the abnormal object was detected. In addition, the abnormal object detection unit 124 may, for example, display a moving image acquired from the camera 50 on a display device, and display the position where the abnormal object was detected by a rectangular area or the like, superimposed on the moving image.

[0084] [Step S35] The abnormal object detection unit 124 determines whether to end the abnormal object detection process. For example, when an operation to end the detection process is performed by the user, it is determined that the detection process is to be ended. When it is determined that the detection process is to be continued, the process proceeds to step S11 in Fig. 8, and when it is determined that the detection process is to be ended, the processes in Figs. 8 to 10 are ended.

[0085] As described above, by the moving object erasure process shown in Fig. 12, even if an abnormal object exists in a position close to the moving object, only the moving object area can be identified and distinguished from the abnormal object area. Therefore, the moving object can be accurately erased from the original image frame while leaving the abnormal object in its vicinity. Then, by detecting the abnormal object using the erased frame generated in this way, even if an abnormal object exists in a position close to the moving object, the abnormal object can be detected with high accuracy.

[0086] Fig. 13 is a diagram showing an example of an abnormal object detection process using erased frames. In step S32 in Fig. 10, the abnormal object detection unit 124 performs contour detection on, for example, the current erased frame 241 and the past erased frame 242 (one second ago in Fig. 13). Fig. 13 shows a contour image 241a generated by contour detection from the current erased frame 241, and a contour image 242a generated by contour detection from the erased frame 242 one second ago. In the contour images 241a and 242a, the pixel values ​​of the pixels at the contour positions of the pixels of the original image are set to 1, and the pixel values ​​of the pixels at positions other than the contour are set to 0.

[0087] The abnormal object detection unit 124 calculates the difference in pixel values ​​at the same pixel between the contour images 241a and 242a generated in this way. Pixels whose pixel values ​​are different between the contour images 241a and 242a are detected as pixels in an abnormal object region. As an example, Fig. 13 shows an image 243 in which a rectangular region 244 including the detected abnormal object region is superimposed on the current erased frame 241.

[0088] 10 and 13, an abnormal object is automatically detected based on the difference between the current erased frame and the past erased frame. However, as another example, the abnormal object detection unit 124 may output a video including the erased frames stored in the erased image storage unit 113 and display it on a display device without performing such automatic detection of an abnormal object. In this case, a user can recognize an abnormal object by visually checking the displayed video.

[0089] The following can be applied as the moving body and the abnormal object in the second embodiment. For example, when used for detecting an abnormal object on a highway, a vehicle such as a car or a motorcycle is detected as the moving body, and an object that has fallen from a vehicle, an object that has flown from outside the road, a person who has gotten off a vehicle, a person who has entered from outside the road, etc. are detected as the abnormal object. When used for detecting a fallen object on a roadway or a sidewalk, a vehicle or a person is detected as the moving body, and an object that has fallen from a vehicle or a person is detected as the abnormal object. When used for detecting scattered garbage or illegal dumping on a roadway or a park, a person is detected as the moving body, and scattered garbage or illegally dumped objects are detected as the abnormal object.

[0090] Furthermore, the processing functions of the image processing device 1, 100 shown in each of the above-mentioned embodiments can be realized by a computer. In this case, a program describing the processing contents of the functions that each device should have is provided, and the above-mentioned processing functions are realized on the computer by executing the program on a computer. The program describing the processing contents can be recorded on a computer-readable recording medium. Examples of computer-readable recording media include magnetic storage devices, optical disks, and semiconductor memories. Examples of magnetic storage devices include hard disk drives (HDDs) and magnetic tapes. Examples of optical disks include CDs (Compact Discs), DVDs (Digital Versatile Discs), and Blu-ray Discs (BD, registered trademark).

[0091] When distributing a program, for example, the program is recorded on a portable recording medium such as a DVD or CD and then sold. The program can also be stored in a storage device of a server computer and transferred from the server computer to other computers via a network.

[0092] A computer that executes a program stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. The computer then reads the program from its own storage device and executes processing according to the program. The computer can also read the program directly from a portable recording medium and execute processing according to the program. The computer can also execute processing according to the received program each time a program is transferred from a server computer connected via a network. [Explanation of symbols]

[0093] 1 Image processing device 2 Processing section 10. Captured images 10a,10b Images 11 Detection Area 21 Object 21a,22a closed region 21b area 22 Object

Claims

1. A detection area of ​​a predetermined shape including a predetermined type of object is detected from the captured image; detecting one or more closed regions from the detection region by performing contour detection on the detection region; a processing unit that identifies a first closed region having a maximum area among the one or more closed regions as a region in which the object is captured; An image processing device comprising:

2. The processing unit further replaces a first partial image of the first closed region in the captured image with a second partial image generated by an interpolation calculation using image information of a periphery of the first closed region.

2. The image processing device according to claim 1.

3. The processing unit further comprises: generating a first corrected image by replacing the first partial image in the captured image with the second partial image; obtaining a second corrected image generated based on a second frame preceding the first frame of a moving image including the captured image as a first frame; detecting an object of a type different from the target object from the first corrected image by calculating a difference between the first corrected image and the second corrected image; 3. The image processing device according to claim 2.

4. The processing unit includes: searching for a second frame in which the detection area in which the object is detected does not overlap with the detection area in the first frame from past frames in a moving image including the captured image as a first frame; If the second frame is found, replace the image of the detection area in the first frame with the image of the detection area in the second frame; If the second frame is not searched, the one or more closed regions are detected from the detection region of the first frame, and the first partial image of the first closed region among the one or more closed regions is replaced with the second partial image.

3. The image processing device according to claim 2.

5. The computer A detection area of ​​a predetermined shape including a predetermined type of object is detected from the captured image; detecting one or more closed regions from the detection region by performing contour detection on the detection region; identifying a first closed region having a maximum area among the one or more closed regions as a region in which the object is captured; Image processing methods.

6. On the computer, A detection area of ​​a predetermined shape including a predetermined type of object is detected from the captured image; detecting one or more closed regions from the detection region by performing contour detection on the detection region; identifying a first closed region having a maximum area among the one or more closed regions as a region in which the object is captured; An image processing program that performs the processing.

Images