Image processing system, image processing method, and program

Abstract

To provide an image processing system, an image processing method, and a program for suitably detecting a moving object.SOLUTION: The image processing system receives inputs to image frames at several different times among a plurality of image frames constituting a video image in which the inputs indicate that any one or more selected pixels in the image frames at the processing time are pixels reflecting a moving object, outputs the input information to learning means for learning a parameter for detecting the moving object based on the inputs, and receives the inputs by an operation for arranging an icon of a first graphic in a moving object region including pixels reflecting the moving object in the image frame.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Some aspects of the present invention relate to an image processing system, an image processing method, and a program storage medium. [Background technology]

[0002] In recent years, there has been a growing need to detect and track moving objects such as people and vehicles in applications like video surveillance. In response to this growing need, numerous methods for detecting and tracking moving objects have been proposed. Here, "moving object" is not limited to objects that are constantly moving within the image, but also includes objects that are temporarily stopped (also called stationary or lingering). In other words, a moving object refers to any object that appears in the image other than what is considered the background. For example, people and vehicles, which are common targets of video surveillance, are not constantly moving; they are often stationary, such as temporarily stopped or parked. Therefore, being able to detect objects even when they are temporarily stopped is crucial for applications like video surveillance.

[0003] One known method for detecting moving objects is the background subtraction method (see, for example, Non-Patent Documents 1 and 2). The background subtraction method compares an image stored as the background with an image captured by a camera and extracts areas with differences as moving objects. When detecting moving objects using background subtraction, accurate background extraction at the time of analysis is necessary. If the data from the start of observation is simply used as the fixed background, many false detections will occur due to the influence of background changes caused by changes in the environment, such as changes in lighting. Therefore, to avoid this problem, the background at the time of analysis is often estimated by methods such as calculating the average value for each pixel from the most recent observation images within a certain period of time. For example, Non-Patent Document 1 discloses a method of applying the background subtraction method while sequentially updating the background.

[0004] On the other hand, there are also techniques that extract only temporarily stationary objects, such as objects left behind or people who remain for a certain period of time (see, for example, Patent Document 1). Patent Document 1 discloses a method for analyzing movement in a scene using multiple background models with different time intervals. In this method, a long-term background model analyzed over a long range and a short-term background model analyzed over a short range are created. If a moving object is not detected by the background difference based on the short-term background model, but is detected by the background difference based on the long-term background model, for a predetermined number of times, then it is assumed that a temporarily stationary object exists, and that stationary object is detected as a moving object. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Patent No. 5058010 [Non-patent literature]

[0006] [Non-Patent Document 1] Atsushi Kawabata, Shinya Tanifuji, and Yasuo Morooka. "Technology for Extracting Moving Object Images," Information Processing Society of Japan, vol.28, no.4, pp.395-402, 1987. [Non-Patent Document 2] C. Stauffer and WEL Grimson, “Adaptive background mixture models for real-time tracking”, Proceedings of CVPR, vol.2, pp. 246-252, 1999 [Overview of the project] [Problems that the invention aims to solve]

[0007] Here, we consider a method, as described in Non-Patent Document 1, that performs background subtraction (also called "background subtraction") by sequentially updating the background image, where moving objects such as people or cars remain for a longer period than the time span during which the background image is analyzed. In this case, the moving object is judged to be part of the background image and therefore cannot be detected. On the other hand, if the time span for analysis is increased in order to detect temporary stationary objects, it becomes more susceptible to background changes caused by external noise such as lighting fluctuations, resulting in the problem of many temporary changes in the background image other than stationary objects being misdetected.

[0008] Furthermore, while Patent Document 1 is intended to detect temporary stationary objects, it assumes that background subtraction based on a long-term background model can represent the true background at the time of image acquisition. Therefore, in environments where the background changes moment by moment, such as with lighting fluctuations, the difference between the long-term background model and the true background at the time of image acquisition becomes large, making it difficult to sufficiently suppress false detections.

[0009] Some aspects of the present invention have been made in view of the aforementioned problems, and one of their objectives is to provide an image processing system, an image processing method, and a program storage medium for suitably detecting moving objects. [Means for solving the problem]

[0010] One image processing system according to the present invention includes an input means that receives a specification to arrange a plurality of first icons indicating pixels in which a target object is shown and a second icon indicating pixels in which the target object is not shown in an image, and a identification means that identifies the region of the target object from the image based on the specification, wherein the input means further receives a specification for at least one of the arrangement of the first icons or the arrangement of the second icons after the region of the target object has been identified by the identification means, the first icons are arranged in plurality for a single region of the target object in the image, the plurality of first icons are the same shape and size as each other, and the first icons and the second icons have different displays.

[0011] In the first image processing method according to the present invention, a computer receives a specification to arrange a plurality of first icons in an image that indicate pixels in which a target object is visible, and second icons that indicate pixels in which the target object is not visible. Based on the specification, the computer identifies the region of the target object from the image, and after the region of the target object has been identified, it receives a specification for at least one of the arrangement of the first icons or the arrangement of the second icons. The first icons are arranged in a plurality for a single region of the target object in the image, the plurality of first icons are the same shape and size as each other, and the first icons and the second icons have different displays.

[0012] The first program according to the present invention causes a computer to perform the following processes: receiving a specification to arrange a plurality of first icons in an image that indicate pixels in which a target object is visible, and second icons that indicate pixels in which the target object is not visible; identifying the region of the target object from the image based on the specification; and, after the region of the target object has been identified, receiving a specification to arrange at least one of the first icons or the second icons, wherein a plurality of first icons are arranged for a single region of the target object in the image, the plurality of first icons are the same shape and size as each other, and the first icons and the second icons have different displays.

[0013] In this invention, "part," "means," "apparatus," and "system" do not merely mean physical means, but also include cases where the functions of such "part," "means," "apparatus," and "system" are realized by software. Furthermore, even if the functions of one "part," "means," "apparatus," or "system" are realized by two or more physical means or devices, the functions of two or more "parts," "means," "apparatus," and "systems" may be realized by one physical means or device. [Effects of the Invention]

[0014] According to the present invention, an image processing system, an image processing method, and a program storage medium for suitably detecting moving objects can be provided. [Brief explanation of the drawing]

[0015] [Figure 1] It is a diagram for explaining the relationship between the background model and the input image frame. [Figure 2] It is a diagram for explaining a specific example of the display screen. [Figure 3] It is a diagram for explaining the moving body detection method according to the present embodiment. [Figure 4] It is a functional block diagram showing the schematic configuration of the image processing system according to the first embodiment. [Figure 5] It is a functional block diagram showing the schematic configuration of the image processing system according to the first embodiment. [Figure 6] It is a flowchart showing the processing flow of the image processing system shown in FIG. 4. [Figure 7] It is a flowchart showing the processing flow of the image processing system shown in FIG. 5. [Figure 8] It is a block diagram showing the configuration of hardware capable of implementing the image processing system shown in FIGS. 4 and 5. [Figure 9] It is a functional block diagram showing the schematic configuration of the image processing system according to the second embodiment. [Figure 10] It is a functional block diagram showing the schematic configuration of the image processing system according to the second embodiment.

Embodiments for Carrying Out the Invention

[0016] Embodiments of the present invention will be described below. In the following description and the description of the accompanying drawings, the same or similar configurations are respectively assigned the same or similar reference numerals.

[0017] (1 First Embodiment) (1.1 Overview) FIGS. 1 to 7 are diagrams for explaining the embodiments. The following will be described while referring to these diagrams.

[0018] This embodiment relates to an image processing system for detecting moving objects, such as people and cars, that repeatedly move or temporarily linger, from images captured by a camera or other imaging device. In particular, the image processing system according to this embodiment can suitably detect moving objects such as people and cars even when the environment changes moment by moment, such as due to fluctuations in lighting.

[0019] To this end, the image processing system according to this embodiment generates three background models, each created based on image frames extracted from the video at different time points, as shown in Figure 1, and uses these background models to detect moving objects. These three background models differ in the time span (the time span of the analysis target) at which the multiple image frames that form the basis of the background model are captured. Hereinafter, these three background models will be referred to as the long-term background model, the medium-term background model, and the short-term background model.

[0020] In this embodiment, the image processing system determines the region containing a moving object (moving object region) and the background region without a moving object by applying a nonlinear function to the short-term background model, the medium-term background model, and the long-term background model. More specifically, the image processing system in this embodiment uses a CNN (Convolutional Neural Network) as the nonlinear function to distinguish the moving object region. This method is broadly divided into two phases: (1) a phase of learning (supervised learning) a moving object detection model (parameters) for distinguishing moving objects, and (2) a phase of detecting moving objects using the generated moving object detection model.

[0021] First, we will explain the method for generating ground truth data to create a moving object detection model. The image processing system receives the user's specification of which pixels belong to the moving object region and which pixels belong to the background region for each image frame taken at the input capture time, as ground truth data. Figure 2 shows an example of a GUI (Graphical User Interface) screen for receiving the specification of which pixels belong to the moving object region and which pixels belong to the background region.

[0022] In the example in Figure 2, a cursor 21 is displayed on image frame 20. The user manipulates the cursor 21 using a pointing device such as a mouse to place icons 23 in the background area and icons 25 in the area containing moving objects such as people. The user does not need to specify whether every pixel in image frame 20 is in the moving object area or the background area. The image processing system generates a CNN moving object detection model using the pixels that have been specified as either moving object areas or background areas in this way.

[0023] Figure 3 shows a specific example of a CNN that can be used with the image processing system according to this embodiment. In the example shown in Figure 3, first, for a pixel position where it is desired to determine whether or not it is a moving object, a 5x5 pixel image centered on that pixel position is extracted from the difference image between the short-term background model and the medium-term background model, the difference image between the short-term background model and the long-term background model, and the difference image between the medium-term background model and the long-term background model.

[0024] Then, using eight types of 3x3x3 filters, eight types of convolution operations are performed to generate eight 3x3 pixel images. Furthermore, a nonlinear transformation is performed by applying the following formula f(x) to the pixel value x within each image.

[0025]

number

[0026] Here, 'a' is a parameter determined for each pixel of the image obtained by the eight types of filters, and is determined by supervised learning. The generated 3x3x8 images correspond to the nodes of the neural network.

[0027] Similarly, 15 different convolution operations are performed on these 3x3x8 images using 15 different 3x3x8 filters to generate 15 1x1 images. Then, the above f(x) is applied to the pixel value x within each of these images. The parameter a included in f(x) is a parameter determined for each pixel of the image obtained by the 15 types of filters, similar to the above case, and is determined by the above supervised learning. The generated 1-pixel × 1-pixel × 15 images correspond to the nodes of the neural network.

[0028] Finally, convolution processing is performed on these 1-pixel × 1-pixel × 15 images using one type of 1×1×15 filter to calculate one value. Then, f(x) is applied to the one value. The parameter a included in f(x) is a parameter determined for the value obtained by the one filter, similar to the above case, and is determined by the above supervised learning.

[0029] The value obtained by such processing is here referred to as the moving object-likeness v. The determination as to whether it is a moving object is made by comparing v with a predetermined threshold T. If v≥T, the pixel position of the processing target is determined to be a moving object, and if v<T, it is determined to be the background area. The value of the threshold T is a predetermined parameter.

[0030] As described above, parameters such as a and T used in the CNN are estimated by supervised learning and stored in the moving object detection parameter dictionary described later. By performing parameter learning, for example, without using a background model with a time width that is easy to detect movement such as the swaying of trees by wind, a background model with a time width that is easy to correctly detect the movement of specific moving objects such as people and cars is used. It is possible to construct a moving object detection model that is easy to detect specific moving objects.

[0031] Therefore, the image processing system according to the present embodiment can stably detect moving objects such as people and cars even in an environment affected by background changes due to external noises such as illumination fluctuations and wind.

[0032] In the example of the moving object detection model in Figure 3, eight and fifteen intermediate images are generated from the difference images of three background models, and the final moving object likelihood v is calculated. However, this is not the only method. For example, the input background model difference images could be four or more, and the number of intermediate images could also be increased or decreased.

[0033] (1.2 System Configuration) The system configuration of the image processing system according to this embodiment will be described below with reference to Figures 4 and 5. Figure 4 shows the system configuration of the image processing system 100 that performs learning related to the generation of a moving object detection model (parameters) for detecting moving objects. Figure 5 shows the system configuration of the image processing system 200 that detects moving objects using the generated moving object detection model. The image processing systems 100 and 200 shown in Figures 4 and 5 may be implemented on the same device or on different devices.

[0034] (1.2.1 System configuration of the image processing system 100 used for learning) First, with reference to Figure 4, the system configuration of the image processing system 100 for generating a CNN moving object detection model for moving object detection will be described. The image processing system 100 includes an image input unit 110, a region specification unit 120, a background model acquisition unit 130, a background model update unit 140, a background model database (DB) 150, a background model distance calculation unit 160, a moving object detection model construction unit 170, and a moving object detection parameter dictionary 180.

[0035] (1.2.1.1 Image input section 110) The image input unit 110 receives image frames that make up a video from a shooting device such as a camera (not shown), that is, image frames that are each taken at different times. Here, the image frames may be monochrome or color images. If it is a monochrome image, each pixel in the image frame contains one value. If it is a color image, each pixel in the image frame has three values ​​(for example, a color representation such as RGB or YCbCr). Alternatively, the image frame may have four or more values ​​per pixel, such as distance information obtained from a TOF (Time of Flight) camera.

[0036] (1.2.1.2 Area specification part 120) The region designation unit 120 provides the user with a GUI for inputting correct data for an image frame, and, in response to user input, designates the pixels contained within the image frame as either a moving object region or a background region. A specific example of the display screen that the region designation unit 120 displays on the display device is shown in Figure 2.

[0037] This allows the region selection unit 120 to prepare correct data (distinguishing between moving object regions and background regions) for the pixels selected by the user within the image frame.

[0038] Furthermore, since the designation of the moving object region and the background region is done at the pixel level, it is input as points at various positions on the screen. By inputting as points at various positions, diverse training data is generated even with a small number of inputs, improving training efficiency. In addition, the designation of the moving object region and the background region is performed on images at various time points. This generates even more diverse training data (ground truth data), improving training efficiency.

[0039] (1.2.1.3 Background Model Acquisition Section 130) The background model acquisition unit 130 reads the image frame input from the image input unit 110, as well as three background models stored in the background model DB 150: a short-term background model, a medium-term background model, and a long-term background model.

[0040] (1.2.1.4 Background Model DB150) The background model DB150 stores multiple background models, including short-term, medium-term, and long-term background models with different time intervals for the capture time of the image frames used for analysis. Here, various formats are possible for each background model, but for example, it can be the same image format as the image frame input from the image input unit 110. For example, if the background model is a monochrome image, one value will be included for each pixel, and if it is a color image, three values ​​will be included for each pixel.

[0041] Alternatively, the background model can be a pixel-by-pixel distribution function that shows the likelihood of the pixel value in the original image frame for each pixel. Here, the distribution function could be, for example, a histogram, or it could be a distribution function that is a sum of multiple Gaussians.

[0042] As mentioned above, the short-term, medium-term, and long-term background models differ in the time span of the original image frames, with the time span increasing in the order of short-term, medium-term, and long-term background models. In particular, for the short-term background model, it is conceivable to use the image frame input from the image input unit 110 directly as the short-term background model. In that case, it is also conceivable that the background model DB 150 does not manage the short-term background model.

[0043] (1.2.1.5 Background Model Update Section 140) The background model update unit 140 generates short-term, medium-term, and long-term background models that take into account the image frame at the processing time (the most recent image frame) from the image frame at the processing time acquired by the background model acquisition unit 130 and the background models stored in the background model DB 150. The generated background models are stored in the background model DB 150.

[0044] In this embodiment, the short-term background model, the medium-term background model, and the long-term background model each have different time intervals for the capture of the original image frames. As shown in Figure 1, the short-term background model is generated from image frames captured within the shortest time interval from the processing time, the medium-term background model is generated from image frames captured within a longer time interval, and the long-term background model is generated from image frames captured within the longest time interval.

[0045] One method for generating a background model is to take the mean or mode of pixel values ​​for image frames within a time span defined for each background model. Alternatively, if the background model is defined as a pixel-by-pixel distribution function as described above, then the distribution function of the pixel values ​​for each image frame contained within the time span can be generated.

[0046] In this embodiment, the short-term background model, medium-term background model, and long-term background model are described as having different time spans for the capture times of the original image frames, but this is not the only way to describe them. The short-term background model, medium-term background model, and long-term background model can also be understood as background models in which the influence of the image frame at the processing time (the most recent time) differs. That is, the short-term background model is most influenced by the image frame at the processing time, while the long-term background model is least influenced by the image frame at the processing time. Therefore, instead of using the concept of time span, the concept of an update coefficient may be introduced, and the update coefficient used to update the background model using the image frame input from the image input unit 110 may be changed for the short-term background model, medium-term background model, and long-term background model.

[0047] In this case, for example, if the background model is I bg Therefore, if we denote the image frame input from the image input unit 110 as I,

[0048]

number

[0049] The background model can be updated as follows. In this equation, a is a constant between 0 and 1, and takes different values ​​for the short-term, medium-term, and long-term background models. If the constants for the short-term, medium-term, and long-term background models are a1, a2, and a3, then,

[0050]

number

[0051] The following relationship holds: When a1=1, the short-term background model is always replaced with a new image frame. Also, when a3=0, it means that a fixed background model is used for the long-term background model. Even when using a fixed background model, it can be updated in the same manner.

[0052] (1.2.1.6 Background Model Distance Calculation Unit 160) The background model distance calculation unit 160 calculates a distance value for each pixel that numerically represents the difference between the three background models acquired by the background model acquisition unit 130. Specifically, for each pixel, the background model distance calculation unit 160 calculates the distance between the short-term background model and the medium-term background model, the distance between the short-term background model and the long-term background model, and the distance between the medium-term background model and the long-term background model.

[0053] For example, if the background model is in image format, the background model distance calculation unit 160 may calculate the difference value or difference vector of the pixel values ​​of each pixel, and then calculate the absolute value or magnitude of this difference as the distance. If the background model has multiple values ​​for each pixel, for example, in the case of a color image format such as RGB, YCbCr, or HSV, it may be possible to calculate the difference value for each value and then use the sum of the absolute values ​​of these difference values ​​as the distance for each pixel. Alternatively, neighboring sub-images such as a 3x3 pixel image or a 5x5 pixel image centered on the pixel position to be processed may be extracted, and the pixel values ​​of the two extracted neighboring sub-images may be treated as two vectors, and the vector distance or normalized correlation r of these two vectors may be calculated. In this case, for example, if the background model is in monochrome image format and the distance is calculated using a 3x3 neighboring image, the distance between 9-dimensional vectors will be calculated. Furthermore, when calculating distance using 5x5 neighboring images in an RGB color image, the distance between 75-dimensional (5x5x3) vectors will be calculated.

[0054] When using normalized correlation r as the distance, the maximum value of correlation r is 1, and the closer r is to 1, the closer the images are to being identical. Therefore, to convert it to a distance scale, 1-r can be used as the value representing the distance. Alternatively, the distance may be calculated after preprocessing the above neighboring area images with an edge enhancement filter or similar.

[0055] Furthermore, if a distribution function such as a histogram is used in the background model, the background model distance calculation unit 160 can calculate the distance between the background models using the area of ​​the common part of the two histograms or a histogram distance calculation method such as the Bhattacharya distance.

[0056] In the method described above, the background model distance calculation unit 160 was explained as calculating the distance for each pixel, but this is not the only method. For example, it is also possible to divide the image into several regions called meshes and then calculate the distance for each mesh unit. The distance may also take a negative value.

[0057] Furthermore, the short-term, medium-term, and long-term background models may each be in different formats. For example, the short-term background model could be an image format, while the medium-term background model could be a pixel-by-pixel distribution function. In this case, one possible method for calculating the distance is to generate a histogram of a normal distribution with a predetermined standard deviation, centered on the pixel values ​​held in the short-term background model. Then, considering this histogram as the distribution function in the short-term background model, the distance could be calculated by comparing this histogram with the histogram of the medium-term background model. Alternatively, the mean value could be calculated for each pixel from the distribution function of each pixel in the medium-term background model, and the distance could be calculated by comparing the medium-term background model, which is generated as a set of these mean values ​​in an image format, with the short-term background model.

[0058] (1.2.1.7 Mobile Object Detection Model Construction Section 170) The moving object detection model construction unit 170 generates a moving object detection model (parameters corresponding to the CNN) for detecting moving objects using a CNN, based on the ground truth data provided by the region specification unit 120. More specifically, the moving object detection model construction unit 170 is configured to be as close as possible to the given ground truth data. Specifically, N pixels x i For each of the following (where i is an integer of 1 ≤ i ≤ N), the correct data y i Let's assume that the following is given. Here, for example, pixel x i If it is the background area, then y i = 0, if it is in the moving object region, then y i Let = 1.

[0059] Each parameter of the CNN can be initially set to a random value. Then, the CNN is used to calculate the likelihood of a moving object appearing in the above N pixels. Here, pixel x i The estimated result (likelihood of movement) indicating whether it is a moving object region or a background region is v i Let's assume that the following evaluation value S is considered.

[0060]

number

[0061] The evaluation value S becomes smaller the closer the CNN's estimation result is to the ground truth data. Therefore, the CNN parameters should be determined using a gradient method, such as stochastic descent, to minimize S.

[0062] Note that different methods may be used to calculate the evaluation value S. For example, a value equivalent to cross-entropy.

[0063]

number

[0064] This is also a possibility.

[0065] The method for generating correct data using the region specification unit 120 described above is performed by the user at random locations. In such methods, the training data may not be sufficient, and the estimation accuracy at specific locations may be poor. Therefore, it is also possible to generate a moving object detection model by training with the correct data provided by the region specification unit 120, then examine the estimation results, provide additional correct data for areas with low estimation accuracy, and repeat the training process using that new correct data.

[0066] The mobile object detection parameters generated by the mobile object detection model construction unit 170 through this process are stored in the mobile object detection parameter dictionary 180.

[0067] (1.2.2 System configuration of the image processing system 200 used for moving object detection) Next, the system configuration of the image processing system 200, which detects moving objects using the moving object detection model generated by the image processing system 100, will be described. The image processing system 200 includes an image input unit 210, a background model acquisition unit 220, a background model update unit 230, a background model DB 240, a background model distance calculation unit 250, a moving object detection unit 260, a moving object detection parameter dictionary 270, and a result output unit 280.

[0068] Here, the functions of the image input unit 210, background model acquisition unit 220, background model update unit 230, background model DB 240, background model distance calculation unit 250, and moving object detection parameter dictionary 270 are the same as those of the image input unit 110, background model acquisition unit 130, background model update unit 140, background model DB 150, background model distance calculation unit 160, and moving object detection parameter dictionary 180, respectively, so their explanation is omitted.

[0069] The moving object detection unit 260 determines whether an object is moving or not using a CNN, which is a moving object detection model that uses parameters stored in the moving object detection parameter dictionary 270. The specific method for detecting moving objects has been described above with reference to Figure 3, so it will not be explained here. Alternatively, the moving object detection unit 260 may detect stationary moving objects using a moving object detection model that uses parameters stored in the moving object detection parameter dictionary 270, and detect moving objects based on the difference between the medium-term background model and the short-term background model.

[0070] The result output unit 280 outputs information about the moving object obtained by the moving object detection unit 260. Various output methods are possible, but for example, it can be output as a binary image in which the moving object region is set to 1 and the other regions are set to 0. Alternatively, it is also possible to generate connected parts by applying a labeling process to the binary image and output a bounding rectangle for each connected part.

[0071] Alternatively, consider a case where the moving object detection unit 260 can detect both moving objects and temporarily stationary moving objects. In this case, for example, it is conceivable to output a 3-value signal, assigning a pixel value of 1 to pixels detected as moving objects, a pixel value of 2 to pixels detected as temporarily stationary moving objects, and 0 to all other pixels. In some cases, it may be difficult to determine whether an object is moving or stationary; in such cases, a value of 1 may be output for moving objects. Alternatively, it may be possible to allow for 4 values ​​overall and output a pixel value of 3 for pixels where it is unclear whether they are moving or stationary.

[0072] (1.3 Processing Flow) The processing flow of image processing system 100 and image processing system 200 will be explained below with reference to Figures 6 and 7. Figure 6 is a flowchart showing the processing flow of image processing system 100, and Figure 7 is a flowchart showing the processing flow of image processing system 200.

[0073] Furthermore, the processing steps described below can be arbitrarily rearranged or executed in parallel, provided that no inconsistencies arise in the processing content. Additional steps may also be added between each processing step. Moreover, steps described as a single step for convenience can be divided into multiple steps for execution, and steps described as multiple steps for convenience can be executed as a single step.

[0074] (1.3.1 Processing flow of image processing system 100) First, the processing flow of the image processing system 100, which learns parameters for detecting moving objects, will be explained with reference to Figure 6.

[0075] The image input unit 110 receives the input of a new image frame (the image frame at the processing time) (S601). Further, the region designating unit 120 displays the display screen shown in FIG. 2 as a specific example in order to receive the input of the correct data for the input image frame, and for one or more pixels at random positions within the image frame, receives from the user a designation as to whether it is a moving object region or a background region (S603). Based on this designation, the region designating unit 120 generates correct data indicating whether it is a moving object region or not. Here, the reception of the input by the image input unit 110 is performed for a plurality of different image frames at a predetermined number of times. Here, the predetermined number is arbitrary and may be determined in advance, or may be received by the user's designation.

[0076] Further, the background model acquisition unit 130 reads the short-term background model, the medium-term background model, and the long-term background model stored in the background model DB 150 (S605). The background model distance calculation unit 160 calculates, for each pixel, the distance between the short-term background model and the medium-term background model, the distance between the medium-term background model and the long-term background model, and the distance between the short-term background model and the long-term background model (S607).

[0077] The moving object detection model construction unit 170 applies a CNN to the pixels of each background model calculated by the background model distance calculation unit 160 for which correct data is prepared, and obtains the parameters of the CNN such that the evaluation value S becomes small (S609). That is, a moving object detection model is constructed by learning.

[0078] The image processing system 100 displays the moving object detection result of the sample data on the display screen by the learned moving object detection model. When displaying, the image processing system 100 may display, as the reliability for the moving object detection result, the above-described estimation result (likelihood of moving object) v i near the detected moving object region. Alternatively, it may be displayed like a heat map according to the value of v i . Specifically, the image processing system 100, v iThe higher the value, the redder it is displayed, and the lower the value, the bluer it is displayed. In this way, a person visually judges the motion detection results obtained by the trained motion detection model. If the motion detection accuracy of the motion detection model is sufficient (Yes in S611), the motion detection model construction unit 170 outputs the calculated parameters to the motion detection parameter dictionary 180 (S613).

[0079] If the accuracy of the moving object detection model is insufficient (No. in S611), the region selection unit 120 receives input from the user on a display screen as shown in Figure 2, asking whether a random pixel in a region with particularly low detection accuracy is a moving object region (S615). After that, the process returns to S605 to construct a new moving object detection model using the new ground truth data.

[0080] (1.3.2 Processing flow of image processing system 200) Next, the processing flow related to moving object detection using the CNN parameters generated by the image processing system 100 will be explained with reference to Figure 7.

[0081] The image input unit 210 receives a new image frame (the image frame at the processing time) as input (S701). The background model acquisition unit 220 reads the short-term background model, medium-term background model, and long-term background model stored in the background model DB 240 (S703).

[0082] The background model distance calculation unit 250 calculates the distance between the short-term background model and the medium-term background model, the distance between the medium-term background model and the long-term background model, and the distance between the short-term background model and the long-term background model for each pixel (S705). The moving object detection unit 260 uses the parameters generated by the image processing system 100 and stored in the moving object detection parameter dictionary 270, and takes the distances between each background model calculated by the background model distance calculation unit 250 as input to determine whether each pixel is in the region where a moving object is visible (S707). The result output unit 280 outputs the detection result (S709).

[0083] Furthermore, the background model update unit 230 updates each background model using the image frames input from the image input unit 210 and stores the updated background models in the background model DB 240 (S711).

[0084] (1.4 Specific examples of hardware configurations) The following describes an example of a hardware configuration when the image processing systems 100 and 200 described above are implemented using a computer 800, with reference to Figure 8. As mentioned above, the image processing systems 100 and 200 may be implemented on the same computer, or the functions of the image processing systems 100 and 200 may be implemented on multiple computers.

[0085] As shown in Figure 8, the computer 800 includes a processor 801, memory 803, storage device 805, input interface (I / F) unit 807, data I / F unit 809, communication I / F unit 811, and display device 813.

[0086] The processor 801 controls various processes of the image processing systems 100 and 200 by executing programs stored in memory 803. For example, the processes related to the image input unit 110, region specification unit 120, background model acquisition unit 130, background model update unit 140, background model distance calculation unit 160, and moving object detection model construction unit 170 shown in Figure 4 can be implemented as programs that are temporarily stored in memory 803 and then mainly run on the processor 801. Similarly, the processes related to the image input unit 210, background model acquisition unit 220, background model update unit 230, background model distance calculation unit 250, moving object detection unit 260, and result output unit 280 shown in Figure 5 can also be implemented as programs that are temporarily stored in memory 803 and then mainly run on the processor 801.

[0087] Memory 803 is a storage medium such as RAM (Random Access Memory). Memory 803 temporarily stores the program code of the program executed by the processor 801, as well as data required during program execution.

[0088] The storage device 805 is a non-volatile storage medium such as a hard disk or flash memory. The storage device 805 can store the operating system, various programs for realizing the functions of the image processing systems 100 and 200, background model databases 150 and 240, and various data including the moving object detection parameter dictionaries 180 and 270. The programs and data stored in the storage device 805 are loaded into memory 803 as needed and referenced by the processor 801.

[0089] The input interface unit 807 is a device for receiving input from the user. For example, in a display screen like the one shown in Figure 2 provided by the area selection unit 120, user operations to specify whether an area is a background area or a moving object area are input via the input interface unit 807. Specific examples of the input interface 807 include keyboards, mice, and touch panels. The input interface 807 may also be connected to the computer 800 via an interface such as USB (Universal Serial Bus).

[0090] The data I / F unit 809 is a device for inputting data from outside the computer 800. Specific examples of the data I / F unit 809 include drive devices for reading data stored in various storage devices. The data I / F unit 809 may be located outside the computer 800. In that case, the data I / F unit 809 is connected to the computer 800 via an interface such as USB.

[0091] The communication interface unit 811 is a device for wired or wireless data communication between the computer 800 and external devices, such as imaging devices (video cameras, surveillance cameras, digital cameras, etc.). The communication interface unit 811 may be located outside the image processing system 100. In that case, the communication interface unit 811 is connected to the computer 800 via an interface such as USB.

[0092] The display device 813 is a device for displaying, for example, a display screen for specifying the background area / moving object area as illustrated in Figure 2, or the detection results of the moving object output by the result output unit 280. Specific examples of the display device 813 include liquid crystal displays and organic EL (Electro-Luminescence) displays. The display device 813 may be located outside the computer 800. In that case, the display device 813 is connected to the computer 800 via, for example, a display cable.

[0093] (1.5 Effects according to this embodiment) As described above, the image processing systems 100 and 200 according to this embodiment detect moving objects using a CNN based on the differences between the short-term, medium-term, and long-term background models. In particular, by using learning, it is possible to detect specific moving objects while suppressing false detections by increasing the contribution of background models with time intervals that are easily detectable for detecting moving objects such as people and cars, while minimizing the use of background models with time intervals that are easily detectable for detecting movement such as the swaying of trees due to wind. Furthermore, it is possible to stably detect moving objects even in environments affected by background changes due to external noise such as fluctuations in lighting.

[0094] (2. Second Embodiment) The second embodiment will be described below with reference to Figures 9 and 10. Figure 9 is a block diagram showing the functional configuration of the image processing system 900 according to this embodiment. The image processing system 900 includes an input unit 910 and a learning unit 920.

[0095] The input unit 910 receives input for several image frames from multiple image frames that make up the video at different time points, and for any one or more pixels selected from the image frame at the processing time, it receives input indicating whether the pixel shows a moving object or not.

[0096] The learning unit 920 learns parameters for detecting moving objects based on input from the input unit 910, which indicates whether a pixel contains a moving object or not.

[0097] The image processing system 1000 also includes an input unit 1010 and a detection unit 1020.

[0098] The input unit 1010 receives input from multiple image frames taken at different times.

[0099] The detection unit 1020 uses a first background model generated based on the image frame at the processing time, a second background model that is less influenced by the image frame at the processing time than the first background model, and a third background model that is less influenced by the image frame at the processing time than the second background model, and performs a convolution calculation one or more times using the values ​​of the background models in the neighborhood region of the target pixel to detect a moving object.

[0100] By implementing it in this way, the image processing system 900 and the image processing system 1000 according to this embodiment can suitably detect moving objects.

[0101] (3. Additional Notes) Furthermore, the configurations of the embodiments described above may be combined or some of the components may be replaced. Also, the configuration of the present invention is not limited to the embodiments described above, and various modifications may be made without departing from the spirit of the present invention.

[0102] Furthermore, some or all of the embodiments described above may also be described as follows, but are not limited to these. In addition, the program of the present invention may be any program that causes a computer to execute the operations described in each of the embodiments above.

[0103] (Note 1) An image processing system comprising: an input means that receives input to several image frames at different time points among a plurality of image frames constituting a video, wherein for any one or more selected pixels in the image frame at the processing time, input whether a pixel shows a moving object or not; and a learning means that learns parameters for detecting a moving object based on the input.

[0104] (Note 2) The image processing system according to Appendix 1 further comprises a calculation means for calculating the difference between a first background model generated based on the image frame at the processing time, a second background model having less influence from the image frame at the processing time than the first background model, and a third background model having less influence from the image frame at the processing time than the second background model, wherein the learning means learns parameters for detecting a moving object using the first background model, the second background model, and the third background model based on the input.

[0105] (Note 3) The image processing system according to Appendix 1 or Appendix 2, wherein the learning means learns the parameters used in a detection model for detecting a moving object by performing one or more convolution calculations using the values ​​of the background model in the neighborhood region of one or more pixels.

[0106] (Note 4) The image processing system according to any one of Appendix 1 to Appendix 3, wherein the learning means learns the convolution calculation and a threshold value that is compared with the value obtained as a result of the convolution calculation as parameters.

[0107] (Note 5) An image processing system comprising: an input means for receiving multiple image frames taken at different times; and a detection means for detecting a moving object by performing one or more convolution calculations using the values ​​of the background models in the neighborhood region of a target pixel, using a first background model generated based on the image frame at the processing time, a second background model having less influence from the image frame at the processing time than the first background model, and a third background model having less influence from the image frame at the processing time than the second background model.

[0108] (Note 6) The image processing system described in Appendix 5, wherein the first background model, the second background model, and the third background model differ in the time interval of the image frame being considered.

[0109] (Note 7) The image processing system according to Appendix 5 or Appendix 6, which uses the image frame at the processing time as the first background model.

[0110] (Note 8) An image processing method in which a computer receives input to several image frames from multiple image frames constituting a video at different time points, and for any one or more pixels selected from the image frames at the processing time, receives input indicating whether a pixel contains a moving object or not; and learns parameters for detecting a moving object based on the input.

[0111] (Note 9) The image processing method according to Appendix 8 further comprises a calculation means for calculating the difference between a first background model generated based on the image frame at the processing time, a second background model having less influence from the image frame at the processing time than the first background model, and a third background model having less influence from the image frame at the processing time than the second background model, wherein the learning means learns parameters for detecting a moving object using the first background model, the second background model, and the third background model based on the input.

[0112] (Note 10) The image processing method according to Appendix 8 or Appendix 9, wherein the parameters used in a detection model for detecting a moving object are learned by performing one or more convolution calculations using the values ​​of the background model in the neighborhood region of one or more pixels.

[0113] (Note 11) The image processing method according to any one of the appendices 8 to 10, wherein the convolution calculation and a threshold value compared with the value obtained as a result of the convolution calculation are learned as parameters.

[0114] (Note 12) An image processing method in which a computer performs the following steps: receiving input of multiple image frames taken at different times; detecting a moving object by performing one or more convolution calculations using the values ​​of the background models in the neighborhood region of a target pixel, using a first background model generated based on the image frame at the processing time, a second background model having less influence from the image frame at the processing time than the first background model, and a third background model having less influence from the image frame at the processing time than the second background model.

[0115] (Note 13) The image processing method described in Appendix 12, wherein the first background model, the second background model, and the third background model differ in the time interval of the image frame being considered.

[0116] (Note 14) The image processing method according to Appendix 12 or Appendix 13, wherein the image frame at the processing time is used as the first background model.

[0117] (Note 15) The input consists of several image frames from a plurality of image frames constituting a video, each at a different time, and the processing involves receiving input for any one or more selected pixels in the image frame at the processing time, indicating whether the pixel contains a moving object or not. A program that causes a computer to perform a process of learning parameters for detecting a moving object based on the aforementioned input.

[0118] (Note 16) The program further comprises a calculation means for calculating the difference between a first background model generated based on the image frame at the processing time, a second background model having less influence from the image frame at the processing time than the first background model, and a third background model having less influence from the image frame at the processing time than the second background model, wherein the learning means learns parameters for detecting a moving object using the first background model, the second background model, and the third background model based on the input, as described in Appendix 15.

[0119] (Note 17) The program described in Appendix 15 or Appendix 16, which learns the parameters used in a detection model for detecting a moving object by performing one or more convolution calculations using the values ​​of the background model in the neighborhood region of one or more pixels.

[0120] (Note 18) A program according to any one of the appendices 15 to 17, which learns the convolution calculation and a threshold value to be compared with the value obtained as a result of the convolution calculation as the parameters.

[0121] (Note 19) A program that causes a computer to perform the following steps: receive input from multiple image frames taken at different times; and detect a moving object by performing one or more convolution calculations using the values ​​of the background models in the neighborhood region of a target pixel, using a first background model generated based on the image frame at the processing time, a second background model that is less influenced by the image frame at the processing time than the first background model, and a third background model that is less influenced by the image frame at the processing time than the second background model.

[0122] (Note 20) The program described in Appendix 19, wherein the first background model, the second background model, and the third background model differ in the time interval of the image frame being considered.

[0123] (Note 21) The program described in Appendix 19 or Appendix 20, which uses the image frame of the processing time as the first background model.

[0124] The present invention has been described above using the embodiments described above as exemplary examples. However, the present invention is not limited to the embodiments described above. That is, the present invention can be applied in various forms that can be understood by those skilled in the art within the scope of the present invention.

[0125] This application claims priority based on Japanese Patent Application No. 2014-115205, filed on 30 June 2014, and incorporates all of its disclosures herein. [Explanation of symbols]

[0126] 20: Image Frame 21: Cursor 23, 25: Icons 100: Image processing system 110: Image input section 120: Area specification part 130: Background model acquisition section 140: Background Model Update Section 150: Background Model Database 160: Background Model Distance Calculation Unit 170: Mobile Object Detection Model Construction Unit 180: Mobile Object Detection Parameter Dictionary 200: Image Processing System 210: Image input section 220: Background model acquisition section 230: Background Model Update Section 240: Background Model Database 250: Background model distance calculation unit 260: Moving object detection unit 270: Mobile Object Detection Parameter Dictionary 280: Result output section 800: Computer 801: Processor 803: Memory 805 :Storage device 807: Input Interface Unit 809: Data Interface Section 811: Communication Interface Unit 813:Display device 900: Image processing system 910: Input section 920: Learning Department 1000: Image processing system 1010: Input section 1020: Detection unit

Claims

1. An input means that receives a specification to arrange, in the image, a plurality of first icons indicating pixels in which the target object is visible, and second icons indicating pixels in which the target object is not visible. A means for identifying the region of the target object from the video using a detection model obtained based on the above designation, Equipped with, The input means, after the region of the target object has been identified by the identification means, receives a specification for at least one of the placement of the first icon or the placement of the second icon for pixels in the region where the detection model has determined that the detection accuracy of the target object is insufficient. The first icons are arranged in multiples for a region of one of the target objects in the video, and the multiple first icons are identical in shape and size, while the first icons and the second icons have different displays. Image processing system.

2. The aforementioned designation is a designation made by the user regarding the aforementioned video. The image processing system according to claim 1.

3. The input means receives a designation for an image frame that constitutes the video. The image processing system according to claim 1.

4. The aforementioned object is a moving object. The image processing system according to claim 1.

5. In the aforementioned video, multiple second icons are arranged, and the multiple second icons are identical in shape and size to one another. The image processing system according to claim 1.

6. Computers In the video, the specification is to place multiple first icons indicating pixels in which the target object is visible, and second icons indicating pixels in which the target object is not visible. Using the detection model obtained based on the above specification, the region of the target object is identified from the video. An image processing method in which, after the region of the target object has been identified, at least one of the placement of the first icon or the placement of the second icon is specified for the pixels of the region in which the detection model has determined that the detection accuracy of the target object is insufficient, The first icons are arranged in multiples for a region of one of the target objects in the video, and the multiple first icons are identical in shape and size, while the first icons and the second icons have different displays. Image processing methods.

7. On the computer, A process that receives a specification to arrange, in a video, multiple first icons indicating pixels in which the target object is visible, and second icons indicating pixels in which the target object is not visible. Using the detection model obtained based on the above specification, a process is performed to identify the region of the target object from the video. A program that, after the region of the target object has been identified, causes the program to perform a process of specifying at least one of the placement of the first icon or the placement of the second icon for pixels in the region where the detection accuracy of the target object by the detection model is determined to be insufficient, The first icons are arranged in multiples for a region of one of the target objects in the video, and the multiple first icons are identical in shape and size, while the first icons and the second icons have different displays. program.

Images