Image processing system, image processing method, and program

Abstract

To provide an image processing system, an image processing method, and a program for suitably detecting moving objects.SOLUTION: An image processing system places an icon 25 of a first figure in a moving object area that includes a pixel reflecting a moving object for an image frame, as an input that is an input to image frames at several different times among a plurality of image frames 20 constituting a video, and that indicates the pixel reflecting the moving object for one or more selected arbitrary pixels among image frames at processing time, receives the input by an operation placing an icon 23 of a second figure in a background area, and outputs the input information to learning means that learns parameters for detecting the moving object based on the input.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] Several aspects of the present invention relate to an image processing system, an image processing method, and a program storage medium. [Background technology]

[0002] In recent years, there has been an increasing need to detect and track moving objects such as people and cars in applications such as video surveillance. In response to this growing need, many methods have been proposed for detecting moving objects and tracking the detected moving objects. Here, a moving object is not limited to an object that is constantly moving among objects captured in an image, but also includes objects that temporarily stop (also called stationary / stay). In other words, a moving object refers to all objects captured in an image other than the part considered to be the background. For example, people and cars, which are common targets of surveillance in video surveillance, do not always move, but may temporarily stop or park, resulting in stationary states. Therefore, in applications such as video surveillance, it is important to be able to detect them even when they are temporarily stopped.

[0003] Background subtraction is known as one of the methods for detecting moving objects (see, for example, Non-Patent Document 1 and Non-Patent Document 2). The background subtraction is a method for comparing an image stored as the background with an image captured by a camera, and extracting a region with a difference as a moving object. Here, when detecting a moving object using background subtraction, accurate background extraction at the time of analysis is required. This is because, if data at the start of observation is simply used as a fixed background, many false detections occur due to the influence of background changes accompanying changes in the environment such as changes in lighting. Therefore, in order to avoid such problems, the background at the time of analysis is often estimated by a method such as calculating an average value for each pixel from the most recent observation image within a certain period of time. For example, Non-Patent Document 1 discloses a method for applying background subtraction while updating the background successively.

[0004] On the other hand, there is also a technique for extracting only temporary stationary objects such as objects left behind or people staying for a certain period of time (for example, see Patent Document 1). Patent Document 1 discloses a method for analyzing movements in a scene using background models with multiple different time widths. In this method, a long-term background model analyzed over a long range and a short-term background model analyzed over a short range are created. Then, if a state in which a moving object is not detected by background subtraction based on the short-term background model but is detected by background subtraction based on the long-term background model continues for a predetermined number of times, it is determined that a temporary stationary object exists, and the stationary object is detected as a moving object. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 5058010 [Non-patent literature]

[0006] [Non-Patent Document 1] Atsushi Kawabata, Shinya Tanito, Yasuo Morooka. "Extraction Technology of Moving Object Images", Information Processing Society of Japan, vol.28, no.4, pp.395-402, 1987 [Non-Patent Document 2] C. Stauffer and WEL Grimson, “Adaptive background mixture models for real-time tracking”, Proceedings of CVPR, vol.2, pp. 246-252, 1999 Summary of the Invention [Problem to be solved by the invention]

[0007] Here, in a method for calculating the difference between an image to be analyzed and a background image (also called "background difference") while successively updating the background image as in Non-Patent Document 1, consider a case where a moving object such as a person or a car stays there for a longer period of time than the time width for analyzing the background image. In this case, there is a problem that the moving object is determined to be part of the background image and cannot be detected. On the other hand, if the time width for analysis is extended to detect temporary stationary objects, the object becomes susceptible to background changes due to external noise such as lighting fluctuations, and there is a problem that many temporary changes in the background image other than stationary objects are falsely detected.

[0008] In addition, although Patent Document 1 is intended to detect temporary stationary objects, it is premised on the fact that background subtraction based on a long-term background model can express the true background at the time of acquisition of an observed image. Therefore, in an environment in which the background changes from moment to moment due to lighting fluctuations, the difference between the long-term background model and the true background at the time of acquisition of an observed image becomes large, making it difficult to sufficiently prevent false detections.

[0009] Some aspects of the present invention have been made in consideration of the above-mentioned problems, and one of the objects is to provide an image processing system, an image processing method, and a program storage medium for suitably detecting a moving object. [Means for solving the problem]

[0010] One image processing system according to the present invention comprises: In the captured image, pixels that include the target object and pixels that do not include the target object are specified. input means and, a specifying unit for specifying a region of the target object from the captured image based on the designation; Equipped with The input means further receives designation of at least one of pixels showing the target object and pixels not showing the target object after the region of the target object is specified by the specification means. .

[0011] In one image processing method according to the present invention, a computer In a captured image, pixels showing a target object and pixels not showing the target object are designated, an area of ​​the target object is identified from the captured image based on the designation, and after the area of ​​the target object is identified, at least one of the pixels showing the target object and the pixels not showing the target object is designated. .

[0012] A program according to the present invention is provided for a computer to: A process of receiving designation of pixels in which a target object is shown and pixels in which the target object is not shown in a captured image, a process of identifying an area of ​​the target object from the captured image based on the designation, and a process of receiving designation of at least one of the pixels in which the target object is shown and the pixels in which the target object is not shown after the area of ​​the target object has been identified; Execute the command.

[0013] In the present invention, the terms "unit", "means", "device", and "system" do not simply mean physical means, but also include cases where the functions of the "unit", "means", "device", and "system" are realized by software. Also, the functions of one "unit", "means", "device", or "system" may be realized by two or more physical means or devices, and the functions of two or more "units", "means", "device", or "system" may be realized by one physical means or device. Effect of the Invention

[0014] According to the present invention, it is possible to provide an image processing system, an image processing method, and a program storage medium for suitably detecting a moving object. [Brief description of the drawings]

[0015] [Figure 1] FIG. 2 is a diagram for explaining the relationship between a background model and an input image frame. [Diagram 2] FIG. 11 is a diagram for explaining a specific example of a display screen. [Diagram 3] FIG. 2 is a diagram for explaining a moving object detection method according to the embodiment. [Figure 4] 1 is a functional block diagram showing a schematic configuration of an image processing system according to a first embodiment. [Diagram 5] 1 is a functional block diagram showing a schematic configuration of an image processing system according to a first embodiment. [Figure 6] 5 is a flowchart showing a process flow of the image processing system shown in FIG. [Figure 7] 6 is a flowchart showing a process flow of the image processing system shown in FIG. 5. [Figure 8] FIG. 6 is a block diagram showing a hardware configuration capable of implementing the image processing system shown in FIGS. 4 and 5. [Figure 9] FIG. 11 is a functional block diagram showing a schematic configuration of an image processing system according to a second embodiment. [Figure 10] FIG. 11 is a functional block diagram showing a schematic configuration of an image processing system according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following description and in the drawings, the same or similar components are denoted by the same or similar reference numerals.

[0017] (1 First embodiment) (1.1 Overview) 1 to 7 are diagrams for explaining the embodiment. The following description will be given with reference to these diagrams.

[0018] This embodiment relates to an image processing system for detecting moving objects such as people and cars that repeatedly move or temporarily stay in place from an image captured by a photographing device such as a camera. In particular, the image processing system according to this embodiment suitably detects moving objects such as people and cars even when the environment changes from moment to moment due to, for example, lighting fluctuations.

[0019] To this end, the image processing system according to this embodiment generates three background models based on image frames extracted from a video at each time, as shown in Fig. 1, and detects moving objects using these background models. These three background models differ in the time span (time span to be analyzed) at which the multiple image frames that are the basis of the background models are captured. Hereinafter, these three background models are referred to as the long-term background model, the medium-term background model, and the short-term background model.

[0020] Here, in the image processing system according to this embodiment, a nonlinear function is applied to the short-term background model, the medium-term background model, and the long-term background model to determine an area in which a moving object is captured (moving object area) and a background area in which no moving object is present. More specifically, the image processing system according to this embodiment uses a CNN (Convolution Neural Network, also called a convolutional neural network) as a nonlinear function to determine the moving object area. This method is broadly divided into two phases: (1) a phase in which a moving object detection model (parameters) for determining a moving object is learned (supervised learning), and (2) a phase in which a moving object is detected using the generated moving object detection model.

[0021] First, a method for generating correct answer data for generating a moving object detection model will be described. The image processing system receives from the user, as correct answer data, designations of pixels in a moving object region and pixels in a background region for each image frame at each shooting time input. Figure 2 shows an example of a GUI (Graphical User Interface) screen that receives designations of pixels in a moving object region and pixels in a background region.

[0022] In the example of FIG. 2, a cursor 21 is displayed on an image frame 20. The user operates the cursor 21 with a pointing device such as a mouse to place an icon 23 in the background region and an icon 25 in the moving object region in which a person appears. Note that the user does not need to specify whether all pixels in the image frame 20 are moving object regions or background regions. The image processing system generates a CNN moving object detection model using pixels that have been specified as being moving object regions or background regions in this way.

[0023] A specific example of a CNN that can be used by the image processing system according to this embodiment is shown in Fig. 3. In the example shown in Fig. 3, first, for a pixel position where it is desired to determine whether or not it is a moving object, a 5 pixel x 5 pixel image centered on the pixel position is extracted from each of the difference image between the short-term background model and the medium-term background model, the difference image between the short-term background model and the long-term background model, and the difference image between the medium-term background model and the long-term background model.

[0024] Then, eight types of 3x3x3 filters are used to perform eight types of convolution processing to generate eight images of 3 pixels x 3 pixels. Furthermore, a nonlinear transformation is performed by applying the following formula f(x) to the pixel value x in each image.

[0025]

number

[0026] Here, a is a parameter determined for each pixel of the image obtained by the eight types of filters, and is determined by supervised learning. The generated 3 pixel x 3 pixel x 8 images correspond to the nodes of the neural network.

[0027] Similarly, 15 types of convolution processing are performed on these 3 pixel x 3 pixel x 8 images using 15 types of 3 x 3 x 8 filters to generate 15 1 pixel x 1 pixel images. Then, the above f(x) is applied to the pixel value x in each image. The parameter a included in f(x) is a parameter determined for each pixel of the image obtained by the 15 types of filters, as in the above case, and is determined by the above supervised learning. The generated 1 pixel x 1 pixel x 15 images correspond to the nodes of the neural network.

[0028] Finally, these 1 pixel x 1 pixel x 15 images are convolved using one type of 1 x 1 x 15 filter to calculate one value. Then, the above f(x) is applied to that one value. The parameter a included in f(x) is a parameter determined for the value obtained by that one filter, as in the above case, and is determined by the above supervised learning.

[0029] The value obtained by such processing is here referred to as the "mobility-like value" v. The determination as to whether or not the object is a moving object is made by comparing v with a predetermined threshold value T. If v ≥ T, the pixel position of the processing target is determined to be a moving object, and if v < T, it is determined to be the background area. The value of the threshold value T is a parameter determined in advance.

[0030] As described above, parameters such as the parameters a and T used in the CNN are estimated by supervised learning and stored in a moving object detection parameter dictionary described later. By performing learning of the parameters, for example, a background model with a time width that is easy to detect movement, such as the swaying of trees due to wind, is not used much, and a background model with a time width that is easy to correctly detect the movement of specific moving objects such as people and cars is used. It is possible to construct a moving object detection model that is easy to detect specific moving objects.

[0031] Therefore, the image processing system according to the present embodiment can stably detect moving objects such as people and cars even in an environment affected by background changes due to external noises such as illumination fluctuations and wind.

[0032] Note that in the example of the moving object detection model in FIG. 3, eight and fifteen intermediate images are generated from the difference images of three background models, and the final mobility-like value v is calculated, but the present invention is not limited thereto. For example, the difference image of the input background model may be four or more, and the number of intermediate images may be larger or smaller.

[0033] (1.2 System Configuration) Hereinafter, the system configuration of the image processing system according to the present embodiment will be described with reference to FIGS. 4 and 5. FIG. 4 shows the system configuration of an image processing system 100 that performs learning related to the generation of a moving object detection model (parameters) for moving object detection. FIG. 5 shows the system configuration of an image processing system 200 that detects a moving object using the generated moving object detection model. Note that the image processing systems 100 and 200 according to FIGS. 4 and 5 may be realized on the same device or on different devices.

[0034] (1.2.1 System configuration of image processing system 100 used for learning) First, the system configuration of an image processing system 100 for generating a CNN moving object detection model for moving object detection will be described with reference to Fig. 4. The image processing system 100 includes an image input unit 110, an area designation unit 120, a background model acquisition unit 130, a background model update unit 140, a background model database (DB) 150, a background model distance calculation unit 160, a moving object detection model construction unit 170, and a moving object detection parameter dictionary 180.

[0035] (1.2.1.1 Image input unit 110) The image input unit 110 receives input of image frames constituting a video from a photographing device such as a camera (not shown), that is, image frames photographed at different times. Here, the image frames may be monochrome images or color images. In the case of monochrome images, the image frame contains one value for each pixel. In the case of color images, the image frame has three values ​​(e.g., color representations such as RGB, YCbCr, etc.) for each pixel. Alternatively, the image frame may have four or more values ​​for each pixel, such as distance information obtained by a TOF (Time of Flight) camera, etc.

[0036] (1.2.1.2 Area specification part 120) The area designation unit 120 provides the user with a GUI for inputting correct answer data for the image frame, and designates a moving object area and a background area for pixels contained in the image frame in response to the user's input. A specific example of a display screen displayed on a display device by the area designation unit 120 is as shown in FIG.

[0037] This enables the area designation unit 120 to prepare correct answer data (distinguishing between a moving object area and a background area) for the pixel selected by the user in the image frame.

[0038] As can be seen from the fact that the designation of moving object regions and background regions is performed for pixels, they are input as points at various positions on the screen. By inputting points at various positions, diverse learning data can be generated even with a small number of inputs, improving learning efficiency. In addition, the designation of moving object regions and background regions is performed for images taken at various times. This generates more diverse learning data (correct answer data), improving learning efficiency.

[0039] (1.2.1.3 Background model acquisition unit 130) The background model acquisition unit 130 reads the image frame input from the image input unit 110 and three background models stored in the background model DB 150: a short-term background model, a medium-term background model, and a long-term background model.

[0040] (1.2.1.4 Background model DB150) The background model DB 150 stores a plurality of background models including a short-term background model, a medium-term background model, and a long-term background model that differ in time span of the capture time of the image frame that is the analysis source. Here, various formats of the background models are possible, but for example, they may be in the same image format as the image frame input from the image input unit 110. For example, if the background model is a monochrome image, each pixel contains one value, and if it is a color image, each pixel contains three values.

[0041] Alternatively, the background model can be a distribution function for each pixel that indicates, for each pixel, the likelihood of the pixel value in each of the image frames from which it originates (e.g., a histogram or a sum-of-Gaussians distribution function).

[0042] As described above, the short-term background model, medium-term background model, and long-term background model have different time spans of the capture times of the original image frames, with the time spans increasing in the order of short-term background model, medium-term background model, and long-term background model. In particular, with regard to the short-term background model, it is possible to adopt the image frame input from the image input unit 110 as the short-term background model as it is. In that case, it is possible not to manage the short-term background model in the background model DB 150.

[0043] (1.2.1.5 Background model update unit 140) The background model update unit 140 generates a short-term background model, a medium-term background model, and a long-term background model taking into consideration the image frame at the processing time (the image frame at the most recent time) from the image frame at the processing time acquired by the background model acquisition unit 130 and the background model stored in the background model DB 150. The generated background models are stored in the background model DB 150.

[0044] In this embodiment, the short-term background model, the medium-term background model, and the long-term background model are generated from an image frame captured at the shortest time interval from the processing time, the medium-term background model is generated from an image frame captured at a longer time interval, and the long-term background model is generated from an image frame captured at the longest time interval, as shown in Fig. 1.

[0045] A method for generating a background model may involve, for example, taking the average or most frequent pixel value for image frames over a time span determined for each background model. Alternatively, if the background model is a distribution function for each pixel as described above, a distribution function of pixel values ​​for each image frame included within the time span may be generated.

[0046] In this embodiment, the short-term background model, the medium-term background model, and the long-term background model are described as having different time widths of the capture times of the original image frames, but the present invention is not limited to this. The short-term background model, the medium-term background model, and the long-term background model can also be understood as background models having different magnitudes of influence of the image frame at the processing time (latest time). That is, the short-term background model is most influenced by the image frame at the processing time, and the long-term background model is least influenced by the image frame at the processing time. Therefore, instead of using the concept of time width, the concept of update coefficient may be introduced, and the update coefficient when updating the background model using the image frame input from the image input unit 110 may be changed for the short-term background model, the medium-term background model, and the long-term background model.

[0047] In this case, for example, if the background model is I bg If the image frame input from the image input unit 110 is I, then

[0048]

number

[0049] In this formula, a is a constant between 0 and 1, and takes different values ​​for the short-term background model, the medium-term background model, and the long-term background model. If the constants for the short-term background model, the medium-term background model, and the long-term background model are a1, a2, and a3, respectively, then

[0050]

number

[0051] The relationship is established. When a1 = 1, the short-term background model is always replaced with a new image frame. When a3 = 0, it means that a fixed background model is used as the long-term background model. Even if a fixed background model is used as the background model, it can be updated in the same manner.

[0052] (1.2.1.6 Background model distance calculation unit 160) The background model distance calculation unit 160 calculates, for each pixel, a distance value that numerically indicates the difference between the three background models acquired by the background model acquisition unit 130. Specifically, the background model distance calculation unit 160 calculates, for each pixel, the distance between the short-term background model and the medium-term background model, the distance between the short-term background model and the long-term background model, and the distance between the medium-term background model and the long-term background model.

[0053] For example, when the background model is in an image format, the background model distance calculation unit 160 may calculate the difference value or difference vector of the pixel values ​​of each pixel, and then calculate the absolute value or magnitude of the difference value as the distance. When the background model has multiple values ​​for each pixel, for example, in a color image format such as RGB, YCbCr, or HSV, it may calculate the difference value for each value, and then use the sum of the absolute values ​​of the difference values ​​as the distance for each pixel. Alternatively, after extracting a neighboring partial image such as a 3 pixel x 3 pixel image or a 5 pixel x 5 pixel image centered on the pixel position to be processed, the pixel values ​​of the two extracted neighboring partial images may be regarded as two vectors, and the inter-vector distance or normalized correlation r between the two vectors may be calculated. In this case, when calculating the distance using a neighboring 3 x 3 image with a monochrome image format background model, the distance between 9-dimensional vectors is calculated. Furthermore, when calculating distance using neighboring 5×5 images in an RGB color image, the distance between 75-dimensional (5×5×3) vectors is calculated.

[0054] When normalized correlation r is used for distance, correlation r has a maximum value of 1, and the closer r is to 1, the closer the images are to being identical. Therefore, in order to convert to a distance scale, 1-r can be used as a value indicating distance. Alternatively, the distance may be calculated after preprocessing the neighboring partial images using an edge enhancement filter or the like.

[0055] In addition, when a distribution function such as a histogram is used for the background model, the background model distance calculation unit 160 can calculate the distance between the background models by using the area of ​​the common part of two histograms or a histogram distance calculation method such as the Bhattacharyya distance.

[0056] In the above-mentioned method, the background model distance calculation unit 160 is described as calculating the distance for each pixel, but this is not limited to the above. For example, a method of dividing an image into meshes, which are several area units, and calculating the distance for each mesh unit can be used. The distance may be a negative value.

[0057] In addition, the short-term background model, the medium-term background model, and the long-term background model may be in different formats. For example, the short-term background model may be in an image format, and the medium-term background model may be in a distribution function for each pixel. In this case, as a method of calculating the distance, for example, a histogram of a normal distribution with a predetermined standard deviation is generated with the pixel value held in the short-term background model as the center. Then, a method of calculating the distance by considering the histogram as a distribution function in the short-term background model and comparing the histogram with the histogram of the medium-term background model may be adopted. Alternatively, a mean value may be calculated for each pixel from the distribution function of each pixel of the medium-term background model, and the distance may be calculated by comparing the medium-term background model in an image format generated as a set of the mean values ​​with the short-term background model.

[0058] (1.2.1.7 Moving object detection model construction unit 170) The moving object detection model construction unit 170 uses the correct answer data provided by the region designation unit 120 to generate a moving object detection model (parameters corresponding to CNN) for detecting a moving object using CNN. More specifically, the moving object detection model construction unit 170 is set so as to be closest to the given correct answer data. Specifically, N pixels x i (i is an integer 1≦i≦N) for each correct data y i For example, pixel xi If is background area, y i = 0, if it is a moving object area, y i =1.

[0059] The initial values ​​of each parameter of the CNN can be set to random values. Then, the likelihood of a moving object is calculated for the above N pixels using the CNN. Here, i The estimation result (moving object likelihood) indicating whether the region is a moving object or a background region for v i In this case, consider the following evaluation value S.

[0060]

number

[0061] The closer the estimation result by CNN is to the correct data, the smaller the evaluation value S. Therefore, the parameters of the CNN can be calculated using a gradient method such as the stochastic descent method so that S is as small as possible.

[0062] The evaluation value S may be calculated using a different method. For example,

[0063]

number

[0064] It is also possible to consider the following.

[0065] The above-mentioned method of generating correct answer data by the area designation unit 120 is performed by the user for random positions. In such a method, there may be cases where the learning data is insufficient and the estimation accuracy of a specific location is poor. Therefore, after generating a moving object detection model by performing learning using the correct answer data provided by the area designation unit 120, the estimation result may be examined, and additional correct answer data may be provided for areas with low estimation accuracy, and learning may be performed again using the correct answer data. This process may be repeated.

[0066] The moving object detection parameters generated by the moving object detection model construction unit 170 through such processing are stored in the moving object detection parameter dictionary 180.

[0067] (1.2.2 System configuration of image processing system 200 used for moving object detection) Next, a description will be given of a system configuration of an image processing system 200 that detects a moving object using the moving object detection model generated by the image processing system 100. The image processing system 200 includes an image input unit 210, a background model acquisition unit 220, a background model update unit 230, a background model DB 240, a background model distance calculation unit 250, a moving object detection unit 260, a moving object detection parameter dictionary 270, and a result output unit 280.

[0068] Here, the functions of the image input unit 210, the background model acquisition unit 220, the background model update unit 230, the background model DB 240, the background model distance calculation unit 250, and the moving body detection parameter dictionary 270 are similar to those of the image input unit 110, the background model acquisition unit 130, the background model update unit 140, the background model DB 150, the background model distance calculation unit 160, and the moving body detection parameter dictionary 180, respectively, and therefore will not be described.

[0069] The moving object detection unit 260 determines whether or not an object is a moving object by CNN, which is a moving object detection model using parameters stored in the moving object detection parameter dictionary 270. A specific method for detecting a moving object has been described above with reference to Fig. 3, and therefore a description thereof will be omitted here. Note that the moving object detection unit 260 may detect a stationary moving object by a moving object detection model using parameters stored in the moving object detection parameter dictionary 270, and detect a moving moving object based on the difference between a medium-term background model and a short-term background model.

[0070] The result output unit 280 outputs the information on the moving object obtained by the moving object detection unit 260. There are various possible output methods, but for example, the information can be output as a binary image in which the moving object region is set to 1 and the other regions are set to 0. Alternatively, it is also possible to generate connected parts by performing a labeling process on the binary image, and output a circumscribing rectangle for each connected part.

[0071] Alternatively, consider a case where the moving body detection unit 260 can detect both moving bodies and temporarily stationary moving bodies. In this case, for example, it is possible to output a ternary value, with the pixel value of a pixel detected as a moving moving body being 1, the pixel value of a pixel detected as a temporarily stationary moving body being 2, and other pixel values ​​being 0. In some cases, it may be difficult to distinguish whether an object is moving or stationary, in which case a value of 1 may be output for a moving object. Alternatively, it is possible to allow four values ​​overall, and output a pixel value of 3 for pixels where it is unclear which one it is.

[0072] (1.3 Processing flow) Hereinafter, the process flow of the image processing system 100 and the image processing system 200 will be described with reference to Fig. 6 and Fig. 7. Fig. 6 is a flowchart showing the process flow of the image processing system 100, and Fig. 7 is a flowchart showing the process flow of the image processing system 200.

[0073] The processing steps described below can be executed in any order or in parallel as long as no inconsistency occurs in the processing content, and other steps can be added between the processing steps. Furthermore, a step described as one step for convenience can be executed as multiple steps, and a step described as multiple steps for convenience can be executed as one step.

[0074] (1.3.1 Processing flow of the image processing system 100) First, the flow of processing by the image processing system 100 for learning parameters for moving object detection will be described with reference to FIG.

[0075] The image input unit 110 receives an input of a new image frame (image frame at a processing time) (S601). The area designation unit 120 displays a display screen, a specific example of which is shown in FIG. 2, in order to receive input of correct answer data for the received image frame, and receives designation from the user as to whether one or more pixels at random positions in the image frame are a moving object area or a background area (S603). Based on the designation, the area designation unit 120 generates correct answer data indicating whether or not the area is a moving object area. Here, the image input unit 110 receives input for a predetermined number of image frames at different times. Here, the predetermined number is arbitrary and may be determined in advance, or may be designated by the user.

[0076] Furthermore, the background model acquisition unit 130 reads the short-term background model, the medium-term background model, and the long-term background model stored in the background model DB 150 (S605). The background model distance calculation unit 160 calculates, for each pixel, the distance between the short-term background model and the medium-term background model, the distance between the medium-term background model and the long-term background model, and the distance between the short-term background model and the long-term background model (S607).

[0077] The moving object detection model construction unit 170 applies CNN to pixels for which correct answer data is prepared, of each background model calculated by the background model distance calculation unit 160, and obtains CNN parameters that reduce the evaluation value S (S609). That is, a moving object detection model is constructed by learning.

[0078] The image processing system 100 displays the moving object detection result of the sample data on the display screen using the learned moving object detection model. When displaying, the image processing system 100 uses the aforementioned estimated result (moving object likelihood) v i may be displayed near the detected moving object area. Or, iThe image processing system 100 may display the image in a heat map format according to the value of v i The higher the value, the redder the display, and the lower the value, the bluer the display. In this way, the moving object detection result by the learned moving object detection model is visually judged by a person. If the moving object detection accuracy by the moving object detection model is sufficient (Yes in S611), the moving object detection model construction unit 170 outputs the calculated parameters to the moving object detection parameter dictionary 180 (S613).

[0079] If the moving object detection accuracy of the moving object detection model is insufficient (No in S611), the area designation unit 120 receives input from the user on a display screen such as that shown in Fig. 2 as to whether random pixels in areas with particularly low detection accuracy are moving object areas (S615). After that, the process returns to S605 and a moving object detection model is constructed using new ground truth data.

[0080] (1.3.2 Processing flow of the image processing system 200) Next, a process flow for detecting a moving object using CNN parameters generated by the image processing system 100 will be described with reference to FIG.

[0081] The image input unit 210 receives an input of a new image frame (an image frame at a processing time) (S701). Furthermore, the background model acquisition unit 220 reads the short-term background model, the medium-term background model, and the long-term background model stored in the background model DB 240 (S703).

[0082] The background model distance calculation unit 250 calculates, for each pixel, the distance between the short-term background model and the medium-term background model, the distance between the medium-term background model and the long-term background model, and the distance between the short-term background model and the long-term background model (S705). The moving object detection unit 260 uses the parameters generated by the image processing system 100 and stored in the moving object detection parameter dictionary 270 to input the distances between the background models calculated by the background model distance calculation unit 250 and determines whether each pixel is an area in which a moving object is reflected (S707). The result output unit 280 outputs the detection result (S709).

[0083] Furthermore, the background model update unit 230 updates each background model using the image frame input from the image input unit 210, and stores the updated background models in the background model DB 240 (S711).

[0084] (1.4 Specific examples of hardware configuration) An example of a hardware configuration in which the above-described image processing systems 100 and 200 are realized by a computer 800 will be described below with reference to Fig. 8. As described above, the image processing systems 100 and 200 may be implemented on the same computer, or the functions of the image processing systems 100 and 200 may be implemented on multiple computers.

[0085] As shown in FIG. 8, a computer 800 includes a processor 801, a memory 803, a storage device 805, an input interface (I / F) unit 807, a data I / F unit 809, a communication I / F unit 811, and a display device 813.

[0086] The processor 801 executes programs stored in the memory 803 to control various processes of the image processing systems 100 and 200. For example, the processes of the image input unit 110, the area designation unit 120, the background model acquisition unit 130, the background model update unit 140, the background model distance calculation unit 160, and the moving object detection model construction unit 170 shown in Fig. 4 can be realized as programs that are temporarily stored in the memory 803 and run mainly on the processor 801. Similarly, the processes of the image input unit 210, the background model acquisition unit 220, the background model update unit 230, the background model distance calculation unit 250, the moving object detection unit 260, and the result output unit 280 shown in Fig. 5 can also be realized as programs that are temporarily stored in the memory 803 and run mainly on the processor 801.

[0087] The memory 803 is a storage medium such as a RAM (Random Access Memory), etc. The memory 803 temporarily stores the program code of the program executed by the processor 801 and data required when the program is executed.

[0088] The storage device 805 is a non-volatile storage medium such as a hard disk or a flash memory. The storage device 805 can store an operating system, various programs for implementing the functions of the image processing systems 100 and 200, and various data including the background model DBs 150 and 240, and the moving object detection parameter dictionaries 180 and 270. The programs and data stored in the storage device 805 are loaded into the memory 803 as necessary, and are referred to by the processor 801.

[0089] The input I / F unit 807 is a device for receiving input from a user. For example, on a display screen such as that shown in FIG. 2 provided by the region designation unit 120, a user operation for designating whether a region is a background region or a moving object region is input via the input I / F unit 807. Specific examples of the input I / F unit 807 include a keyboard, a mouse, a touch panel, etc. The input I / F unit 807 may be connected to the computer 800 via an interface such as a USB (Universal Serial Bus).

[0090] The data I / F unit 809 is a device for inputting data from outside the computer 800. A specific example of the data I / F unit 809 is a drive device for reading data stored in various storage devices. The data I / F unit 809 may be provided outside the computer 800. In that case, the data I / F unit 809 is connected to the computer 800 via an interface such as a USB.

[0091] The communication I / F unit 811 is a device for performing wired or wireless data communication with an external device of the computer 800, such as an image capturing device (a video camera, a security camera, or a digital camera). The communication I / F unit 811 may be provided outside the image processing system 100. In that case, the communication I / F unit 811 is connected to the computer 800 via an interface such as a USB.

[0092] The display device 813 is, for example, a device for displaying a display screen for designating the background region / moving object region illustrated in Fig. 2, a moving object detection result output by the result output unit 280, and the like. Specific examples of the display device 813 include, for example, a liquid crystal display and an organic EL (Electro-Luminescence) display. The display device 813 may be provided outside the computer 800. In that case, the display device 813 is connected to the computer 800 via, for example, a display cable or the like.

[0093] (1.5 Effects of this embodiment) As described above, the image processing systems 100 and 200 according to the present embodiment detect moving objects by CNN using the differences between the short-term background model, the medium-term background model, and the long-term background model. In particular, by using learning, it is possible to detect a specific moving object while suppressing false detection by increasing the contribution of a background model with a time width that is easy to detect moving objects such as people and cars, without making much use of a background model with a time width that is easy to detect movement due to, for example, the swaying of trees caused by wind. In addition, it is possible to stably detect a moving object even in an environment affected by background changes due to external noise such as lighting fluctuations.

[0094] (2. Second embodiment) The second embodiment will be described below with reference to Fig. 9 and Fig. 10. Fig. 9 is a block diagram showing the functional configuration of an image processing system 900 according to this embodiment. The image processing system 900 includes an input unit 910 and a learning unit 920.

[0095] The input unit 910 receives input for image frames at different times among the multiple image frames that make up the video, and receives input as to whether any one or more selected pixels in the image frame at the processing time are pixels that show a moving object or pixels that do not show a moving object.

[0096] The learning unit 920 learns parameters for detecting a moving object based on the input from the input unit 910 indicating whether a pixel represents a moving object or not.

[0097] The image processing system 1000 also includes an input unit 1010 and a detection unit 1020 .

[0098] The input unit 1010 receives as input a plurality of image frames captured at different times.

[0099] The detection unit 1020 detects a moving object by performing one or more convolution calculations using values ​​of the background models in the vicinity of the target pixel, using a first background model generated based on the image frame at the processing time, a second background model that is less influenced by the image frame at the processing time than the first background model, and a third background model that is less influenced by the image frame at the processing time than the second background model.

[0100] By implementing it in this way, the image processing system 900 and the image processing system 1000 according to this embodiment can suitably detect a moving object.

[0101] (3 Notes) The configurations of the above-described embodiments may be combined or some of the components may be replaced. Furthermore, the configuration of the present invention is not limited to the above-described embodiments, and various modifications may be made without departing from the scope of the present invention.

[0102] In addition, a part or all of each of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. In addition, the program of the present invention may be a program that causes a computer to execute each operation described in each of the above embodiments.

[0103] (Appendix 1) An image processing system comprising: an input means for receiving input for image frames at different times among a plurality of image frames constituting an image, wherein an input indicating whether any one or more selected pixels among the image frames at the processing time are pixels that show a moving object or pixels that do not show a moving object; and a learning means for learning parameters for detecting moving objects based on the input.

[0104] (Appendix 2) The image processing system of claim 1, further comprising a calculation means for calculating a difference between a first background model generated based on the image frame at the processing time, a second background model having a smaller influence of the image frame at the processing time than the first background model, and a third background model having a smaller influence of the image frame at the processing time than the second background model, wherein the learning means learns parameters for detecting a moving object using the first background model, the second background model, and the third background model based on the input.

[0105] (Appendix 3) 3. The image processing system according to claim 1, wherein the learning means learns the parameters used in a detection model for detecting a moving object by performing one or more convolution calculations using values ​​of a background model of a region neighboring the one or more pixels.

[0106] (Appendix 4) 4. The image processing system according to claim 1, wherein the learning means learns the convolution calculation and a threshold value to be compared with a value obtained as a result of the convolution calculation as the parameters.

[0107] (Appendix 5) An image processing system comprising: an input means for receiving input of a plurality of image frames captured at different times; and a detection means for detecting a moving object by performing one or more convolution calculations using a first background model generated based on the image frames at the processing time, a second background model having a smaller influence of the image frames at the processing time than the first background model, and a third background model having a smaller influence of the image frames at the processing time than the second background model, using values ​​of the background models in a region adjacent to a target pixel.

[0108] (Appendix 6) 6. The image processing system according to claim 5, wherein the first background model, the second background model, and the third background model have different time spans of shooting times of image frames to be taken into account.

[0109] (Appendix 7) 7. The image processing system according to claim 5 or 6, wherein an image frame at the processing time is used as the first background model.

[0110] (Appendix 8) An image processing method in which a computer performs the steps of receiving input for several image frames at different times among a plurality of image frames constituting a video, and input indicating whether any one or more selected pixels among the image frames at the processing time are pixels that show a moving object or pixels that do not show a moving object, and learning parameters for detecting moving objects based on the input.

[0111] (Appendix 9) The image processing method according to claim 8, further comprising a calculation means for calculating a difference between a first background model generated based on the image frame at the processing time, a second background model having a smaller influence of the image frame at the processing time than the first background model, and a third background model having a smaller influence of the image frame at the processing time than the second background model, wherein the learning means learns parameters for detecting a moving object using the first background model, the second background model, and the third background model based on the input.

[0112] (Appendix 10) 10. The image processing method according to claim 8 or 9, further comprising: learning the parameters used in a detection model for detecting a moving object by performing one or more convolution calculations using values ​​of a background model in a neighborhood area of ​​the one or more pixels.

[0113] (Appendix 11) 11. The image processing method according to claim 8, further comprising learning the convolution calculation and a threshold value to be compared with the value obtained as a result of the convolution calculation as the parameters.

[0114] (Appendix 12) An image processing method performed by a computer, comprising the steps of: receiving input of a plurality of image frames captured at different times; and detecting a moving object by performing one or more convolution calculations using a first background model generated based on the image frames captured at the processing time, a second background model having a smaller influence of the image frames captured at the processing time than the first background model, and a third background model having a smaller influence of the image frames captured at the processing time than the second background model, using background model values ​​of an area near a target pixel.

[0115] (Appendix 13) The image processing method according to claim 12, wherein the first background model, the second background model, and the third background model have different time spans of shooting times of image frames to be considered.

[0116] (Appendix 14) 14. The image processing method according to claim 12 or 13, wherein an image frame at the processing time is used as the first background model.

[0117] (Appendix 15) A process of receiving input for image frames at different times among a plurality of image frames constituting a video, and inputting whether any one or more selected pixels of the image frames at the processing time are pixels showing a moving object or pixels not showing a moving object; and learning parameters for detecting a moving object based on the input.

[0118] (Appendix 16) The program according to claim 15, further comprising a calculation means for calculating a difference between a first background model generated based on the image frame at the processing time, a second background model having a smaller influence of the image frame at the processing time than the first background model, and a third background model having a smaller influence of the image frame at the processing time than the second background model, wherein the learning means learns parameters for detecting a moving object using the first background model, the second background model, and the third background model based on the input.

[0119] (Appendix 17) 17. The program according to claim 15 or 16, wherein the parameters used in a detection model for detecting a moving object are learned by performing one or more convolution calculations using values ​​of a background model in a neighborhood area of ​​the one or more pixels.

[0120] (Appendix 18) 18. The program according to any one of claims 15 to 17, which learns the convolution calculation and a threshold value to be compared with the value obtained as a result of the convolution calculation as the parameters.

[0121] (Appendix 19) A program that causes a computer to execute a process of receiving input of multiple image frames captured at different times, and a process of detecting a moving object by performing one or more convolution calculations using a first background model generated based on the image frames at the processing times, a second background model that is less influenced by the image frames at the processing times than the first background model, and a third background model that is less influenced by the image frames at the processing times than the second background model, using values ​​of the background models in a region near a target pixel.

[0122] (Appendix 20) 20. The program according to claim 19, wherein the first background model, the second background model, and the third background model have different time ranges of shooting times of image frames to be considered.

[0123] (Appendix 21) 21. The program according to claim 19 or 20, wherein an image frame at the processing time is used as the first background model.

[0124] The present invention has been described above by taking the above-mentioned embodiment as an exemplary example. However, the present invention is not limited to the above-mentioned embodiment. That is, the present invention can be applied in various aspects that can be understood by a person skilled in the art within the scope of the present invention.

[0125] This application claims priority based on Japanese Patent Application No. 2014-115205, filed on June 30, 2014, the disclosure of which is incorporated herein by reference in its entirety. [Explanation of symbols]

[0126] 20: Image frame 21: Cursor 23, 25: Icon 100: Image processing system 110: Image input unit 120: Area specification part 130: Background model acquisition unit 140: Background model update section 150: Background model database 160: Background model distance calculation unit 170: Moving object detection model construction unit 180: Moving object detection parameter dictionary 200: Image processing system 210: Image input unit 220: Background model acquisition unit 230: Background model update section 240: Background model database 250: Background model distance calculation section 260: Moving object detection unit 270: Moving object detection parameter dictionary 280: Result output section 800: Computer 801: Processor 803: Memory 805 :Storage device 807: Input interface section 809: Data interface section 811: Communication interface unit 813:Display device 900: Image processing system 910: Input section 920: Learning Department 1000: Image processing system 1010: Input section 1020: Detector

Claims

1. An input means for receiving designation for arranging a plurality of first icons representing pixels in which a target object is captured and a plurality of second icons representing pixels in which the target object is not captured in a plurality of captured images constituting a video; a specifying unit for specifying a region of the target object from the captured image based on the designation; Equipped with the input means further receives designation of at least one of pixels reflecting the target object or pixels not reflecting the target object after the region of the target object is specified by the specification means, a plurality of the first icons are arranged in a region of one of the target objects in the captured image, the plurality of first icons have the same shape and size, and the first icon and the second icon are displayed differently; Image processing system.

2. The designation is a designation made by a user to the captured image. The image processing system according to claim 1 .

3. the captured images are a plurality of image frames constituting a video; The input means receives designation for the plurality of captured images.

3. The image processing system according to claim 1.

4. The input means receives designation of a plurality of the pixels reflecting the target object and a plurality of the pixels not reflecting the target object. The image processing system according to any one of claims 1 to 3.

5. The target object is a moving object. The image processing system according to any one of claims 1 to 4.

6. The identification means identifies a region of the target object from the captured image by using a detection model obtained based on the specification. The image processing system according to any one of claims 1 to 5.

7. In the captured image, a plurality of the second icons are arranged, and the plurality of the second icons are the same shape and size. The image processing system according to claim 1 .

8. The computer receiving a designation to arrange a plurality of first icons representing pixels in which a target object is captured and a second icon representing pixels in which the target object is not captured in a plurality of captured images constituting a video; Identifying a region of the target object from the captured image based on the designation; An image processing method, comprising: after a region of the target object is specified, at least one of pixels showing the target object and pixels not showing the target object is designated, a plurality of the first icons are arranged in a region of one of the target objects in the captured image, the plurality of first icons have the same shape and size, and the first icon and the second icon are displayed differently; Image processing methods.

9. In the captured image, a plurality of the second icons are arranged, and the plurality of the second icons are the same shape and size. The image processing method according to claim 8.

10. On the computer, A process of receiving a designation to arrange a plurality of first icons representing pixels in which a target object is captured and a second icon representing pixels in which the target object is not captured in a plurality of captured images constituting a video; A process of identifying a region of the target object from the captured image based on the designation; a process of receiving designation of at least one of pixels reflecting the target object and pixels not reflecting the target object after the region of the target object is specified, a plurality of the first icons are arranged in a region of one of the target objects in the captured image, the plurality of first icons have the same shape and size, and the first icon and the second icon are displayed differently; program.

11. In the captured image, a plurality of the second icons are arranged, and the plurality of the second icons are the same shape and size. The program according to claim 10.

Images