Image processing system, image processing method, and program

Abstract

To provide an image processing system, an image processing method, and a program storage medium capable of generating an image in which personal information is preferably protected.SOLUTION: An image processing system, for example, comprises: an image input unit 110 for receiving input of a plurality of image frames forming a video from a photographing apparatus; an edge detection unit 150 for detecting a feature point included in the image frame to be processed among the plurality of image frames; and an output unit 180 for outputting an output image in which the image frame to be processed in the area detected as the feature point is superimposed on a background image generated from at least part of the plurality of image frames.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] Some aspects of the present invention relate to an image processing system, an image processing method, and a program storage medium.

Background Art

[0002] Videos captured by imaging devices such as surveillance cameras are not only very useful for crime prevention and criminal investigations, but can also be used for various purposes such as in the marketing field through customer flow analysis. However, when using videos, it is desirable to protect personal information and portrait rights by abstracting the people shown. <00XXXXXX

[0003] On the other hand, for purposes such as marketing, in order to analyze the actions of people shown in videos, it is also desirable to be able to understand how the people are moving and what kind of interaction they are having with the background objects.

[0004] To meet both of these needs, it is required to make it impossible to identify who the people shown in the video are, while at the same time making the specific movements of the people visible and also showing what is in the background.

[0005] To meet at least some of such requirements, Patent Document 1 discloses a technique of detecting a face and applying mosaic processing to that part. Also, Patent Document 2 discloses a technique of applying mosaic processing or painting black to portions having a difference from the background. Patent Document 3 discloses a blurring processing technique as a means for protecting personal information. Patent Document 4 discloses a technique of synthesizing a foreground image in an image with a background image or performing mask processing or the like according to the authority for the foreground image of the user. Patent Document 5 discloses a technique of specifying a person area from an image and changing the person area to another image. Further, Non-Patent Document 1 discloses techniques such as see-through expression and expressing contours in a specific color.

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Patent No. 4036051 [Patent Document 2] Japanese Patent Publication No. 2000-000216 [Patent Document 3] Patent No. 4701356 [Patent Document 4] Japanese Patent Publication No. 2009-225398 [Patent Document 5] Japanese Patent Publication No. 2013-186838 [Non-patent literature]

[0007] [Non-Patent Document 1] Noboru Babaguchi, "Video Surveillance Considering Privacy," Information Processing, Special Issue, Vol. 48, No. 1, pp. 30-36, January, 2007. [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] However, simply applying mosaic processing to the face, as in the method described in Patent Document 1, does not guarantee complete protection of personal information because information about clothing and belongings other than the face remains. Patent Documents 2 and 3 disclose techniques for applying mosaic, blur, or blackout to the entire person, but since information within or around the person is greatly reduced, much information about the person's actions, such as picking up an object, is lost. With see-through representations, as in the method in Non-Patent Document 1, individuals can be easily identified. Furthermore, methods that represent individuals using outlines require high-precision extraction of the person, but accurately extracting a person's outline is technically difficult.

[0009] Some aspects of the present invention have been made in view of the aforementioned problems, and one of their objectives is to provide an image processing system, an image processing method, and a program storage medium that can suitably generate images that protect personal information. [Means for solving the problem]

[0010] An image processing system according to one aspect of the present disclosure includes means for detecting parts of a person in an image frame, means for detecting a plurality of feature points from the parts, and means for outputting an output image which is an image in which information about the feature points is superimposed on a background image at the positions of the detected plurality of feature points, wherein the display mode of the information about the feature points is at least one of two types, and the parts include the head and hands of the person.

[0011] An image processing method according to one aspect of the present disclosure includes detecting a part of a person in an image frame, detecting a plurality of feature points from the part, outputting an output image which is an image in which information about the feature points is superimposed on a background image at the positions of the detected plurality of feature points, wherein the display mode of the information about the feature points is one of at least two types, and the part includes the head and hands of the person.

[0012] A program according to one aspect of the present disclosure causes a computer to perform the following processes: a process to detect parts of a person in an image frame; a process to detect a plurality of feature points from the parts; and a process to output an output image, which is an image in which information about the feature points is superimposed on a background image at the positions of the detected plurality of feature points, wherein the display mode of the information about the feature points is one of at least two types, and the parts include the head and hands of the person.

[0013] In this invention, "part," "means," "apparatus," and "system" do not merely mean physical means, but also include cases where the functions of such "part," "means," "apparatus," and "system" are realized by software. Furthermore, even if the functions of one "part," "means," "apparatus," or "system" are realized by two or more physical means or devices, the functions of two or more "parts," "means," "apparatus," and "systems" may be realized by one physical means or device. [Effects of the Invention]

[0014] According to the present invention, it is possible to provide an image processing system, an image processing method, and a program storage medium that can preferably generate an image that protects personal information.

Brief Description of the Drawings

[0015] [Figure 1] It is a diagram for explaining an example of the relationship between an input image frame and a background image. [Figure 2] It is a diagram for explaining the outline of the processing of the image processing system according to the present embodiment. [Figure 3] It is a functional block diagram showing the schematic configuration of the image processing system according to the first embodiment. [Figure 4] It is a flowchart showing the processing flow of the image processing system shown in FIG. 3. [Figure 5] It is a block diagram showing the configuration of hardware capable of implementing the detection system shown in FIG. 3. [Figure 6] It is a functional block diagram showing the schematic configuration of the image processing system according to the second embodiment.

Modes for Carrying Out the Invention

[0016] Embodiments of the present invention will be described below. In the following description and the description of the accompanying drawings, the same or similar components are denoted by the same or similar reference numerals, respectively.

[0017] (1 First Embodiment) (1.1 Outline) FIGS. 1 to 5 are diagrams for explaining the first embodiment. The following description will be made with reference to these drawings.

[0018] The present embodiment relates to an image processing system that generates an image showing a moving object such as a person from an image taken by a photographing device such as a surveillance camera. In particular, the image processing system according to the present embodiment takes into account personal information and also takes into account the ability to perform behavior analysis of moving objects such as when picking up a product or which direction the object is facing.

[0019] To this end, the image processing system according to this embodiment extracts only the edge regions from the image frame to be processed (hereinafter also referred to as the original image). The image processing system then superimposes the original image of the edge region with a background image stored in the memory unit and outputs the superimposed image. As a result, the faces, clothing, etc. of moving objects such as people are not displayed in the output image to a degree that would allow for recognition, thus protecting personal information. On the other hand, since the outlines of moving objects such as people are included in the output image as edges, it is possible to analyze the orientation of people and their relationship to the background, such as products. Furthermore, even if edges other than those of moving objects are detected, the edges that fall on parts other than moving objects correspond to the background region in the original image, so there is no significant difference from the superimposed background image, and users viewing the image will not feel any particular discomfort.

[0020] The image processing system according to this embodiment detects edge regions in the image frame (original image) to be processed, but is not limited to this. For example, the image processing system may extract singular points such as corner points.

[0021] Figure 1 shows an example of image frames input to the image processing system according to this embodiment. In the example in Figure 1, image frames that make up the video, each taken at a different time, are input sequentially. The background image can be generated by methods such as averaging the pixel values ​​or taking the mode for each pixel of the multiple image frames taken at different times. Alternatively, a single fixed image frame that does not show a moving object can be used as the background image.

[0022] The following outlines the process up to generating the output image in this embodiment, with reference to Figure 2. In the example in Figure 2, it is assumed that the image frame to be processed (original image) containing the moving object and the background image are prepared in advance.

[0023] The image processing system first identifies the difference region between the image frame to be processed and the background image. More specifically, for example, it compares the pixel values ​​of the image frame to be processed with the pixel values ​​of the background image pixel by pixel, and then identifies the region where the difference between the two is greater than or equal to a threshold. This region usually corresponds to the area in the image frame where a moving object is captured.

[0024] Next, the image processing system performs edge detection on the difference region from the background image in the image frame to be processed, which corresponds to the area in which the moving object is captured. Then, the image processing system extracts only the edge region portion detected from the image frame to be processed. Finally, the image processing system generates the output image by superimposing the extracted portion of the image frame to be processed (original image) onto the background image.

[0025] The output image allows viewers to see the relationship between the moving object and the background, as well as the movements the moving object is performing. In particular, the outline of the moving object is visible, making it easier to visually observe or infer its gaze, facial direction, posture, etc. Furthermore, since the background image is displayed in areas other than the outline of the moving object, faces, clothing, etc., cannot be identified from the output image, thus protecting personal information.

[0026] The image processing system of this embodiment, having such functions, can be applied to safety fields such as surveillance using video from surveillance cameras, and to marketing fields such as analyzing the behavior of moving objects.

[0027] (1.2 System Configuration) The system configuration of the image processing system 100 according to this embodiment will be described below with reference to Figure 3. Figure 3 shows the system configuration of the image processing system 100 for generating images that take into consideration the protection of personal information. The image processing system 100 includes an image input unit 110, a background image generation unit 120, a background image database (DB) 130, a difference region extraction unit 140, an edge detection unit 150, an edge region extraction unit 160, an overlay unit 170, and an output unit 180. The image input unit 110 receives image frames that make up a video, that is, image frames taken at different times, from a shooting device such as a camera (not shown).

[0028] The background image generation unit 120 generates a background image from one or more image frames that are sequentially input from the image input unit 110. As mentioned above, for example, the background image may be generated by averaging the pixel values ​​of each pixel of image frames input in a certain period of time prior to the processing time in the time direction or by taking the mode. Alternatively, for example, it is conceivable to use a single image frame in which no moving objects are thought to be present as the background image. Note that if a fixed image frame is used as the background image, the image processing system 100 does not necessarily need to have the background image generation unit 120. The generated background image is stored in the background image DB 130.

[0029] The difference region extraction unit 140 extracts difference regions from the image frame to be processed input from the image input unit 110, where the difference between the image frame and the background image generated by the background image generation unit 120 is greater than or equal to a certain threshold. As mentioned above, these difference regions generally correspond to the areas in the image frame to be processed that contain moving objects.

[0030] The edge detection unit 150 performs edge detection on the difference regions in the image frame to be processed, which have been extracted by the difference region extraction unit 140. Various existing edge detection methods can be applied to the edge detection method used by the edge detection unit 150.

[0031] The edge region extraction unit 160 extracts only the regions detected as edges by the edge detection unit 150 from the image frame to be processed. In this case, for example, only a portion of the detected edge regions may be extracted. For example, it is conceivable to detect parts such as the head or hands of a moving object and then extract only those parts, or to change the transparency of the extracted parts depending on the part. Alternatively, it is conceivable to divide the detected predetermined edge region into units of a certain length and output them in units so that the edges or the entire edge corresponding to a predetermined part of the body are represented by dashed lines. In this way, the edge region extraction unit 160 may extract and output the regions detected as edge parts from the image frame to be processed in different ways depending on the corresponding part of the moving object, or it may output the entire detected edge by representing it with a dashed line, etc. In other words, the output method of the edge region extraction unit 160 is not limited to a specific one.

[0032] Alternatively, the transparency can be changed according to the strength (importance) of the edges detected by the edge detection unit 150. This improves visibility in the output image later generated by the superimposition unit 170, as the moving object (foreground) will appear darker in areas with strong edges considered to be of high importance, and lighter in areas with weak edges considered to be of low importance.

[0033] The superposition unit 170 generates an output image by superimposing the image frames to be processed from the edge region extraction unit 160 onto the background image. For example, if the original image (the image frame to be processed) is fg, the background image is bg, and (x,y) are the coordinates of each image, the output image out can be expressed as follows.

[0034] [Mathematics 1] TIFF0007845412000001.tif6150

[0035] Here, if (x,y) belongs to the edge region, α=1; otherwise, α=0. Furthermore, if transparency is set for the superimposed area of ​​the original image depending on the edge portion, as mentioned above, then α should be set as a real value in the range of 0≦α≦1 according to that transparency.

[0036] The output unit 180 outputs the generated output image. Various output methods are possible, but for example, the output images could be displayed sequentially on a device to show them to the user as a video, or these output images could be compiled and recorded as a video file on a storage medium.

[0037] In this embodiment, the difference region extraction unit 140 extracts a difference region from the background image in the image frame that corresponds to the moving object, and then performs edge detection only on that difference region. By performing edge detection only on the region where the moving object is presumed to be captured, it is possible to prevent the output image from including image frames of the background region that are not the target of analysis.

[0038] However, this is not the only option, and edge detection can also be performed without extracting the difference region. In such a case, the edge detection unit 150 will perform edge detection on the entire image frame, but since the pixel values ​​of areas other than the moving object in the image frame are not expected to differ significantly from the pixel values ​​of the background image, it is unlikely that the user viewing the output image will feel any discomfort. In this case, the image processing system 100 does not necessarily have to include the difference region extraction unit 140.

[0039] (1.3 Processing Flow) The processing flow of the image processing system 100 will be explained below with reference to Figure 4. Figure 4 is a flowchart showing the processing flow of the image processing system 100.

[0040] Furthermore, the processing steps described below can be arbitrarily rearranged or executed in parallel, provided that no inconsistencies arise in the processing content. Additional steps may also be added between each processing step. Moreover, steps described as a single step for convenience can be divided into multiple steps for execution, and steps described as multiple steps for convenience can be executed as a single step.

[0041] The image input unit 110 receives a new image frame (the image frame at the processing time) as input (S401). The background image generation unit 120 generates a background image from the input image frame and the background images stored in the background image DB 130, which were generated from image frames input before the processing time (S403).

[0042] The difference region extraction unit 140 extracts the difference region from the background image of the input image frame (S405). This difference region is identified, for example, by comparing the pixel value of each pixel in the input image frame with the pixel value of each pixel in the background image, and then identifying the region of pixels where the difference between the two is greater than or equal to a threshold. As mentioned above, the difference region corresponds to the region of the moving object in which the moving object is captured.

[0043] The edge detection unit 150 performs edge detection on the moving object region (difference region) extracted by the difference region extraction unit 140 (S407). The edge region extraction unit 160 extracts only the region detected as an edge from the image frame to be processed (S409). The superposition unit 170 generates an output image by superimposing the image frame to be processed, which was detected as an edge region, onto the background image stored in the background image DB 130 (S411).

[0044] The output unit 180 outputs the generated output image to, for example, a display device or a storage medium (S413). The image processing system 100 performs the processes S401 to S413 sequentially, enabling it to display or store the output image as video.

[0045] (1.4 Specific examples of hardware configurations) The following describes an example of a hardware configuration when the image processing system 100 described above is implemented using a computer, with reference to Figure 5. As shown in Figure 5, the image processing system 100 includes a processor 501, memory 503, storage device 505, input interface (I / F) unit 507, data I / F unit 509, communication I / F unit 511, and display device 513.

[0046] The processor 501 controls various processes of the image processing system 100 by executing programs stored in the memory 503. For example, the processes related to the image input unit 110, background image generation unit 120, difference region extraction unit 140, edge detection unit 150, edge region extraction unit 160, superimposition unit 170, and output unit 180 shown in Figure 3 can be implemented as programs that are temporarily stored in the memory 503 and then mainly run on the processor 501.

[0047] Memory 503 is a storage medium such as RAM (Random Access Memory). Memory 503 temporarily stores the program code of the program executed by the processor 501, as well as data required during program execution.

[0048] The storage device 505 is a non-volatile storage medium such as a hard disk or flash memory. The storage device 505 can store the operating system, various programs for realizing the functions of the image processing system 100, and various data including the background image DB 130. The programs and data stored in the storage device 505 are loaded into the memory 503 as needed and accessed by the processor 501.

[0049] The input interface unit 507 is a device for receiving input from the user. Specific examples of the input interface unit 507 include a keyboard, mouse, and touch panel. The input interface unit 507 may be connected to the image processing system 100 via an interface such as USB (Universal Serial Bus).

[0050] The data I / F unit 509 is a device for inputting data from outside the image processing system 100. Specific examples of the data I / F unit 509 include a drive device for reading data stored in various storage devices. The data I / F unit 509 may be located outside the image processing system 100. In that case, the data I / F unit 509 is connected to the image processing system 100 via an interface such as USB.

[0051] The communication interface unit 511 is a device for wired or wireless data communication between the image processing system 100 and external devices, such as imaging devices (video cameras, surveillance cameras, digital cameras, etc.). The communication interface unit 511 may be located outside the image processing system 100. In that case, the communication interface unit 511 is connected to the image processing system 100 via an interface such as USB.

[0052] The display device 513 is a device for displaying, for example, an output image output by the output unit 180. Specific examples of the display device 513 include liquid crystal displays and organic EL (Electro-Luminescence) displays. The display device 513 may be located outside the image processing system 100. In that case, the display device 513 is connected to the image processing system 100 via, for example, a display cable.

[0053] (1.5 Effects according to this embodiment) As described above, the image processing system 100 according to this embodiment generates an output image by superimposing only the image frame (original image) to be processed in the edge region onto the background image. This allows users viewing the output image to confirm the shape of the moving object corresponding to the detected edge region and its relationship to the background. Furthermore, since the area of ​​the moving object other than the edge region is not included in the output image, it is not possible to determine what kind of user is depicted from the output image. In other words, the output image generated by the image processing system 100 is designed with consideration for personal information.

[0054] Methods that detect moving objects and display their outlines have a problem: if the detection accuracy of the moving object is insufficient, parts other than the moving object may be displayed in the output image. In contrast, the method according to this embodiment displays the image frame itself that is the target of edge region processing. Therefore, even if parts other than the moving object are detected as edge regions, they will blend in almost with the background image, making it less likely for users to feel any discomfort when viewing the output image.

[0055] (2. Second Embodiment) The second embodiment will be described below with reference to Figure 6. Figure 6 is a block diagram showing the functional configuration of the image processing system 600 according to this embodiment. The image processing system 600 includes an input unit 610, a detection unit 620, and an output unit 630. The input unit 610 receives input of multiple image frames that make up an image from a shooting device such as a camera (not shown). The detection unit 620 detects feature points contained in the image frame to be processed among multiple image frames. Here, feature points may include edges such as the outline of a moving object, such as a person, or singular points such as corner points. The output unit 630 outputs an output image in which the image frame to be processed, which is the region detected as a feature point, is superimposed on a background image generated from at least a portion of multiple image frames. By implementing it in this way, the image processing system 600 according to this embodiment can suitably generate images that protect personal information.

[0056] (3. Additional Notes) Furthermore, the configurations of the embodiments described above may be combined or some of the components may be replaced. Also, the configuration of the present invention is not limited to the embodiments described above, and various modifications may be made without departing from the spirit of the present invention.

[0057] Furthermore, some or all of the embodiments described above may also be described as follows, but are not limited to these. In addition, the program of the present invention may be any program that causes a computer to execute the operations described in each of the embodiments above.

[0058] (Note 1) An image processing system comprising: means for receiving input of multiple image frames constituting an image from a shooting device; detection means for detecting feature points included in the image frame to be processed among the multiple image frames; and means for outputting an output image in which the image frame to be processed in the region detected as a feature point is superimposed on a background image generated from at least a portion of the multiple image frames.

[0059] (Note 2) The aforementioned characteristic feature is the edge portion, as described in Appendix 1 of the image processing system.

[0060] (Note 3) The image processing system according to Appendix 2, wherein the detection means detects the edge portion in a region of the image frame to be processed where the difference from the background image is greater than or equal to a threshold.

[0061] (Note 4) The image processing system according to Appendix 2 or Appendix 3, wherein the output image is obtained by superimposing a portion of the image frames to be processed, which are part of the region detected as the edge portion, onto the background image.

[0062] (Note 5) The image processing system according to any one of Appendix 2 to Appendix 4, wherein the output image is an image frame of the region detected as the edge portion superimposed on the background image in a manner that differs depending on the position.

[0063] (Note 6) The image processing system according to Appendix 5, wherein the output image is obtained in which the image frames of the region detected as the edge portion are superimposed on the background image in different ways depending on the part of the moving object.

[0064] (Note 7) An image processing method in which an image processing system performs the steps of: receiving input of multiple image frames constituting an image from a shooting device; detecting feature points included in the image frame to be processed among the multiple image frames; and outputting an output image in which the image frame to be processed in the region detected as a feature point is superimposed on a background image generated from at least a portion of the multiple image frames.

[0065] (Note 8) The aforementioned characteristic feature is the edge portion, as described in Appendix 7 of the image processing method.

[0066] (Note 9) The image processing method according to Appendix 8, which detects the edge portion in a region of the image frame to be processed in which the difference from the background image is greater than or equal to a threshold.

[0067] (Note 10) The image processing method according to Appendix 8 or Appendix 9, wherein the output image is obtained by superimposing a portion of the image frames to be processed from the region detected as the edge portion onto the background image.

[0068] (Note 11) The image processing method according to any one of Appendix 8 to Appendix 10, wherein the output image is obtained by superimposing image frames of the region detected as the edge portion onto the background image in a manner that differs depending on the position.

[0069] (Note 12) The image processing method according to Appendix 11, wherein the output image is obtained in which the image frames of the region detected as the edge portion are superimposed on the background image in different manner depending on the part of the moving object.

[0070] (Note 13) A program that causes a computer to perform the following steps: receiving input of multiple image frames constituting an image from a shooting device; detecting feature points included in the image frame to be processed among the multiple image frames; and outputting an output image in which the image frame to be processed in the region detected as a feature point is superimposed on a background image generated from at least a portion of the multiple image frames.

[0071] (Note 14) The aforementioned characteristic feature is the edge portion, as described in Appendix 13 of the program.

[0072] (Note 15) The program described in Appendix 14, which detects the edge portion in the region of the image frame to be processed in which the difference from the background image is greater than or equal to a threshold.

[0073] (Note 16) The program according to Appendix 14 or Appendix 15, wherein the output image is obtained by superimposing a portion of the image frames to be processed, which are part of the region detected as the edge portion, onto the background image.

[0074] (Note 17) The program according to any one of Appendix 14 to Appendix 16, wherein the output image is a program in which image frames of the region detected as the edge portion are superimposed on the background image in a manner that differs depending on the position.

[0075] (Note 18) The output image is a program as described in Appendix 17, wherein the image frames of the region detected as the edge portion are superimposed on the background image in different ways depending on the corresponding part of the moving object.

[0076] The present invention has been described above using the embodiments described above as exemplary examples. However, the present invention is not limited to the embodiments described above. That is, the present invention can be applied in various forms that can be understood by those skilled in the art within the scope of the present invention. This application claims priority based on Japanese Patent Application No. 2014-134961, filed on 30 June 2014, and incorporates all of its disclosures herein. [Explanation of Symbols]

[0077] 100: Image processing system 110: Image input section 120: Background image generation unit 130: Background Image Database 140: Difference area extraction part 150: Edge detection unit 160: Edge region extraction unit 170: Overlapping section 180: Output section 501: Processor 503: Memory 505: Storage device 507: Input Interface Section 509: Data Interface Section 511: Communication Interface Unit 513:Display device 600: Image Processing System 610: Input section 620: Detection unit 630: Output section

Claims

1. A means for detecting parts of a person in an image frame, Means for detecting multiple feature points from the aforementioned part, means for outputting an output image which is an image in which the detected feature points are superimposed on a background image at positions corresponding to the positions of the plurality of feature points with transparency according to the importance of the feature points, Includes, The aforementioned parts include the head and hands of the person. Image processing system.

2. The means for outputting the output image outputs the output image sequentially. The image processing system according to claim 1.

3. The aforementioned feature point is an edge. The image processing system according to claim 1 or 2.

4. The aforementioned feature is the outline of the moving object that appears in the image frame. The image processing system according to any one of claims 1 to 3.

5. The moving object is the person. The image processing system according to claim 4.

6. The system further comprises means for analyzing the behavior of the aforementioned person. The image processing system according to claim 5.

7. Detects parts of a person in the image frame, Multiple feature points are detected from the aforementioned region, An output image is produced in which the detected feature points are superimposed on the background image at positions corresponding to the locations of the multiple feature points, with transparency corresponding to the importance of each feature point. The aforementioned parts include the head and hands of the person. Image processing methods.

8. A process for detecting parts of a person in an image frame, A process for detecting multiple feature points from the aforementioned part, The computer is instructed to perform the following process: output an output image, which is an image in which the detected feature points are superimposed on the background image at positions corresponding to the positions of the multiple feature points, with transparency corresponding to the importance of the feature points. The aforementioned parts include the head and hands of the person. program.

Images