Image masking device and image masking method

Abstract

To provide an image masking device and method for automatically masking, in order to protect personal information, an object which may identify a face or an individual captured by a monitoring camera system or image sensor installed in various locations, the image masking device and method retaining a function for removing the masking when it is necessary to remove the masking in case of a criminal investigation or accident.SOLUTION: An image masking device includes: an image input reception unit that receives an input of an image; a masking portion extraction unit that automatically extracts a portion to be masked from the input image; a masking unit that masks the extracted portion to be masked; and an image output unit that outputs the masked image. The image masking device may also include an unmasking unit that removes the mask on the image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image masking device and an image masking method for masking a specific portion of an image. Masking refers to covering up a part of an image. [Background technology]

[0002] 2. Description of the Related Art In recent years, surveillance cameras have been widely installed for the purpose of preventing crimes or grasping the circumstances when crimes occur, and have been functioning effectively for those purposes.

[0003] However, the images contain information that is unrelated to crime but that violates personal privacy, such as facial images of individuals and vehicle license plates, which is causing problems.

[0004] To prevent such problems from occurring, Patent Document 1 discloses a technical concept in which a surveillance camera stores mask data that masks privacy zones that appear in an image, and is configured to mask parts of the image in accordance with this data.

[0005] Furthermore, the technical idea of ​​removing the masking can only be done by specific people who have a password, preventing arbitrary operation by the general public, is also disclosed. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2001-69494 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the privacy zone in Patent Document 1 requires the operator to specify the center position and zone frame, which is time-consuming as it requires manual operation, and there is a problem that information outside the range is not masked, meaning that personal information is not protected.

[0008] In consideration of the above-mentioned problems, an object of the present invention is to provide an image masking device and method that automatically masks faces or other identifying objects from the perspective of protecting personal information when they are captured in surveillance camera systems and image sensors that will be installed in various locations in the future.Furthermore, an object of the present invention is to provide an image masking device and method that has a function for removing masking when it becomes necessary to do so in the event of a criminal investigation or incident. [Means for solving the problem]

[0009] In order to solve this problem, an image masking device according to the present invention comprises: A) an image input receiving unit that receives an input of an image; B) a masking target portion extraction unit that automatically extracts a masking target portion from an input image; C) a masking applying unit that applies a mask to the extracted masking target portion; D) an image output unit that outputs the masked image; The present invention is characterized by having the following.

[0010] Here, images include both still images and moving images, and the image input receiving unit receives inputs such as those input directly from an imaging unit such as a surveillance camera, those received by the image receiving unit via a wired or wireless network or the Internet, and those that have already been photographed or processed and stored in the image storage unit.

[0011] Furthermore, outputting an image from the image output unit includes displaying the image on the image display unit, storing the image in a storage device, and transmitting the image to an external device such as a network via the image transmission unit.

[0012] In addition, the image masking device of the present invention may be characterized in that the part to be masked is one or more of the face of an animal including a human, the outer shape of an animal including a human, a vehicle number, the exterior of a vehicle, a nameplate, the exterior of a house, and an area containing personal information.

[0013] Here, the part containing personal information refers to the part of a document or card containing personal information, such as a driver's license, passport, or personal identification number card.

[0014] Furthermore, the target images include not only images from surveillance cameras, but also all types of moving and still images, such as television broadcast images, images taken with smartphones and portable cameras, etc.

[0015] For example, in television broadcasts and movies, the faces of people in the background are sometimes masked (or pixelated) during interviews, but in such cases, it is possible to perform the masking automatically.

[0016] In this way, the masking target portion can be extracted and masked without manual intervention, greatly expanding the range of applications for surveillance camera images and the like.

[0017] The image masking device according to the present invention further comprises: E) a masking target attribute detection unit that detects attribute information of the masking target portion; The present invention may be characterized by having the following.

[0018] In other words, if the part to be masked is a human face, attribute information such as gender, age, clothing, and facial features (glasses, beard, amount of hair, etc.) can be detected from the image, and this attribute information can be linked to the part to be masked using techniques such as tagging and output, allowing it to be effectively used for various subsequent purposes.

[0019] It should be noted that attributes may be manually assigned without automatically detecting attributes, or in combination with automatic detection, which may increase the accuracy of attribute information.

[0020] Furthermore, the image masking device according to the present invention may have a function of searching for the portion to be masked using the attribute information.

[0021] In this way, it is possible to extract a specific image from a vast amount of image information based on the combination of attribute information in the attached tags (for example, male, wearing glasses, etc.), and further, even when unmasking becomes necessary, the image to be unmasked can be identified and processed efficiently.

[0022] The image masking device according to the present invention further comprises: F) a masking target statistical processing unit that performs statistical processing on the masked image using the attribute information; The present invention may be characterized by having the following.

[0023] In this way, in the case of facial images, statistical data compiled by customer attributes (gender, age, etc.) necessary for marketing purposes can be obtained without identifying individuals.

[0024] Furthermore, the image masking device according to the present invention may be characterized in that, when specific information that does not require masking is recognized in image information, the image masking device masks a masking target portion related to the specific information.

[0025] In this way, it is possible to mask other information that appears in the same image in relation to information that is desired to be used without masking, and it is possible to use image information while protecting personal information, etc.

[0026] In particular, the image masking device according to the present invention may be characterized in that the specific information is a vehicle number, and the portion to be masked is all or part of the driver and / or passengers of the vehicle.

[0027] This is extremely effective when masking the vehicle number is not necessary for purposes such as toll collection, but the faces of the driver and passengers of the vehicle must be masked for the purpose of protecting personal information.

[0028] Here, the image masking device according to the present invention may be characterized in that the masking target portion extraction unit extracts the masking target portion by image processing using machine learning.

[0029] In this way, the masking target portion can be extracted with high accuracy without manual intervention.

[0030] Furthermore, the image masking device according to the present invention may be characterized in that the machine learning utilizes a convolutional neural network.

[0031] This method is an excellent machine learning method for image processing, making it possible to extract the masking target portion with higher accuracy.

[0032] Alternatively, the image masking device according to the present invention may be characterized in that the masking target portion extraction section extracts the masking target portion by image processing using edge extraction processing.

[0033] In this way, depending on the target image, it may be possible to extract the masking target portion simply and with high accuracy.

[0034] Alternatively, the image masking device according to the present invention may be characterized in that the masking target portion extraction section extracts the masking target portion by image processing using OCR processing.

[0035] In this way, particularly when the portion to be masked contains text information, it becomes easier to extract that portion.

[0036] Furthermore, the image masking device according to the present invention comprises: G) an unmasking unit that removes the masking from the image to which the masking has been applied; The present invention may be characterized by having the following.

[0037] In this way, the masked image information can be unmasked for a specific purpose, such as criminal investigation, and the image information can be used effectively.

[0038] The image masking device according to the present invention may be characterized in that the part from which the masking is to be removed is determined based on attribute information of the part to be masked.

[0039] In this way, for example, specific images that require unmasking can be extracted from surveillance camera images using attribute information written on tags attached to the images, thereby reducing manual intervention and enabling highly accurate unmasking without oversight.

[0040] The unmasking unit may be characterized in that it can unmask only by a specific procedure.

[0041] For example, the specific procedure may be characterized by storing the masked image in association with the encrypted original image of the masked portion, and upon unmasking, restoring the original image of the masked portion and fitting it into the masked image.

[0042] Specifically, only the police or persons with specific authority can decrypt the data using biometric authentication such as an ID card, password, fingerprint, or iris. Alternatively, the operation can be restricted to only be performed from a specific device.

[0043] For example, it is conceivable that approved masking methods, similar to the TSA locks used on suitcases in the United States, could be installed on surveillance cameras around the world, preventing private citizens from seeing certain information on the surveillance camera, but allowing only police or other authorized personnel to unmask it.

[0044] In the case of the aforementioned television broadcast footage, it is possible to unmask only the people being interviewed during editing.

[0045] Furthermore, the present invention may be embodied in such a manner that the functions of the image masking device are embodied as an image masking method. The effects of the present invention can be realized regardless of the configuration of the device.

[0046] The present invention may also be in the form of an image masking unit having a masking target portion extraction unit that automatically extracts a masking target portion from an input image, and a masking application unit that applies masking to the extracted masking target portion.

[0047] Furthermore, the image masking unit may be configured to include at least one of a masking target attribute detection unit that detects attribute information of the masking target portion, a masking target statistical processing unit that performs statistical processing on the masked image using the attribute information, and an unmasking unit that removes the masking from the masked image.

[0048] In this way, other peripheral units can be combined without restrictions, and the effect of the present invention can be achieved with only the image masking unit. [Effects of the Invention]

[0049] According to the image masking device and image masking method of the present invention, it is possible to automatically mask the masking target parts in images from surveillance cameras, etc., so that personal information is protected, and when it becomes necessary to remove the masking during a criminal investigation, etc., the masking can be removed through a specific, controlled procedure, which has the effect of enabling the use of images from a public interest perspective. [Brief explanation of the drawings]

[0050] [Figure 1] 1 is a block diagram of an embodiment of an image masking device of the present invention; [Figure 2] 1 is a flowchart showing a process up to masking in an embodiment of the image masking device of the present invention. [Figure 3] 1 is a flowchart of facial image extraction in an embodiment of the device of the present invention. [Figure 4] FIG. 1 is an illustration of an original image of an embodiment of the device of the present invention. [Figure 5] FIG. 10 is an explanatory diagram of an image in which a masking target region is extracted by the device according to the embodiment of the present invention. [Figure 6] FIG. 10 is an illustration of an image after applying masking in one embodiment of the device of the present invention. [Figure 7] 1 is a flowchart of unmasking in an embodiment of the image masking device of the present invention. [Figure 8] 10 is an explanatory diagram of an original image of another embodiment of the image masking device of the present invention. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0051] An image masking device according to one embodiment of the present invention will be described below with reference to the drawings. Note that the scope necessary for the explanation to achieve the object of the present invention will be schematically shown below, and the scope necessary for explaining the relevant parts of the present invention will be mainly explained, and the parts that are omitted from the explanation will be based on publicly known techniques.

[0052] FIG. 1 is a block diagram of an embodiment of an image masking device 1 of the present invention.

[0053] The image masking device 1 is preferably configured as a PC in terms of hardware, and is composed of a CPU, memory, a communication interface, a display, and the like.

[0054] Functionally, the image masking device 1 includes an image masking unit 10, a control unit 20, an image input receiving unit 30, and an image output unit 40, which are realized using these hardware components, and the image masking unit 10 further includes functional elements of a masking target portion extraction unit 110, a masking target attribute detection unit 120, a masking target statistical processing unit 130, a masking application unit 140, and an unmasking unit 150.

[0055] The control unit 20 is responsible for overall control, and controls the reception of image input from the image input reception unit 30 , the output of images to the image output unit 40 , and the masking and unmasking of images using the image masking unit 10 .

[0056] The image input receiving unit 30 receives input of an image that requires masking. For example, the image input is received from an image receiving unit 60 that receives images transmitted from an imaging unit 50 such as a camera via wired or wireless communication or the Internet, or from an image storage unit 70 that stores previously captured or processed images.

[0057] Here, the imaging unit 50 is typically a surveillance camera installed in a fixed location, but it can also be anything that can capture video and still images, such as an in-vehicle camera, a television or movie camera, a smartphone, or a camera.

[0058] The image masking unit 10 includes a masking target portion extraction unit 110, a masking target attribute detection unit 120, a masking target statistical processing unit 130, a masking applying unit 140, and an unmasking unit 150, and processes the image transmitted from the control unit 20.

[0059] The masking target portion extraction unit 110 extracts the masking target portion from within the image.

[0060] The masking target attribute detection unit 120 detects the attributes of the masking target portion, for example, in the case of a human face, attributes such as gender and age.

[0061] The masking target statistical processing unit 130 performs statistical processing of the masking target by attribute, by day, by time period, etc.

[0062] The masking applying unit 140 applies masking to the extracted masking target portion.

[0063] The unmasking unit 150 unmasks all or part of the masked portion of the masked image.

[0064] The image output unit 40 is connected to the control unit 20 and outputs the image captured by the imaging unit 50, the image processed by the image masking unit 10, attributes associated with the image, and further information required for unmasking.

[0065] The output destinations include an image display unit 90 that displays the image, an image storage unit 70 that stores the image, or an image transmission unit 80 that transmits the image via wired or wireless communication or the Internet.

[0066] It should be noted that the image masking device is not limited to the configuration described above, and some or all of the imaging unit, image receiving unit, image storage unit, image transmission unit, and image display unit may be built into it, or conversely, some of the functions of the image masking device may be built into the imaging unit, or the functions of the image masking device may be distributed differently, or even the functions of the image masking device may be remotely distributed, such as in cloud computing.

[0067] Next, the operation of the image masking device of the present invention configured as above will be described. Figure 2 is a flowchart showing the process up to masking in one embodiment of the image masking device 1 of the present invention.

[0068] First, the image input receiving unit 30 receives an input of an image (step S01). Here, for example, it is assumed that a video image continuously captured by a surveillance camera that monitors a street as the imaging unit 50 is input.

[0069] Next, the masking target portion extraction unit 110 extracts the masking target portion (S02). Here, from the viewpoint of protecting personal privacy, the masking target portion is a person's face image in a surveillance camera image.

[0070] Specifically, a captured image is subjected to machine learning, particularly deep learning using an artificial neural network, to extract what appears to be a facial image. Figure 3 is a flowchart of facial image extraction in one embodiment of the image masking device 1 of the present invention, and Figures 4 to 6 are explanatory diagrams of images from facial image extraction to masking in one embodiment of the image masking device 1 of the present invention, where Figure 4 is the original image, Figure 5 is an image from which the masking target area has been extracted, and Figure 6 is the image after masking has been applied. Note that the diagram of the extracted masking target area corresponding to Figure 5 does not need to be displayed as an image in actual use.

[0071] First, classified learning data (in the case of facial images, whether the image is a facial image or not, and the position of the outline of a rectangle or the like that contains the facial image) is collected from a surveillance camera or the like (S11). The larger the amount, the better.

[0072] Next, the learning data is divided into training data and validation data (S12).

[0073] For this, a neural network is used to learn using training data, weight adjustment is performed, and the current learning status is verified using verification data (S13).

[0074] This process is repeated until the error rate during verification becomes smaller than a desired value, at which point the learning process is completed (S14).

[0075] Next, a new image, for example, an image as shown in FIG. 4 captured by the image capturing unit 50, is input to the neural network that has completed learning (S15).

[0076] Then, the portion to be masked, for example, a person's face image, is extracted from this (S16). As shown in Fig. 5, a woman's face 101 facing forward in the upper right corner of the screen, a man's face 102 facing sideways in the lower right corner, and a child's face 103 in the center are extracted and enclosed in a rectangle. Note that this shape is not limited to this and may be any shape, such as a circle, an ellipse, or a polygon.

[0077] The machine learning method is not limited to neural networks or deep learning, but may be any machine learning method such as decision tree learning or support vector machine (SVM), although additional manual input may be required.

[0078] Methods that do not rely on machine learning can also be used. For example, edge extraction processing can be used. This method regards sharp changes in image brightness as edges, or contours, and specific methods have been put to practical use.

[0079] Alternatively, for images containing text, such as vehicle license plates or address information, the locations to be masked may be determined using optical character recognition (OCR).

[0080] Here, when extracting the masking target portion, the masking target attribute detection unit 120 detects the required attributes from the masking target (S03).

[0081] For example, in the case of a human face, the necessary attributes include gender, age, and characteristics (body shape, glasses, beard, hair volume, etc.). This is useful for later unmasking or statistical processing.

[0082] In addition, the masking target portion extraction unit 110 and the masking target attribute detection unit 120 can also apply commercially available technologies, such as FieldAnalyst (registered trademark) from NEC Solution Innovators, Ltd. and the face authentication system from Glory Co., Ltd.

[0083] The detected attributes are stored by the control unit 20 in association with the extracted masking target portion, for example, by tagging the masking target portion.

[0084] Next, the masking unit 140 encrypts the original image of the masking target portion (S04). This is to enable unmasking, but is a process that prevents unmasking without performing a specific operation.

[0085] At the same time, the masking applying unit 140 applies masking to the extracted target portions 201, 202, and 203 (S04), as shown in Fig. 6. The masking is performed by painting the entire area opaquely with gray, but is not limited to this and may be any type of masking, such as a color, a degree of transparency, or a mosaic pattern instead of a solid color, and may have a frame, or may be an achromatic color or a chromatic color other than gray.

[0086] After masking is performed, the image information of the masked portion is overwritten and lost, and cannot be restored from the image.

[0087] Next, the masked image, the encrypted original image of the masked portion, and the attributes of the masked portion are stored in association with each other in the image storage unit 70 via the image output unit 40 (S06). This is to ensure that unmasking can be performed reliably if it becomes necessary later.

[0088] All of the operations up to this point (S01 to S06) are performed automatically, so no human intervention is required, making the process extremely efficient and eliminating the risk of infringing personal privacy due to human intervention.

[0089] If necessary, the masked image is displayed on the image display unit 90 via the image output unit 40 (S07).

[0090] Furthermore, if necessary, the masking target statistical processing unit 130 can perform statistical aggregation processing in association with the detected attributes (S08).

[0091] For example, for images of people's faces, statistics such as 56% male, 44% female, 32% under 40 years old, 54% between 40 and 60 years old, and 14% over 60 years old can be obtained. This is effective for utilizing the results of classification by attributes for marketing purposes without disclosing personal information.

[0092] At present, it is desirable to use a convolutional neural network (CNN) as a neural network for image processing such as that of the present invention. For example, see IEEE Signal Processing, August 26, 2016, Joint Face Detection and Alignment Using Letters The method described in Multitask Cascaded Convolutional Networks is extremely effective. It has a network structure in which multiple (3 or 4) shallow deep learnings are cascaded, enabling it to operate at high speed.

[0093] Because it can process at high speed, it is possible to extract the masking target area for each frame and apply masking. Therefore, it is possible to handle images not only from fixed cameras but also from moving objects such as in-vehicle cameras. Note that it is also possible to use a method in which the masking area is determined by the difference from the previous frame.

[0094] Next, a description will be given of the operation of the unmasking unit 150. Fig. 7 is a flowchart relating to unmasking in one embodiment of the image masking device 1 of the present invention.

[0095] The unmasking unit 150 reads out the masked image and the original image of the encrypted masking target portion, which is stored in association with the masked image, from the image storage unit 70 via the image input receiving unit 30 (S21).

[0096] Next, the original image of the masked portion is restored (S22). This can be done by predetermining an operator or device with specific authority and starting the operation by inputting a password, personal ID, or biometric authentication such as a fingerprint or iris.

[0097] Next, the restored image of the masking target portion is fitted into the target portion of the masked image (S23).

[0098] In this way, an unmasked image can be obtained, and if necessary, the unmasked image can be stored in the image storage unit 70 via the image output unit 40, displayed on the image display unit 90, or remotely transmitted from the image transmission unit 80.

[0099] In the explanation so far, it has been stated that after masking is performed, the image information of the masked portion is overwritten and lost, and cannot be restored from that image, and the original image of the masked portion is encrypted and stored, but it is also possible to maintain the image information of the masked portion even after masking is performed, and to encrypt and store only information such as the coordinates of the masked portion.

[0100] In this case, when unmasking, the coordinate information of the portion to be masked is decoded and used to restore the original image of the masked image.

[0101] In this way, since the amount of encrypted information is small, it is expected that high-speed processing will be possible.

[0102] In addition, both the image before and after masking may be stored. Privacy of the image before masking is protected by strict access management (encryption, passwords, etc.).

[0103] In this way, a certain degree of privacy protection can be achieved without requiring complex processing.

[0104] In the explanation so far, the part to be masked has been described as a person's face, but it may also be other personal information such as an animal's face, the outline of an animal including a human, a vehicle number, the exterior of a vehicle, a nameplate, the exterior of a house, or an area containing personal information.

[0105] Here, the part containing personal information refers to the part of a document or card containing personal information, such as a driver's license, passport, or personal identification number card.

[0106] In addition, when specific information that does not require masking is recognized in the image information, such as a necessary vehicle number on a road or in a parking lot, the parts that require masking related to that specific information, such as the driver or passengers of the vehicle, may be masked.

[0107] 8 is an explanatory diagram of an original image of another embodiment of the image masking device 1 of the present invention, which targets vehicle license plates 401 and 402. In the case of license plates, the symbols and numbers on the license plates can be automatically read during unmasking, which can be useful in criminal investigations. [Industrial Applicability]

[0108] The image masking device of the present invention can be used to process a wide range of images that require privacy protection, not just surveillance camera images, and therefore has great industrial applicability. [Explanation of symbols]

[0109] 1 Image masking device 10 Image Masking Unit 20 Control Unit 30 Image input reception unit 40 Image output unit 50 Imaging unit 60 Image receiving unit 70 Image storage unit 80 Image transmission unit 90 Image display section 110 Masking target portion extraction unit 120 Masking target attribute detection unit 130 Masking target statistical processing section 140 Masking section 150 Unmasking Department 101-103 Masking target area 201-203 Masking section 401-402 license plate

Claims

[Claim 1] A) an image input receiving unit that receives an input of an image; B) a masking target portion extraction unit that automatically extracts a masking target portion from an input image; C) a masking applying unit that applies a mask to the extracted masking target portion; D) an image output unit that outputs the masked image; An image masking device comprising:

Images