Image processing apparatus and image processing method

Abstract

In an image processing device, personal information within an image is altered while the image is processed to create a natural-looking image without any unnatural appearance. [Solution] The image processing device maintains a face feature table that links face features with whether or not consent has been given to use those face features, and an image processing method rule table that defines personal relevance, which quantitatively evaluates the connection between personal information features and individuals, and rules for image processing methods. The device detects images of people and the facial regions of those people from the input images, and determines the processing area according to the consent given to use the image of the person authenticated from the face region features. Then, it obtains personal information features from the image of the area, and processes the image of the processing area based on the personal relevance regarding the personal information features and the processing method defined in the image processing method rule table.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and an image processing method, and more particularly to an image processing apparatus and an image processing method suitable for processing an image into a natural image without a sense of incongruity while changing personal-related information in the image.

Background Art

[0002] In recent years, due to the progress of digital technology, changes in economic and social situations such as globalization, and the increasing awareness of personal information in society, the need to protect personal information has been emphasized. Therefore, the "Personal Information Protection Law" has been enacted for the purpose of protecting the rights and interests of individuals while considering the usefulness of personal information. In this law, "personal information" is defined as information related to living individuals, such as information that can identify a specific individual by name, date of birth, address, face photo, etc. (Article 2, Paragraph 1). In addition, "personal-related information" is also defined as a protected target in this law. "Personal-related information" corresponds to information related to living individuals other than personal information (Article 2, Paragraph 7). Information related to an individual is, for example, information representing facts, judgments, and evaluations regarding the attributes of an individual's body, property, occupation, title, etc.

[0003] Regarding such personal-related information, recently, in images and videos taken in public places, there are sometimes people who have not consented to the shooting included in the background. When using these images and videos, it is required to consider not being able to identify individuals. In addition to biometric information such as faces that identify individuals, it is also possible to estimate from combinations of clothing, hairstyles, body shapes, possessions, etc. for individual identification. As in the Personal Information Protection Law shown above, there are particularly strict regulations regarding the handling of personal information and privacy. There are risks of lawsuits and penalties if data with unprotected privacy is publicly disclosed in places where an unspecified large number of people can see it, such as on the Internet, without obtaining the consent of the individual.

[0004] To prevent the identification of personal information from such images and videos, it is common practice to blur, mosaic, or black out the target video area using video filtering. However, video processing requires skilled workers and editing equipment, making it difficult for ordinary people to use images and videos they have taken directly on social networking services (SNS). Therefore, as a mechanism to automate the process of processing personal information from videos, for example, Patent Document 1 proposes an information processing device that detects the face image area of ​​a person and protects privacy by replacing that image area with another image according to the face attributes. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2014-85796 [Overview of the project] [Problems that the invention aims to solve]

[0006] Generally, when taking photographs with a camera, people may appear in the background. If the photographer shares photos of people who appear in the background without permission using services such as social media, the identity of those people may be revealed, leading to privacy violations and the leakage of personal information. To avoid these problems, it is necessary to mask faces or blur backgrounds from a privacy protection standpoint. However, such processing can leave an unnatural appearance in the image.

[0007] Filtering techniques such as blurring, mosaic, and blacking out personal information from video footage requires manual editing and is time-consuming. Furthermore, because these techniques also process background information and other non-personally relevant details, they degrade the quality of videos intended to convey the atmosphere of a location.

[0008] The information processing device described in Patent Document 1 above protects privacy by detecting the facial image region of a person and replacing that image region with another image corresponding to the facial attributes. This makes it possible to protect privacy by transforming a person's facial image in a way that does not cause any unnaturalness.

[0009] However, while the method described in Patent Document 1 can protect a certain degree of privacy by transforming only a part of a person, it does not completely prevent the identification of an individual. If personal information is present in the image, it may still be possible to identify the individual from that information.

[0010] The object of the present invention is to provide an image processing apparatus that can process an image to create a natural-looking image without any sense of incongruity, while simultaneously altering personal information within the image. [Means for solving the problem]

[0011] The configuration of the image processing apparatus of the present invention is preferably an image processing apparatus that takes an image as input and processes the image to change the personal information of an individual appearing in the image, and holds a first table that links facial features associated with an individual with whether or not consent has been given to use the facial features, and a second table that defines an image processing method relating to the characteristics of personal information, a processing target detection unit that detects a person and the region of that person's face from an input image, a facial recognition unit that extracts facial features from the region of the face and performs facial recognition by comparing them with the facial features held in the first table, and based on the first table and the facial features of the person detected by the processing target detection unit, The system includes a processing / non-processing separation unit that determines whether the image of a person and the area of ​​that person's face is a processing area or a non-processing area; a personal-related information feature acquisition unit that acquires personal-related information features from the image of a person and the area of ​​that person's face; an image processing method determination unit that determines an image processing method for the processing area based on the processing method defined in a second table and the personal-related information features acquired from the image of a person and the area of ​​that person's face; and an image processing unit that processes the image of a person and the area of ​​that person's face detected by the processing target detection unit according to the image processing method determined by the image processing method determination unit.

[0012] More specifically, in the image processing apparatus described above, the second table further holds a personal relevance score that quantitatively evaluates the characteristics of personal information, and the image processing unit changes the degree of image processing for the person detected by the processing target detection unit and the area of ​​that person's face, based on the personal relevance score. [Effects of the Invention]

[0013] According to the present invention, it is possible to provide an image processing device that can modify personal information within an image while processing it into a natural-looking image without any sense of incongruity. [Brief explanation of the drawing]

[0014] [Figure 1] This is a functional configuration diagram of an image processing device. [Figure 2]It is a hardware-software configuration diagram of an image processing apparatus. [Figure 3] It is a diagram showing an example of a face feature amount table. [Figure 4] It is a diagram showing an example of an image processing method rule table. [Figure 5A] It is a flowchart showing details of an image processing method of an image processing apparatus (Part 1). [Figure 5B] It is a flowchart showing details of an image processing method of an image processing apparatus (Part 2). [Figure 6] It is a diagram showing a specific example of image processing in an image processing apparatus. [Figure 7A] It is a diagram showing a user interface screen related to image processing of an image processing apparatus (Part 1). [Figure 7B] It is a diagram showing a user interface screen related to image processing of an image processing apparatus (Part 2). [Figure 7C] It is a diagram showing a user interface screen related to image processing of an image processing apparatus (Part 3). [Figure 7D] It is a diagram showing a user interface screen related to image processing of an image processing apparatus (Part 4).

Embodiments for Carrying Out the Invention

[0015] Hereinafter, an embodiment of the present invention will be described with reference to FIGS. 1 to 7D. In this embodiment, when a person who has not consented to use appears in the background of a group photo, an example will be described in which the personal related information of the person is erased by replacing the information related to the person with unrelated information, and the group photo is processed so as not to damage the atmosphere.

[0016] First, the configuration of an image processing apparatus according to an embodiment will be described with reference to FIGS. 1 and 2. FIG. 1 is a functional configuration diagram of an image processing apparatus. FIG. 2 is a hardware-software configuration diagram of an image processing apparatus.

[0017] As shown in FIG. 1, the image processing apparatus 100 includes an imaging unit 101, a processing target detection unit 102, a face authentication unit 103, a processed / unprocessed segmentation unit 104, a personal-related information feature acquisition unit 105, an image processing method determination unit 106, an image processing unit 107, a display unit 110, and a storage unit 120. The imaging unit 101 is a functional unit that captures an image using a camera or the like. The processing target detection unit 102 is a function for detecting regions of a face and a person to be recognized from the image input from the imaging unit 101. Specific examples of the detection of the authentication target will be described later.

[0018] The face authentication unit 103 is a functional unit that extracts face feature amounts from an image, compares them with the face feature amounts held in a face feature amount table (described later), and determines whether or not the person is the same. Here, the targets of face authentication include both a person who can be a main subject and a person who can be a background.

[0019] The processed / unprocessed segmentation unit 104 is a functional unit that refers to the authentication result by the face authentication unit 103 and a face feature amount table (described later), and determines whether to process or leave it unprocessed according to whether the target person agrees to the use of the image. Examples of image processing / non-processing will be described later.

[0020] The personal-related information feature acquisition unit 105 is a functional unit that acquires features of personal-related information from an image. The personal-related information feature acquisition unit 105 acquires features such as the hairstyle, hair color, height, body shape, and other physical features, face features, and clothing features of the person shown in the processed area of the image by, for example, an object detection method, and features such as the name, color, shape, and state of personal-related information. Here, object detection is a technology for specifying "the position, type, and number of objects" from the captured image, and various AI algorithms such as Faster R-CNN, YOLO, and SSD are known.

[0021] Personal information obtained from images is information that can potentially identify an individual. Here, while biometric authentication such as facial recognition provides highly reliable information for identifying an individual, it should be noted that identifying an individual from belongings or clothing depends on factors such as the rarity of the object and the probability of its presence in a given location. For example, if there is a handmade garment of which only a few exist in the world, and a mass-produced garment sold in a clothing store with hundreds of thousands of units sold, the individual wearing the handmade garment is clearly more likely to be identified. On the other hand, for a person wearing the mass-produced garment, it is difficult to identify the individual because many people wear it, and its rarity is low, making it less likely to identify the individual. As in these examples, in this specification, a quantitative value representing the likelihood of identifying an individual in personal information is defined as "personal relevance," and the higher the personal relevance, the greater the likelihood of identifying the individual.

[0022] The image processing method determination unit 106 is a functional unit that determines the processing method according to the degree of personal relevance of the person recognized in the image. By utilizing object recognition technology that determines the region and type of an object from an image, it is possible to obtain the type of each part within the region containing a person. Based on these types, the image processing method determination unit 106 determines the image processing method based on the degree of personal relevance of the image. The process for determining the specific image processing method will be explained later.

[0023] The image processing unit 107 is a functional unit that processes an image that has been determined to be processed in order to change the personal information in question. Image processing can utilize image generation technologies such as GAN (Generative Adversarial Network), VAE (Variational Autoencoder), and CNN (Convolutional Neural Network). As an image generation technology, for example, an image generation AI mechanism can be used that takes a person image in the processing area as an input image and generates a video from prompt-like words indicating the type of personal information (described later) and the processing method. Then, using such technology, an image is generated to match the area containing the person included in the target area, and the image processing is performed by replacing the original image with the generated image. As an image generation AI technology, for example, Stable Diffusion can be used. Specific examples of image processing will be explained later.

[0024] The image processing device 100 can be implemented using a PC (Personal Computer) having the hardware configuration shown in Figure 2. In this embodiment, as shown in Figure 2, the image processing device 100 has a configuration in which a CPU (Central Processing Unit) 10, ROM (Read Only Memory) 11, main memory 12, auxiliary storage device 20, display I / F 30, communication I / F 31, and input / output I / F 32 are connected by a bus 7.

[0025] The CPU 10 is a processor that controls the image processing unit 100, references work data in the main memory 12, and executes the loaded program. Main memory 12 is a volatile, high-speed memory composed of semiconductor elements such as RAM (Random Access Memory). ROM11 is a non-volatile semiconductor memory device that stores programs for starting the hardware of the image processing device 100, such as firmware (BIOS).

[0026] The auxiliary storage device 20 is a large-capacity magnetic storage device or semiconductor storage device such as an HDD (Hard Disk Drive) or an SDD (Solid State Drive). An image processing program 21 is installed on the auxiliary storage device 20 in this embodiment. The image processing program 21 is a program that executes the functions of each part shown in Figure 1. The auxiliary storage device 20 in this embodiment also stores image data 121, a face feature quantity table 122, and an image processing method rule table 123. Details of each table will be described later.

[0027] The display I / F30 is an interface for connecting a display device 40 such as an LCD (Liquid Crystal Display). Communication I / F31 is the interface for connecting network 5. Network 5 can be wired or wireless. If network 5 is wired, communication I / F31 becomes a network card. If network 5 is wireless, it becomes a wireless communication device conforming to standards such as IEEE 802.11b, g, a, n, ac, or ax. The input / output interface 31 is an interface for connecting input / output devices. In this embodiment, a keyboard 51, a mouse 52, and a camera 50 are connected. The keyboard 51 and mouse 52 are devices for inputting commands and necessary information to the image processing device 100. The camera 50 is a device for capturing images and acquiring still images and videos.

[0028] The image processing device 100 may be a general-purpose PC, or a portable information processing device such as a smartphone or tablet that performs the functions shown in Figure 2. Alternatively, it may be a client-server system that performs the functions shown in Figure 2 in the cloud. For example, in a cloud environment, video data may be transmitted from the client to a remote server via the internet, and the client may receive the results of the image processing performed on that server.

[0029] In the above, the image processing device 100 has been described as an information processing device that executes a program. However, the present invention also covers computer-readable recording media that store software program code (computer program) that realizes the functions of such embodiments.

[0030] Next, we will describe the data structures used in the image processing device using Figures 3 and 4. Figure 3 shows an example of a facial feature table. Figure 4 shows an example of an image processing method rule table.

[0031] The facial feature table 122 is a table that stores facial features associated with an individual and whether or not consent has been given to the use of those facial features. As shown in Figure 3, the facial feature table 122 consists of the fields: individual ID 122a, facial feature 122b, and consent flag 122c. The individual ID 122a stores a unique identifier for identifying an individual. The facial feature 122b stores features that evaluate the characteristics of the individual's face. Facial features include, for example, features extracted using deep learning and landmark features that focus on specific points. The consent flag 122c stores a flag indicating whether or not an individual has consented to the publication of their image. For example, it is set to "1" when the individual has explicitly consented to the publication of their image, and to "0" when the individual refuses to publish their image or when no response has been received.

[0032] The image processing method rule table 123 is a table that defines the rules for image processing with respect to personal information and personal relevance. As shown in Figure 4, the image processing method rule table 123 consists of the following fields: personal information type 123a, personal information features 123b, personal relevance evaluation score 123c, and image processing method 123d.

[0033] The Personal Information Type 123a stores information categorized as personal information that can be inferred from the image. Examples include "hairstyle," "body type," and "clothing." This is because these are considered to be personal information that can identify or infer the person by knowing them. The Personal Information Feature 123b stores a string that identifies the specific feature corresponding to the Personal Information Type 123a. For example, when the value of Personal Information Type 123a is "hairstyle," values ​​such as "short," "medium," "long," "shaved head," and "perm" are set. When the feature changes depending on gender, such as "hairstyle," separate entries for male and female may be used. The Personal Relevance Score 123c stores a score that evaluates the degree of personal relevance to the personal information feature described in the Personal Information Feature 123b column. The Image Processing Method 123d stores a specific processing method related to the personal information feature described in the Personal Information Feature 123b column when processing the image. The image processing method 123d can be changed as shown in Figure 4, according to each personal information feature 123b of the personal information type 123a.

[0034] The personal relevance score 123c is increased for personal relevance information features that are statistically considered to be common in the target group. For example, suppose that among Japanese company employees, the suit colors are approximately 30% for "black" and "navy," approximately 10% for "gray," and approximately 50% for "brown." In this case, as shown in Figure 4, the personal relevance score 123c is set to 0 when the personal relevance information feature 123b is "black" or "navy," 3 when it is "gray," and 10 when it is "brown." When the personal relevance score indicates that the personal relevance of the person in the image is low, no processing is performed or the degree of processing is reduced. Conversely, when the personal relevance is high, the degree of processing is increased. By not processing or minimizing the processing of images of individuals with statistically high proportions of personal information characteristics 123b, and by increasing the processing of images of individuals with statistically low proportions of personal information characteristics 123b, personal information can be effectively concealed. Processing based on personal relevance will be explained in detail later.

[0035] Furthermore, the personal relevance score (123c) can be adjusted depending on the time and location of the photograph to appropriately conceal personal information and define appropriate image processing methods. For example, regarding suit colors, in business settings, especially during job interviews for company recruitment, personal relevance scores for colors like "black" and "navy" are set lower, while those for "brown" and other bright colors are set higher. On the other hand, in images taken in casual settings, even for "brown" and other bright colors, those with lower personal relevance scores are selected.

[0036] Furthermore, in the example shown in Figure 4, the value of the personal relevance score 123c for personal relevance feature 123b, "uniform," is "20." If this is, for example, the score for a person in a uniform at a theme park, then the personal relevance score for a person in a uniform around a school on a weekday morning will be lower.

[0037] Furthermore, regarding image processing methods, trendy clothing and hairstyles may be registered in advance as processing methods, and adjustments may be made to reduce the degree of personal relevance. The rarity of objects and types in the target area may be automatically generated by performing trend analysis from the total number of SNS posts obtained from internet search information, and using this as statistical information such as rarity, thereby creating the image processing method rule table 123. Alternatively, a questionnaire survey may be conducted for each type of SNS, and the personal relevance evaluation score for personal relevance information characteristics may be customized to determine the content of the image processing method rule table 123.

[0038] Next, we will explain the details of the image processing method of the image processing device using Figures 5A and 5B. Figure 5A is a flowchart detailing the image processing method of the image processing device (part 1). Figure 5B is a flowchart detailing the image processing method of the image processing device (part two).

[0039] First, the image processing device 100 acquires the image to be processed from the captured image captured by the imaging unit 101 (S101).

[0040] Next, the processing target detection unit 102 of the image processing device 100 detects a person and the region of that person's face from the image as a processing target detection step (S102). The detection of a person and the region of that person's face can be done using methods such as detecting from image features such as Haar basis features that apply object recognition, or using object detection methods learned by deep learning.

[0041] Next, the image processing device 100 determines whether or not it has processed the regions of all people in the image (S103). If it has processed the regions of all people (S103:YES), it terminates the image processing. If there are regions of people that have not been processed (S103:NO), it proceeds to S104.

[0042] Next, the image processing device 100 acquires the next person and the region of that person's face (S104).

[0043] Next, the face recognition unit 103 of the image processing device 100 performs face recognition by matching the acquired feature quantities of the face region of the person with the feature quantities of the face feature quantity table 122 shown in Figure 3 (S105).

[0044] Referencing the facial feature table 122, if the authenticated person has consented to the use of their image (S106: YES), the region of that person is considered an unprocessed region, and the process returns to S103. If the facial features of the acquired person do not match any of the features in the facial feature table 122, or if the person whose features match has not consented to the use of their image (S106: NO), the region of that person is considered a processed region, and the process proceeds to S107.

[0045] Next, when the region containing a person in the acquired image is designated as the processing region, the personal information feature acquisition unit 105 of the image processing device 100 acquires personal information features of the region containing a person in the acquired image as a personal information feature acquisition step (S107). Personal information features concerning objects and their types in the region of the image containing personal information can be acquired using deep learning-based image recognition methods. Specifically, this can be achieved by using image recognition methods such as R-CNN, which performs object detection by combining region search and image recognition algorithms, or YOLO, which estimates region candidates as a regression problem.

[0046] Next, the image processing method determination unit 106 of the image processing device 100 calculates the personal relevance score by taking the sum of the personal relevance evaluation scores of the personal relevance information features that span the region of the person and are matched with the image processing method rule table 123 (S108).

[0047] For example, in the example shown in Figure 4, when the person in the acquired image has "long hair," "golden hair color," "uniform clothing," and "a bag," the personal relevance score is 0 + 5 + 20 + 3 = 28.

[0048] Next, if the personal relevance calculated in S108 is equal to or greater than a predetermined threshold Th1 (S109: YES in Figure 5B), the process proceeds to S110; otherwise, it is less than the predetermined threshold Th1 (S109: NO). If the predetermined threshold Th1 is, for example, 23, then in the example given above, the personal relevance is 28, which corresponds to the case where the process in S110 is performed.

[0049] When the degree of personal relevance is equal to or greater than a predetermined threshold Th1, the image processing unit 107 of the image processing device 100 processes the image related to the personal relevance information features of the entire person and the image of the face region as an image processing step, according to the image processing method defined in Figure 4 (S110).

[0050] Image manipulation of the facial region can utilize AI technology that has advanced significantly in recent years, which can transform faces to resemble those of different people. Examples include deepfake technology, face swapping, and StyleGAN. Furthermore, by using image generation AI that takes posture into account, the image can be processed to overwrite the target region while maintaining the person's posture. This makes it possible to maintain the naturalness of the image while avoiding the risk of leaking personal information by replacing the person's image with a generated image.

[0051] If the personal relevance score is less than the predetermined threshold Th1, the next step is to determine whether there is a portion of each personal relevance information feature whose personal relevance score is equal to or greater than the predetermined threshold Th2. If such a portion exists (S111:YES), proceed to S112; otherwise, proceed to S113 (S111:NO).

[0052] If there is a portion that is above the threshold Th2, the image processing unit 107 of the image processing device 100 processes the image of the face region and the portion of the personal relevance evaluation score of the personal relevance information features that is above the threshold Th2, according to the image processing method specified in Figure 4 (S112).

[0053] If the predetermined threshold Th1 is set to, for example, 7, then in the example given above, the personal relevance score for "clothing" being "uniform" is 20, which corresponds to the example of processing S112.

[0054] If the personal relevance score of each personal information feature does not exceed the threshold Th2, the image of the face region is processed (S113). This measure is taken considering that individuals can be identified by their faces relatively often, even if they lack personal features such as hairstyle or clothing. However, if the image resolution is low, the image size is small, and the face region is relatively small, it is also possible to choose not to process the image of the face region.

[0055] Then, once the image processing of the acquired person's region (S110, S112, S113) is complete, the process returns to S103.

[0056] Furthermore, the processed images can be transmitted over a network and used as a print service for output. It is also possible to send the images via a communication network to individuals or systems that require them. For example, in a system where customers can view surveillance camera footage to indicate store congestion, the system can remove any potentially leaked personal information from the displayed footage, transmit the processed image, and display it on a digital signage screen.

[0057] Next, we will explain a specific example of image processing in an image processing device using Figure 6. Figure 6 shows a specific example of image processing in an image processing device.

[0058] Image 401, as shown in Figure 6, shows a photograph of several people taken as a group on the right, with a theme park building in the background, and two students in uniform on the left as passersby.

[0059] Image 402 shows the stage in step S102 of Figure 5A where the person and the region of their face have been extracted from the image. The person and the region of their face extracted from the image are shown as a rectangular region, for example, as shown in Figure 6.

[0060] Assume that the group of people photographed on the right side of the image have given their consent to the publication of the image (corresponding to S106: YES), while the two students in uniform passing by on the left side have not given their consent to the publication of the image (corresponding to S106: NO). In this case, the image area of ​​the group of people photographed on the right side becomes the unprocessed area 405, and the two students in uniform passing by on the left side becomes the processed area 406.

[0061] Image 404 is an image obtained by replacing the image area with the face area and clothing area of ​​the two people in the processing area 406 (corresponding to S110).

[0062] Next, we will describe the user interface related to image processing of the image processing device using Figures 7A to 7D. Figure 7A shows the user interface screen related to image processing of an image processing device (part 1). Figure 7B shows the user interface screen related to image processing of an image processing device (part two). Figure 7C shows the user interface screen related to image processing of an image processing device (part three). Figure 7D shows the user interface screen related to image processing of an image processing device (part four).

[0063] First, when a user selects an image to process from multiple image data, the display device 40 of the image processing device 100 displays the image selection screen 601 shown in Figure 7A.

[0064] When the user selects an image using a pointing device such as a mouse 52 and clicks the OK button 610, the image processing target image details screen 602 shown in Figure 7B is displayed, and the image processing device 100 allows the user to confirm the details of the image to be processed.

[0065] Next, when the user approves the image processing, they click the processing button 611 on the image details screen 602. Accordingly, the image processing is performed on the target image according to the flowcharts shown in Figures 5A and 5B, and the image processing device 100 displays the processed image screen 603 in Figure 7C.

[0066] Next, if the user finds any problems with the image displayed on the processed screen 603, they click the back button 612 to return to the image before processing. If there are no problems with the processing, they click the OK / Save button 613.

[0067] If there are problems with the processing, it is possible to customize the processed image by updating the contents of the image processing method rule table 123 shown in Figure 4 or by changing the image processing method in the image processing unit 206.

[0068] If there are no problems with the processing and the user clicks the OK / Save button 613, the user proceeds to the processed image selection screen 604 shown in Figure 7D. The filenames of the processed screen thumbnails on the processed image selection screen 604 should be modified to make the correspondence clearer, by adding modifiers to the filenames of the original screen thumbnails.

[0069] As described above, the image processing device of this embodiment calculates a personal relevance score, which evaluates the characteristics of personal information appearing in an image, and processes the image area of ​​a person by changing the degree of processing based on this value. As a result, even when an image is made public, it is possible to provide a natural image while changing the personal information of the person, thereby increasing the usability of the image.

[0070] (Composition 1) An image processing device that takes an image as input and processes the image to change the personal information of individuals appearing in the image, A first table that links facial features associated with an individual with whether or not consent was given for the use of the said facial features, It maintains a second table that defines how images are processed regarding the characteristics of personal information, A processing target detection unit that detects a person and the area of ​​that person's face from the input image, A face recognition unit extracts face features from the facial region and compares them with the face features held in the first table to perform face recognition. A processing / unprocessing separation unit determines whether the image of the person and the region of the person's face is to be processed or unprocessed, based on the first table and the facial feature quantities of the person detected by the processing target detection unit. A personal information feature acquisition unit that acquires personal information features from images of a person and the facial region of that person, The processing method defined in the second table, and the image processing method determination unit which determines the image processing method for the processing area based on the characteristics of personal information obtained from the image of the person and the area of ​​the person's face, An image processing apparatus comprising: an image processing unit that processes an image of a person detected by the processing target detection unit and the area of ​​that person's face, in accordance with the image processing method determined by the image processing method determination unit.

[0071] (Configuration 2) Furthermore, the second table holds a personal relevance score for quantitatively evaluating the characteristics of personal information, and the image processing unit changes the degree of image processing for the person detected by the processing target detection unit and the area of ​​that person's face, as described in Configuration 1.

[0072] (Composition 3) The image processing apparatus according to configuration 2, characterized in that the image processing unit changes whether to process an image relating to all the features of personal information of the person and the facial region of that person detected by the processing target detection unit, or to process an image relating to some of the features of personal information, based on the degree of personal relevance.

[0073] (Composition 4) The image processing apparatus according to either configuration 1 or configuration 2, characterized in that the features of the aforementioned personal information include hairstyle, hair color, physical features including height and body shape, facial features, and clothing features.

[0074] (Composition 5) The image processing apparatus according to configuration 2, characterized in that the degree of personal relevance is determined based on the statistical proportion of the characteristics of the personal relevance information of the target group.

[0075] (Composition 6) The image processing apparatus according to configuration 2, characterized in that the degree of personal relevance is determined according to the rarity of the characteristics of personal relevance information that appear depending on the time and place.

[0076] (Method 1) An image processing method using an image processing device that takes an image as input and processes the image to change personal information of individuals appearing in the image, The aforementioned image processing device is A first table that links facial features associated with an individual with whether or not consent was given for the use of the said facial features, It maintains a second table that defines how images are processed regarding the characteristics of personal information, The image processing apparatus performs a processing target detection step in which it detects a person and the region of the person's face from the input image, The image processing device performs a facial recognition step by extracting facial features from the facial region and comparing them with the facial features held in the first table, The image processing apparatus performs a processing / unprocessing separation step, which determines whether the image of the person and the region of the person's face is to be processed or unprocessed, based on the first table and the facial feature quantities of the person detected by the processing target detection step. The aforementioned image processing device performs a personal information feature acquisition step, which involves acquiring personal information features from images of a person and the region of that person's face, The image processing apparatus includes an image processing method determination step in which it determines an image processing method for the processing area based on the processing method defined in the second table and the characteristics of personal information obtained from the image of the person and the area of ​​the person's face, The image processing step includes processing the image of the person detected in the processing target detection step and the area of ​​the person's face, according to the image processing method determined in the image processing method determination step. Furthermore, the second table holds a personal relevance score for quantitatively evaluating the characteristics of personal information, and the image processing method is characterized in that, in the image processing step, the degree of image processing for the person detected in the processing target detection step and the area of ​​that person's face is changed based on the personal relevance score. [Explanation of Symbols]

[0077] 5...Network, 7...Bus, 10...CPU, 11...ROM, 12...Main Memory, 20...Auxiliary Storage, 30...Display I / F, 31...Communication I / F, 32...Input / Output I / F 100...Image processing device, 101...Imaging unit, 102...Processing target detection unit, 103...Face recognition unit, 104...Processed / unprocessed separation unit, 105...Personal information feature acquisition unit, 106...Image processing method determination unit, 107...Image processing unit, 110...Display unit, 120...Storage unit, 121...Image data, 122...Facial feature table, 123...Image processing method rule table

Claims

1. An image processing device that takes an image as input and processes the image to change the personal information of individuals appearing in the image, A first table that links facial features associated with an individual with whether or not consent was given for the use of the said facial features, It maintains a second table that defines how images are processed regarding the characteristics of personal information, A processing target detection unit that detects a person and the area of ​​that person's face from the input image, A face recognition unit extracts face features from the facial region and compares them with the face features held in the first table to perform face recognition. A processing / non-processing separation unit determines whether the image of the person and the region of the person's face is to be processed or not, based on the first table and the facial feature quantities of the person detected by the processing target detection unit. A personal information feature acquisition unit that acquires personal information features from images of a person and the facial region of that person, The processing method defined in the second table, and the image processing method determination unit which determines the image processing method for the processing area based on the characteristics of personal information obtained from the image of the person and the area of ​​the person's face, An image processing apparatus comprising: an image processing unit that processes an image of a person detected by the processing target detection unit and the area of ​​that person's face, in accordance with the image processing method determined by the image processing method determination unit.

2. Furthermore, the second table holds a personal relevance score for quantitatively evaluating the characteristics of personal information, and the image processing unit changes the degree of image processing for the person detected by the processing target detection unit and the area of ​​that person's face, as described in claim 1.

3. The image processing unit is characterized in that, based on the degree of personal relevance, it changes whether to process an image relating to all the features of personal information of the person and the facial region of that person detected by the processing target detection unit, or to process an image relating to some of the features of personal information, as described in claim 2.

4. The image processing apparatus according to claim 1, characterized in that the characteristics of the personal information include hairstyle, hair color, physical characteristics including height and body shape, facial features, and clothing features.

5. The image processing apparatus according to claim 2, characterized in that the degree of personal relevance is determined based on the statistical proportion of the characteristics of the personal relevance information of the target group.

6. The image processing apparatus according to claim 2, characterized in that the degree of personal relevance is determined according to the rarity of the characteristics of personal relevance information that appear depending on the time and place.

7. An image processing method using an image processing device that takes an image as input and processes the image to change personal information of individuals appearing in the image, The aforementioned image processing device is A first table that links facial features associated with an individual with whether or not consent was given for the use of the said facial features, It maintains a second table that defines how images are processed regarding the characteristics of personal information, The image processing apparatus performs a processing target detection step in which it detects a person and the region of the person's face from the input image, The image processing device performs a facial recognition step by extracting facial features from the facial region and comparing them with the facial features held in the first table, The image processing apparatus performs a processing / unprocessing separation step, which determines whether the image of the person and the region of the person's face is to be processed or unprocessed, based on the first table and the facial feature quantities of the person detected by the processing target detection step. The aforementioned image processing device performs a personal information feature acquisition step, which involves acquiring personal information features from images of a person and the region of that person's face, The image processing apparatus includes an image processing method determination step in which it determines an image processing method for the processing area based on the processing method defined in the second table and the characteristics of personal information obtained from the image of the person and the area of ​​the person's face, The image processing step includes processing the image of the person detected in the processing target detection step and the area of ​​the person's face, according to the image processing method determined in the image processing method determination step. Furthermore, the second table holds a personal relevance score for quantitatively evaluating the characteristics of personal information, and the image processing method is characterized in that, in the image processing step, the degree of image processing for the person detected in the processing target detection step and the area of ​​that person's face is changed based on the personal relevance score.

Images