Information processing apparatus, information processing method, and program

Abstract

To reduce a load on a user when using content on the basis of consent information of the user.SOLUTION: The information processing apparatus determines the necessity of consent to the use of the edited content on the basis of the editing history of the content, acquires consent information according to the necessity of consent, and restricts the use of the edited content on the basis of the consent information.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing technique for handling content such as images. [Background technology]

[0002] In recent years, advances in deep learning have made it easier for anyone to edit content such as images. By incorporating such editing functions into devices such as smartphones, users of the devices can perform various edits, such as erasing people from images or replacing their facial expressions with those from the best shot. Meanwhile, cases in which malicious users post images processed using so-called deepfakes on social networking services (SNS) have become a social problem. To address this issue, a technology has been proposed to ensure authenticity by embedding historical information about the photo and editing process in photos and videos, such as who took the photo and what editing was performed. Patent Document 1 also discloses a technology that prevents editing of user information until consent is obtained from all parties involved in sharing the user's information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-162154 Summary of the Invention [Problem to be solved by the invention]

[0004] However, while the above-mentioned authenticity assurance technology can record whether editing has been performed manually or by AI, it is not possible to tell at a glance that content such as an image has been edited, and therefore it is not possible to completely prevent unauthorized use. Patent Document 1 can control so that user information cannot be edited until the consent of all parties involved is obtained, but obtaining consent from all parties involved is a difficult task and places a significant burden on the user who performs the editing.

[0005] Therefore, an object of the present invention is to make it possible to reduce the burden on the user when using content based on user consent information. [Means for solving the problem]

[0006] The information processing device of the present invention is characterized by having a decision means for deciding whether consent is necessary for use of edited content based on the editing history of the content, a consent acquisition means for acquiring consent information according to the need for consent, and a restriction means for restricting use of the edited content based on the consent information. [Effects of the Invention]

[0007] According to the present invention, it is possible to reduce the burden on the user when using content based on the user's consent information. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. [Figure 2] 1 is a diagram illustrating an example of a functional configuration of an information processing device according to a first embodiment. [Figure 3] FIG. 10 is a diagram showing an example of a GUI when capturing an image. [Figure 4] FIG. 10 is a diagram showing an example of a person detection result displayed on an image. [Figure 5] FIG. 10 is a diagram illustrating an example of a GUI for image editing. [Figure 6] FIG. 10 is a diagram illustrating an example of an editing history. [Figure 7] FIG. 10 is a diagram illustrating an example of functions to be used and the number of functions. [Figure 8] FIG. 10 is a diagram showing an example of an edit history corresponding to a person to whom consent is required. [Figure 9] FIG. 10 is a diagram showing an example of a GUI for an editor and a consenter. [Figure 10] FIG. 10 is a diagram showing an example of a face photograph taken when consent is obtained. [Figure 11] FIG. 10 is a diagram showing an example of a GUI after usage restrictions are lifted. [Figure 12] 4 is a flowchart of information processing according to the first embodiment. [Figure 13] FIG. 10 is a diagram illustrating an example of a functional configuration of an information processing device according to a second embodiment. [Figure 14] 10 is a flowchart of information processing according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the features described in the present embodiments are necessarily essential to the solution of the present invention; these features may be combined as desired. The configuration of the embodiments may be modified or changed as appropriate depending on the specifications of the device to which the present invention is applied and various conditions (such as usage conditions and usage environment). Furthermore, the present invention may be configured by appropriately combining parts of the embodiments described below.

[0010] First Embodiment In the first embodiment, an example will be described in which the necessity for the individual's consent is determined based on the function or editing area used when editing content such as an image, and the use of the image is restricted until the individual's consent is obtained. In the following embodiment, an example will be given in which image content is edited, but the editing is not limited to image content, and the function according to this embodiment can also be applied to cases in which various content such as audio and text is edited.

[0011] FIG. 1 is a diagram showing an example of a hardware configuration applicable to an information processing apparatus 100 according to this embodiment. The CPU 104 reads out a control program stored in the ROM 106, loads it into the RAM 105, and executes various control processes. The CPU 104 also executes an information processing program according to this embodiment stored in the mass storage unit 107, thereby realizing information processing, which will be described later. The information processing program according to this embodiment may be stored in the ROM 106. The mass storage unit 107 has, for example, an HDD or an SSD, and stores image data, various setting parameters, various programs such as the information processing program according to this embodiment, and the like. The RAM 105 is used as a temporary storage area, such as a work memory, where programs read out from the ROM 106 are expanded.

[0012] Network I / F unit 103 transmits and receives various information, image data, etc. to and from external devices via the Internet. Image data, etc. received from external devices via network I / F unit 103 is stored in RAM 105 or mass storage unit 107 via data bus 108, and is read and processed by CPU 104 as needed. Network I / F unit 103 can also transmit and receive programs to and from external devices, etc.

[0013] The input unit 101 includes, for example, a keyboard, a mouse, a touch panel, etc., and receives instructions from a user and transmits them to the CPU 104. The display unit 102 includes a display device such as a liquid crystal display, and displays images and GUI (Graphical User Interface) images generated by the CPU 104 through program execution. The input unit 101 and display unit 102 are also connected to the data bus 108.

[0014] 2 is a functional block diagram showing an example of the functional configuration of an information processing device 100 according to this embodiment. The information processing device 100 may be, for example, a device such as a smartphone, a tablet terminal, or a personal computer, and the following embodiment will be described using a smartphone as an example of a device. In this embodiment, the functional units shown in FIG. 2 are realized by the CPU 104 in FIG. 1 executing the information processing program of this embodiment, but at least some or all of the functional units may be realized by a configuration such as a circuit.

[0015] The image acquisition unit 201 acquires an image captured by a built-in camera (not shown) of the smartphone as image content. The image content acquired by the image acquisition unit 201 may be an image recorded in a storage device such as a hard disk, or an image received via a network such as the Internet. In other words, the source of the image content is not limited to a specific device. For simplicity, the image content will be simply referred to as an image below. The image acquired by the image acquisition unit 201 is sent to the detection unit 202 and the editing unit 203.

[0016] 3 is a diagram showing an example of a UI (User Interface) when capturing an image with a built-in camera (e.g., an outer camera) not shown in the figure of smartphone 300. CPU 104 displays an image being captured by the built-in camera of smartphone 300 on display screen 301 (display unit 102), and when the user inputs a capture instruction by operating button 302, for example, the image at the time the capture instruction was input is stored in mass storage unit 107. Image acquisition unit 201 acquires the image stored in mass storage unit 107.

[0017] The detection unit 202 detects the image area of ​​a person's face (hereinafter simply referred to as face) and the image area of ​​the entire person (hereinafter referred to as person area) from the image acquired by the image acquisition unit 201, and generates a person detection result list according to the detection results. The person detection result list is a list of information linking the face detection frame (center position and size of the frame) containing the face with the person area, as the person detection result. The method of detecting the face and person area from the image can be a method such as that disclosed in Reference 1 below. Reference 1: S. Kiu, "Path Aggregation Network for Instance segmentation", VPR2018.

[0018] The method for detecting faces and person regions in the detection unit 202 is not limited to the method in Reference 1, and may be, for example, a method for detecting faces and person regions using a machine learning method that uses handmade features. For example, the detection unit 202 may detect faces and person regions using a model trained with SVM (Support Vector Machines) using HOG (Histograms of Oriented Gradients) features. In this way, the detection of faces and person regions by the detection unit 202 is not limited to a specific method. Furthermore, a list of person detection results may be displayed on the display screen of the smartphone, but this is not essential and may not be displayed.

[0019] 4 is a diagram showing an example of a diagram illustrating face detection frames and person regions in the person detection result list generated by the detection unit 202, corresponding to an image 400 acquired by the image acquisition unit 201. The detection unit 202 detects a face detection frame and person region for each person appearing in the image 400, links the face detection frame with the person region corresponding to the center coordinates of the face detection frame, and assigns a person ID to uniquely identify each person, resulting in a person detection result. In the case of image 400 shown in FIG. 4, four people appear in the image, so the detection unit 202 detects face detection frames 401a to 401d and person regions 402a to 402d for each person. Furthermore, the detection unit 202 links the face detection frames 401a to 401d with the person regions 402a to 402d corresponding to the center coordinates of each face detection frame, respectively, and assigns person IDs ID:1 to ID:4 to uniquely identify each person, thereby generating a person detection result list. Then, the detection unit 202 sends the above-mentioned person detection result list to the determination unit 204.

[0020] The editing unit 203 acquires an edited image by editing the image acquired by the image acquisition unit 201. The image editing method may be a method such as that disclosed in Reference 2 below. Reference 2: R. Tolosana, "DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection", CVPR2020.

[0021] Note that image editing is not limited to editing techniques using GAN (Generative Adversarial Network) that utilizes deep learning, and may involve, for example, manual image editing or image generation using generative AI. Image editing may be any process that modifies the image itself, and may include simple processes such as brightness correction or color correction. In this way, image editing is not limited to a specific technique.

[0022] In this embodiment, examples of image editing include face replacement, facial expression change, feature change, and background person removal. Face replacement is editing in which the face of a person in an image to be edited is replaced with the face of a person in another image. For example, by replacing the face of person A in an image containing person A with the face of person B in another image, the image containing person A can be edited to look like an image containing person B. Facial expression manipulation is editing in which the face of a person in an image to be edited is changed to a face with a different expression. For example, an angry face of person A can be edited to look like a smiling face. Feature change is editing in which the features of a person in an image to be edited are changed. For example, an image of person A's bare face without any clothing can be edited to an image in which person A is wearing a mask or sunglasses. Background person removal is editing in which the area of ​​a person selected as a background person to be edited is removed, and a background generated based on the surrounding image features is inserted into the removed area. For example, by removing the area of ​​person B from an image containing person A and person B, it is possible to edit the image to look as if person B were not present.

[0023] The editing unit 203 selects from these editing examples in accordance with instructions from the user and performs editing. In addition to editing the image, the editing unit 203 also creates an editing history indicating what kind of editing was performed on which area of ​​the image each time an edit is performed. The editing history includes an edited area indicating the image area in the image where the edit was performed, and information indicating what kind of editing was performed, such as face replacement, facial expression change, feature change, or background person deletion.

[0024] Fig. 5 is a diagram showing an example of a GUI 501 used when the user instructs one of the editing examples described above. In this embodiment, the editing unit 203 generates and presents the GUI 501 for image editing as shown in Fig. 5, allowing the user to instruct which editing to perform. The GUI 501 for image editing shown in Fig. 5 is displayed on the display screen of the smartphone 500. The image editing GUI 501 illustrated in FIG. 5 includes an editing function list 502, an image to be edited 503, and a person to be edited list 504. The editing function list 502 displays a list of the editing functions described above, namely, face replacement, facial expression change, feature change, and background person deletion. The image to be edited 503 displays an image acquired by the image acquisition unit 201, and a face detection frame and person area detected from the image. The person to be edited list 504 displays, for example, an image of a person to be replaced by the face replacement function. By operating the input unit 101, the user can select any editing function from the editing function list 502, and can also select any person to be replaced from the person to be edited list 504. The editing unit 203 edits the image based on instructions from the user via this image editing GUI 501.

[0025] 6 is a diagram showing an example in which editing is performed by the editing unit 203 based on user instructions via the image editing GUI 501, and the editing history is diagrammed in association with the image acquired by the image acquisition unit 201. Fig. 6 shows an example in which the face replacement function is applied to a face area 601 of the second person from the left (person with person ID: 2 in Fig. 4), and the background person removal function is applied to a person area 602 of the third person from the left (person with person ID: 3 in Fig. 4). The editing unit 203 sends the image before editing, the image after editing, and the editing history to the determination unit 204.

[0026] The determination unit 204 determines whether consent is necessary and the person to whom consent is required, using the person detection result list generated by the detection unit 202, and the images before and after editing and the editing history by the editing unit 203. For example, the determination unit 204 determines from which person in the person detection result list consent is required, based on the correspondence between the person detection result list and the edited area included in the editing history.

[0027] First, the determination unit 204 finds an overlapping area between each person area included in the person detection result list and the edited area included in the edit history. Then, the determination unit 204 calculates the agreement requirement score C from the number of pixels n in the overlapping area, the number of pixels N in the original image, and the function score F using the following formula (1):

[0028] C=(n / N)×F formula (1)

[0029] The function score is a parameter set for a function used for editing (hereinafter referred to as a "used function") among the aforementioned editing functions (face replacement, facial expression change, feature change, and background person removal). The higher the risk of fraudulent use, the higher the score. FIG. 7 shows an example of function scores for each used function. In FIG. 7, the function scores for each used function among the editing functions are set as follows: face replacement is set to "5," facial expression change is set to "3," feature change is set to "2," and background person removal is set to "1." The determination unit 204 then determines whether or not the consent of the person is required based on the result of comparing the consent score C calculated using Equation (1) with a predetermined threshold. For example, if the consent score C is equal to or greater than the predetermined threshold, the determination unit 204 determines that the consent of the person is required, and sets the person detection result of the person determined to require the consent of the person in the person detection result list as consent-required person information. On the other hand, if the consent score C is less than the threshold, the determination unit 204 determines that the consent of the person is not required.

[0030] Furthermore, the determination unit 204 cuts out an image area corresponding to the face detection frame included in the consent subject person information from the image acquired by the image acquisition unit 201, and sets the cut-out image area as a consent subject face image.The determination unit 204 then manages the consent subject face image in association with the person ID of the person determined to require the consent of the person, and adds the consent subject person information to the editing history of the person determined to require the consent of the person.The determination unit 204 sends the consent subject face image to the consent acquisition unit 206, and sends the image before editing, the image after editing, and the editing history to the storage unit 205.

[0031] Fig. 8 is a diagram showing an example of the editing history in the example shown in Fig. 6 to which consent-requiring person information has been added as described above, diagrammed in association with the image acquired by the image acquisition unit 201. The example in Fig. 8 shows a state in which it has been determined that consent is required for the person with person ID: 2, but consent has not been obtained from that person (consent: not provided), and an example in which it has been determined that consent is not required for the person with person ID: 3 (consent: not provided).

[0032] Note that the judgment unit 204 calculated the consent score C using the number of pixels n in the overlapping area, the number of pixels N in the original image, and the function score F, as shown in equation (1), but it may also determine the consent score by weighting each part of the person that has been edited. For example, the determination unit 204 may assign a weight so that the consent-requiring score is high when an edit has been made to the face, and may assign a weight so that the consent-requiring score is low when an edit has been made to a part other than the face. In this way, by weighting the consent-requiring score so that it is high for edits to the face, the editing unit 203 is more likely to determine that the consent of the person in question is required when edits to the face have been made.

[0033] Furthermore, for example, a model for measuring the inappropriateness of image content can be trained using deep learning, and the determination unit 204 can use this trained model to make a determination. The inappropriateness of image content indicates the degree to which the image or video causes discomfort to users who view it, such as expressions that lead to racial discrimination or human rights violations, extreme, cruel, or violent expressions, or sexual expressions. In this case, the trained model is a model that is trained so that the consent requirement score increases when editing is performed to increase the inappropriateness of the image content. As a result, when editing is performed to increase the inappropriateness of the image content, the determination unit 204 will determine that the consent of the individual is required.

[0034] Furthermore, for example, when an editing operation to change a facial expression has been performed, the determination unit 204 may assign a weight according to the comparison result between the emotion expressed by the facial expression of the person after the editing operation and the emotion expressed by the facial expressions of people around the person, and calculate the agreement score. For example, if the editing operation for the target person changes the facial expression to one that expresses roughly the same emotion as the emotions expressed by the facial expressions of other people around the person, the determination unit 204 assigns a weight so that the agreement score becomes lower. On the other hand, if the editing operation for the target person changes the facial expression to one that expresses an emotion different from the emotions expressed by the people around the person, the determination unit 204 assigns a weight so that the agreement score becomes higher. As a result, if the editing operation changes the facial expression of the target person to an emotion that is different from the emotions expressed by the people around the person, and is inappropriate for the scene, the determination unit 204 determines that the person's consent is required.

[0035] As described above, the determination unit 204 can determine the need for consent based on at least one of the number of pixels in the edited area included in the editing history, the part of the human body corresponding to the edited area, the editing function used for editing, and the inappropriateness of the content used for editing.

[0036] Note that the persons for whom consent is to be obtained by the determination unit 204 may be all persons surrounding the editing area. In this case, the determination unit 204 may calculate the consent requirement score C from the distance D between the center of the editing area and the center of the surrounding person area and the function score F using the following formula (2).

[0037] C=(1 / D)×F formula (2)

[0038] The storage unit 205 stores the unedited image, the edited image, the consent-receiving facial image, the editing history, and the digital signature. First, the storage unit 205 generates a hash value from the unedited image, generates a private key / public key pair from the hash value, and issues a digital signature. Then, the storage unit 205 stores the editing history and the digital signature inside the edited image data to create a signed edited image. Here, in this embodiment, the image file is stored as a file that complies with, for example, the C2PA (Coalition for Content Provenance and Authenticity) standard. The C2PA standard allows for records to be kept, in addition to EXIF ​​(Exchangeable image file format) information, such as which camera was used to capture the image, and which tool was used to edit the image. In this embodiment, the storage unit 205 stores the editing history and digital signature information in a "history recording function" based on the C2PA standard. Then, the storage unit 205 sends the unedited image and the signed edited image to the restriction unit 207. The editing history may be stored on an external server or database by the storage unit 205. The digital signature structure is not essential, and the method is not limited to a specific one as long as the pre-edited and post-edited images and the editing history can be stored in association with each other.

[0039] The consent acquisition unit 206 identifies the consent acquisition subject based on the consent acquisition subject facial image stored in the storage unit 205, and performs face recognition processing using the facial image of the consent acquisition subject himself / herself to acquire the consent information of the consent acquisition subject. The consent information of the consent acquisition subject is information indicating whether the consent acquisition subject himself / herself has consented to the editing content. Although details will be described later, in the example of this embodiment, the consent acquisition unit 206 acquires the consent information of the consent acquisition subject based on a facial image of the consent acquisition subject himself / herself captured with a built-in camera of a smartphone used by the consent acquisition subject.

[0040] FIG. 9(a) is a diagram showing an example of an editor's GUI 901 generated by the consent acquisition unit 206 when acquiring consent information on a smartphone 900 used by an editor who edited an image. The editor's GUI 901 shown in FIG. 9(a) includes a consent target face image 902 of the consent target person, a current consent status 903, a consent URL copy button 904, and an edited image 905 whose use is currently restricted. The editor using this smartphone 900 can recognize the consent target person by looking at the consent target face image 902 displayed in the editor's GUI 901. Then, when the editor presses the consent URL copy button 904 and inputs an instruction to send the consent URL to the smartphone used by the consent target person, the smartphone 900 sends the consent URL to the smartphone used by the consent target person. This allows the editor's smartphone 900 to share the consent URL with the smartphone of the consent target person.

[0041] FIG. 9(b) is a diagram showing an example of a GUI 921 for consenters displayed on the smartphone 920 of the person to whom consent is to be made, which has shared a consent URL with the smartphone 900 of the editor shown in FIG. 9(a). The GUI 921 for consenters is assumed to be a GUI prepared in response to the consent URL. The GUI 921 for consenters shown in FIG. 9(b) displays a facial image 922 to be consented to, an edited facial image 923 thereof, an agree button 924 that the person to whom consent is to be made presses when consenting, and a disagree button 925 that the person to whom consent is to be made presses when disagreeing. The person to whom consent is to be made views the facial image 922 to be consented to and the edited facial image 923, and if the person to whom consent is to be made presses the disagree button 925 to disagree if the person does not agree to the image editing. On the other hand, if the person to whom consent is to be made presses the agree button 924 to agree to the image editing such as the edited facial image 923.

[0042] Furthermore, when the consent button 924 of the GUI 921 for consenters is pressed, a facial image capture assist screen, as shown in FIG. 10 , for use in facial recognition processing to determine whether the person is the person to be consented to, is displayed on the smartphone 920 of the person to be consented to. As shown in FIG. 10 , the capture assist screen for facial recognition processing displays an instruction image requesting the person to orient their face in a predetermined direction or in multiple directions and then capture a facial image. The person to be consented to adjust their facial orientation in accordance with the instruction image in FIG. 10 and then operates a built-in camera (e.g., an in-camera) (not shown) of their smartphone 920 to capture a facial image. At this time, the editor's smartphone 900 and the smartphone 920 of the person to be consented to share a consent URL, so the consent acquisition unit 206 of the editor's smartphone 900 can acquire a facial image of the person to be consented to with their face oriented in accordance with the instruction image in FIG. 10 .

[0043] The consent acquisition unit 206 of the editor's smartphone 900 performs face recognition processing to determine whether the face image acquired via the consent URL is the face image of the person to be consented to, based on the face image acquired via the consent URL and the face image of the person to be consented to acquired from the judgment unit 204. First, the consent acquisition unit 206 determines whether the facial direction indicated on the shooting assistance screen of Fig. 10 via the consent URL matches the facial direction in the captured facial image. This makes it possible to prevent, for example, fraud such as someone impersonating the consent-obtained person by using a facial image of the consent-obtained person captured in advance. Note that the method disclosed in Reference 3 below can be used to estimate the facial direction of the subject from the facial image. Reference 3: A. Gupta, "Nose, Eyes and Ears: Head Pose Estimation by Locating Facial Keypoints," CVPR2018.

[0044] Furthermore, if the face direction indicated via the consent URL matches the face direction of the captured face image, the consent acquisition unit 206 performs face recognition processing to determine whether or not the face image is the person to be consented to, based on that face image and the consent target face image acquired from the determination unit 204. For the face recognition processing, a method such as that disclosed in Reference 4 below can be used. Reference 4: J. Deng, et al, "Arcface: Additive angular margin loss for deep face recognition", CVPR2018.

[0045] The consent acquisition unit 206 may acquire the consent information by having the person to consent present a photo ID, such as a passport or driver's license, and acquiring the photo of the person's face from the ID. Alternatively, the consent acquisition unit 206 may perform facial recognition using, for example, facial information registered for unlocking a smartphone, and acquire the consent information based on the results of the facial recognition. In this case, the consent acquisition unit 206 may determine the upper limit of the consent score based on the information used when acquiring the consent information. For example, the consent acquisition unit 206 may increase the consent score when using strictly managed information such as an ID, whereas it may decrease the consent score when using simple information such as facial information registered on a smartphone. In other words, the consent information in this case can be used as information indicating up to which consent score the person has consented to editing. That is, by determining the condition value for the consent score based on the information used when acquiring the consent information, it is possible to determine the extent to which the person can consent to editing. In this way, the consent acquisition unit 206 can also determine the degree of image editing for which consent can be obtained from the individual based on the information used when acquiring the individual's consent information. The consent acquisition unit 206 then sends the individual's consent information to the restriction unit 207.

[0046] The restriction unit 207 determines whether to restrict or release the use of the edited image based on the unedited image, the signed edited image, and the user's consent information. In this embodiment, the restriction unit 207 first restricts the use of the edited image based on the signed digital signature of the edited image. Image use restriction refers to a measure such as adding a watermark to the edited image displayed on a smartphone to make it clear that the image has been edited. Note that image use restriction may include adding a watermark to the image, or one or more measures such as blurring the faces of non-consenting persons or blacking out the edited areas to prevent the edited image from being saved on the smartphone as is. Image use restriction may also include restricting (prohibiting) saving the image to the smartphone, restricting (prohibiting) attaching the image to an email, restricting (prohibiting) public disclosure such as uploading to the web, and restricting (prohibiting) screenshots.

[0047] When determining whether to restrict or lift the restriction on the use of an image, the restriction unit 207 first restores the hash value of the image from the public key included in the digital signature and checks whether it matches the hash value of the signed edited image. If the image has been edited, the hash value will be different, and the restriction unit 207 will restrict the use of the image. Furthermore, the restriction unit 207 checks whether personal consent information exists for any edit history that requires the consent of the individual, among the edit history included in the signed edited image. If corresponding consent information exists, the restriction unit 207 updates the consent-receiving person information included in the edit history to a state with the consent of the individual. Then, if all consent-receiving person information included in the edit history has the consent of the individual or does not require consent, the restriction unit 207 lifts the restriction on the use of the edited image.

[0048] 11 is a diagram showing an example of a GUI 1102 displayed on the smartphone 1100 of a user on the editor side after the use restrictions on the image have been lifted. The editor's GUI 1102 shown in FIG. 11 displays a consent-requested face image 1103 of the person who was the subject of image editing, the person's consent status 1104 for the image, and an edited image 1106 for which the use restrictions have been lifted. The GUI 1102 also provides a download button 1107 for the edited image, which is enabled when the use restrictions on the image are lifted. When the user presses the download button 1107 via the input unit 101, the smartphone 1100 becomes able to download the edited image via the consent URL.

[0049] 12 is a flowchart showing the flow of information processing performed by the information processing device 100 of this embodiment. In the following flowcharts, the symbol S represents a processing step (process). First, in S1201, the image acquisition unit 201 acquires an image. For example, the image acquisition unit 201 acquires an image stored in the mass storage unit 107. In this embodiment, the image stored in the mass storage unit 107 is assumed to be an image captured by a built-in camera of a smartphone, but as described above, the image may be an image recorded in a storage device such as a hard disk or an image received via the Internet.

[0050] Next, in S1202, the detection unit 202 detects faces and human regions from the image acquired by the image acquisition unit 201, and generates a human detection result list. Information equivalent to the human detection result list may be displayed in the GUI 901, but this is not essential and may not be displayed.

[0051] Next, in S1203, the editing unit 203 edits the image in accordance with instructions from the user. In this embodiment, the editing unit 203 enables the user to edit the image by generating an image editing GUI 501 such as the example in FIG. 5 described above. That is, in S1203, the editing unit 203 edits the image based on instructions from the user via the image editing GUI 501.

[0052] Next, in S1204, the determination unit 204 determines whether or not consent is necessary for each edit history by the editing unit 203, and adds consent recipient information to the edit history. Next, in S1205, the storage unit 205 stores the pre-edited image, the edited image, the consent target face image, the editing history, and the digital signature. Next, in S1206, the restriction unit 207 restricts the use of the edited image based on the unedited image, the edited image with the signature, and the individual's consent information. Next, in S1207, the consent acquisition unit 206 generates a GUI for acquiring the principal's consent information. The consent acquisition unit 206 acquires the principal's consent information as described above with reference to FIGS. 9(a) and 9(b).

[0053] Next, in S1208, the consent acquisition unit 206 determines whether or not the necessary consent information of the person concerned has been acquired for each editing history, and if it is determined that the consent information of the person concerned has been acquired, the process proceeds to S1209. On the other hand, if the consent information of the person concerned has not been acquired, the consent acquisition unit 206 proceeds to S1210. If the process proceeds to S1209 because the consent information of the person concerned has been acquired, the restriction unit 207 removes the restriction on the use of the image. After that, the restriction unit 207 proceeds to S1210. When the process proceeds to S1210, the consent acquisition unit 206 determines whether to continue the process, and returns the process to step S1207 unless, for example, an instruction to end the process is input from the user. On the other hand, if an instruction to end the process is input from the user, the consent acquisition unit 206 determines not to continue the process, and ends the process of the flowchart in FIG.

[0054] As described above, according to the information processing device 100 of the first embodiment, it is possible to easily obtain consent from the person in question for image editing, and also to impose restrictions on images for which consent has not been obtained from the person in question, thereby reducing the risk of unauthorized use of edited images.

[0055] <Second embodiment> In the second embodiment, an example will be described in which image editing is restricted outside of an area for which the user's consent has been obtained. Note that, in the information processing device 100 of the second embodiment, the hardware configuration is the same as the configuration in FIG. 1 described above, and therefore illustration and description thereof will be omitted.

[0056] FIG. 13 is a functional block diagram showing an example of the functional configuration of an information processing device 100 according to the second embodiment. In the second embodiment, the information processing device 100 may be, for example, a smartphone, a tablet terminal, a personal computer, or other device, and the following description will be given using a smartphone as an example. The functional units shown in FIG. 13 are realized by the CPU 104 in FIG. 1 executing an information processing program according to the second embodiment, but at least some or all of the functional units may be realized by a circuit or other configuration. The image acquisition unit 1301 and the detection unit 1302 have the same functions as the image acquisition unit 201 and the detection unit 202 described in the first embodiment, and therefore detailed description thereof will be omitted.

[0057] The storage unit 1305 stores the person detection result list together with the digital signature in the image. The method of issuing the digital signature is performed using the same procedure as in the first embodiment. The storage unit 1305 stores the person detection result list and the digital signature in the image data in the same manner as in the first embodiment, creating a signed image. The storage unit 1305 then sends the signed image to the consent acquisition unit 1306.

[0058] The consent acquisition unit 1306 obtains consent from the person to be consented to based on the person detection result list included in the signed image. The method for obtaining consent from the person to be consented to can be the same method as described in the first embodiment. Furthermore, the consent acquisition unit 1306 adds information indicating whether or not consent has been given to each person detection result, and updates the data of the signed image. Then, the consent acquisition unit 1306 sends the signed image to the editing unit 1303. Note that the consent acquisition unit 1306 may determine the need for consent based on the ratio of the number of pixels in the person area of ​​the person detection result to the number of pixels in the original image.

[0059] The editing unit 1303 edits the signed image, and the same editing method as in the first embodiment can be used for editing the signed image. When the area to be edited by the editing unit 1303 includes a person area in the person detection result, the restriction unit 1307 checks information indicating whether or not the person's consent is obtained, which is included in the corresponding person detection result. If the person's consent is not obtained, the restriction unit 1307 restricts the target editing work. Restricting the editing work means that when editing an area that requires the person's consent, if it is determined that the person's consent has not been obtained, the restriction unit 1307 suspends the editing by displaying an error message, for example. In this case, the restriction unit 1307 may determine the range of editing functions to be released from the restriction, based on information used by the consent acquisition unit 206 when acquiring the person's consent.

[0060] 14 is a flowchart showing the flow of information processing performed by the information processing device 100 according to the second embodiment. The processes in S1401 and S1402 are the same as those in S801 and S802 in FIG. 8, and therefore description thereof will be omitted. In S1403, the consent acquisition unit 1306 acquires information indicating whether or not consent is given for each person detection result in the person detection result list. Next, in S1404, the storage unit 1305 stores the person detection result including the digital signature and information indicating whether consent is given for the image. Next, in S1405, the editing unit 1303 selects a person or a body part to be edited. Furthermore, in S1406, the editing unit 1303 determines whether or not there is consent based on information indicating whether or not there is consent included in the person detection result corresponding to the person or body part to be edited. If the editing unit 1303 determines that there is consent, the processing proceeds to S1408, and if the editing unit 1303 determines that there is no consent, the processing proceeds to S1407.

[0061] If the process proceeds to S1407 because there is no personal consent information, the restriction unit 1307 restricts the corresponding editing function. On the other hand, if the process proceeds to S1408 because there is consent information from the person, the restriction unit 1307 permits the corresponding editing function. After S1407 or S1408, in S1409, the editing unit 1303 determines whether to continue the processing, and returns the processing to S1405 unless, for example, an instruction to end the processing is input from the user. On the other hand, if an instruction to end the processing is input from the user, the editing unit 1303 determines not to continue the processing, and ends the processing of the flowchart in FIG.

[0062] As described above, the information processing device 100 of the second embodiment restricts or does not restrict the editing function depending on whether or not the person concerned has given consent for each editing target. The information processing device 100 of the second embodiment can reduce the risk of unauthorized use of images while also reducing the burden on the editor.

[0063] <Examples of application to other content> In the above-described embodiment, an example of editing image content was given. However, in the case of editing audio content, for example, the functional configuration of FIG. 2 can include functional units that perform the following processes. The audio acquisition unit, provided instead of the image acquisition unit 201, acquires audio content in which people are conversing, and the detection unit 202 detects only the human conversation from the audio acquired by the audio acquisition unit, excluding background sounds, etc. The editing unit 203 may, for example, change the audio of a person's conversation to the audio of a different conversation, or to the audio of a different person's conversation. The determination unit 204 determines the necessity of consent and the person to whom consent is required based on the audio detected by the detection unit 202, the audio edited by the editing unit 203, and the editing history. The storage unit 205 stores the audio, and the consent acquisition unit 206 acquires consent based on the determination result of the necessity of consent. The restriction unit 207 restricts or removes the restriction on the edited audio based on whether or not consent is obtained. It is possible to impose not only the same restrictions on such edited audio content as in the first embodiment, but also the same editing restrictions as in the second embodiment. Similarly, it is possible to impose the same post-editing restrictions as in the first embodiment and the same editing restrictions as in the second embodiment on text content.

[0064] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more of the functions. The above-described embodiments are merely examples of specific implementations of the present invention, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0065] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) A determination means for determining whether consent is required for use of the edited content based on the edit history of the content; consent acquisition means for acquiring consent information according to the need for consent; a restriction means for restricting use of the edited content based on the consent information; An information processing device comprising: (Configuration 2) The information processing device described in configuration 1, characterized in that the determination means determines the need for consent based on at least one of the editing area of ​​the content, the editing function used for editing, and the inappropriateness of the content. (Configuration 3) the content is an image content including a person, The information processing device described in configuration 2, wherein the determination means determines the need for consent based on at least one of the number of pixels in the edited area of ​​the image, the edited part of the person, the editing function, and the inappropriateness of the content. (Configuration 4) The information processing device described in configuration 3 is characterized in that the determination means calculates a consent requirement score to be used in determining the need for consent based on the number of pixels in the overlapping area between the person's area and the edited area, the number of pixels in the image before editing, and a score set for each editing function used in the editing, and determines the need for consent for use of the edited content based on a comparison of the consent requirement score with a predetermined threshold. (Configuration 5) 5. The information processing device according to configuration 4, wherein the determining means calculates the agreement-requiring score by assigning a weight to each of the body parts of the person. (Configuration 6) 6. The information processing device according to configuration 5, wherein the determining means assigns the weight so that the agreement-requiring score becomes high when the edited part is a person's face. (Configuration 7) The information processing device described in any one of configurations 4 to 6, characterized in that when editing is performed using the editing function to change a person's facial expression, the determination means calculates the agreement requirement score by assigning a weight according to the person's facial expression. (Configuration 8) The information processing device described in configuration 7, characterized in that the determination means assigns the weight so that the required agreement score is higher when editing is performed to change the facial expression of the person to an expression different from the facial expressions of other people around them. (Configuration 9) 9. The information processing device according to any one of configurations 4 to 8, wherein the determining means calculates the consent-requiring score by assigning a weight according to the inappropriateness of the content. (Configuration 10) 10. The information processing device according to configuration 9, wherein the determining means assigns the weighting such that the higher the inappropriateness of the content, the higher the consent-requiring score. (Configuration 11) The information processing device described in any one of configurations 3 to 10, characterized in that the determination means calculates a consent requirement score to be used in determining the need for consent based on the distance between the center of the edited area and the center of the person's area and a score set for each editing function used in the editing, and determines the need for consent for use of the edited image based on a comparison of the consent requirement score with a predetermined threshold. (Configuration 12) 12. The information processing device according to any one of configurations 3 to 11, wherein the determination means determines from which person in the image the consent is required based on a correspondence between the person included in the image and the edited area included in the edit history. (Configuration 13) 13. The information processing device according to any one of configurations 2 to 12, wherein the determining means determines the degree of editing of the content based on information used when the consent information is obtained. (Configuration 14) 14. An information processing device according to any one of configurations 1 to 13, further comprising a storage means for storing at least one of the editing history including information on the editing area where editing was performed and the consent information in the content. (Configuration 15) The information processing device according to any one of configurations 1 to 14, wherein the restriction means restricts the use of the edited content by at least one of prohibiting the saving of the edited content, prohibiting the public release of the edited content, and prohibiting the transmission of the edited content. (Configuration 16) If the content is an image content, 16. The information processing device according to configuration 15, wherein the restriction on the use of the edited content further includes at least one of adding a watermark to the image and blurring the image. (Configuration 17) editing means for editing the content; consent acquisition means for acquiring consent information for editing the content; a restriction unit that restricts editing of the content by the editing unit based on the consent information; An information processing device comprising: (Configuration 18) editing means for editing the content; a storage means for storing an editing history including information on an editing area for the content in data of the content; An information processing device comprising: (Method 1) a determination step of determining whether consent is required for use of the edited content based on the edit history of the content; a consent acquisition step of acquiring consent information according to the need for consent; a restriction step of restricting use of the edited content based on the consent information; An information processing method comprising: (Direction 2) an editing step of editing the content; a consent acquisition step of acquiring consent information for editing the content; a limiting step of limiting editing of the content in the editing step based on the consent information; An information processing method comprising: (Method 3) an editing step of editing the content; a storing step of storing an editing history including information on an editing area for the content in data of the content; An information processing method comprising: (Program 1) A program that causes a computer to function as the information processing device according to any one of configurations 1 to 18. [Explanation of symbols]

[0066] 201: Image acquisition unit, 202: Detection unit, 203: Editing unit, 204: Determination unit, 205: Storage unit, 206: Consent acquisition unit, 207: Restriction unit

Claims

1. A determination means for determining whether consent is required for use of the edited content based on the edit history of the content; consent acquisition means for acquiring consent information according to the need for consent; a restriction means for restricting use of the edited content based on the consent information; An information processing device comprising:

2. 2. The information processing device according to claim 1, wherein the determining means determines the necessity of consent based on at least one of an editing area of ​​the content, an editing function used for editing, and a degree of inappropriateness of the content.

3. the content is an image content including a person, The information processing device according to claim 2, characterized in that the determination means determines the need for consent based on at least one of the number of pixels in the edited area of ​​the image, the edited part of the person, the editing function, and the inappropriateness of the content.

4. The information processing device described in claim 3, characterized in that the determination means calculates a consent requirement score to be used in determining the need for consent based on the number of pixels in the overlapping area between the person's area and the edited area, the number of pixels in the image before editing, and a score set for each editing function used in the editing, and determines the need for consent for the use of the edited content based on a comparison of the consent requirement score with a predetermined threshold.

5. 5. The information processing apparatus according to claim 4, wherein the determining means calculates the agreement-requiring score by assigning a weight to each of the body parts of the person.

6. 6. The information processing apparatus according to claim 5, wherein said determining means assigns the weight so that the consent-requiring score becomes high when the edited part is a human face.

7. The information processing device according to claim 4, characterized in that, when editing is performed using the editing function to change a person's facial expression, the determination means calculates the agreement requirement score by assigning a weight according to the facial expression of the person.

8. The information processing device according to claim 7, characterized in that the determination means assigns the weight so that the agreement score is higher when editing is performed to change the facial expression of the person to an expression different from the facial expressions of other people around them.

9. 5. The information processing apparatus according to claim 4, wherein the determining means calculates the consent-requiring score by assigning a weight according to the inappropriateness of the content.

10. 10. The information processing apparatus according to claim 9, wherein the determining means assigns the weights such that the higher the inappropriateness of the content, the higher the consent-requiring score.

11. The information processing device described in claim 3, characterized in that the determination means calculates a consent requirement score to be used in determining the need for consent based on the distance between the center of the edited area and the center of the person's area and a score set for each editing function used in the editing, and determines the need for consent for the use of the edited image based on a comparison of the consent requirement score with a predetermined threshold.

12. 4. The information processing device according to claim 3, wherein the determining means determines from which person in the image the consent is required based on a correspondence between the person included in the image and the edited area included in the edit history.

13. 3. The information processing apparatus according to claim 2, wherein the determining means determines the degree of editing of the content based on information used when the consent information is obtained.

14. 2. The information processing apparatus according to claim 1, further comprising a storage unit for storing at least one of the editing history including information on an editing area where editing has been performed and the consent information in the content.

15. The information processing device according to claim 1, characterized in that the restriction means restricts the use of the edited content by at least one of prohibiting the saving of the edited content, prohibiting the public release of the edited content, and prohibiting the transmission of the edited content.

16. 16. The information processing device according to claim 15, wherein, when the content is an image content, restrictions on the use of the edited content further include at least one of adding a watermark to the image and blurring the image.

17. editing means for editing the content; consent acquisition means for acquiring consent information for editing the content; a restriction unit that restricts editing of the content by the editing unit based on the consent information; An information processing device comprising:

18. editing means for editing the content; a storage means for storing an editing history including information on an editing area for the content in data of the content; An information processing device comprising:

19. a determination step of determining whether consent is required for use of the edited content based on the edit history of the content; a consent acquisition step of acquiring consent information according to the need for consent; a restriction step of restricting use of the edited content based on the consent information; An information processing method comprising:

20. an editing step of editing the content; a consent acquisition step of acquiring consent information for editing the content; a limiting step of limiting editing of the content in the editing step based on the consent information; An information processing method comprising:

21. an editing step of editing the content; a storing step of storing an editing history including information on an editing area for the content in data of the content; An information processing method comprising:

22. Computer, A determination means for determining whether consent is required for use of the edited content based on the edit history of the content; consent acquisition means for acquiring consent information according to the need for consent; a restriction means for restricting use of the edited content based on the consent information; A program that causes the device to function as an information processing device having the above.

23. Computer, editing means for editing the content; consent acquisition means for acquiring consent information for editing the content; a restriction unit that restricts editing of the content by the editing unit based on the consent information; A program that causes the device to function as an information processing device having the above.

24. Computer, editing means for editing the content; a storage means for storing an editing history including information on an editing area for the content in data of the content; A program that causes the device to function as an information processing device having the above.

Images