Information processing device and its control method

Abstract

We help assess the risk of content infringement. [Solution] The information processing device includes an acquisition means for acquiring products generated by a generation AI, an identification means for identifying risky objects that pose a risk of infringing rights among the objects included in the product, and a recording means for recording the location information of the risky objects in the product as risk information.

Description

Technical Field

[0001] The present invention relates to a technology for assisting in the evaluation of the risk of infringement of rights regarding products (content).

Background Art

[0002] With the spread of pre-trained models (generative AI) trained for data generation, an environment is being prepared where a large amount of various types of content (text, images, videos, audio, 3D models, etc.) can be generated. On the other hand, products generated by generative AI (AI products) may include content that has a risk of infringing on the rights of third parties (trademark rights, copyrights, portrait rights, etc.). In particular, in generative AI, there are some that arbitrarily generate detailed parts not included in the content (prompt) specified by the user. Therefore, when using AI products, it is necessary to correctly grasp the risk of infringement of rights of AI products.

[0003] In Patent Document 1, a technique is disclosed for obtaining related information (such as copyright license conditions and portrait right license conditions) of an image group used for training an image generation AI, and attaching the related information as metadata to an AI product generated by the model. Also, in Non-Patent Document 1, the utilization of Internet searches (text searches and image searches) has been proposed as a method for checking whether an AI product is similar to an existing copyrighted work.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

[0005]

Non-Patent Document 1

Summary of the Invention

[0006] However, with the method described in Patent Document 1, if relevant information related to the image set used to train the generating AI cannot be obtained, it is not possible to assign appropriate metadata to the AI ​​product. As a result, it is not possible to assess the risk of intellectual property infringement of the AI ​​product.

[0007] Furthermore, the method described in Non-Patent Document 1 may overlook objects with a high risk of infringement if the AI-generated image contains a large number of objects. In particular, when an AI-generated image contains many objects, it is easy to fail to assess the infringement risk of relatively small objects. [Means for solving the problem]

[0008] To solve the above-mentioned problems, the information processing apparatus according to the present invention has the following configuration. That is, the information processing apparatus is A means for obtaining the generated product from the generation AI, An identification means for identifying risky objects among the objects contained in the aforementioned product that pose a risk of infringing rights, A recording means for recording the positional information of the risk object in the product as risk information, It has. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a technology that supports the assessment of the risk of rights infringement with respect to content. [Brief explanation of the drawing]

[0010] [Figure 1] This diagram shows the hardware configuration of an information processing device. [Figure 2] This is a diagram illustrating the usage scenarios of information processing devices. [Figure 3] This diagram shows the functional configuration of an information processing device. [Figure 4]It is a flowchart for calculating and recording risk values. [Figure 5] It is a diagram showing a table for managing category names and selection targets. [Figure 6] It is a diagram showing a table for managing the risk values of objects (Modification Examples 1-3). [Figure 7] It is a diagram for explaining the usage scenario of the information processing apparatus (Second Embodiment). [Figure 8] It is a diagram showing the functional configuration of the information processing apparatus (Second Embodiment). [Figure 9] It is a flowchart for calculating and recording risk values (Second Embodiment). [Figure 10] It is a diagram for explaining the usage scenario of the information processing apparatus (Third Embodiment). [Figure 11] It is a diagram showing the functional configuration of the information processing apparatus (Third Embodiment). [Figure 12] It is a flowchart for evaluating a learned model.

Modes for Carrying Out the Invention

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0012] (First Embodiment) As a first embodiment of the information processing apparatus according to the present invention, an information processing apparatus for evaluating the risk of copyright infringement for an image generated using an image generation AI will be described below as an example.

[0013] <Summary> In the present embodiment, when evaluating the risk of copyright infringement for an image generated using an image generation AI, an image area with a high risk of copyright infringement in the image is presented to the user. Specifically, objects included in the image are extracted, a risk value for copyright infringement is calculated for each object, and the position information in the image, the risk value, and the metadata of the image are recorded. The risk value is not essential as metadata. For example, there may be a case of information regarding the presence or absence of risk. Then, when displaying the image, the content of the metadata is also displayed. For example, the area recorded in the metadata is superimposed and displayed on the image.

[0014] FIG. 2 is a diagram for explaining the usage scenario of the information processing apparatus. The calculation of the position information and the risk value, and the recording into the metadata are performed by the information processing apparatus 201. In FIG. 2, the position information and the risk value are calculated for the objects 203 to 205 included in the image 202. Further, the calculated position information and risk value are recorded as metadata 206 in the image.

[0015] The user can check the content recorded in the metadata 206 on the display screen 207 displayed via the display unit 107 at an arbitrary timing. For example, it can be checked when considering commercially using the image 202. Further, by checking the content recorded in the metadata 206, the user can easily identify the position of the object with a risk of copyright infringement and the magnitude of the risk of copyright infringement. Thereby, the burden on the user for checking the risk of copyright infringement of the image 202 can be reduced.

[0016] <Regarding terms> The image generation AI is, for example, an image processing model that artificially generates a new image based on specific input data (also called a prompt) such as text or an image.

[0017] Objects that pose a risk of rights infringement include, for example, characters such as people and animals, buildings, symbols, and logos—objects included in an image. Furthermore, there are no particular limitations on the proportion or quantity of these objects that occupy the image. In other words, an object may be depicted in only a small area of ​​the image, or it may occupy the majority of the image. Also, there may be only one object or multiple objects in the image.

[0018] Location information refers to the area within an image where an object is depicted. The shape of this area is not particularly limited, as long as it includes the depiction range of the object. For example, it may be rectangular or follow the contour of the object.

[0019] A risk value is an arbitrary index that indicates the degree to which an object is likely to be an infringing object that infringes on the rights of existing copyrighted works, etc., which are protected by rights (trademark rights, copyrights, portrait rights, etc.).

[0020] Metadata refers to supplementary information about an image, and in this embodiment, it is the location information and risk value mentioned above. Metadata can be embedded, for example, as the Exif (Exchangeable image file format) header of an image file. The specific method of recording metadata is not particularly limited, as long as the recorded content can be viewed by the user at any time.

[0021] <Device configuration> Figure 1 shows the hardware configuration of the information processing device. The CPU 101 controls various devices connected to the bus 102 and executes the information processing of the present invention. CPU is an abbreviation for Central Processing Unit. ROM 103 stores the basic input / output (BIOS) program and the boot program. ROM is an abbreviation for Read Only Memory. RAM 104 is used as the main memory of the CPU 101. RAM is an abbreviation for Random Access Memory.

[0022] External storage 105 is a storage device such as an HDD or SSD that stores programs to be processed by the information processing device 201. HDD is an abbreviation for Hard Disk Drive, and SSD is an abbreviation for Solid State Drive.

[0023] The input unit 106 is a keyboard or mouse, and is hardware that accepts user input. The display unit 107 is hardware that outputs the calculation results of the information processing device 201 to the display device according to instructions from the CPU 101. The display device can be of any type, such as a liquid crystal display, projector, or LED indicator. LED is an abbreviation for Light Emitter Diode. The I / O 108 is an interface for communicating with a management server (not shown) that stores a given set of rights-protected data. I / O is an abbreviation for Input / Output. The communication connection can be wired or wireless. The bus 102 is a bus that connects the above-mentioned parts in a manner that allows them to communicate with each other.

[0024] Figure 3 is a diagram showing the functional configuration of the information processing device. Details of the processing operation of each functional unit will be described later with reference to Figure 4. The information processing device 301 includes a first acquisition unit 302, an identification unit 303, a first feature quantity calculation unit 304, a second acquisition unit 305, a second feature quantity calculation unit 306, a risk value calculation unit 307, and a recording unit 308.

[0025] While it is assumed that each functional unit included in the information processing device 301 is implemented by software (by the CPU 101 executing a program), some or all of them may be implemented by hardware such as application-specific integrated circuits (ASICs). Furthermore, they may be implemented by a single information processing device, or by multiple information processing devices working together.

[0026] The first acquisition unit 302 acquires the generated products (content such as text, images, and audio) generated by the generation AI. In this embodiment, an image 202 generated using the image generation AI is acquired.

[0027] The identification unit 303 identifies one or more objects (risky objects with a risk of infringement) included in the product acquired by the first acquisition unit 302. In this embodiment, it identifies objects 203 to 205, which are objects included in the image 202, and calculates the positional information (coordinates) of each object within the image.

[0028] The first feature calculation unit 304 calculates the feature quantities of the objects identified by the identification unit 303. In this embodiment, the feature quantities of objects 203 to 205 are calculated separately.

[0029] The second acquisition unit 305 acquires data (images) from a management server (not shown) where rights-protected data is stored, which will serve as the basis for the data (features) used for comparison by the risk value calculation unit 307 described later. In this embodiment, the rights-protected data is a group of images known to be protected by copyright, trademark rights, portrait rights, etc.

[0030] The second feature calculation unit 306 calculates the features of the rights-protected data acquired by the second acquisition unit 305. In this embodiment, the features are calculated from the group of rights-protected images acquired by the second acquisition unit 305. The second feature calculation unit 306 calculates features in the same way as the first feature calculation unit 304. The second feature calculation unit 306 and the first feature calculation unit 304 may be configured as a single common functional unit.

[0031] The risk value calculation unit 307 calculates the risk value by comparing the features obtained by the first feature calculation unit 304 and the second feature calculation unit 306. In this embodiment, the risk value is calculated by comparing the features of each of the objects 203 to 205 calculated by the first feature calculation unit 304 with the features of each rights-protected image calculated by the second feature calculation unit 306. Details of the risk value calculation method will be described later.

[0032] The recording unit 308 associates the generated product acquired by the first acquisition unit 302, the location information of the object acquired by the identification unit 303, and the risk value calculated by the risk value calculation unit 307 and records them as risk information. In this embodiment, the risk information (location information and risk value for each of the objects 203 to 205) is recorded for the metadata portion of the image 202 acquired by the first acquisition unit 302.

[0033] <Device Operation> Figure 4 is a flowchart for calculating and recording the risk value for the generated product. Here, the generated product is assumed to be an image generated by the generation AI based on user instructions (such as prompts), and the following processing starts when the acquisition of the said image is instructed. However, processing S406 and S407 may be configured to be performed prior to the acquisition of the image in S401.

[0034] In S401, the first acquisition unit 302 acquires the image generated by the generation AI. In this embodiment, the image 202 generated using the image generation AI is acquired. The data acquired here is not particularly limited as long as it is a product generated using the generation AI, and may include, for example, video, audio, text, 3D models, etc. Also, multiple data may be acquired.

[0035] In S402, the identification unit 303 identifies (detects) objects included in the generated data acquired in S401. For example, it identifies objects 203 to 205 included in image 202. The process can also be simplified by identifying (detecting) only objects of a specific category (type) that needs to be checked for infringement.

[0036] Figure 5 shows a table that manages category names and selected objects. This table records category names that indicate the category of the object, and the type of object to be selected. Users can select one or more categories from the object management table to narrow down the objects to be identified in the generated data.

[0037] The specific method for identifying an object is not particularly limited, as long as it can extract the characteristic region of the object. For example, by calculating the feature quantities of an arbitrary range in an image and comparing them with the general feature quantities of each object recorded in the object management table, it can be determined whether that range corresponds to an object. The comparison of feature quantities can be performed, for example, by inputting the feature quantity data of an arbitrary range in an image into a pre-trained classification model.

[0038] Classification models include, specifically, models using neural networks, models using decision trees such as Random Forest and Gradient Boosting, and models using the k-nearest neighbor method. Alternatively, the results of comparing features can be output numerically, and a user-defined threshold can be used as a criterion to determine whether to set the feature region of the target object.

[0039] Image features used to extract the feature regions of an object include, for example, broad features such as color histograms and color distributions. Other features include those generated by SIFT (Scale-Invariant Feature Transform), SURF (Speed-Up Robust Features), and features generated by neural networks. Methods for extracting the feature regions of an object based on image features include bounding boxes (BB), instance segmentation, and semantic segmentation. Users may also manually define the feature regions.

[0040] In this embodiment, it is assumed that feature generation and comparison will be performed using a neural network, and the feature region of the target object will be extracted using a broadband (BB).

[0041] In S403, the identification unit 303 acquires the positional information of the feature region of the object acquired in S402. The positional information of the feature region of the object refers to information that identifies the location and range of the feature region of the object in the image. In this embodiment, the vertex coordinates in the image of the rectangular feature region corresponding to the object are acquired.

[0042] In S404, the risk value calculation unit 307 determines whether one or more objects were identified in S402. If one or more objects are identified, the process proceeds to S405; otherwise, the entire process shown in Figure 4 is terminated.

[0043] In S405, the first feature calculation unit 304 calculates the feature quantities for each region of the target object. Methods for calculating feature quantities include, for example, a neural network-based calculation method and a principal component analysis-based calculation method. In this embodiment, it is assumed that feature calculation will be performed using a convolutional neural network.

[0044] In S406, the second acquisition unit 305 connects to a management server where the rights protection data group is stored and acquires the rights protection data group. The management server where the rights protection data is stored is not particularly limited as long as it can store the rights protection data group, and may be an external storage 105 or a network server such as a cloud that communicates via a network using I / O 108.

[0045] In S407, the second feature calculation unit 306 calculates the features of each of the rights protection data group acquired in S406. The method for calculating the features is not particularly limited, as long as it can be compared with the features of the target object calculated in S405. In this embodiment, it is assumed that the same method as the feature calculation method used in S405 is used.

[0046] The stored rights protection data may be pre-featured, and the acquisition unit 305 may acquire the feature-rich data. Furthermore, if there are multiple data points with similar features among the rights protection data, only a portion of them may be acquired.

[0047] In S408, the risk value calculation unit 307 calculates the risk value of the object by comparing the feature quantities of the object acquired in S405 with the feature quantities of each of the rights protection data acquired in S407. Various methods are possible for comparing the feature quantities of the object with the feature quantities of the rights protection data. For example, Euclidean distance and cosine similarity can be used. Furthermore, the method of expressing the risk value is not particularly limited as long as it is a method that can compare the magnitude of the risk. For example, it may be a numerical value or it may be indicated by categories such as high, medium, and low. In this embodiment, it is assumed that the metadata 206 uses the maximum value of the cosine similarity obtained by comparing it with each of the feature quantities of the rights protection data as the risk value of the object. Furthermore, it is assumed that the risk value of the object takes the range of 0 to 1, and the closer it is to 1, the higher the risk of rights infringement.

[0048] In S409, the recording unit 308 records the location information of the feature region of the object acquired in S403 and the risk value calculated in S408 in the external storage 105 for each object. It is also possible to compare the risk value with a threshold and record those that exceed the threshold as having a risk, and all others as not having a risk. The user can then find out whether an object has a risk or not.

[0049] In S410, the risk value calculation unit 307 checks whether a risk value has been calculated for all objects identified in S402. If there are any objects for which a risk value has not been calculated, the unit performs the processes in S405 and S408-S409 for the unprocessed objects to calculate and record their risk values. Once a risk value has been calculated and recorded for all objects, the entire process shown in Figure 4 is terminated.

[0050] As described above, according to the first embodiment, one or more objects contained in an image are identified, and the location information and risk value (or presence or absence of risk) of each object are calculated and recorded as metadata. When a user evaluates the rights infringement risk of an image (such as a product generated by an image generation AI), the image to be evaluated is displayed, and the image areas with a high risk of rights infringement are displayed within that image. This makes it possible to reduce the possibility of overlooking objects with a high risk of rights infringement, even if the image to be evaluated contains many objects or small objects. Even without explicitly showing the risk value or the presence or absence of risk, image areas that are displayed in a way that makes them distinguishable from other areas as a result of the rights infringement check have a risk of rights infringement.

[0051] Furthermore, by associating location information with risk values ​​(presence or absence of risk) and recording this information in metadata for objects with a high risk of infringement, users can easily check information related to the risk of infringement at any time.

[0052] (Extreme Variation 1-1) In the above explanation, the method for specifying the category of the object included in the generated product, which is an identification condition for identifying the object, is configured to be selected by the user from a table (Figure 5). However, this is not limited to this, as long as the category of the object can be determined. For example, the category of the object may be determined from the user input information (prompt) when generating the product from the generation AI. Alternatively, the category of the object may be determined by performing scene analysis of the product. This eliminates the need for the user to determine the category of the object depending on the product.

[0053] (Variations 1-2) In the above explanation, the risk value was calculated using the maximum value obtained by calculating the cosine similarity between the features of the object and each feature of the rights protection data set. However, any value that indicates the magnitude of the rights infringement risk may be calculated using other derivation methods. For example, the risk value could be a value obtained by comparing the features of the object and each feature of the rights protection data set, weighted by information related to each rights protection data or information related to the object. Information related to rights protection data includes, for example, the "remaining rights period" and "whether or not a license is granted" for the rights protection data. Information related to the object includes the "occupation rate" and "center coordinates" of the object in the generated product. This allows users to obtain a rights infringement risk assessment result using complex information.

[0054] (Variations 1-3) In the above explanation, the location information and risk value of the object's feature region were recorded in the object's metadata (such as Exif). However, the data can be recorded in any other format, as long as it allows for the association between the generated data, the location information of the object's feature region, and the object's risk value. For example, a channel showing the distribution of risk values ​​could be added to the generated data's channels (e.g., RGB color channels) and recorded. Alternatively, the data could not be recorded together with the generated data, but managed in a separate database (table).

[0055] Figure 6 shows a table for managing the risk value of an object. For example, as shown in the management table in Figure 6, the identification information (ID) of the product, the location information of the characteristic region of the object (maximum and minimum values ​​of the X and Y coordinates), and the risk value of the object may be recorded in association with each other. Note that the items recorded in the management table are not limited to these items. Also, the storage location (storage unit) of the data containing the information such as the management table is not particularly limited as long as the user can check it at any time. That is, it may be an external storage 105, or a network server such as a cloud that can communicate via I / O 108.

[0056] (Variations 1-4) The above explanation assessed the risk of rights infringement when the generated product is an image. On the other hand, the type of data generated is not particularly limited, as long as a specific area within the generated product can be represented as location information. For example, it may be video, audio, or text. Therefore, we will now explain specific examples of location information of the target object when video, audio, or text is acquired.

[0057] If the output is a video, the object's location information includes, for example, location information that specifies the location and range within a specific frame image of the video. If it is audio, the object's location information includes, for example, time information based on the elapsed time from the start of the audio. The object's location and range can be specified by specifying the start and end times of the target range. If it is text, the object's location information includes, for example, information based on a number obtained by counting the word position from the start of the document within the text data. The object's location and range can be specified by specifying the starting word position and ending word position of the target range. Note that other location representation methods are also acceptable as long as the object's location within the output can be identified.

[0058] (Second Embodiment) In the second embodiment, a form is described in which additional information regarding rights protection is presented when the user assesses the risk of image rights infringement.

[0059] <Overview> In this embodiment, in addition to the screen display in the first embodiment (Figure 2), relevant rights protection data and supplementary information, as well as alternative images, are presented for objects whose risk value for rights infringement exceeds an arbitrary threshold. Here, relevant rights protection data refers to data from rights protection data managed by a management server, etc., that has been determined to have a high probability of infringing rights in the object. Supplementary information refers to information related to the rights pertaining to the rights protection data, and specifically includes information on the rights holder, the publication date of the rights protection data, and whether or not a license has been granted. The alternative images presented will be described in more detail later, but they are images with a low risk of rights infringement that can be replaced with the object.

[0060] Figure 7 illustrates a usage scenario of the information processing device in the second embodiment. Figure 7 shows a screen in which the information processing device 701 calculates location information and risk values ​​for objects contained in the generated image, and presents objects that exceed the risk value threshold set by the user.

[0061] Specifically, the information processing device 701 displays the image to be evaluated (an image depicting three faces) in the display area 702. It also overlays highlighting 703 on objects that exceed the threshold specified in the threshold setting user interface (UI) 707. Furthermore, it displays images of rights-protecting data and related information similar to the objects displayed in the display area 704 in the display area 705. In addition, it displays alternative images of the objects displayed in the display area 704 (images with a low risk of rights infringement) in the alternative image UI 706.

[0062] By checking display area 702, users can identify the location of objects with a high risk of copyright infringement within the image. Furthermore, by checking display area 704, they can view detailed information about these objects. Additionally, by checking display area 705, they can view the relevant copyright protection data and any associated information.

[0063] Furthermore, the alternative image UI706 presents one or more alternative images suggested as replacements for the object with a high risk of infringement. The user can perform the replacement of the object with an alternative image by selecting one of these alternative images to replace the object in question.

[0064] In the threshold setting UI 707, the user can arbitrarily set a threshold for the risk value, and the user can arbitrarily determine the sensitivity of notifications for rights infringement risk depending on the situation. In the display area 708, the generation conditions used when generating the image to be evaluated displayed in the display area 702 (for example, information on the generation model and prompts set during generation) are presented.

[0065] This allows users to easily identify objects posing a risk of copyright infringement, as well as similar copyright protection data and related information, within the image being evaluated. Furthermore, it makes it easy to generate images in which objects posing a risk of copyright infringement have been replaced with alternative images.

[0066] <Device configuration> Figure 8 shows the functional configuration of the information processing device in the second embodiment. Note that the hardware configuration is the same as in the first embodiment (Figure 1), so a description is omitted.

[0067] The information processing device 801 includes a first acquisition unit 302, an identification unit 303, a first feature quantity calculation unit 304, a second feature quantity calculation unit 306, and a risk value calculation unit 307, as described in the first embodiment (Figure 3). The information processing device 801 further includes an input unit 802, a second acquisition unit 803, a condition acquisition unit 804, a risk determination unit 805, a substitute acquisition unit 806, and a display unit 807.

[0068] The input unit 802 selects the product to be acquired by the first acquisition unit 302 and acquires thresholds used by the risk determination unit 805. In this embodiment, it selects the image to be displayed in the display area 702 and acquires the thresholds entered by the user in the threshold setting UI 707.

[0069] The second acquisition unit 803 acquires comparison data and supplementary information from the management server where the rights protection data is stored. In this embodiment, if the risk determination unit 805 determines that the risk value exceeds a threshold, the acquired data is displayed in the display area 705.

[0070] The condition acquisition unit 804 acquires the generation conditions used when generating the product acquired by the first acquisition unit 302. In this embodiment, the acquired generation conditions are displayed in the display area 708.

[0071] The risk determination unit 805 compares the risk value calculated by the risk value calculation unit 307 with the threshold value obtained by the input unit 802 to determine the risk of infringement. In this embodiment, the determination is made by comparing the threshold value entered in the threshold setting UI 707 with the risk value calculated by the risk value calculation unit 307.

[0072] The substitute acquisition unit 806 acquires an image to serve as a substitute for the object that the risk determination unit 805 has determined to have a high risk of infringement. For example, the image may be acquired from the external storage 105, or from a network server such as a cloud that can communicate via the I / O 108. In this embodiment, the acquired substitute image is displayed in the substitute image UI 706. The user can arbitrarily decide which image to use as a substitute for the object by operating the substitute image UI 706.

[0073] The display unit 807 generates and displays the screen shown in Figure 7. Specifically, it displays and controls the information acquired by the first acquisition unit 302, the identification unit 303, the second acquisition unit 803, the condition acquisition unit 804, and the substitute acquisition unit 806, as well as the judgment results determined by the risk judgment unit 805.

[0074] <Device Operation> Figure 9 is a flowchart for calculating and recording the risk value for the generated product in the second embodiment. Here, the generated product is assumed to be an image generated by the generation AI based on user instructions (such as prompts), and the following processing starts when the acquisition of the image is instructed. However, the processing in S406 and S407 may be configured to be performed prior to the acquisition of the image in S401. Note that S401 to S408 and S410 are the same as in the first embodiment, so their explanation is omitted.

[0075] In S901, the input unit 802 acquires the input content entered by the user. In this embodiment, the input content shown in Figure 7 (file path of image data, threshold) is acquired. Note that it is sufficient that the data includes data related to determining the risk of rights infringement, such as the threshold, and other information is not particularly limited.

[0076] In S902, the condition acquisition unit 804 acquires the product generation conditions acquired in S401. Product generation conditions are the conditions set to create the product acquired in S401. Examples include Model Name, Prompt, Negative Prompt, CFG Scale, and Seed. In this embodiment, it is assumed that the Model Name and Prompt are acquired, but the generation conditions to be acquired are not limited to these.

[0077] In S903, the risk assessment unit 805 compares the risk value of the object calculated in S408 with the threshold obtained in S901 to evaluate the infringement risk of the object and presents the result. The method for evaluating the infringement risk is not limited to a specific method. For example, the evaluation may be based on the relationship between the risk value of the object calculated by one model and the threshold. Alternatively, the risk value of the object may be calculated using multiple models, and the evaluation may be based on the relationship between the mean or median value and the threshold. Furthermore, supplementary information from the rights protection data may also be taken into consideration when performing the evaluation.

[0078] The display unit 107 may display one or more evaluation results. Furthermore, all results may be displayed, or the results displayed may be limited by filtering based on arbitrary conditions. In this embodiment (Figure 7), if the risk value of the object calculated in S408 is greater than the threshold, it is determined that the risk of infringement is high, and the object with a high risk of infringement is displayed using the highlighting 703 and display area 704.

[0079] In S904, the substitute acquisition unit 806 acquires substitute data for the object that was determined to have a high risk of infringement in S903 and presents it to the user. If an instruction is received to replace the object determined to have a high risk of infringement with a substitute image (for example, by pressing the "Replace" button in Figure 7), the unit controls the generation of a substitute product in which the object has been replaced with a substitute image. The substitute data is not particularly limited as long as it is data that can replace the object determined to have a high risk of infringement among the products acquired in S401. For example, a group of data with a low risk of infringement can be acquired in advance, recorded on a management server, etc., and retrieved at any time. The method for retrieving the recorded group of data is also not particularly limited, but for example, data with a high degree of similarity to the object determined to have a high risk of infringement can be retrieved.

[0080] Alternatively, data may be selected from prompt information when generating products from the generation AI, or the user may select arbitrary data by referring to tag data of the recorded data set. Only one alternative data set may be acquired, or multiple sets may be acquired and the user may be allowed to arbitrarily select which data to use as an alternative. In this embodiment (Figure 7), multiple images with high similarity to the object determined to have a high risk of infringement are acquired from a pre-prepared group of images with a low risk of infringement and displayed in the display area 708.

[0081] As described above, the second embodiment displays information on objects with a high risk of infringing rights contained in the product (image) being evaluated, as well as related rights protection data. This reduces the user burden when checking the details of rights protection data.

[0082] Furthermore, it becomes easier to replace objects with a high risk of infringement with similar alternative images in the evaluated product (image), making it easier to obtain images with a low risk of infringement.

[0083] (Variation 2-1) In the above explanation, the threshold was arbitrarily determined and set by the user by inputting into the threshold setting UI707. However, the method of setting the threshold is not limited to this. For example, it can be determined using a set of rights-protected data. For example, the maximum distance in the feature space of data sets recorded as the same category in the rights-protected data set can be determined as the threshold for rights infringement risk. Alternatively, the minimum distance between data sets recorded as different categories in the rights-protected data set can be determined as the threshold. Here, a category in the rights-protected data set refers to data that has the same rights information; for example, different poses or art styles of the same character are classified as the same category.

[0084] This eliminates the need for users to manually determine thresholds. Furthermore, it allows for the adaptive determination of thresholds based on the content of the rights protection data set being compared to the object at risk of infringement.

[0085] (Variation 2-2) In the above explanation, the alternative image was obtained from a pre-prepared group of images with a low risk of infringement. However, the method for obtaining an image with a low risk of infringement that can replace an object with a high risk of infringement is not particularly limited. For example, the image may be regenerated based on the generation conditions obtained by the condition acquisition unit 804, and any part with a low risk of infringement may be obtained. Alternatively, an object with a high risk of infringement may be used as a generation condition for the generation AI to generate a new image, and then an image with a low risk of infringement may be obtained from the generated image.

[0086] This eliminates the need for users to prepare a set of images with a low risk of copyright infringement in advance. It also reduces the amount of storage space required to record these low-risk images.

[0087] (Third embodiment) In the third embodiment, an information processing device for evaluating trained models that generate images will be described. In particular, a configuration will be described in which the risk values ​​described in the first embodiment are derived for multiple images generated using various trained models, recorded in relation to the generation conditions, and each trained model is evaluated.

[0088] <Overview> In this embodiment, for each trained model, the locations (regions) in the generated images that tend to have relatively high risk values ​​are displayed. Furthermore, the user is presented with the details of the generation conditions that tend to result in relatively high risk values ​​at those locations.

[0089] Figure 10 illustrates a usage scenario of the information processing device in the third embodiment. In Figure 10, the information processing device 1001 records the location information of objects with a risk of rights infringement in the generated products (images) of each trained model, along with the associated risk value and generation conditions, in the management table 1002. The information processing device 1001 also displays the contents recorded in the management table 1002, as well as the contents calculated from the recording results, on the display screen 1003. For example, users can check this information when retraining a trained model, making improvements specific to a trained model, or considering commercial use.

[0090] Display area 1004 shows the trend of risk values ​​according to location in the image generated by the trained model (in this case, ID=001). For example, areas with a particularly high risk of infringement are displayed as high-risk areas 1005. The risk value of the area and the generation conditions that affect the risk value are also displayed. By checking these displays, users can understand the locational trends of the products generated by each trained model. This allows users to, for example, make appropriate improvements to the trained model according to the locational trends.

[0091] <Device configuration> Figure 11 shows the functional configuration of the information processing device in the third embodiment. Note that the hardware configuration is the same as in the first embodiment (Figure 1), so its explanation is omitted.

[0092] The information processing device 1101 includes a first acquisition unit 302, an identification unit 303, a first feature quantity calculation unit 304, a second acquisition unit 305, a second feature quantity calculation unit 306, and a risk value calculation unit 307, as described in the first embodiment (Figure 3). The information processing device 1101 further includes a condition acquisition unit 1102, a condition evaluation unit 1103, a recording unit 1104, and a display unit 1105.

[0093] The condition acquisition unit 1102 acquires the generation conditions (model used, prompt) used when generating the product acquired by the first acquisition unit 302. In this embodiment, the acquired generation conditions are recorded in the management table 1002.

[0094] The condition evaluation unit 1103 evaluates the rights infringement risk associated with location information using the results from the risk determination unit 805 and the generation conditions acquired by the condition acquisition unit 1102. The recording unit 1104 records the contents acquired by the condition acquisition unit 1102 and the condition evaluation unit 1103 in the management table 1002, etc. As a result, information like the management table 1002 in Figure 10 is managed. The display unit 1105 displays the evaluation results of the trained model based on the contents of the management table 1002 recorded by the recording unit 1104.

[0095] Figure 12 is a flowchart for evaluating a trained model. Here, the output is an image generated by the generating AI based on user instructions (such as prompts), and the following processing starts when the acquisition of the image is instructed. However, processing in S406 and S407 may be configured to be performed prior to the acquisition of the image in S401.

[0096] In S1201, the condition acquisition unit 1102 acquires the product generation conditions acquired in S401. In this embodiment, the trained model ID and the prompt are acquired. The trained model ID is an ID that identifies the model file and the training data used for training. Note that the generation conditions to be acquired are not limited to these.

[0097] In S1202, the risk value calculation unit 307 checks whether the infringement risk assessment has been performed for all generated products (images) instructed by the user. If there are any generated products that have not been assessed for infringement risk, the process from S1201 onwards is performed on the unassessed products to assess their infringement risk. Once the infringement risk assessment has been completed for all generated products, the process proceeds to S1203.

[0098] In S1203, the condition evaluation unit 1103 records the model used, the product, the location information of each object in the product, the risk value of each object, and the generation conditions for multiple products whose infringement risk has been evaluated, in the recording unit 1104. Then, the display unit 1105 displays the evaluation results of the model from the recorded results.

[0099] Here, the positional information of each object in the product refers to the positional information of each object obtained in S403. The risk value is the risk value calculated in S408 for that object. The generation conditions are those related to the characteristics of the object among the generation conditions obtained in S1201. Examples include prompts, negative prompts, CFG scale, and seeds. In this embodiment, the trained model ID that identifies the model used, the product ID that identifies the product, the region range of the object in the product, the risk value of the object, and the prompt used when creating the product are recorded in the management table 1002.

[0100] The evaluation of a model is calculated from the location information, risk value, and generation conditions of each object generated by the model used. For example, by linking the infringement risk value of the objects generated by the model used with location information, and then accumulating and normalizing it, the trend of the distribution of infringement risk of the model used can be calculated.

[0101] Furthermore, for example, each word used in the prompt, which is one of the generation conditions, is linked to the risk value of the object included in the generated product and the location information of the object, and the summation is performed. By comparing the summation results, it is possible to calculate the words that tend to have a high risk of infringement (high-risk words) in the model used, and their location information. In this embodiment, the distribution trend of the infringement risk of the model used, and the prompt words with a high risk of infringement and their location information are calculated. Note that the evaluation method is not limited to the method described above, as long as it can evaluate the infringement risk of the model.

[0102] The display unit 107 presents the evaluation results. The method of presentation is not particularly limited, as long as it is a method that the user can recognize. The items to be presented are not particularly limited, as long as they are items that allow the user to confirm the risk of infringement of rights of the model used in conjunction with location information. For example, some or all of the items recorded in the management table 1002 may be presented. Alternatively, some or all of the evaluation results of the model used may be presented.

[0103] In the display screen 1003 in Figure 10 above, the trend of risk values ​​according to location, which is part of the evaluation results of the model used (ID=001), is displayed, and the high-risk area 1005 is highlighted. In addition, the risk value and high-risk word in the high-risk area 1005 are also displayed.

[0104] As described above, according to the third embodiment, the risk values ​​described in the first embodiment are derived for multiple images generated using various trained models and recorded in association with the generation conditions. In particular, the risk values ​​and generation conditions are recorded in association with the positional information within the images. Then, based on the recorded information, an evaluation linked to the position in the images generated by each model is calculated and presented to the user. This allows the user to grasp the trends regarding rights infringement risk in each trained model and to make appropriate improvements to the trained models according to the trends.

[0105] (Variation 3-1) In the above explanation, the risk value linked to location information was calculated by accumulating the risk value of the object in the generated product created by the trained model, linked to the object's location information. However, the method is not particularly limited as long as it can evaluate the rights infringement risk of the trained model. For example, weighting may be performed based on the location information within the image of the object. Specifically, if the user intends to use only a part of the generated product, weighting may be applied to reduce the rights infringement risk of areas that are less likely to be used. Alternatively, the risk value may be calculated by accumulating only the risk values ​​of objects that exceed an arbitrary threshold. Furthermore, the linking of the object's risk value to the object's location information may be done on a pixel-by-pixel basis in the generated product, or it may be linked to a region formed by connecting multiple pixels.

[0106] (Variation 3-2) In the above explanation, the risk of infringement of rights for prompt words was evaluated by linking the risk value of the object included in the generated product for each prompt word with the location information of the object and accumulating them. The risk of infringement of rights for prompt words was then evaluated from the result of this accumulation. However, the method for evaluating the risk of infringement of rights for prompt keywords is not limited to the method described above. For example, it is possible to limit the association to a select few prompt keywords that are highly relevant to the object among the multiple prompt words used to generate the product, and then accumulate them. The determination of prompt keywords highly relevant to the object can be made by user selection. Alternatively, all prompt words used to generate the object and product can be input into, for example, a pre-trained neural network to extract features. By comparing the extracted features using, for example, cosine similarity, prompt keywords highly relevant to the object can be selected.

[0107] The disclosures herein include the following information processing devices, control methods, and programs. (Item 1) A means for obtaining the generated product from the generation AI, An identification means for identifying risky objects among the objects contained in the aforementioned product that pose a risk of infringing rights, A recording means for recording the positional information of the risk object in the product as risk information, An information processing device characterized by having the following features. (Item 2) The identification means identifies objects of a specific category included in the product as the risk objects. The information processing device described in item 1, characterized by the features described herein. (Item 3) The system further includes a calculation means for calculating a risk value that indicates the degree of possibility that the aforementioned risk object is an object subject to infringement of rights, The recording means associates the location information and the risk value with respect to the risk object and records them in the risk information. An information processing device according to item 1 or 2, characterized by the features described herein. (Item 4) The calculation means calculates the risk value for the risk object based on the similarity between the risk object and given rights protection data. The information processing device described in item 3, characterized by the features described herein. (Item 5) The system further includes a display control means for displaying the product and the risk information on a display unit. An information processing device according to any one of items 1 to 4, characterized by the above. (Item 6) The display control means further displays information on the display unit regarding rights protection data similar to the risk object. The information processing device described in item 5, characterized by the features described herein. (Item 7) Substitute acquisition means for acquiring a substitute that can replace the aforementioned risk object, A receiving means for receiving instructions to replace the risky substance in the product with the substitute, A generating means for generating a substitute product in which the risk object is replaced with the substitute based on the above instructions, It further possesses An information processing device according to item 5 or 6, characterized by the features described herein. (Item 8) The system further includes a derivation means for deriving locations where there is a relatively high probability of a risk object being generated among the products generated by the generation AI, based on the risk information for the multiple products generated by the generation AI. An information processing device according to any one of items 1 to 7, characterized by the above. (Item 9) The generation AI further includes a condition acquisition means for acquiring the generation conditions when it generates each of the multiple products, The derivation means further derives the generation conditions under which a risk object is likely to be generated in the product generated by the generation AI. The information processing device described in item 8, characterized by the features described herein. (Item 10) A method for controlling an information processing device, The acquisition process involves obtaining the generated product using the generation AI, An identification step to identify risky objects that pose a risk of infringement among the objects contained in the aforementioned product, A recording step of recording the positional information of the risk object in the product as risk information in the storage unit, A control method characterized by including (Item 11) A program to cause a computer to execute the control method described in item 10.

[0108] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0109] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0110] 301 Information processing device; 302 First acquisition unit; 303 Identification unit; 304 First feature calculation unit; 305 Second acquisition unit; 306 Second feature calculation unit; 307 Risk value calculation unit; 308 Recording unit

Claims

1. A means for obtaining the product generated by the generation AI, An identification means for identifying risky objects among the objects contained in the aforementioned product that pose a risk of infringing rights, A recording means for recording the positional information of the risk object in the product as risk information, An information processing device characterized by having the following features.

2. The identification means identifies objects of a specific category included in the product as the risk objects. The information processing apparatus according to feature 1.

3. The system further includes a calculation means for calculating a risk value that indicates the degree of possibility that the aforementioned risk object is an object subject to infringement of rights, The recording means associates the location information and the risk value with respect to the risk object and records them in the risk information. The information processing apparatus according to feature 1.

4. The calculation means calculates the risk value for the risk object based on the similarity between the risk object and given rights protection data. The information processing apparatus according to claim 3.

5. The system further includes a display control means for displaying the product and the risk information on a display unit. The information processing apparatus according to feature 1.

6. The display control means further displays information on the display unit regarding rights protection data similar to the risk object. The information processing apparatus according to feature 5.

7. Substitute acquisition means for acquiring a substitute that can replace the aforementioned risk object, A receiving means for receiving instructions to replace the risky substance in the product with the substitute, A generating means for generating a substitute product in which the risk object is replaced with the substitute based on the above instructions, It further possesses The information processing apparatus according to claim 5 or 6, characterized by the above.

8. The system further includes a derivation means for deriving a location where there is a relatively high probability of a risk object being generated among the products generated by the generation AI, based on the risk information for the multiple products generated by the generation AI. The information processing apparatus according to feature 1.

9. The generation AI further includes a condition acquisition means for acquiring the generation conditions when it generates each of the plurality of products, The derivation means further derives the generation conditions under which a risk object is likely to be generated in the product generated by the generating AI. The information processing apparatus according to feature 8.

10. A method for controlling an information processing device, The acquisition process involves obtaining the products generated by the generation AI, An identification step to identify risky objects that pose a risk of infringement among the objects contained in the aforementioned product, A recording step of recording the positional information of the risk object in the product as risk information in the storage unit, A control method characterized by including

11. A program for causing a computer to execute the control method described in claim 10.

Images