Information processing system

Abstract

To appropriately calculate a counter value that should be refunded to a creator of each image used for learning of an image generation model.SOLUTION: An image generation model storage section 100 stores one or more image generation models that have been learned based on a plurality of first learning images I1 created by a first creator and a plurality of second learning images I2 created by a second creator. A use request reception section 101 receives a use request for the one or more image generation models, that is transmitted from a client terminal 2. An image generation section 102 executes image generation processing P for generating one or more images based on one or more image generation models, in response to the reception of a use request, and outputs a result image obtained as a result of the image generation processing P. A contribution calculation section 103 calculates a first contribution indicating a contribution of the first creator with regard to the image generation processing P, and a second contribution indicating a contribution of the second creator with regard to the image generation processing P, on the basis of the contents of the image generation processing P.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing system. [Background technology]

[0002] In recent years, image generation models have become known that accept inputs of sentences called prompts or source images and output images that correspond to the content indicated by these prompts or images. In particular, a technology that retrains a general-purpose image generation model trained with general images for each work, such as manga or anime, so that it is more likely to output images that reflect the art style of that work has attracted attention. Summary of the Invention [Problem to be solved by the invention]

[0003] Some websites distribute image generation models that have been trained on images without the permission of the original copyright holders of the works. Website users can obtain the images they want by using these unlicensed image generation models without any restrictions, such as paying a fee.

[0004] Under the circumstances described above, it is impossible to properly calculate the compensation to be returned to the original authors of each image used in training the image generation model. It can be said that the generation of an image by an image generation model involves the creative contribution of the original authors of the works (images) used in training the image generation model. Therefore, the original authors of each image used in training the image generation model should be compensated for the image generation by the image generation model (i.e., the use of the image generation model). However, in a situation where unauthorized image generation models are distributed as described above, it is difficult to grasp information such as the extent to which each image generation model is being used. In other words, under such circumstances, it is difficult to calculate the compensation to be returned to each original author.

[0005] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide an information processing system that can appropriately calculate the compensation to be returned to the original creator of each image used in training an image generation model. [Means for solving the problem]

[0006] An information processing system according to one embodiment of the present invention comprises an image generation model storage means for storing one or more image generation models trained based on a plurality of first training images created by a first author and a plurality of second training images created by a second author; a usage request receiving means for receiving a request to use the one or more image generation models sent from a client terminal; an image generation means for executing an image generation process to generate one or more images based on the one or more image generation models in response to the reception of the usage request, and outputting a resultant image obtained as a result of the image generation process; and a contribution degree calculation means for calculating, based on the content of the image generation process, a first contribution degree indicating the contribution of the first author to the image generation process and a second contribution degree indicating the contribution of the second author to the image generation process. [Effects of the Invention]

[0007] According to the present invention, it is possible to properly calculate the compensation to be returned to the original creator of each image used in training an image generation model. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of a hardware configuration of a copyright management system according to an embodiment of the present invention. [Figure 2] 1 is a functional block diagram showing an example of functions realized by a copyright management system according to an embodiment of the present invention; [Figure 3] 10A and 10B are diagrams illustrating image generation and contribution degree calculation in the case of usage mode A. FIG. [Figure 4] 10A and 10B are diagrams illustrating image generation and contribution degree calculation in the case of a usage mode B. FIG. [Figure 5]10A and 10B are diagrams illustrating image generation and contribution degree calculation in the case of a usage pattern C. FIG. [Figure 6] FIG. 10 is a diagram illustrating a similarity determination. [Figure 7] FIG. 10 is a diagram illustrating a determination of a violation of public order and morals. [Figure 8] FIG. 10 is a flow diagram showing an example of processing executed by the copyright management system in the case of usage form A. [Figure 9] FIG. 10 is a flowchart showing an example of processing executed by the copyright management system in the case of usage mode B. [Figure 10] FIG. 10 is a flowchart showing an example of processing executed by the copyright management system in the case of a usage pattern C. [Figure 11] FIG. 10 is a flow diagram illustrating an example of processing executed by a server according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] [1. Hardware Configuration of the Copyright Management System According to the Present Embodiment] Fig. 1 is a diagram showing an example of the hardware configuration of a copyright management system S according to an embodiment of the present invention. As shown in Fig. 1, the copyright management system S according to this embodiment includes a server (information processing system) 1 and a client terminal 2. The server 1 and the client terminal 2 are connected to each other via a network N such as the Internet.

[0011] The server 1 is a server computer. The server 1 includes a control unit 10, a storage unit 11, and a communication unit 12. Although the present embodiment illustrates a case where the server 1 is configured by one computer, the server 1 may be configured by multiple computers.

[0012] The control unit 10 is, for example, a program-controlled device such as a microprocessor that operates according to a program installed in the server 1. The control unit 10 includes at least one processor.

[0013] The storage unit 11 is, for example, a storage element such as a ROM or a RAM, a hard disk drive (HDD), a solid state drive (SSD) including a flash memory, etc. The storage unit 11 stores programs executed by the control unit 10, etc.

[0014] The communication unit 12 is a communication interface for wired or wireless communication, such as a network interface card, and transmits and receives data to and from other computers or terminals via a computer network such as the Internet.

[0015] The client terminal 2 is a computer of a client (user). The client terminal 2 includes a control unit 20, a storage unit 21, a communication unit 22, an operation unit 23, and a display unit 24. The client terminal 2 is, for example, a smartphone, a personal computer, a tablet, or a wearable terminal. The physical configurations of the control unit 20, the storage unit 21, and the communication unit 22 are similar to those of the control unit 10, the storage unit 11, and the communication unit 12, respectively.

[0016] The operation unit 23 is an input device and includes, for example, a touch panel, a pointing device such as a mouse, a keyboard, etc. The operation unit 23 transmits the operation content to the control unit 20. The display unit 24 is, for example, a display such as a liquid crystal output unit or an organic EL output unit.

[0017] [2. Functions realized by the copyright management system according to this embodiment] Fig. 2 is a functional block diagram showing an example of functions realized by a copyright management system S according to one embodiment of the present invention. Each function shown in Fig. 2 is realized by control units 10 and 20 executing programs stored in storage units 11 and 21. The programs may also be stored in a computer-readable information storage medium.

[0018] [2-1. Server] 2, the server 1 functionally includes a model storage unit 100, a use request receiving unit 101, an image generation unit 102, a contribution degree calculation unit 103, a similarity determination unit 104, an image storage unit 105, a public order / morals violation determination unit 106, and an image display restriction unit 107. The use request receiving unit 101, the image generation unit 102, the contribution degree calculation unit 103, the similarity determination unit 104, the public order / morals violation determination unit 106, and the image display restriction unit 107 are mainly implemented by the control unit 10. The model storage unit 100 and the image storage unit 105 are mainly implemented by the storage unit 11.

[0019] The model storage unit 100 stores a first image generation model M1, a second image generation model M2, a third image generation model M3, and a contribution estimation model M4.

[0020] In the following description, the first image generation model M1, the second image generation model M2, and the third image generation model M3 may be collectively referred to as "image generation models." An image generation model is a machine learning model trained to accept input of at least one of text and an image and output an image corresponding to the content of at least one of the text and the image. In this embodiment, as an example, the well-known model Stable Diffusion is used as the image generation model. However, similarly well-known models such as DALL-E may also be used as the image generation model. In addition to the above-mentioned examples, the image generation model may be any model, such as a generative adversarial network (GAN), a variational autoencoder (VAE), or a diffusion model.

[0021] The first image generation model M1 is an image generation model trained based on a plurality of first training images. Specifically, the first image generation model M1 is a pre-trained image generation model that has been retrained based on a plurality of first training images using a known additional training method such as LoRA (Low-Rank Adaptation). The additional training method is not limited to LoRA, and other methods such as Dreambooth and Textual Inversion may also be used. Furthermore, the retraining is not limited to additional training, and may be performed using other methods such as fine tuning and transfer learning.

[0022] The multiple first training images are images created by a first author. In the present embodiment, as an example, each first training image is a drawing of a cartoon or animation character, but the content of each first training image is not limited to this example. Because the first image generation model M1 is trained based on the multiple first training images, the style of the first author is reflected in the images generated by the first image generation model M1.

[0023] The second image generation model M2 is an image generation model trained based on a plurality of second training images. The plurality of second training images are images created by a second author, who is different from the first author. The details of the second image generation model M2 are the same as those of the first image generation model M1, and therefore will not be described here.

[0024] The third image generation model M3 is an image generation model trained based on a plurality of first training images and a plurality of second training images. That is, unlike the first image generation model M1 and the second image generation model M2, the third image generation model M3 is an image generation model trained based on two works (a work by a first author and a work by a second author). Therefore, the images generated by the third image generation model M3 reflect the styles of the first author and the second author.

[0025] The contribution estimation model M4 is a machine learning model that learns the relationship between an input image and label data indicating the probability that the input image was created by a first author (first probability) and the probability that the input image was created by a second author (second probability). In this embodiment, the contribution estimation model M4 outputs both the first probability and the second probability, but the contribution estimation model M4 may output only either the first probability or the second probability. In this embodiment, a convolutional neural network (CNN) is used as the contribution estimation model M4, but the contribution estimation model M4 may be any model such as a neural network, a support vector machine, or a decision tree.

[0026] The usage request receiving unit 101 receives a usage request for an image generation model sent from the client terminal 2. The usage request for an image generation model includes, for example, information indicating the usage form of the image generation model (the type of image generation model to be used and the number of times each image generation model is used, etc.) and input data (ID1 and ID2 in FIG. 3, and IDs in FIGS. 4 and 5).

[0027] The following describes the processing of the image generation unit 102 and the contribution calculation unit 103, assuming the three usage patterns shown in Figures 3 to 5. That is, the following description assumes three usage patterns: (1) Usage pattern A (Figure 3) in which the first image generation model M1 and the second image generation model M2 are each used multiple times so as to be independent of each other, (2) Usage pattern B (Figure 4) in which the output of the first image generation model M1 is input to the second image generation model M2, and the output of the second image generation model M2 is input to the first image generation model M1, and (3) Usage pattern C (Figure 5) in which the third image generation model M3 is used once.

[0028] [(1) Usage form A] FIG. 3 is a diagram illustrating image generation and contribution calculation in the case of usage mode A. In FIG.

[0029] 3, in response to the reception of a usage request, the image generation unit 102 executes a first image generation process P1 for generating a first image I1 based on a first image generation model M1 and a second image generation process P2 for generating a second image I2 based on a second image generation model M2, multiple times each. The first image generation process P1 and the second image generation process P2 are included in the image generation process P executed by the image generation unit 102. The image generation unit 102 outputs the first image I1 and the second image I2 as resultant images (hereinafter simply referred to as "resultant images") obtained as a result of the image generation process P.

[0030] In the first image generation process P1, first input data ID1 is input to the first image generation model M1, and a first image I1 showing the content of the first input data ID1 is output from the first image generation model M1. Similarly, in the second image generation process P2, second input data ID2 is input to the second image generation model M2, and a second image I2 showing the content of the second input data ID2 is output from the second image generation model M2. In this embodiment, both the first input data ID1 and the second input data ID2 include text (prompts). Note that the first input data ID1 or the second input data ID2 may include an image, in which case the first input data ID1 or the second input data ID2 does not have to include text.

[0031] The contribution calculation unit 103 calculates a first contribution rate and a second contribution rate based on the number of times the first image generation process P1 was performed and the number of times the second image generation process P2 was performed. Specifically, the contribution calculation unit 103 calculates the first contribution rate based on the ratio of the number of times the first image generation process P1 was performed to the sum of the number of times the first image generation process P1 and the number of times the second image generation process P2 was performed. The contribution calculation unit 103 also calculates the second contribution rate by subtracting the first contribution rate from 1. The contribution calculation unit 103 may also calculate the second contribution rate based on the ratio of the number of times the second image generation process P2 was performed to the sum of the number of times the first image generation process P1 and the number of times the second image generation process P2 was performed. In the example shown in FIG. 3, the first image generation process P1 was performed 20 times and the second image generation process P2 was performed 30 times, so the first contribution rate is calculated to be 40% and the second contribution rate is calculated to be 60%.

[0032] [(2) Usage form B] FIG. 4 is a diagram illustrating image generation and contribution calculation in the case of usage mode B. In FIG.

[0033] 4, in response to a usage request, the image generation unit 102 executes a first mixed image generation process P1' to generate a first mixed image I1' based on the first image generation model M1 and a second mixed image generation process P2' to generate a second mixed image I2' based on the second image generation model M2 multiple times. The first mixed image generation process P1' and the second mixed image generation process P2' are included in the image generation process P executed by the image generation unit 102.

[0034] In the first mixed image generation process P1' (included in the first image generation process P1), the second image I2 is input to the first image generation model M1, and the first mixed image I1' (which is the first image I1) based on the second image I2 is output from the first image generation model M1. The first mixed image I1' reflects the style of the first author and the style of the second author. Similarly, in the second mixed image generation process P2' (included in the second image generation process P2), the first image I1 is input to the second image generation model M2, and the second mixed image I2' (which is the second image I2) based on the first image I1 is output from the second image generation model M2. The second mixed image I2' also reflects the style of the first author and the style of the second author.

[0035] The image generation unit 102 executes the first mixed image generation process P1' and the second mixed image generation process P2' multiple times, and outputs one final mixed image MI as a resultant image. The final mixed image MI is either the first mixed image I1' or the second mixed image I2'.

[0036] The contribution degree calculation unit 103 calculates the first contribution degree and the second contribution degree based on the number of times the first mixed image generation process P1' is performed and the number of times the second mixed image generation process P2' is performed. In this embodiment, the image generation process P first includes the first image generation process P1 as shown in FIG. 4, so the contribution degree calculation unit 103 calculates the first contribution degree and the second contribution degree based on the number of times the first image generation process P1 is performed, the number of times the first mixed image generation process P1' is performed, and the number of times the second mixed image generation process P2' is performed. Note that the contribution degree calculation unit 103 may calculate the first contribution degree and the second contribution degree based on the number of times the first mixed image generation process P1' is performed and the number of times the second mixed image generation process P2' is performed, without using the number of times the first image generation process P1 is performed.

[0037] Specifically, the contribution calculation unit 103 calculates the first contribution based on the ratio of the sum of the number of times the first image generation process P1 and the number of times the first mixed image generation process P1' are performed to the sum of the number of times the first image generation process P1, the number of times the first mixed image generation process P1', and the number of times the second mixed image generation process P2' are performed. The contribution calculation unit 103 also calculates the second contribution by subtracting the first contribution from 1. The contribution calculation unit 103 may also calculate the second contribution based on the ratio of the sum of the number of times the second image generation process P2 and the number of the second mixed image generation process P2' to the sum of the number of times the first image generation process P1, the number of the first mixed image generation process P1', and the number of the second mixed image generation process P2'. In the example shown in FIG. 4, the first image generation process P1 is performed once, the first mixed image generation process P1' is performed six times, and the second mixed image generation process P2' is performed three times. Therefore, in the example shown in FIG. 4, the first contribution rate is calculated to be 70%, and the second contribution rate is calculated to be 30%.

[0038] [(3) Usage form C] FIG. 5 is a diagram illustrating image generation and contribution calculation in the case of usage mode C. In FIG.

[0039] 5, the image generation unit 102 executes a third image generation process P3 that generates a third image I3 based on a third image generation model M3. The third image generation process P3 is included in the image generation process P executed by the image generation unit 102. In the third image generation process P3, input data ID is input to the third image generation model M3, and a third image I3 representing the contents of the input data ID is output from the third image generation model M3. As described above, the third image I3 reflects at least one of the style of the first author and the style of the second author.

[0040] The contribution calculation unit 103 calculates the first contribution and the second contribution based on the output of the contribution estimation model M4 when the third image I3 is input to the contribution estimation model M4. The contribution estimation model M4 receives the third image I3 as input and outputs the probability that the third image I3 was created by the first author (first probability) and the probability that the third image I3 was created by the second author (second probability). The contribution calculation unit 103 acquires the first probability and the second probability as the first contribution and the second contribution, respectively. Note that if the contribution estimation model M4 outputs only the first probability or the second probability, the contribution calculation unit 103 may acquire the value obtained by subtracting the first probability or the second probability from 1 as the second probability or the first probability (i.e., the second contribution or the first contribution).

[0041] [Processing related to display restrictions] Next, with reference to Figures 6 and 7, functions (similarity determination unit 104, public order and morals violation determination unit 106, and image display restriction unit 107) for restricting the display of an image generated by image generation process P when the image satisfies predetermined conditions will be described. Note that the following description will be given assuming the above-mentioned usage pattern A, but the same description applies to usage patterns B and C. Figure 6 is a diagram for explaining the similarity determination. Figure 7 is a diagram for explaining the public order and morals violation determination.

[0042] The similarity determination unit 104 calculates a first similarity, which is the similarity between the first image I1 and an image created by the first author, and determines whether the value of the first similarity is equal to or greater than a predetermined threshold. Specifically, as shown in FIG. 6 , the similarity determination unit 104 acquires a comparison image CI, which is an image created by the first author, from the image storage unit 105 and calculates the similarity between the first image I1 and the comparison image CI as the first similarity. In this embodiment, the similarity determination unit 104 calculates the cosine similarity between the first image I1 or its feature amount and the comparison image CI or its feature amount as the first similarity. However, a known similarity other than cosine similarity, such as Euclidean distance, may also be calculated as the first similarity. The similarity determination unit 104 then determines whether the calculated first similarity is equal to or greater than a predetermined threshold (90% in the example shown in FIG. 6 ). In this embodiment, one of the multiple first training images used in training the first image generation model M1 is used as the comparison image CI, but the comparison image CI may be any image created by the first author. Note that the similarity determination unit 104 may obtain the comparison image CI from an external device or information storage medium.

[0043] Although not shown, the similarity determination unit 104 further calculates a second similarity, which is the similarity between the second image I2 and the image created by the second author, and determines whether the value of the second similarity is equal to or greater than a predetermined threshold. Note that the similarity determination unit 104 does not necessarily have to process the second image I2.

[0044] The public order and morals violation determination unit 106 determines whether the content of the first image I1 is contrary to public order and morals based on whether the proportion of the skin-colored area (exposed pixels, which are pixels having a color that represents a person's skin) in the first image I1 is equal to or greater than a predetermined proportion (70% in the example shown in FIG. 7). Although not shown, the public order and morals violation determination unit 106 further determines whether the content of the second image I2 is contrary to public order and morals based on whether the proportion of the skin-colored area in the second image I2 is equal to or greater than a predetermined proportion. Note that the public order and morals violation determination unit 106 does not have to process the second image I2.

[0045] When the first image I1 satisfies a predetermined condition, the image display restriction unit 107 restricts the display of the first image I1 on the client terminal 2. Furthermore, when the second image I2 satisfies a predetermined condition, the image display restriction unit 107 restricts the display of the second image I2 on the client terminal 2. Note that the image display restriction unit 107 does not have to perform processing on the second image I2.

[0046] In this embodiment, the image display restriction unit 107 restricts the display of the first image I1 or the second image I2 on the client terminal 2 by deleting the first image I1 or the second image I2. Note that the manner of display restriction is not limited to this example. For example, the image display restriction unit 107 may restrict the display of the first image I1 or the second image I2 on the client terminal 2 by blacking out the first image I1 or the second image I2. Alternatively, the image display restriction unit 107 may restrict the display of the first image I1 or the second image I2 on the client terminal 2 by not granting the client terminal 2 access authority to the first image I1 or the second image I2.

[0047] Specifically, when the similarity determination unit 104 determines that the value of the first similarity is equal to or greater than a threshold, the image display restriction unit 107 restricts the display of the first image I1 on the client terminal 2 (see FIG. 6). Also, when the public order and morals violation determination unit 106 determines that the first image I1 is contrary to public order and morals, the image display restriction unit 107 restricts the display of the first image I1 on the client terminal 2.

[0048] Similarly, when the similarity determination unit 104 determines that the value of the second similarity is equal to or greater than the threshold, the image display restriction unit 107 restricts the display of the second image I2 on the client terminal 2 (see FIG. 7). Furthermore, when the public order and morals violation determination unit 106 determines that the second image I2 is contrary to public order and morals, the image display restriction unit 107 restricts the display of the second image I2 on the client terminal 2.

[0049] [2-2. Client terminal] The client terminal 2 functionally includes a usage request transmission unit 200. The usage request transmission unit 200 transmits a usage request for an image generation model to the server 1.

[0050] 3. Processing Executed by the Copyright Management System According to the Present Embodiment 8 to 10, the processing executed by the copyright management system S will be explained separately for each of the usage patterns A to C. The processing shown in Figs. 8 to 10 is executed by the control units 10 and 20 operating in accordance with the programs stored in the storage units 11 and 21.

[0051] [(1) Usage form A] FIG. 8 is a flow diagram showing an example of processing executed by the copyright management system S in the case of usage mode A. As shown in FIG. 8, the client terminal 2 transmits a usage request for the first image generation model M1 and the second image generation model M2 to the server 1 (S800), and ends the processing. The server 1, which has accepted the usage request from the client terminal 2 (S801), executes the first image generation processing P1 (S802). The server 1 determines whether to end the first image generation processing P1 in S803, and if it determines not to end the first image generation processing P1 (S803; N), it executes the processing of S802 again. If it determines to end the first image generation processing P1 (S803; Y), it executes the second image generation processing P2 (S804). The server 1 determines whether to end the second image generation processing P2 in S805, and if it determines not to end the second image generation processing P2 (S805; N), it executes the processing of S804 again. If the server 1 determines to terminate the second image generation process P2 (S805; Y), it calculates the first contribution rate and the second contribution rate based on the number of times the first image generation process P1 is performed and the number of times the second image generation process P2 is performed (S806), and terminates the processing.

[0052] 8, the server 1 executes the first image generation process P1 and the second image generation process P2 in that order, but the server 1 may execute the second image generation process P2 and the first image generation process P1 in that order. Also, the server 1 may execute the first image generation process P1 and the second image generation process P2 alternately.

[0053] [(2) Usage form B] 9 is a flow diagram showing an example of processing executed by the copyright management system S in the case of usage mode B. As shown in FIG. 9, the client terminal 2 transmits a usage request for the first image generation model M1 and the second image generation model M2 to the server 1 (S900), and ends the processing. The server 1, which has accepted the usage request from the client terminal 2 (S901), executes the first image generation processing P1 (S902). Next, the server 1 executes the second mixed image generation processing P2' (S903), and executes the first mixed image generation processing P1' (S904). In S905, the server 1 determines whether to end the image generation processing P, and if it determines not to end the image generation processing P (S905; N), it executes the processing of S903 and S904 again. If the server 1 determines to terminate the image generation process P (S905; Y), it calculates the first contribution rate and the second contribution rate based on the number of times the first image generation process P1 was performed, the number of times the first mixed image generation process P1' was performed, and the number of times the second mixed image generation process P2' was performed (S906), and terminates the processing.

[0054] 9, the server 1 starts the image generation process P from the first image generation process P1, but may start the image generation process P from the second image generation process P2. In that case, the server 1 may execute the first mixed image generation process P1' and the second mixed image generation process P2' in that order. Also, in the example shown in FIG. 9, the second mixed image generation process P2' and the first mixed image generation process P1' are executed alternately, but at least one of the second mixed image generation process P2' and the first mixed image generation process P1' may be executed consecutively.

[0055] [(3) Usage form C] 10 is a flow diagram showing an example of processing executed by the copyright management system S in the case of usage pattern C. As shown in FIG. 10, the client terminal 2 transmits a usage request for the third image generation model M3 to the server 1 (S1000), and ends the processing. The server 1, having received the usage request from the client terminal 2 (S1001), executes the third image generation process P3 (S1002). The server 1 inputs the third image I3 generated by the third image generation process P3 to the contribution estimation model M4 (S1004), obtains the output (first probability and second probability) of the contribution estimation model M4 as the first contribution and the second contribution, respectively (S1005), and ends the processing.

[0056] [Processing related to display restrictions] Finally, the processing related to the display restriction (processing of the similarity determination unit 104, the public order and morals violation determination unit 106, and the image display restriction unit 107) will be described. Note that the following description will be given assuming the use mode C, but the same description also applies to the use mode A and the use mode B.

[0057] 11 is a flow diagram showing an example of processing executed by the server 1 according to an embodiment of the present invention. As shown in FIG. 11, the server 1 first calculates a first similarity (S1100) and determines whether the first similarity is equal to or greater than a predetermined threshold (S1101). If the server 1 determines that the first similarity is equal to or greater than the predetermined threshold (S1101; Y), it restricts the display of the third image I3 on ​​the client terminal 2 (S1106).

[0058] If the server 1 determines that the first similarity is less than a predetermined threshold (S1101; N), it calculates a second similarity (S1102) and determines whether the second similarity is equal to or greater than a predetermined threshold (S1103).If the server 1 determines that the second similarity is equal to or greater than a predetermined threshold (S1103; Y), it restricts the display of the third image I3 on ​​the client terminal 2 (S1106).

[0059] If the server 1 determines that the second similarity is less than a predetermined threshold (S1103; N), it determines whether the content of the third image I3 is offensive to public order and morals based on whether the proportion of the skin-colored area in the third image I3 is equal to or greater than a predetermined proportion (S1104). If the server 1 determines that the content of the third image I3 is offensive to public order and morals (S1104; Y), it restricts the display of the third image I3 on ​​the client terminal 2 (S1106). If the server 1 determines that the content of the third image I3 is not offensive to public order and morals (S1104; N), it displays the third image I3 on ​​the client terminal 2 (S1105) and ends the process.

[0060] 11, the server 1 executes the process for the first similarity and the process for the second similarity in that order, but may execute the process for the second similarity and the process for the first similarity in that order. Also, in the example shown in FIG. 11, the process executes the similarity determination and the violation of public order and morals determination in that order, but may execute the similarity determination and the violation of public order and morals determination in that order.

[0061] [4. Summary] The server 1 (information processing system) of this embodiment described above calculates a first contribution degree indicating the contribution of a first author to the image generation process P and a second contribution degree indicating the contribution of a second author to the image generation process P. That is, the server 1 can clarify the contribution of each author to the image generation process P using an image generation model, and therefore can properly calculate the compensation to be returned to the author of each image used in training the image generation model.

[0062] Furthermore, when an image generation model is used, a phenomenon called backflow may occur, in which the image used to train the image generation model is output almost as is. In this regard, according to the server 1 of this embodiment, when it is determined that the value of the first similarity (or the second similarity), which is the similarity between the first image I1 (or the second image I2) and an image created by the first author (or the second author), is equal to or greater than a threshold, the display of the first image I1 (or the second image I2) on the client terminal 2 is restricted, thereby preventing the backflow.

[0063] Furthermore, one possible misuse of the image generation model is to use the image generation model to reflect the style of a specific author in an image that violates public order and morals (for example, a pornographic image). In this regard, according to the server 1 of this embodiment, whether the content of the first image I1 (or the second image I2) violates public order and morals is determined based on whether the proportion of the skin-colored area in the first image I1 (or the second image I2) is equal to or greater than a predetermined proportion, and if it is determined that the first image I1 (or the second image I2) is violates public order and morals, the display of the first image I1 (or the second image I2) on the client terminal 2 is restricted, thereby preventing the above-mentioned misuse.

[0064] [5. Modifications] It should be noted that the present invention is not limited to the above-described embodiment. Furthermore, the specific character strings and numerical values ​​described above and the specific character strings and numerical values ​​in the drawings are merely examples, and the present invention is not limited to these character strings and numerical values.

[0065] For example, in the present embodiment, the first image generation process P1 and the second image generation process P2 are each executed multiple times in usage mode A, but at least one of the first image generation process P1 and the second image generation process P2 may be executed only once. Also, in the present embodiment, in usage mode B, one final mixed image MI obtained by executing the first mixed image generation process P1' and the second mixed image generation process P2' multiple times is output as a resultant image, but at least one of the first mixed image generation process P1' and the second mixed image generation process P2' may be executed only once.

[0066] In this embodiment, the public order and morals violation determination unit 106 determines whether the content of the first image I1 is offensive to public order and morals based on whether the proportion of the skin-colored area in the first image I1 is equal to or greater than a predetermined proportion. However, the public order and morals violation determination unit 106 may also determine whether the content of the first image I1 is offensive to public order and morals based on the output of a public order and morals violation determination model that has been trained to determine the relationship between an input image and label data indicating whether the content of the input image is offensive to public order and morals. Such a public order and morals violation determination model may be any model, such as a convolutional neural network (CNN), a neural network, a support vector machine, or a decision tree. Specifically, the label data used to train the public order and morals violation determination model may indicate whether the proportion of the skin-colored area in the input image is equal to or greater than a predetermined proportion.

[0067] In the present embodiment, the public order and morals violation determination unit 106 determines whether the content of the first image I1 is contrary to public order and morals based on whether the proportion of the skin-colored area in the first image I1 is equal to or greater than a predetermined proportion, i.e., whether the content of the first image I1 is obscene, but the public order and morals violation determination unit 106 may also determine whether the content of the first image I1 is contrary to public order and morals based on whether the content of the first image I1 is discriminatory or insulting. Alternatively, the public order and morals violation determination unit 106 may determine whether the content of the first image I1 is contrary to public order and morals based on whether the content of the first image I1 is cruel. [Explanation of symbols]

[0068] S copyright management system, N network, 1 server, 2 client terminal, 10, 20 control unit, 11, 21 memory unit, 12, 22 communication unit, 23 operation unit, 24 display unit, 100 model memory unit, 101 usage request reception unit, 102 image generation unit, 103 contribution calculation unit, 104 similarity determination unit, 105 image memory unit, 106 public order and morals violation determination unit, 107 image display restriction unit, 200 usage request transmission unit, P image generation process, P1 first mixed image generation process, P2 second image generation process, P1' first mixed image generation process, P2' second mixed image generation process, P3 third image generation process, M1 first image generation model, M2 second image generation model, M3 third image generation model, M4 contribution estimation model, I1 first image, I1' first mixed image, I2 second image, I2' second mixed image, I3 Third image, MI final blended image, CI comparison image, ID input data.

Claims

1. an image generation model storage means for storing one or more image generation models trained based on a plurality of first training images created by a first author and a plurality of second training images created by a second author; a usage request receiving means for receiving a usage request for the one or more image generation models transmitted from a client terminal; an image generation means for executing an image generation process to generate one or more images based on the one or more image generation models in response to the reception of the usage request, and outputting a resultant image obtained as a result of the image generation process; a contribution degree calculation means for calculating a first contribution degree indicating the contribution of the first author to the image generation process and a second contribution degree indicating the contribution of the second author to the image generation process based on the content of the image generation process; An information processing system having the above.

2. the one or more image generation models include a first image generation model trained based on the plurality of first training images and a second image generation model trained based on the plurality of second training images; the one or more images include a first image generated based on the first image generation model and a second image generated based on the second image generation model; the image generation process includes a first image generation process that generates the first image based on the first image generation model, and a second image generation process that generates the second image based on the second image generation model, the image generating means executes each of the first image generating process and the second image generating process at least once and outputs the resulting image; the contribution degree calculation means calculates the first contribution degree and the second contribution degree based on a value corresponding to the number of times the first image generation process is performed and a value corresponding to the number of times the second image generation process is performed. The information processing system according to claim 1 .

3. the first image generation process includes a first mixed image generation process of inputting the second image into the first image generation model and generating a first mixed image that is the first image; the second image generation process includes a second mixed image generation process of inputting the first image into the second image generation model and generating a second mixed image that is the second image, the image generating means outputs, as the resultant image, an image obtained by executing at least one of the first mixed image generating process and the second mixed image generating process one or more times; the contribution degree calculation means calculates the first contribution degree and the second contribution degree based on a value corresponding to the number of times the first mixed image generation process has been performed and a value corresponding to the number of times the second mixed image generation process has been performed. The information processing system according to claim 2 .

4. the contribution degree calculation means calculates the first contribution degree and the second contribution degree based on an output of a contribution degree estimation model that has learned a relationship between an input image and label data indicating the probability that the input image was created by the first author and the probability that the input image was created by the second author, when the resultant image is input to the contribution degree estimation model. The information processing system according to claim 1 .

Images