Information processing system and information processing method

Abstract

The present invention provides an information processing system that carries out a process related to subjective evaluation of data which has been generated using a machine learning model.　The information processing system comprises: a system feedback acquisition unit that acquires generated data and evaluation information based on an evaluation model with respect to the generated data; a system feedback presentation unit that presents the generated data and the evaluation information; a user feedback acquisition unit that acquires user evaluation with respect to the generated data or to the evaluation information; and an output unit that outputs the user evaluation which has been acquired by the user feedback acquisition unit.

Description

[Technical Field]

[0001] The technology disclosed in this specification (hereinafter referred to as "the present disclosure") relates to an information processing system and an information processing method that perform processing related to the evaluation of data generated using a machine learning model. [Background technology]

[0002] Machine learning is a technology that allows computers to learn from large amounts of data and automatically construct models and algorithms that perform operations such as data classification and prediction. For example, machine learning can be used to obtain discriminative models that analyze data such as images, audio, and text, as well as generative models that generate new data such as images, audio, and text. Models are composed of neural networks, for example. Recently, there has been significant progress in technology related to deep neural networks (DNNs), which are neural networks that are developed through deep learning.

[0003] For example, GAN (Generative Adversarial Network) is known as a data generation technology that uses a machine learning model (see, for example, Non-Patent Document 1). GAN consists of a generator that generates data and a discriminator that distinguishes whether the data is true or false, and through mutual learning between the generator and discriminator, the generator can generate data that cannot be distinguished as true or false by the discriminator.

[0004] On the other hand, there are cases where data that reflects the user's subjectivity and preferences is required. Although the classifier in a GAN can determine the truth of data, it cannot evaluate the user's subjectivity. For this reason, GAN cannot generate data that reflects the user's subjectivity and preferences.

[0005] Furthermore, although LPIPS (Learned Perceptual Image Patch Similarity) is known as an index for evaluating the image quality of a generated image, it is not an index that indicates whether the generated image reflects the user's subjectivity or preferences. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] I. Goodfellow et al., "Generative adversarial nets", Advances in neural information processing systems, pp. 2672-2680, 2014 Summary of the Invention [Problem to be solved by the invention]

[0007] An object of the present disclosure is to provide an information processing system and an information processing method that perform processing related to subjective evaluation of data generated using a machine learning model. [Means for solving the problem]

[0008] The present disclosure has been made in consideration of the above problems, and a first aspect thereof is: a system feedback acquisition unit that acquires generated data and evaluation information based on an evaluation model for the generated data; a system feedback presentation unit that presents the generated data and the evaluation information; a user feedback acquisition unit that acquires user evaluations on the generated data or the evaluation information; an output unit that outputs the user evaluation acquired by the user feedback acquisition unit; It is an information processing system including:

[0009] However, the term "system" used here refers to a logical collection of multiple devices (or functional modules that realize specific functions), regardless of whether each device or functional module is contained within a single housing. In other words, both a single device consisting of multiple parts or functional modules and a collection of multiple devices are considered "systems."

[0010] The information processing system according to a first aspect includes an interface presenting unit that presents an interface for inputting the user evaluation.

[0011] The system feedback acquisition unit acquires the generated data and the evaluation information from one or more devices that generate data using a generative model and evaluate the generated data using the evaluation model, and the output unit outputs the user evaluation to a device that updates the generative model that generates data and the evaluation model that evaluates the generated data.

[0012] Furthermore, a second aspect of the present disclosure is a system feedback acquisition step of acquiring evaluation information based on generated data and an evaluation model for the generated data; a system feedback presentation step of presenting the generated data and the evaluation information; a user feedback acquisition step of acquiring a user evaluation on the generated data or the evaluation information; an output step of outputting the user evaluation acquired in the user feedback acquisition step; The information processing method has the following features.

[0013] Furthermore, a third aspect of the present disclosure is an evaluation unit that generates evaluation information for the generated data using an evaluation model; an acquisition unit that acquires user evaluation information regarding the evaluation information; a rating model update unit that updates the rating model based on the user rating information; It is an information processing system including:

[0014] An information processing device according to a third aspect includes a generation unit that generates data using a generative model, and a generative model update unit that updates the generative model based on the user evaluation information, and the evaluation unit generates evaluation information for the data generated by the generation unit.

[0015] The evaluation unit outputs the evaluation information to an information terminal, and the acquisition unit acquires, from the information terminal, user evaluation information input through an interface presented on the information terminal.

[0016] Furthermore, a fourth aspect of the present disclosure is an evaluation step of generating evaluation information for the generated data using an evaluation model; an acquisition step of acquiring user evaluation information for the evaluation information; a rating model updating step of updating the rating model based on the user rating information; The information processing method has the following features. [Effects of the Invention]

[0017] According to the present disclosure, it is possible to provide an information processing system and an information processing method for acquiring a user's evaluation of a simulated subjective evaluation using a machine learning model, and an information processing system and an information processing method for generating, using a machine learning model, a subjective evaluation of subjective data generated using a machine learning model.

[0018] It should be noted that the effects described in this specification are merely examples, and the effects brought about by the present disclosure are not limited to these. Furthermore, the present disclosure may also bring about additional effects in addition to the effects described above.

[0019] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description based on the embodiments and accompanying drawings. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a data generation system 100. As shown in FIG. [Figure 2] FIG. 2 is a flowchart showing the data generation, evaluation, and learning procedures in the data generation system 100. [Figure 3] FIG. 3 is a diagram showing the generator 101 in the data generation phase. [Figure 4] FIG. 4 is a diagram showing the generator 101 and the evaluator 102 in the evaluation phase. [Figure 5] FIG. 5 is a diagram showing the generative model and the evaluation model in the model update phase. [Figure 6] FIG. 6 is a diagram showing a configuration example (first example) of an interface used for system feedback and user feedback. [Figure 7] FIG. 7 is a diagram showing a configuration example (second example) of an interface used for system feedback and user feedback. [Figure 8] FIG. 8 is a diagram showing a configuration example (third example) of an interface used for system feedback and user feedback. [Figure 9] FIG. 9 is a diagram showing a configuration example (fourth example) of an interface used for system feedback and user feedback. [Figure 10] FIG. 10 is a diagram showing a configuration example (fifth example) of an interface used for system feedback and user feedback. [Figure 11] FIG. 11 is a diagram showing a configuration example (sixth example) of an interface used for system feedback and user feedback. [Figure 12] FIG. 12 is a diagram showing a configuration example (seventh example) of an interface used for system feedback and user feedback. [Figure 13] FIG. 13 is a diagram showing the paths along which data is transferred between the phases. [Figure 14] FIG. 14 is a diagram showing a specific data flow between each phase. [Figure 15] FIG. 15 is a diagram showing an example of the configuration of an information processing system 1500. DETAILED DESCRIPTION OF THE INVENTION

[0021] The present disclosure will be described below in the following order with reference to the drawings.

[0022] A. Overview B. Data Generation, Evaluation, and Training Procedures C. Interface configuration example D. Details of Data Generation, Evaluation, and Training Procedures E. Information Processing Systems

[0023] A. Overview The present disclosure relates to a technology for generating data that reflects a user's subjectivity and preferences using a machine learning model. The data may be various types such as images, audio (including music), and text. For convenience, the following describes an embodiment in which the data is limited to images.

[0024] GAN is a technology that generates data that is difficult to distinguish between true and false through mutual learning between a generator that generates data and a classifier that distinguishes the true or false of the generated data. In contrast, the present disclosure is a technology that generates data that reflects the subjectivity and preferences of a user by using a generator that generates data and an evaluator that subjectively evaluates the generated data of a specific user.

[0025] FIG. 1 shows a schematic configuration example of a data generation system 100 that generates data that reflects the subjectivity and preferences of a user.

[0026] The data generation system 100 is composed of a generator 101 that generates images, and an evaluator 102 that evaluates the images generated by the generator 101 based on the user's subjectivity and preferences. The generator 101 and the evaluator 102 are each machine learning models made up of neural networks, and set parameters (coefficients of each neuron) acquired during learning to generate images and subjectively evaluate the generated images, respectively.

[0027] 1 shows a generator 101 in the data generation phase and an evaluator 102 in the data evaluation phase. The generator 101 is made up of a neural network with coefficients acquired during learning set, and receives random numbers as input to output a newly generated image. The evaluator 102 is also made up of a neural network with coefficients acquired during learning set, and receives the image generated by the generator 101 as input to output a simulated subjective evaluation that estimates a specific user's subjective evaluation of the generated image.

[0028] The generator 101 updates the neural network coefficients so that the simulated subjective evaluation of the data generated by the generator 101 by the evaluator 102 is improved. If the evaluator 102 has a subjective evaluation model created in a form that reflects the user's original subjectivity and preferences, this update process allows the generator 101 to learn so that it can generate data that reflects the user's subjectivity and preferences.

[0029] Here, a neural network is used to model the subjective evaluation in the evaluator 102, but there is a problem that the evaluation index is greatly affected by the purpose and individual differences. For this reason, it is desirable to be able to create different subjective evaluation models for each purpose in a short time and at low cost, and it would be even better if a method for creating subjective evaluation models could be provided in a form that can reflect the subjectivity and preferences of each user.

[0030] For example, conventional data generation technologies, such as the classifiers used in GANs and LPIPS as an image quality evaluation index, do not have interfaces or evaluation indexes to reflect user intentions in data generation.

[0031] Therefore, in the present disclosure, data generated by the generator 101 is separated into various elements, mathematically modeled as evaluation indices, and an interface is provided that allows a user to adjust each evaluation indices. Furthermore, in the present disclosure, new generated data related to the data generated by the generator 101 is generated, and these generated data are relatively subjectively evaluated and the results are modeled. The data (e.g., images) generated by the generator 101 have various characteristics. According to the present disclosure, detailed user feedback can be provided, in which the user appropriately adjusts the gain of each feature of the generated data via the interface, or simple user feedback can be provided, in which the user assigns a good or bad rating to related generated data. Therefore, according to the present disclosure, a model evaluation index that largely reflects the user's intentions can be realized, and automatic generation of data that is considered good by the user (i.e., data that reflects the user's subjectivity and preferences) can be easily realized.

[0032] B. Learning Procedures FIG. 2 shows in the form of a flowchart the data generation, evaluation, and learning procedures in the data generation system 100 to which the present disclosure is applied.

[0033] In the data generation phase, the generator 101 receives a random number as an input and outputs a newly generated image (step S201).

[0034] Next, in the generated data evaluation phase, the evaluator 102 inputs the image generated by the generator 101 and outputs a simulated subjective evaluation that simulates a specific user's subjective evaluation of the generated image (step S202).

[0035] Next, in the feedback phase from the data generation system 100 to the user (System Feedback), the data newly generated by the generator 101 in the data generation phase and the simulated subjective evaluation output by the evaluator 102 for the generated data in the data evaluation phase are presented to the user (step S203).

[0036] Next, in the user feedback phase, the user provides feedback to the data generation system 100 regarding the generated data by the generator 101 and the simulated subjective evaluation by the evaluator 102 (step S204).

[0037] In the preceding data evaluation phase, the evaluator 102 outputs a simulated subjective evaluation of the generated data using evaluation indices that separate the characteristics of the generated data into multiple elements. In the subsequent system feedback phase, the generated data and the simulated subjective evaluation for each evaluation index are presented to the user. Then, in this user feedback phase, the user provides feedback of their own subjective evaluation for each evaluation index to the data generation system 100. In steps S203 and S204, an interface that presents the generated data and the simulated subjective evaluation to the user, and an interface for inputting the user's original subjective evaluation of the generated data are prepared. The interface for inputting the user's own subjective evaluation may be an interface that allows the user to adjust the evaluation index for each element of the simulated subjective evaluation. Details of these interfaces will be provided later.

[0038] Then, in the model update phase of the generator 101 and the evaluator 102 (Generator & Evaluator update), the coefficients of each neural network constituting the generator 101 and the evaluator 102 are updated based on feedback from the user in the preceding user feedback phase (step S205).

[0039] Figure 3 shows the generator 101 in the data generation phase. In the data generation phase, the neural network that constitutes the generator 101 is set with coefficients acquired during learning, and receives random numbers as inputs and outputs newly generated data.

[0040] 4 shows the evaluator 102 in the generated data evaluation (Data Evaluation) phase. In the generated data evaluation phase, the neural network constituting the evaluator 102 has coefficients acquired during learning set, receives the generated data of the generator 101 as input, and outputs a simulated subjective evaluation that estimates the user's subjective evaluation of the generated data.

[0041] Fig. 5 shows the generative model and the evaluation model in the model update phase. In the learning phase, the coefficients of the neural networks constituting the generator 101 and the evaluator 102 are updated (Generator & Evaluator update). In the example shown in Fig. 3, the coefficients of each neural network are updated based on the user's evaluation (User Feedback) of the data newly generated by the generator 101.

[0042] Specifically, the simulated subjective evaluation of the evaluator 102 for the data generated by the generator 101 and user feedback are input to the evaluation model. For example, the simulated subjective evaluation is output using evaluation indices that separate the features of the generated data into multiple elements, and the user feedback is an adjustment to the simulated subjective evaluation for each evaluation indices. Then, the neural network coefficients are updated so that a loss function based on the error of the simulated subjective evaluation with respect to the user feedback is minimized. In this way, the evaluator 102 is trained to obtain a simulated subjective evaluation that reflects the user's subjectivity and preferences for new data generated by the generator 101. The generator 101 also inputs user feedback for its own generated data and updates the neural network coefficients so that the user feedback becomes better. In this way, the generator 101 is trained to generate data that reflects the user's subjectivity and preferences.

[0043] All phases in FIG. 2 may be implemented on a single device, or the processing of each phase may be distributed across multiple devices. For example, the user feedback phase may be implemented on a user-owned information terminal (e.g., a smartphone, tablet, or personal computer) for the convenience of direct user input. On the other hand, phases such as data generation, data evaluation, and coefficient update involve neural networks and involve a heavy computational load, so they may be implemented on a cloud-based device with high computing power. For example, the data generation phase by the generator 101 and the evaluation phase by the evaluator 102 may be implemented on a first device, the system feedback phase that presents the generated data and simulated subjective data to the user may be implemented on a second device, and the data update phase based on user feedback may be implemented on a third device. Furthermore, the second device that presents the generated data and simulated subjective data to the user in the system feedback phase and the information terminal that accepts user feedback in the user feedback phase may be the same device. In this case, user feedback may be provided by the user operating a mouse, keyboard, touch panel, or the like based on the generated data and simulated subjective data displayed on the information terminal.

[0044] C. Interface configuration example In this section, examples of the configuration of interfaces used for system feedback and user feedback will be described with reference to FIGS.

[0045] In the first example shown in FIG. 6, it is assumed that the evaluator 102 divides the features of the data generated by the generator 101 into multiple elements and performs a simulated subjective evaluation for each element. Then, in the system feedback phase shown in the left half of FIG. 6, the data newly generated by the generator 101 (Generator Output) and the simulated subjective evaluation (Evaluator Output) output by the evaluator 102 after dividing the features of the generated data into multiple elements are simultaneously presented to the user. In the user feedback phase shown in the right half of FIG. 6, an interface is provided on the screen of the information terminal used by the user, for example, to allow the user to adjust the simulated subjective evaluation for each element. The user can provide user feedback by adjusting the evaluation values ​​of several elements that characterize the generated data presented in the system feedback via the interface. As shown in the right half of FIG. 6, a screen equipped with interfaces for adjusting the evaluation value a1 of element 1 and the evaluation value a2 of element 2 is presented. The user can change each evaluation value, for example, by manually entering the value, operating the up and down cursor buttons, or operating a dial.

[0046] As shown in FIG. 6, by using an interface for adjusting multiple elements that characterize the generated data, the user's subjectivity and preferences can be reflected in detail in the data generation system 100.

[0047] In the second example shown in FIG. 7 , it is assumed that the evaluator 102 outputs generated data related to the generated data generated by the generator 101 as a simulated subjective evaluation. Here, “related” means, for example, “evaluated as data intended by the user.” In the system feedback phase shown in the left half of FIG. 7 , the data newly generated by the generator 101 (Generator Output) and the related generated data output by the evaluator 102 (Evaluator Output) are simultaneously presented to the user. Then, in the user feedback phase shown in the right half of FIG. 7 , an interface is provided on the screen of the information terminal used by the user, for example, to indicate whether the related generated data as the simulated subjective evaluation is the data intended by the user. The user can provide user feedback by indicating through the interface whether the related generated data presented in the system feedback is the data intended by the user. In the example screen configuration shown in the right half of FIG. 7 , ○ and × buttons are provided for the user to evaluate the quality of the related generated data, and the user can select either button by using a mouse or touching the screen.

[0048] As shown in Figure 7, by presenting generated data related to generated data as a simulated subjective evaluation and using an interface that allows users to easily input their opinions on the related generated data, the user's subjectivity and preferences can be easily reflected in the data generation system 100.

[0049] Figures 8 to 12 show specific examples of interface configurations for each data type of data generated by the generator 101. Of these, Figures 8 and 9 show specific examples of interface configurations when the generator 101 generates a facial image, Figures 10 and 11 show specific examples of interface configurations when the generator 101 generates a voice, and Figure 10 shows a specific example of interface configurations when the generator 101 generates advertising content.

[0050] In the third example shown in FIG. 8 , it is assumed that the evaluator 102 divides the features of the facial image generated by the generator 101 into multiple elements and performs a simulated subjective evaluation for each element. In the system feedback phase shown in the left half of FIG. 8 , the generator 101 simultaneously presents to the user a newly generated facial image (Generator Output) and a simulated subjective evaluation (Evaluator Output) in which the evaluator 102 divides the features of the facial image into multiple elements such as skin color and lips. In the user feedback phase shown in the right half of FIG. 8 , an interface is provided on the screen of the information terminal used by the user, for example, to allow the user to adjust the simulated subjective evaluation for each element such as skin color and lips. The user can provide user feedback by adjusting the evaluation values ​​of several elements that characterize the facial image presented in the system feedback via the interface. The user can provide detailed user feedback to the data generation system 100 by changing the subjective index values ​​on the interface to provide feedback of their own subjective evaluation, such as "make the skin color darker" or "make the lips larger." The subjective index value can be changed by manually inputting a value on the screen, by operating the up and down cursor buttons, by operating the dial, or the like.

[0051] As shown in FIG. 8, by using an interface for adjusting a plurality of elements that characterize the generated face image, the user's subjectivity and preferences can be reflected in detail in the data generation system 100.

[0052] In the fourth example shown in FIG. 9, it is assumed that the evaluator 102 outputs a face image related to the face image generated by the generator 101 as a simulated subjective evaluation. In the system feedback phase shown in the left half of FIG. 9, a face image newly generated by the generator 101 (Generator Output) and a related face image generated by the evaluator 102 (Evaluator Output) are simultaneously presented to the user. Then, in the user feedback phase shown in the right half of FIG. 9, an interface is provided on the screen of an information terminal used by the user, for example, to indicate whether the related face image as the simulated subjective evaluation is the face image intended by the user. The user can easily provide user feedback by indicating through the interface whether the related face image presented in the system feedback is the face image intended by the user. In the example screen configuration shown in the right half of FIG. 9, O and X buttons are provided for the user to evaluate the related face image as good or bad, and the user can select either button by operating the mouse or touching the screen.

[0053] As shown in Figure 9, by presenting facial images related to the generated facial image as simulated subjective evaluations and using an interface for easily inputting opinions on the related facial images, the user's subjectivity and preferences can be easily reflected in the data generation system 100.

[0054] In the fifth example shown in FIG. 10 , it is assumed that the evaluator 102 divides the features of the voice generated by the generator 101 into multiple elements and performs a simulated subjective evaluation for each element. In the system feedback phase shown in the left half of FIG. 10 , the generator 101 simultaneously presents to the user the newly generated voice (Generator Output) and the evaluator 102 simultaneously presenting the features of the voice, such as volume and pitch, to the user. The user can play the generated voice and check the features, such as volume and pitch. In the user feedback phase shown in the right half of FIG. 10 , an interface is provided on the screen of the information terminal used by the user, for example, to allow the user to adjust the simulated subjective evaluation for each element, such as volume and pitch. The user can provide user feedback by adjusting the evaluation values ​​of several elements that characterize the voice presented in the system feedback via the interface. The user can provide detailed user feedback to the data generation system 100 by adjusting the subjective index values ​​on the interface, such as "lower the volume" or "higher the pitch." As shown in the right half of Figure 10, a screen equipped with an interface for adjusting the evaluation values ​​of volume and pitch is presented. The user can change each evaluation value, for example, by manually entering it, operating the up and down cursor buttons, or operating the dial.

[0055] As shown in FIG. 10, by using an interface for adjusting multiple elements that characterize the generated voice, the subjectivity and preferences of the user can be reflected in detail in the data generation system 100.

[0056] In the sixth example shown in FIG. 11 , it is assumed that the evaluator 102 outputs a speech related to the speech generated by the generator 101 as a simulated subjective evaluation. In the system feedback phase shown in the left half of FIG. 11 , the newly generated speech (Generator Output) by the generator 101 and the related speech (Evaluator Output) generated by the evaluator 102 are simultaneously presented to the user. The user can play the generated speech and the related speech and check their respective characteristics, such as volume and pitch. Then, in the user feedback phase shown in the right half of FIG. 11 , an interface is provided on the screen of the information terminal used by the user, for example, to indicate whether the related speech as the simulated subjective evaluation is the speech intended by the user. The user can easily provide user feedback by indicating through the interface whether the related speech presented in the system feedback is the speech intended by the user. In the example screen configuration shown in the right half of FIG. 11 , ○ and × buttons are provided for the user to rate the related speech as good or bad, and the user can select either button by using a mouse or touching the screen.

[0057] As shown in Figure 11, by presenting speech related to the generated speech as a simulated subjective evaluation and using an interface that allows users to easily input their opinions on the related speech, the user's subjectivity and preferences can be easily reflected in the data generation system 100.

[0058] For example, for subjective evaluation when color image content is generated from black and white image content using the data generation system 100, the generator 101 and the evaluator 102 can be trained by feedback of the subjective evaluation through the interfaces shown in Figures 8 and 9.

[0059] Furthermore, for subjective evaluation when sound effects for video content are generated using the data generation system 100, the generator 101 and the evaluator 102 can be trained by feedback of subjective evaluations via the interfaces shown in Figures 10 and 11.

[0060] In addition, for subjective evaluation when animated character images and character sounds are generated using the data generation system 100, the generator 101 and the evaluator 102 can be trained by feedback of subjective evaluations through the interfaces shown in Figures 8 to 11.

[0061] 8 to 11, the user may select a portion of data newly generated by the generator 101, thereby providing user feedback on the selected portion. In the examples shown in FIGS. 8 and 10, for example, when a user selects a portion of content, an interface for evaluating an element corresponding to the selected portion is presented. In the examples shown in FIGS. 9 and 11, for example, when a user selects a portion of content, an interface for evaluating whether the selected portion is the data intended by the user is presented.

[0062] When content corresponding to the Generator Output and Evaluator Output is presented on an information terminal that accepts user feedback, the user may select a portion of the content presented on the information terminal using a mouse, keyboard, touch panel, etc., and an interface for providing user feedback may be presented at a position corresponding to the selected position of the content.

[0063] In the seventh example shown in FIG. 12 , it is assumed that the evaluator 102 performs a simulated subjective evaluation of the portions of the advertising content generated by the generator 101 that the user wants to emphasize. In the system feedback phase shown in the left half of FIG. 12 , the advertising content generated by the generator 101 (Generator Output) and the advertising content (Evaluator Output) including the portion of the advertising content that the user has specified as ...

[0064] When an advertisement is automatically generated using the data generation system 100, the generator 101 and the evaluator 102 can be trained by feedback of subjective evaluations via the interface shown in FIG.

[0065] As shown in Figure 12, advertising content that includes a specification of the parts of the generated advertising content that the user wants to emphasize and has given a simulated subjective evaluation is presented as a simulated subjective evaluation, and an interface is used to easily specify the parts of the advertising content that the user actually wants to emphasize, thereby making it possible to easily reflect the user's subjectivity and preferences in the data generation system 100.

[0066] D. Details of Data Generation, Evaluation, and Training Procedures Section B above provided an overview of the procedures for data generation, evaluation, and learning in the data generation system 100. In this section D, we will provide a detailed description of the procedures for data generation, evaluation, and learning in the data generation system 100, including the data flow.

[0067] Here, the data generation phase by the generator 101 and the evaluation phase by the evaluator 102 are implemented in a first device, the system feedback phase that presents the generated data and simulated subjective data to the user is implemented in a second device, the data update phase based on user feedback is implemented in a third device, and the user feedback phase is implemented in a user terminal.

[0068] 13 also shows paths for transferring data between each phase. As shown in the figure, the data transfer path from the data generation phase to the data evaluation phase is defined as Path 1, the data transfer path from the data evaluation phase to the system feedback phase as Path 2, the data transfer path from the system feedback phase to the user feedback phase as Path 3, the data transfer path from the user feedback phase to the model update phase as Path 4, the data transfer path from the data evaluation phase to the model update phase as Path 5, and the data transfer path from the model update phase to the data generation phase as Path 6. When each phase is implemented in a plurality of physically independent devices, such as the first to third devices and a user information terminal, Paths 1 to 6 are configured by communication media connecting the corresponding devices. Furthermore, data transfer paths between multiple phases implemented in a single device (for example, Path 1 between the data generation phase and the data evaluation phase) are implemented, for example, by communication between applications.

[0069] FIG. 14 shows the specific data flow between each phase.

[0070] In the data generation phase, the neural network constituting the generator 101 is set with coefficients acquired during training, inputs random numbers, and outputs newly generated data. This generated data is transferred to the data evaluation phase via Path 1.

[0071] In the evaluation phase of the generated data, the neural network constituting the evaluator 102 is set with coefficients acquired during learning, inputs the generated data via Path 1, and outputs a simulated subjective evaluation that estimates the user's subjective evaluation of the generated data.

[0072] In the system feedback phase, the second device that performs system feedback acquires, via Path 2, the newly generated data (Generator Output) from the generator 101 and the simulated subjective evaluation (Evaluator Output) that the evaluator 102 outputs by dividing the features of this generated data into multiple elements, and presents these to the user simultaneously.

[0073] In the user feedback phase, the simulated evaluation results obtained via Path 3 are presented on an interface that is displayed on the screen of the information terminal used by the user, for example, and allows the user to adjust the simulated subjective evaluation for each element. The user can then adjust the evaluation values ​​of some elements that characterize the generated data via the interface on the screen of the information terminal.

[0074] In the model update phase, a third device that trains the generative model used by the generator 101 and the evaluation model used by the evaluator 102 acquires a simulated subjective evaluation of the generated data by the evaluator 102 via Path 5 and acquires user feedback on the simulated subjective evaluation via Path 4. Here, the simulated subjective evaluation of the generated data by the evaluator 102 and the user feedback are input to the evaluation model. In the example shown in FIG. 14 , the simulated subjective evaluation is an evaluation index obtained by dividing the features of the generated data into multiple elements, and the user feedback is an adjustment to the simulated subjective evaluation for each evaluation index. The third device then updates the coefficients of the neural network that constitutes the evaluation model so as to minimize a loss function based on the error of the simulated subjective evaluation with respect to the user feedback. The generative model also receives user feedback on the simulated subjective evaluation of the generated data. The third device then updates the coefficients of the neural network that constitutes the generative model so as to improve the user feedback.

[0075] The coefficients of the generative model updated in the model update phase are set in the generative model used by the generator 101 via Path 6 and are used in the next data generation phase. Also, the coefficients of the evaluation model updated in the model update phase are set in the evaluation model used by the evaluator 102 via Path 6 and are used in the next data evaluation phase.

[0076] Although the evaluation index of subjective evaluation is influenced by the purpose and individual differences, according to the present disclosure, different subjective evaluation models can be created in a short time and at low cost by following the procedure shown in Fig. 14. Furthermore, according to the present disclosure, an interface that reflects the user's intention in the evaluation model is provided, and therefore it is possible to provide a data processing system 100 that learns the evaluation model in a manner that can reflect the user's subjectivity and preferences, and generates data as intended by the user.

[0077] E. Information Processing Systems 15 shows an example of the configuration of an information processing system 1500 that is used as, for example, the first to third devices or a user's information terminal. Each element of the information processing system 1500 will be described below.

[0078] A CPU (Central Processing Unit) 1501 is interconnected with a ROM (Read Only Memory) 1502, a RAM (Random Access Memory) 1503, a mass storage device 1504, and an input / output interface 1505 via a bus 1610.

[0079] The CPU 1501 executes programs loaded from the ROM 1502 or the mass storage device 1504 to the RAM 1503, and can perform various processes while temporarily storing working data in the RAM 1503. The programs executed by the CPU 1501 include a basic input / output program stored in the ROM 1502, and an operating system (OS) and application programs installed in the mass storage device 1504. The OS provides an execution environment for the application programs. The application programs include application programs that perform at least one of the following: learning a machine learning model; generating data using a trained machine learning model and estimating subjective evaluations of the generated data; presenting the generated data and simulated subjective evaluations thereof; and obtaining user feedback on the simulated subjective evaluations. By having the CPU 1501 execute the application programs in the execution environment provided by the OS, the information processing system 1500 operates as various devices related to the present disclosure.

[0080] In addition, since processes related to machine learning models, such as learning, require a huge amount of calculation and parallel processing is considered, the information processing system 1500 may be equipped with a GPU (Graphics Processing Unit) or GPGPU (General-Purpose computing on Graphics Processing Units) instead of or in addition to the CPU 1501.

[0081] The ROM 1502 is a read-only memory that permanently stores basic input / output programs, device information, etc. The RAM 1503 is composed of volatile memory such as a DRAM (Dynamic RAM), and is used as a work area for the CPU 1501. The mass storage device 1504 is a hard disk drive (HDD) or solid state drive (SSD), etc., and stores programs and data in file format. The HDD is a storage device that uses one or more magnetic disks fixed within the unit as recording media.

[0082] Various input / output devices such as an output unit 1511, an input unit 1512, a communication unit 1513, and a drive 1514 are connected to the input / output interface 1505. The output unit 1511 is made up of an LCD (Liquid Crystal Display), a speaker, a printer, etc., and outputs, for example, the results of program execution by the CPU 1501. The input unit 1512 is made up of a keyboard, a mouse, a microphone, etc., and receives instructions from the user.

[0083] The communication unit 1513 has a wired or wireless communication interface conforming to a predetermined communication protocol, and performs data communication with an external device. When the information processing system 1500 operates as one of the first to third devices, the communication unit 1513 communicates with the other devices among the first to third devices. When the information processing system 1500 operates as a user's information terminal, the communication unit 1513 communicates with the second device and the third device.

[0084] The communication unit 1513 is also connected to a wide area network such as the Internet. Using the communication unit 1513, an application program can be downloaded from a download site on the Internet and installed in the information processing system 1500.

[0085] The drive 1514 has a removable recording medium 1515 loaded therein and performs read processing from the removable recording medium 1515 and write processing to the removable recording medium 1515 (if the removable recording medium is writable). The removable recording medium 1515 stores programs, data, etc. in file format. Examples of the removable recording medium 1515 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, and a semiconductor memory.

[0086] The information processing system 1500 can operate as a first device by installing, for example, a program that generates data using a generative model and a program that estimates a subjective evaluation of data using an evaluation model.

[0087] Furthermore, the information processing system 1500 can operate as a second device by installing a program that acquires and presents simulated subjective evaluations of data generated by a generative model and data generated by an evaluation model.

[0088] In addition, the information processing system 1500 can operate as a user's information terminal by presenting an interface for inputting user evaluations of simulated subjective evaluations of data generated by a generative model and data generated by an evaluation model, and by installing a program that uploads feedback evaluations from users obtained via the interface.

[0089] In addition, the information processing system 1500 can operate as a third device by installing a program that trains the generative model and the evaluation model (i.e., updates the coefficients of each neural network) based on simulated subjective evaluation of data generated by the evaluation model and evaluation feedback from users. [Industrial Applicability]

[0090] Although the present disclosure has been described in detail above with reference to specific embodiments, it is obvious that those skilled in the art can make modifications or substitutions to the embodiments without departing from the spirit and scope of the present disclosure.

[0091] Although the present specification has mainly described an embodiment in which the present disclosure is applied to a data generation system for generating images, the gist of the present disclosure is not limited thereto. The present disclosure can be applied to the generation of various data other than images, such as audio, music, and text, and the subjective evaluation of the generated data.

[0092] Furthermore, the evaluation model trained based on the present disclosure can be applied to subjective evaluation of color image content generated from black and white image content, subjective evaluation of sound effects generated from video content, and subjective evaluation of automatically generated animated character images and character sounds.

[0093] In short, the present disclosure has been described in the form of examples, and the contents of the specification should not be interpreted as limiting. To determine the gist of the present disclosure, the claims should be taken into consideration.

[0094] The present disclosure may also be configured as follows.

[0095] (1) a system feedback acquisition unit that acquires generated data and evaluation information based on an evaluation model for the generated data; a system feedback presentation unit that presents the generated data and the evaluation information; a user feedback acquisition unit that acquires user evaluations on the generated data or the evaluation information; an output unit that outputs the user evaluation acquired by the user feedback acquisition unit; An information processing system including:

[0096] (2) The information processing system according to (1) above, further comprising an interface presenting unit that presents an interface for inputting the user evaluation.

[0097] (3) The system feedback acquisition unit acquires the generated data and the evaluation information from one or more devices that generate data using a generative model and evaluate the generated data using the evaluation model. The information processing system according to either (1) or (2) above.

[0098] (4) the system feedback acquisition unit acquires, as evaluation information for the generated data, evaluation indexes obtained by dividing the characteristics of the generated data into multiple elements; the system feedback presentation unit presents the generated data and an evaluation index for each of the elements; the interface presenting unit presents an interface for adjusting the evaluation index for each element. The information processing system according to either (2) or (3) above.

[0099] (5) the system feedback acquisition unit acquires generated data related to the generated data as evaluation information for the generated data; the system feedback presentation unit presents the generated data and the related generated data; the interface presentation unit presents an interface for inputting a user's intention regarding the related generated data; The information processing system according to any one of (2) to (4) above.

[0100] (6) The output unit outputs the user evaluation to a device for updating each of a generative model that generates data and an evaluation model that evaluates the generated data. The information processing system according to (1) above.

[0101] (7) a first device including the system feedback acquisition unit and the system feedback presentation unit; a second device including the user feedback acquisition unit and the output unit; The information processing system according to any one of (1) to (6) above,

[0102] (8) a third device for updating each of the generative model for generating the data and the evaluation model for evaluating the generated data; the output unit outputs the user evaluation to the third device. An information processing system according to any one of (1) to (7) above.

[0103] (9) One or more devices that generate data using a generative model and evaluate the generative model using the evaluation model, the system feedback acquisition unit acquires the generated data and an evaluation of the generated data using the evaluation model from the one or more devices. An information processing system according to any one of (1) to (8) above.

[0104] (10) A first device including the system feedback acquisition unit, the system feedback presentation unit, the user feedback acquisition unit, and the output unit; The information processing system according to any one of (1) to (9) above.

[0105] (11) a system feedback acquisition step of acquiring evaluation information based on generated data and an evaluation model for the generated data; a system feedback presentation step of presenting the generated data and the evaluation information; a user feedback acquisition step of acquiring a user evaluation on the generated data or the evaluation information; an output step of outputting the user evaluation acquired in the user feedback acquisition step; An information processing method comprising:

[0106] (12) The information processing method according to claim 11, further comprising an interface presenting step of presenting an interface for inputting the user evaluation.

[0107] (13) In the system feedback acquisition step, evaluation indexes obtained by dividing the characteristics of the generated data into a plurality of elements are acquired as evaluation information for the generated data; the system feedback presentation step presents the generated data and an evaluation index for each element; In the interface presenting step, an interface for adjusting the evaluation index for each element is presented. The information processing method according to (12) above.

[0108] (14) In the system feedback acquisition step, generated data related to the generated data is acquired as evaluation information for the generated data; the system feedback presentation step presents the generated data and the related generated data; the interface presenting step presents an interface for inputting a user's intention regarding the related generated data; The information processing method according to (12) above.

[0109] (15) an evaluation unit that generates evaluation information for the generated data using an evaluation model; an acquisition unit that acquires user evaluation information regarding the evaluation information; a rating model update unit that updates the rating model based on the user rating information; An information processing system including:

[0110] (16) A generation unit that generates data using a generative model; a generation model update unit that updates the generation model based on the user evaluation information; Including, the evaluation unit generates evaluation information for the data generated by the generation unit. The information processing system according to (15) above.

[0111] (17) The evaluation unit outputs the evaluation information to an information terminal; the acquiring unit acquires, from the information terminal, user evaluation information input through an interface presented on the information terminal; The information processing system according to any one of (15) and (16) above.

[0112] (18) The evaluation unit divides the characteristics of the generated data into a plurality of elements and evaluates each element, the acquiring unit acquires user evaluation information including information for adjusting an evaluation of each of the plurality of elements. The information processing system according to any one of (15) to (17) above.

[0113] (19) The evaluation unit generates generated data related to the generated data; the acquiring unit acquires user evaluation information including a user's intention regarding the related generated data. The information processing system according to any one of (15) to (18) above.

[0114] (20) an evaluation step of generating evaluation information for the generated data using an evaluation model; an acquisition step of acquiring user evaluation information for the evaluation information; a rating model updating step of updating the rating model based on the user rating information; An information processing method comprising:

[0115] (21) A generation unit that generates data using a generative model; an evaluation unit that generates a subjective evaluation of the data generated by the generation unit using an evaluation model; A data generation system comprising:

[0116] (22) Further comprising a model update unit that updates at least one of the generative model and the evaluation model based on a user evaluation of system feedback including data generated by the generation unit and a subjective evaluation by the evaluation unit. The data generation system according to (21) above.

[0117] (23) The evaluation unit generates evaluation indices by dividing the characteristics of the generated data into multiple elements as a subjective evaluation of the generated data, the model update unit updates the model based on a result of adjustment by a user to the evaluation index for each element. The data generation system according to (22) above.

[0118] (24) The evaluation unit generates data related to the generated data as a subjective evaluation of the generated data, the model update unit updates the model based on the user's intention regarding the related data; The data generation system according to (21) above. [Explanation of symbols]

[0119] 100...data generation system, 101...generator, 102...evaluator 1500...information processing system, 1501...CPU, 1502...ROM 1503...RAM, 1504...Mass storage device 1505...input / output interface, 1510...bus 1511...output unit, 1512...input unit, 1513...communication unit 1514...Drive, 1515...Removable recording medium

Claims

1. a system feedback acquisition unit that acquires generated data and evaluation information based on an evaluation model for the generated data; a system feedback presentation unit that presents the generated data and the evaluation information; a user feedback acquisition unit that acquires user evaluations on the generated data or the evaluation information; an output unit that outputs the user evaluation acquired by the user feedback acquisition unit to a device that updates each of a generation model that generates data and an evaluation model that evaluates the generated data; An information processing system including:

2. The information processing system according to claim 1 , further comprising an interface presenting unit that presents an interface for inputting the user evaluation.

3. the system feedback acquisition unit acquires the generated data and the evaluation information from one or more devices that generate data using a generative model and evaluate the generated data using the evaluation model; The information processing system according to claim 1 .

4. the system feedback acquisition unit acquires, as evaluation information for the generated data, evaluation indexes obtained by dividing characteristics of the generated data into a plurality of elements; the system feedback presentation unit presents the generated data and an evaluation index for each of the elements; the interface presenting unit presents an interface for adjusting the evaluation index for each element. The information processing system according to claim 2 .

5. the system feedback acquisition unit acquires generated data related to the generated data as evaluation information for the generated data; the system feedback presentation unit presents the generated data and the related generated data; the interface presentation unit presents an interface for inputting a user's intention regarding the related generated data; The information processing system according to claim 2 .

6. A first device including the system feedback acquisition unit and the system feedback presentation unit; a second device including the user feedback acquisition unit and the output unit; The information processing system of claim 1 , comprising:

7. A third device for updating each of the generative model for generating data and the evaluation model for evaluating the generated data; the output unit outputs the user evaluation to the third device. The information processing system according to claim 1 .

8. A system comprising one or more devices for generating data using a generative model and evaluating the generative model using the evaluation model, the system feedback acquisition unit acquires the generated data and an evaluation of the generated data using the evaluation model from the one or more devices. The information processing system according to claim 1 .

9. A first device including the system feedback acquisition unit, the system feedback presentation unit, the user feedback acquisition unit, and the output unit, The information processing system according to claim 1 .

10. A system feedback acquisition step of acquiring generated data and evaluation information based on an evaluation model for the generated data; a system feedback presentation step of presenting the generated data and the evaluation information; a user feedback acquisition step of acquiring a user evaluation on the generated data or the evaluation information; an output step of outputting the user evaluation acquired in the user feedback acquisition step to a device for updating each of a generation model that generates data and an evaluation model that evaluates the generated data; An information processing method comprising:

11. The information processing method according to claim 10, further comprising an interface presentation step of presenting an interface for inputting the user evaluation.

12. In the system feedback acquisition step, evaluation indexes obtained by dividing the characteristics of the generated data into a plurality of elements are acquired as evaluation information for the generated data; the system feedback presentation step presents the generated data and an evaluation index for each element; In the interface presenting step, an interface for adjusting the evaluation index for each element is presented. The information processing method according to claim 11.

13. In the system feedback acquisition step, generated data related to the generated data is acquired as evaluation information for the generated data; the system feedback presentation step presents the generated data and the related generated data; the interface presenting step presents an interface for inputting a user's intention regarding the related generated data; The information processing method according to claim 11.

14. A generation unit that generates data using a generative model; an evaluation unit that generates evaluation information for data generated using the generative model using the evaluation model; an acquisition unit that acquires user evaluation information regarding the evaluation information; a generation model update unit that updates the generation model based on the user evaluation information; a rating model update unit that updates the rating model based on the user rating information; An information processing system including:

15. The evaluation unit outputs the evaluation information to an information terminal, the acquiring unit acquires, from the information terminal, user evaluation information input through an interface presented on the information terminal; The information processing system according to claim 14.

16. The evaluation unit divides the characteristics of the generated data into a plurality of elements and evaluates each element, the acquiring unit acquires user evaluation information including information for adjusting an evaluation of each of the plurality of elements. The information processing system according to claim 14.

17. The evaluation unit generates generated data related to the generated data, the acquiring unit acquires user evaluation information including a user's intention regarding the related generated data. The information processing system according to claim 14.

18. A generation step of generating data using a generative model; an evaluation step of generating, using an evaluation model, evaluation information for the data generated using the generative model; an acquisition step of acquiring user evaluation information for the evaluation information; a generative model updating step of updating the generative model based on the user evaluation information; a rating model updating step of updating the rating model based on the user rating information; An information processing method comprising:

Images