Model generation system, evaluation system, model generation method, and model generation program

Abstract

To efficiently evaluate user interfaces using user biometric information. [Solution] The model generation system comprises at least one processor. The at least one processor obtains a specified prompt for generating a sample image, inputs one or more specified prompts to an image generation model to generate one or more sample images corresponding to one or more sample operation screens, sets an evaluation value for the operation on the screen for each of the one or more sample operation screens, obtains the subject's biometric information at the time of operation on the sample operation screen for each of the one or more sample operation screens generated based on the one or more sample images, and performs machine learning using training data that includes combinations of biometric information and evaluation values ​​for each of the one or more sample operation screens to generate an inference model that infers the evaluation value of the target operation screen from the user's biometric information at the time of operation on the target operation screen.

Description

[Technical Field]

[0001] One aspect of this disclosure relates to a model generation system, an evaluation system, a model generation method, and a model generation program. [Background technology]

[0002] Mechanisms for evaluating user interfaces using user biometric information have been known for some time. For example, Patent Document 1 describes a method for detecting a user's mental state using biosignals and performing a more detailed usability evaluation of specific functions of a device.

[0003] Patent Document 2 describes a device that supports the evaluation of usability related to the operation of a device that presents information at least visually according to the user's operation, such as a device equipped with a graphical user interface (GUI). The evaluation of usability is performed using a record of the user's operation, a record of the user's biometric information, and a questionnaire survey of the user. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2007-52601 [Patent Document 2] Japanese Patent Publication No. 2005-216311 [Overview of the project] [Problems that the invention aims to solve]

[0005] A system is needed to efficiently evaluate user interfaces using user biometric information. [Means for solving the problem]

[0006] A model generation system relating to one aspect of this disclosure comprises at least one processor. The at least one processor obtains a specified prompt for each of the one or more sample operation screens to generate a sample image which is an image of the sample operation screen, inputs each of the one or more specified prompts to an image generation model that generates an image based on the prompts to generate one or more sample images corresponding to the one or more sample operation screens, sets an evaluation value for the operation on the screen for each of the one or more sample operation screens generated based on the one or more sample images, obtains the subject's biometric information at the time of operation on the sample operation screen, and performs machine learning using training data that includes combinations of biometric information and evaluation values ​​for each of the one or more sample operation screens to generate an inference model that infers the evaluation value of the target operation screen from the user's biometric information at the time of operation on the target operation screen.

[0007] In this respect, when generating a sample operation screen for acquiring the subject's biometric information during operation, the sample image, which is an image of the sample operation screen, is generated by an image generation model. Because the screen image that forms the basis of the sample operation screen is automatically obtained, the generation of the sample operation screen becomes easier. Therefore, subsequent processes such as setting evaluation values ​​for the sample operation screen, acquiring biometric information, and generating the inference model can proceed smoothly. As a result, it becomes possible to obtain the inference model quickly, and in turn, to evaluate the operation screen using the user's biometric information more efficiently. [Effects of the Invention]

[0008] According to one aspect of this disclosure, it is possible to efficiently evaluate the user interface using the user's biometric information. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the functional configuration of a model generation system and an evaluation system. [Figure 2]This figure shows an example of a computer hardware configuration used as a model generation system or evaluation system. [Figure 3] This is a flowchart showing an example of the process for generating an inference model. [Figure 4] This figure shows an example of a sample image generated based on a specified prompt. [Figure 5] This figure shows an example of the reference screen. [Figure 6] This figure shows an example of clustering for a set of reference and sample images. [Figure 7] This flowchart shows an example of a process that uses an inference model to infer the evaluation value of a target operation screen. [Modes for carrying out the invention]

[0010] The following describes various examples in this disclosure in detail with reference to the attached drawings. In the description of the drawings, identical or equivalent elements are denoted by the same reference numeral, and redundant descriptions are omitted.

[0011] [System Overview] The model generation system described herein is a computer system that generates an inference model for evaluating the usability of an operation screen. An operation screen refers to a user interface (UI) displayed on a computer and operated by a user, and is composed of one or more user interface components (UI components). The inference model is a computational model that infers an evaluation value related to the operation on the operation screen from the user's biometric information during operation of the operation screen. The evaluation value related to the operation on the operation screen refers to an index that indicates the degree of usability of the operation screen, that is, the degree of ease of operation for the user on the operation screen. Biometric information refers to information that indicates a person's biological responses.

[0012] The evaluation system according to the present disclosure is a computer system that inputs biometric information of a user when operating an operation screen whose evaluation value is unknown to an inference model generated by a model generation system, and estimates the evaluation value of the operation screen. In the present disclosure, the operation screen for which the evaluation value is estimated is also referred to as the "target operation screen".

[0013] As a method for evaluating the usability of an operation screen, evaluation based on a user's questionnaire is known. However, since the evaluation based on the questionnaire strongly depends on the user's subjectivity, it cannot always be said that a correct evaluation is obtained. The methods shown in Patent Documents 1 and 2 above can be said to be methods for quantitatively evaluating the usability. However, it is not easy to establish criteria for performing such quantitative evaluation.

[0014] The model generation system executes machine learning using teacher data including combinations of biometric information and evaluation values for each of one or more sample operation screens to generate an inference model. Machine learning refers to a method of autonomously finding laws or rules by repeatedly learning based on given information. A sample operation screen refers to an operation screen prepared to obtain teacher data. The model generation system generates a sample image, which is an image for generating the sample operation screen, using a predetermined image generation model. The generation of the sample operation screen becomes easier as much as the screen image serving as the basis of the sample operation screen is automatically obtained. As a result, it becomes possible to obtain an inference model used for quantitatively evaluating the usability of the operation screen earlier, and accordingly, the evaluation of the operation screen using the biometric information of the user can be efficiently performed.

[0015] [Configuration of the System] While referring to FIG. 1, the configurations of the model generation system 1 and the evaluation system 2 according to an example will be described. FIG. 1 is a diagram showing an example of the functional configurations of the model generation system 1 and the evaluation system 2.

[0016] The model generation system 1 is connected to the evaluation system 2, the image generation system 3, and the training database 4 via a communication network. The communication network may include the internet or an intranet. The image generation system 3 is a computer system equipped with an image generation model 31 that generates images based on prompts. An example of the image generation model 31 is a generative artificial intelligence (AI) for generating images from text, such as Stable Diffusion. A prompt is an instruction input to the image generation model 31 in order to generate an image. The training database 4 is a device that stores training data. Both the image generation system 3 and the training database 4 may be components of the model generation system 1 or may be located outside of the model generation system 1.

[0017] The model generation system 1 comprises a prompt acquisition unit 11, an image generation unit 12, a training data generation unit 13, and a learning unit 14. The prompt acquisition unit 11 is a functional module that acquires one or more prompts as specified prompts to be used to generate one or more sample images. The image generation unit 12 is a functional module that generates sample images from each of the one or more specified prompts using the image generation model 31 of the image generation system 3. The training data generation unit 13 is a functional module that generates training data including combinations of biometric information and evaluation values ​​for each of the one or more sample operation screens generated based on one or more sample images, and stores the training data in the learning database 4. The learning unit 14 is a functional module that generates an inference model 23 by performing machine learning using the training data and provides the inference model 23 to the evaluation system 2.

[0018] The evaluation system 2 comprises a biometric information acquisition unit 21 and an inference unit 22. The biometric information acquisition unit 21 is a functional module that acquires the user's biometric information when operating the target operation screen. The inference unit 22 is a functional module that inputs this biometric information into an inference model 23 and infers the evaluation value of the target operation screen.

[0019] Figure 2 shows an example of the hardware configuration of a computer 100 that functions as either a model generation system 1 or an evaluation system 2. For example, the computer 100 includes a processor 101, a main memory unit 102, an auxiliary memory unit 103, a communication control unit 104, an input device 105, and an output device 106 as hardware components.

[0020] The processor 101 is an arithmetic circuit that executes the operating system and application programs. The main memory 102 consists of, for example, ROM and RAM. The auxiliary memory 103 consists of, for example, a hard disk or flash memory and generally stores a larger amount of data than the main memory 102. The auxiliary memory 103 stores programs 110 for implementing each functional element of the model generation system 1 or evaluation system 2 in the computer 100. The communication control unit 104 consists of, for example, a network card or a wireless communication module. The input device 105 consists of, for example, at least one of a keyboard, mouse, and touch panel. The output device 106 consists of, for example, at least one of a monitor and speaker.

[0021] Program 110 may be a model generation program that causes computer 100 to function as model generation system 1. Alternatively, program 110 may be an evaluation program that causes computer 100 to function as evaluation system 2. In either case, each functional module of model generation system 1 or evaluation system 2 is realized by program 110, which is pre-stored in auxiliary storage unit 103. Each functional module is realized by loading program 110 onto processor 101 or main memory unit 102 and causing processor 101 to execute program 110. Processor 101 operates communication control unit 104, input device 105, or output device 106 according to program 110 to read and write data to main memory unit 102 or auxiliary storage unit 103.

[0022] The program 110 may be provided on a non-temporary recording medium such as a DVD-ROM or semiconductor memory. Alternatively, the program 110 may be provided via a communication network as a data signal superimposed on a carrier wave. The provided program 110 is stored in the auxiliary storage unit 103.

[0023] Each of the model generation system 1 and evaluation system 2 consists of one or more computers. When multiple computers are used in a single system, the multiple computers are connected to each other via a communication network, thereby logically constructing a single system.

[0024] [System operation] (Generation of inference models) Referring to Figure 3, an example of the operation of the model generation system 1 will be described as an example of the model generation method related to this disclosure. Figure 3 is a flowchart showing this example as the processing flow S1.

[0025] In step S11, the prompt acquisition unit 11 acquires a specified prompt for each of the one or more sample operation screens. For example, each specified prompt indicates various information such as context, conditions, and parameters related to the sample image. Each specified prompt may be represented by at least text, or by a combination of text and a reference image, for example.

[0026] In step S12, the image generation unit 12 inputs each of the one or more specified prompts it has received into the image generation model 31 to generate one or more sample images corresponding to one or more sample operation screens. The image generation unit 12 generates one or more sample images by performing the following process for each of the one or more specified prompts. That is, the image generation unit 12 sends the specified prompt to the image generation system 3. The image generation system 3 inputs the specified prompt into the image generation model 31. The image generation model 31 generates a sample image based on the specified prompt, and the image generation system 3 sends the sample image to the model generation system 1. The image generation unit 12 receives the sample image.

[0027] Figure 4 shows an example of a sample image generated based on a specified prompt. In this example, the specified prompt 200 is a prompt for generating a sample image of a sample operation screen for calculating the physical properties of a product by specifying the respective proportions of multiple materials. The image generation unit 12 transmits the specified prompt 200 to the image generation system 3 for processing by the image generation model 31. The image generation model 31 receives the input of the specified prompt 200 and generates a sample image 210 including an input area 211, a confirmation unit 212, a display unit 213, and instruction buttons 214. The image generation unit 12 receives the sample image 210.

[0028] Returning to Figure 3, in step S13, the training data generation unit 13 acquires reference information for each of the one or more reference operation screens, including the subject's biological information, a reference image, and evaluation values. The reference operation screen is a screen used as the basis for the evaluation values ​​set on the sample operation screen. Reference information is prepared in advance for each reference operation screen. The training data generation unit 13 may read the reference information from a predetermined database, or it may accept the reference information entered by the administrator of the model generation system 1. A subject is a person whose biological information during operation of the operation screen is used as training data. In one example, the training data generation unit 13 acquires the frequency components of the subject's electroencephalogram (EEG) as the subject's biological information. These frequency components are obtained by performing a Fourier transform on the EEG. The reference image is an image of the reference operation screen.

[0029] In step S14, the training data generation unit 13 performs clustering on one or more reference images and one or more sample images, and sets an evaluation value for each of the one or more sample operation screens. In one example, the training data generation unit 13 performs principal component analysis (PCA) on the set of one or more reference images and one or more sample images to reduce the number of dimensions related to the image features. Subsequently, the training data generation unit 13 performs clustering on the distribution of one or more reference images and one or more sample images in the space of the reduced number of dimensions, and classifies these images into multiple clusters. For example, the training data generation unit 13 performs clustering using the k-means method. Based on the results of the clustering and the evaluation values ​​for each of the one or more reference operation screens, the training data generation unit 13 sets an evaluation value for each of the one or more sample operation screens.

[0030] Referring to Figures 5 and 6, an example of the process of setting evaluation values ​​for each sample operation screen using reference information will be explained. Figure 5 is a diagram showing an example of a reference screen. Figure 6 is a diagram showing an example of clustering for a set of reference images and sample images.

[0031] In the example shown in Figure 5, the one or more reference operation screens 300 include a first reference operation screen 310 and a second reference operation screen 320. The training data generation unit 13 acquires reference information for each of these two reference operation screens 300.

[0032] The first reference operation screen 310 includes text 311 explaining the operation of the screen and a set 312 of one or more checkboxes for selecting one or more materials. On the first reference operation screen 310, the operation method is explained in text, and multiple checkboxes for selecting materials are arranged haphazardly. In addition, the numerical input field for entering the mixing ratio of materials and the instruction button for calculating the physical properties of the product are not displayed until the user proceeds with the operation. This first reference operation screen 310 is provided as the most difficult operation screen to use, and correspondingly, the first reference operation screen 310 has a minimum evaluation value P. min They are pre-associated.

[0033] The second reference operation screen 320 includes an input area 321 for inputting materials and their proportions for a predetermined category, an accordion 322 for switching the display of the input area 321 on and off, and an instruction button 323 for calculating the physical properties of the product. On the second reference screen, the selectable materials are categorized, and materials can be selected using a combo box. In addition, the numerical input field for inputting the proportions of materials is displayed from the start, corresponding to the combo box, and the instruction button 323 is also displayed from the start. This second reference operation screen 320 is provided as the most user-friendly operation screen, and accordingly, the second reference operation screen 320 has a maximum evaluation value P. max They are pre-associated.

[0034] In the example of FIG. 6, the teacher data generation unit 13 performs clustering on a plurality of images 410 including the reference image of the first reference operation screen 310, the reference image of the second reference operation screen 320, and a plurality of sample images. The teacher data generation unit 13 performs principal component analysis on the set of the plurality of images 410 to reduce the dimensionality of the features of the plurality of images 410 to two dimensions by the principal components Pa and Pb. The teacher data generation unit 13 performs clustering on the distribution of one or more reference images and a plurality of sample images in the two-dimensional space 400, and classifies these images into three clusters 411 to 413. In this example, it is assumed that the cluster 411 includes the reference image of the first reference operation screen 310, and the cluster 413 includes the reference image of the second reference operation screen 320. In this case, the teacher data generation unit 13 sets the evaluation value of the sample operation screen corresponding to the sample image belonging to the cluster 411 to the evaluation value P min of the first reference operation screen 310. Also, the teacher data generation unit 13 sets the evaluation value of the sample operation screen corresponding to the sample image belonging to the cluster 413 to the evaluation value P max of the second reference operation screen 320. Further, the teacher data generation unit 13 sets the evaluation value of the sample operation screen corresponding to the sample image belonging to the cluster 412 to the evaluation value P mid . However, P min <P mid <P max . As in this example, for each of one or more sample operation screens, the teacher data generation unit 13 may set the evaluation value of the sample operation screen within the range from the minimum evaluation value P min to the maximum evaluation value P max . When one of the plurality of images 410 is the sample image 210 in FIG. 4, the sample image 210 is classified into, for example, the cluster 412. In this case, the teacher data generation unit 13 sets the evaluation value P mid to the sample operation screen corresponding to the sample image 210.

[0035] Returning to Figure 3, in step S15, the training data generation unit 13 acquires the subject's biological information during operation on each of the one or more sample operation screens based on one or more sample images. The generation of the sample operation screens based on sample images may be done manually. Alternatively, the sample operation screens may be automatically generated from the sample images by a generation AI for generating operation screens from images, such as Galileo AI, and the training data generation unit 13 may perform this automatic generation. For each of the one or more generated sample operation screens, the subject's biological information during operation on the sample operation screen is measured by a predetermined measuring instrument. The training data generation unit 13 acquires this biological information for each subject. For example, the training data generation unit 13 acquires the frequency components of the subject's electroencephalogram as biological information for each of the one or more sample operation screens.

[0036] In step S16, the training data generation unit 13 generates training data for each of the one or more reference operation screens and one or more sample operation screens, including combinations of biometric information and evaluation values. The training data generation unit 13 stores this training data in the learning database 4. Each data record in the training data represents a combination of biometric information and evaluation value. The evaluation value is used as the ground truth for machine learning.

[0037] In step S17, the learning unit 14 performs machine learning using the training data stored in the learning database 4 to generate an inference model 23. The inference model 23 may be composed of a neural network, in which case the machine learning may be deep learning.

[0038] In one example, the learning unit 14 performs the following processing for each data record in the training data. The learning unit 14 inputs the biometric information shown in the data record into a machine learning model and obtains an inference result of the evaluation value output from the machine learning model. Based on the error between the inference result and the evaluation value (correct answer) shown in the data record, the learning unit 14 updates the parameters in the machine learning model using a method such as backpropagation. The learning unit 14 repeats machine learning while changing the data record until a predetermined termination condition is met. For example, the termination condition may be that all data records in the training data have been processed, or that the error between the inference result and the correct answer has converged to a predetermined threshold. The learning unit 14 sends the machine learning model at the time the machine learning process is completed as an inference model 23 to the evaluation system 2.

[0039] (Evaluation of the target operation screen) Referring to Figure 7, an example of the operation of the evaluation system 2 will be described as an example of the evaluation method related to this disclosure. Figure 7 is a flowchart showing that example as the processing flow S2.

[0040] In step S21, the biometric information acquisition unit 21 acquires the user's biometric information while operating the target operation screen. The user may be the same person as the subject or a different person. The user's biometric information while operating the target operation screen is measured by a predetermined measuring instrument, and the biometric information acquisition unit 21 acquires this biometric information. For example, the biometric information acquisition unit 21 acquires the frequency components of the user's brainwaves as biometric information.

[0041] In step S22, the inference unit 22 inputs the acquired biometric information into the inference model 23 to estimate the evaluation value of the target operation screen. The inference model 23 infers the evaluation value from the biometric information, and the inference unit 22 obtains that evaluation value. The inference unit 22 outputs the inferred evaluation value. The inference unit 22 may display the evaluation value on a display device, store the evaluation value in a given database, or transmit the evaluation value to another computer or computer system.

[0042] [Differentiation] The technology relating to this disclosure has been described in detail above based on various examples. However, this disclosure is not limited to the examples given above. The technology relating to this disclosure can be modified in various ways without departing from its essence.

[0043] The model generation system may set evaluation values ​​for each of the one or more sample operation screens without using reference information from the reference operation screen. Therefore, the training data generation unit does not need to perform the clustering described above. For example, the training data generation unit may set evaluation values ​​for each of the one or more sample operation screens by simply acquiring the evaluation values ​​manually specified for each of the one or more sample operation screens. In this example, the training data generation unit 13 generates training data that includes combinations of biological information and evaluation values ​​for each of the one or more sample operation screens without using information from the reference operation screen.

[0044] When using a first reference operation screen with the minimum evaluation value and a second reference operation screen with the maximum evaluation value, the model generation system may set evaluation values ​​for each of the one or more sample operation screens without performing clustering. In this example, the prompt acquisition unit acquires a prompt as a specified prompt for generating a sample image corresponding to a sample operation screen in which an evaluation value specified within the range from the minimum evaluation value to the maximum evaluation value is set. This specified prompt is an instruction that includes the specified evaluation value. The image generation unit generates a sample image for each of the one or more sample operation screens by inputting the specified prompt to the image generation model. The training data generation unit sets the specified evaluation value for each of the one or more sample operation screens generated based on the one or more sample images. The training data generation unit also acquires the subject's biological information during operation on each of the one or more sample operation screens. The training data generation unit then generates training data that includes combinations of biological information and evaluation values ​​for each of the one or more sample operation screens.

[0045] In the example above, electroencephalogram (EEG) frequency components were shown as an example of biometric information, but other types of biometric information may be used. For example, the biometric information may include changes in gaze, pupil size, skin potential, heart rate, or changes in cerebral blood flow.

[0046] The processing steps for a method executed by at least one processor are not limited to the examples above. For example, some of the steps described above may be omitted, or each step may be performed in a different order. Also, any two or more of the steps described above may be combined, or some of the steps may be modified or deleted. Alternatively, other steps may be performed in addition to each of the steps described above.

[0047] In comparing the relative magnitudes of two numerical values ​​in this disclosure, either of the two criteria, "greater than or equal to" and "greater than," may be used, or either of the two criteria, "less than or equal to" and "less than," may be used.

[0048] In this disclosure, the expression "at least one processor executes a first process, a second process, ... and the nth process," or a corresponding expression, refers to a concept that includes cases where the entity executing the n processes from the first process to the nth process, i.e., the processor, changes along the way. In other words, this expression refers to a concept that includes both cases where all n processes are executed by the same processor and cases where the processor changes at an arbitrary rate for the n processes.

[0049] [Note] As can be seen from the various examples above, this disclosure includes the following aspects: (Note 1) Equipped with at least one processor, The at least one processor, For each of the one or more sample operation screens, obtain a specified prompt to generate a sample image which is an image of that sample operation screen. The image generation model, which generates images based on prompts, receives one or more of the specified prompts as inputs to generate one or more sample images corresponding to the one or more sample operation screens. For each of the one or more sample operation screens mentioned above, an evaluation value related to the operation on the screen is set. For each of the one or more sample operation screens generated based on the one or more sample images, the subject's biological information is acquired during operation on the sample operation screen. Machine learning is performed using training data that includes combinations of biometric information and evaluation values ​​for each of the one or more sample operation screens, to generate an inference model that infers the evaluation value of the target operation screen from the user's biometric information during operation on the target operation screen. Model generation system. (Note 2) The at least one processor, For each of the one or more pre-prepared reference operation screens, the subject's biological information and the evaluation value of the reference operation screen are acquired during operation on that reference operation screen. The machine learning is performed using the training data, which further includes combinations of the biometric information and the evaluation value for each of the one or more reference operation screens, to generate the inference model. The model generation system described in Appendix 1. (Note 3) The one or more reference operation screens mentioned above include a first reference operation screen and a second reference operation screen. The evaluation value of the first reference operation screen is the minimum evaluation value, The evaluation value on the second reference operation screen is the maximum evaluation value, The at least one processor sets the evaluation value of each of the one or more sample operation screens within a range from the minimum evaluation value to the maximum evaluation value. The model generation system described in Appendix 2. (Note 4) The at least one processor, For each of the one or more reference operation screens mentioned above, a reference image which is an image of the reference operation screen is obtained. Perform clustering on the one or more reference images and the one or more sample images. Based on the results of the clustering and the evaluation values ​​of each of the one or more reference operation screens, the evaluation values ​​are set for each of the one or more sample operation screens. The model generation system described in Appendix 2 or 3. (Note 5) The specified prompt is a prompt for generating the sample image corresponding to the sample operation screen in which the evaluation value specified within the range from the minimum evaluation value to the maximum evaluation value is set, The at least one processor, For each of the one or more sample operation screens, the specified prompt is input to the image generation model to generate the sample image. Set the specified evaluation value for each of the one or more sample operation screens mentioned above. The model generation system described in Appendix 3. (Note 6) The at least one processor acquires the frequency components of the subject's brainwaves as at least part of the biological information. A model generation system described in any one of the appendices 1 to 5. (Note 7) Equipped with at least one processor, The at least one processor, The user's biometric information is acquired when the target operation screen is operated. The user's biometric information is input into the inference model generated by the model generation system described in any one of the appendices 1 to 6, and the evaluation value of the target operation screen is inferred. Evaluation system. (Note 8) A model generation method performed by a model generation system having at least one processor, For each of the one or more sample operation screens, the step is to obtain a specified prompt for generating a sample image which is an image of the said sample operation screen, A step of inputting one or more of the specified prompts into an image generation model that generates images based on prompts, and generating one or more sample images corresponding to the one or more sample operation screens, The steps include setting an evaluation value for each of the one or more sample operation screens, For each of the one or more sample operation screens generated based on the one or more sample images, the step of acquiring the subject's biological information during operation on the sample operation screen, The steps include: performing machine learning using training data that includes combinations of biometric information and evaluation values ​​for each of the one or more sample operation screens, to generate an inference model that infers the evaluation value of the target operation screen from the user's biometric information during operation on the target operation screen; A model generation method that includes this. (Note 9) For each of the one or more sample operation screens, the step is to obtain a specified prompt for generating a sample image which is an image of the said sample operation screen, A step of inputting one or more of the specified prompts into an image generation model that generates images based on prompts, and generating one or more sample images corresponding to the one or more sample operation screens, The steps include setting an evaluation value for each of the one or more sample operation screens, For each of the one or more sample operation screens generated based on the one or more sample images, the step of acquiring the subject's biological information during operation on the sample operation screen, The steps include: performing machine learning using training data that includes combinations of biometric information and evaluation values ​​for each of the one or more sample operation screens, to generate an inference model that infers the evaluation value of the target operation screen from the user's biometric information during operation on the target operation screen; A model generation program that causes a computer to execute a model.

[0050] According to appendices 1, 8, and 9, when generating a sample operation screen for acquiring the subject's biological information during operation, the sample image, which is an image of the sample operation screen, is generated by an image generation model. Because the screen image that forms the basis of the sample operation screen is automatically obtained, the generation of the sample operation screen becomes easier. Therefore, subsequent processes such as setting evaluation values ​​for the sample operation screen, acquiring biological information, and generating the inference model can proceed smoothly. As a result, it becomes possible to obtain the inference model faster, and in turn, to evaluate the operation screen using the user's biological information more efficiently.

[0051] According to Appendix 2, the inference model is generated using training data that further includes combinations of the subject's biometric information and evaluation values ​​related to a pre-prepared reference operation screen. By using information related to the reference operation screen, the bias in the inference results generated by the inference model can be reduced. Therefore, a more accurate inference model can be generated.

[0052] According to Appendix 3, the evaluation value of each sample operation screen is set within the range of the minimum and maximum evaluation values ​​defined by the two reference operation screens, thus preventing the evaluation value of each sample operation screen from becoming an outlier. Since the evaluation value of each sample screen is set appropriately, it becomes possible to generate a more accurate inference model.

[0053] According to Appendix 4, the evaluation value of the sample operation screen is set based on the results of clustering the reference image and sample image, and the evaluation value of the reference operation screen. By using clustering, the sample image can be classified automatically and accurately, and as a result, the evaluation value of the sample operation screen can be set with greater precision.

[0054] According to Appendix 5, sample images are generated such that the evaluation value is set within the range from the minimum evaluation value to the maximum evaluation value defined by the two reference operation screens. Since sample images corresponding to the specified evaluation value are generated, it becomes easier to set the evaluation value on the sample operation screen.

[0055] According to Appendix 6, the frequency components of the subject's brainwaves are used as biometric information to generate the inference model. The frequency components of the brainwaves are expected to reflect the state of the subject operating the control screen. Therefore, by using training data that includes the frequency components of the subject's brainwaves, a more accurate inference model can be generated.

[0056] According to Appendix 7, since an efficiently generated inference model is used, the evaluation of the user interface using the user's biometric information can be performed efficiently. [Explanation of Symbols]

[0057] 1...Model generation system, 2...Evaluation system, 3...Image generation system, 4...Training database, 11...Prompt acquisition unit, 12...Image generation unit, 13...Training data generation unit, 14...Training unit, 21...Biometric information acquisition unit, 22...Inference unit, 23...Inference model, 31...Image generation model.

Claims

1. Equipped with at least one processor, The at least one processor, For each of the one or more sample operation screens, obtain a specified prompt to generate a sample image which is an image of that sample operation screen. Each of the one or more specified prompts is input to an image generation model that generates images based on prompts, and one or more sample images corresponding to the one or more sample operation screens are generated. For each of the one or more sample operation screens mentioned above, an evaluation value related to the operation on the screen is set. For each of the one or more sample operation screens generated based on the one or more sample images, the subject's biological information is acquired during operation on the sample operation screen. Machine learning is performed using training data that includes combinations of biometric information and evaluation values ​​for each of the one or more sample operation screens, to generate an inference model that infers the evaluation value of the target operation screen from the user's biometric information during operation on the target operation screen. Model generation system.

2. The at least one processor, For each of the one or more pre-prepared reference operation screens, the subject's biological information and the evaluation value of the reference operation screen are acquired during operation on that reference operation screen. The machine learning is performed using the training data, which further includes combinations of the biometric information and the evaluation value for each of the one or more reference operation screens, to generate the inference model. The model generation system according to claim 1.

3. The one or more reference operation screens mentioned above include a first reference operation screen and a second reference operation screen. The evaluation value of the first reference operation screen is the minimum evaluation value, The evaluation value on the second reference operation screen is the maximum evaluation value, The at least one processor sets the evaluation value of each of the one or more sample operation screens within a range from the minimum evaluation value to the maximum evaluation value. The model generation system according to claim 2.

4. The at least one processor, For each of the one or more reference operation screens mentioned above, a reference image which is an image of the reference operation screen is obtained. Clustering is performed on the one or more reference images and the one or more sample images. Based on the results of the clustering and the evaluation values ​​of each of the one or more reference operation screens, the evaluation values ​​are set for each of the one or more sample operation screens. The model generation system according to claim 2 or 3.

5. The specified prompt is a prompt for generating the sample image corresponding to the sample operation screen in which the evaluation value specified within the range from the minimum evaluation value to the maximum evaluation value is set, The at least one processor, For each of the one or more sample operation screens, the specified prompt is input to the image generation model to generate the sample image. Set the specified evaluation value for each of the one or more sample operation screens mentioned above. The model generation system according to claim 3.

6. The at least one processor acquires the frequency components of the subject's brainwaves as at least part of the biological information. A model generation system according to any one of claims 1 to 3.

7. Equipped with at least one processor, The at least one processor, The user's biometric information is acquired when the target operation screen is operated. The user's biometric information is input into the inference model generated by the model generation system according to any one of claims 1 to 3, and the evaluation value of the target operation screen is inferred. Evaluation system.

8. A model generation method performed by a model generation system comprising at least one processor, For each of the one or more sample operation screens, the step is to obtain a specified prompt for generating a sample image which is an image of the said sample operation screen, The steps include: inputting one or more of the specified prompts into an image generation model that generates images based on prompts, and generating one or more sample images corresponding to one or more sample operation screens; The steps include setting an evaluation value for the operation on each of the one or more sample operation screens, For each of the one or more sample operation screens generated based on the one or more sample images, the step of acquiring the subject's biological information during operation on the sample operation screen, The steps include: performing machine learning using training data that includes combinations of biometric information and evaluation values ​​for each of the one or more sample operation screens, to generate an inference model that infers the evaluation value of the target operation screen from the user's biometric information during operation on the target operation screen; A model generation method that includes this.

9. For each of the one or more sample operation screens, the step is to obtain a specified prompt for generating a sample image which is an image of the said sample operation screen, The steps include: inputting one or more of the specified prompts into an image generation model that generates images based on prompts, and generating one or more sample images corresponding to one or more sample operation screens; The steps include setting an evaluation value for the operation on each of the one or more sample operation screens, For each of the one or more sample operation screens generated based on the one or more sample images, the step of acquiring the subject's biological information during operation on the sample operation screen, The steps include: performing machine learning using training data that includes combinations of biometric information and evaluation values ​​for each of the one or more sample operation screens, to generate an inference model that infers the evaluation value of the target operation screen from the user's biometric information during operation on the target operation screen; A model generation program that causes a computer to execute a model.

Images