Apparatus, method, and program for determining the validity of messages that promote behavioral change.
The device and method enhance message validity determination by fine-tuning a large-scale language model to generate prompts, addressing low accuracy and data insufficiency issues, providing accurate and reasoned judgments for message validity.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- NIPPON TELEGRAPH & TELEPHONE CORP
- Filing Date
- 2024-10-30
- Publication Date
- 2026-05-15
AI Technical Summary
Conventional methods for determining the validity of messages promoting behavior change face challenges with low classification accuracy for short sentences and insufficient accuracy with limited training data, and lack the ability to output the reason for judgment.
A device and method that utilizes a fine-tuned large-scale language model to generate a message discrimination model, incorporating prompts and user conditions, enabling accurate validity determination and outputting the reason for the judgment, even with limited training data.
Enables accurate evaluation of message validity and provides the reason for the determination, improving classification accuracy for short messages and reducing reliance on extensive training data.
Smart Images
Figure 2026079128000001_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an apparatus, a method, and a program for determining the validity of a message that promotes behavior change, that is, determining whether a message that promotes behavior change can actually motivate a user, which are realized by a computer.
Background Art
[0002] Conventionally, for a certain number of messages, by using data determined by a person as to whether or not it is a message that promotes behavior change, it has been determined whether a message related to behavior change actually promotes a user's behavior change. This problem can be considered as a text classification problem.
[0003] One method for this is to represent a message as a word vector by morphological analysis or the like, and further add user attributes (such as age, gender, behavior change stage, etc.) to the vector as necessary to determine whether the message promotes behavior change.
[0004] As an example thereof, a first method using Naive Bayes can be considered (Non-Patent Document 1). In the first method, text is divided into words using a morphological analyzer, and based on this, the text is represented in a bag-of-words model. Then, the text is classified based on the appearance probability of the words. Using such a first method, the validity of the message is determined.
[0005] Also, as disclosed in Non-Patent Document 2, a second method of vectorizing text and performing text classification using an identification model such as a random forest as disclosed in Non-Patent Document 3 can also be considered. According to the second method, by using the technology disclosed in Non-Patent Document 2, vectorization considering meaning can be performed not only as simple keywords, so it is considered that the classification accuracy in short texts is improved.
Prior Art Documents
Non-licensed literature
[0006]
Non-licensed literature 1
Non-licensed Document 2
Non-licensed Document 4
Non-licensed Document 5
Non-licensed Document 6
[0007] However, conventional technologies have the following problems.
[0008] The first method, being a classification based on keyword vectors, may have low classification accuracy in the case of relatively short sentences, such as behavioral change messages.
[0009] Furthermore, the second method, based on Non-Patent Document 2, represents the message as a vector in the semantic space, which may improve accuracy even for relatively short sentences compared to bag-of-words-based methods.
[0010] However, even with the second method, the accuracy may not be sufficiently high if there is insufficient training data. When a message is judged to be valid or invalid, or even when there is sufficient training data, the reason for that judgment cannot be output.
[0011] This invention has been made in view of these circumstances, and aims to provide a device, method, and program for determining the validity of messages that promote behavioral change, which can evaluate the validity of messages even with relatively little training data, and can also output the judgment result of the evaluation of the validity of the message and the reason for it. [Means for solving the problem]
[0012] A first aspect of the present invention for achieving the above objective is a device for determining the validity of a message that promotes behavioral change, comprising: a tuning execution unit that outputs a message discrimination model for determining a message that has the validity to promote behavioral change based on a specified prompt; a message discrimination prompt construction unit that generates a message that is to be determined for validity and a condition indicating the target to whom the message is conveyed, by inputting the message to be determined for validity and a condition indicating the target to whom the message is conveyed into a pre-prepared template that generates a prompt that is to be determined for validity and a prompt that corresponds to the message to be determined for validity and the condition; and a message validity determination unit that inputs the prompt generated by the message discrimination prompt construction unit into a message discrimination model and outputs a determination result of whether or not the message to be determined for validity has the validity to promote behavioral change and the reason for the determination result.
[0013] A second aspect of the present invention is a message validity discrimination device according to the first aspect, wherein the tuning execution unit tunes a language model that generates text based on a specified prompt using specified training data, and outputs a message discrimination model as a result of the tuning.
[0014] A third aspect of the present invention is a message validity determination device according to the second aspect, wherein the training data includes a combination of a pre-prepared message, conditions indicating the target to whom the message is to be conveyed, and results indicating whether or not the message has validity to promote behavioral change.
[0015] A fourth aspect of the present invention is a message validity determination device of the first aspect, further comprising a message receiving unit that receives input of a message to be determined for validity and conditions indicating the target to which the message is conveyed, wherein a message determination prompt construction unit inputs the message and conditions input to the message receiving unit into a pre-prepared template to generate a prompt corresponding to the message and conditions to be determined for validity.
[0016] A fifth aspect of the present invention is a method for determining the validity of a message that promotes behavior modification, in which a processor outputs a message discrimination model for discriminating a message having validity in promoting behavior modification based on a specified prompt, and inputs a message to be discriminated for validity and a condition indicating a target to whom this message is to be conveyed into a prepared template that generates a prompt according to the message and the condition, to generate a prompt according to the message and the condition to be discriminated for validity, and inputs the generated prompt into the message discrimination model to execute a process of outputting a discrimination result as to whether the message to be discriminated for validity has validity in promoting behavior modification and a reason for the discrimination result.
[0017] A sixth aspect of the present invention is a program for causing a computer to function as each part included in the message validity discrimination device according to any one of the first to fourth aspects.
Advantages of the Invention
[0018] According to the device, method, and program for determining the validity of a message that promotes behavior modification of the present invention, in addition to being able to evaluate the validity of a message even with relatively little training data, it is also possible to output a discrimination result of the evaluation of the validity of the message and the reason therefor.
Brief Description of the Drawings
[0019] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of a message validity discrimination device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a data structure diagram showing an example of training data. [Figure 3] FIG. 3 is a diagram showing an example of a template used for creating a prompt. [Figure 4] FIG. 4 is a schematic diagram showing a general hardware configuration of a computer. [Figure 5]FIG. 5 is a flowchart showing an operation example at the time of model construction performed by the message validity determination device according to the present embodiment. [Figure 6] FIG. 6 is a flowchart showing an operation example at the time of message discrimination performed by the message validity determination device according to the present embodiment. MODE FOR CARRYING OUT THE INVENTION
[0020] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the ratio of the sizes between parts, etc. are not necessarily the same as those in reality. Also, even when representing the same part, there are cases where the dimensions and ratios are represented differently depending on the drawings. In the present specification and each drawing, the same reference numerals are given to elements similar to those described in the previously presented drawings, and detailed descriptions and duplicate descriptions are omitted as appropriate.
[0021] FIG. 1 is a block diagram showing an example of the functional configuration of a message validity determination device according to an embodiment of the present invention.
[0022] The message validity determination device 10 illustrated in FIG. 1 is a device to which the message validity determination method according to an embodiment of the present invention is applied, and discriminates the validity of a message that promotes behavioral modification in response to an input from a user made via a client terminal 90. To achieve this, the message validity determination device 10 includes a fine-tuning execution unit 11, a message discrimination task training data storage unit 12, a discrimination reason estimation unit 13, a large language model storage unit 15, a message discrimination model storage unit 16, a discrimination target message reception unit 30, a message validity management unit 40, a message validity discrimination unit 50, a message discrimination prompt construction unit 55, and a validity discrimination result return unit 60. The discrimination target message reception unit 30 and the validity discrimination result return unit 60 can communicate with an external client terminal 90.
[0023] The message discrimination task training data storage unit 12 stores training data b.
[0024] Figure 2 is a data structure diagram showing an example of training data.
[0025] As illustrated in Figure 2, the training data b includes multiple combinations f (e.g., f1, f2, f3...) of a pre-prepared message c, conditions d indicating the target to whom message c should be conveyed (e.g., attributes and status of the identified user), a result e indicating whether message c has validity to promote behavioral change (hereinafter also referred to as "validity determination result e"), and a reason k for validity determination.
[0026] Note that the validity reason k is not mandatory, and the message validity discriminator 10 can determine the validity of a message even if it is not included in the training data b. However, if the validity reason k is included in the training data b, the message validity discriminator 10 may be able to determine the validity of the message with higher accuracy.
[0027] The number of combinations f may be singular, but the larger the number, the higher the accuracy of the message validity determination of the message ultimately output by the message validity determination device 10.
[0028] The discrimination reason estimation unit 13 queries the large-scale language model 14 based on the message c, condition d, and validity discrimination result e in the training data b contained in the message discrimination task training data storage unit 12. In response, the large-scale language model 14 estimates the discrimination reason k for the validity discrimination result, and the discrimination reason estimation unit 13 registers the estimated discrimination reason k in the validity discrimination reason m column of the training data b contained in the message discrimination task training data storage unit 12. Note that the validity discrimination reason m is not limited to being registered according to the estimation result by the large-scale language model 14, but can also be registered manually.
[0029] The large-scale language model storage unit 15 stores the large-scale language model a. The large-scale language model a is a language model capable of generating text in response to prompts. For example, the large-scale language model a can utilize a model that has been trained to output text in the form of a response to a prompt, based on language models such as GTP (Non-Patent Literature 5) or T5 (Non-Patent Literature 6). The prompt is generated by the message discrimination prompt construction unit 55, as will be described later.
[0030] The fine-tuning execution unit 11 accesses the message discrimination task training data storage unit 12 and obtains message c, condition d, and validity discrimination result e from the training data b stored in the message discrimination task training data storage unit 12. The fine-tuning execution unit 11 further constructs a text representation that includes the information of message c and condition d.
[0031] The fine-tuning execution unit 11 further fine-tunes the large-scale language model a stored in the large-scale language model storage unit 15 based on the text representation, the corresponding validity judgment result e, and the validity judgment reason k, generating a message discrimination model g that enables validity judgment for the original model, and outputs it to the message discrimination model storage unit 16. Note that during fine-tuning, the corresponding validity judgment result e and the validity judgment reason k are handled as a set.
[0032] Thus, the fine-tuning execution unit 11 is a means for modifying the large-scale language model a. Fine-tuning can be performed, for example, using the method described in Non-Patent Document 4. In fine-tuning, multiple models may be constructed for each condition d, or condition d may be included as part of message c and constructed as a single model.
[0033] The message discrimination model storage unit 16 stores the message discrimination model g output from the fine-tuning execution unit 11.
[0034] Thus, the fine-tuning execution unit 11, the message discrimination task training data storage unit 12, the large-scale language model storage unit 15, and the message discrimination model storage unit 16 are the parts used for model construction.
[0035] As mentioned above, the message receiving unit 30 and the validity determination result return unit 60 can communicate with an external client terminal 90. The client terminal 90 will be described below.
[0036] The client terminal 90 receives input from a user or software that requires message discrimination, including a message h1 to be validated and a condition h2 indicating the recipient of message h1 (for example, a male in his 20s). The client terminal 90 then transmits this input h, including message h1 and condition h2, to the message receiving unit 30.
[0037] The message receiving unit 30 receives input h transmitted from the client terminal 90 and outputs the received input h to the message validity management unit 40.
[0038] The message validity management unit 40 receives the input h output from the message receiving unit 30 and outputs it to the message validity determination unit 50.
[0039] The message validity determination unit 50 receives the input h output from the message validity management unit 40 and outputs the message determination prompt construction unit 55.
[0040] The message discrimination prompt construction unit 55 receives the input h output from the message validity discrimination unit 50. The message discrimination prompt construction unit 55 then inputs the message h1 and condition h2 contained in the input h into a pre-prepared template T, as illustrated in Figure 3, to create a prompt i, and outputs it to the message validity discrimination unit 50.
[0041] Figure 3 shows an example of a template used to create a prompt.
[0042] When the message validity determination unit 50 receives a prompt i from the message determination prompt construction unit 55, it accesses the message determination model storage unit 16, inputs the prompt i to the message determination model g stored in the message determination model storage unit 16, and generates a response sentence, which is text information. In this way, the message determination model g is used to determine a message h1 that has validity to promote behavioral change based on the specified prompt i.
[0043] The message validity determination unit 50 further identifies the validity determination result j and the reason for determination k of the input message h1 from the response text, which is text information, and outputs the determination result j and the reason for determination k to the message validity management unit 40.
[0044] In this way, the message validity determination unit 50 and the message determination prompt construction unit 55 construct means for determining the validity of message h1 using the message determination model g.
[0045] The message validity management unit 40 receives the discrimination result j and the reason for discrimination k output from the message validity discrimination unit 50 and outputs them to the validity discrimination result return unit 60.
[0046] The validity determination result return unit 60 receives the determination result j and the reason for determination k output from the message validity management unit 40 and transmits them to the client terminal 90.
[0047] The client terminal 90 receives the judgment result j and the reason for the judgment k transmitted from the validity judgment result return unit 60. The user who made the input h can see the validity judgment result j and the reason for the judgment k for message h1 sent to the client terminal 90 and determine whether the message h1 they entered was valid or not.
[0048] Thus, the message discrimination model storage unit 16, the message to be discriminated receiving unit 30, the message validity management unit 40, the message validity discrimination unit 50, the message discrimination prompt construction unit 55, and the validity discrimination result return unit 60 are the parts that perform message discrimination.
[0049] Such a message validity determination device 10 can be implemented using a computer such as a PC.
[0050] Figure 4 is a simplified diagram showing a typical computer hardware configuration.
[0051] Computer 100 receives some kind of input from an external source, processes it, and outputs the result to an external source. Input is handled by input device 120, and output is handled by output device 140. CPU (Central Processing Unit) 110 controls the entire flow of data and processing, and performs calculations and other processing. In Figure 4, solid arrows represent the flow of data, and dashed arrows represent the flow of control.
[0052] For computer 100 to perform processing, it needs a program that describes the processing steps. The program contains a series of instructions and the data used by those instructions, all of which are temporarily stored in memory 130. CPU 110 retrieves the instructions from memory 130 one by one, interprets them, and operates according to those instructions.
[0053] Therefore, the message validity determination device 10 is implemented by the computer 100, through the operation of the CPU 110 according to a program stored in the memory 130.
[0054] In the message validity determination device 10, the message receiving unit 30 corresponds to the input device 120, and the validity determination result return unit 60 corresponds to the output device 140. Furthermore, the fine-tuning execution unit 11, the message validity management unit 40, the message validity determination unit 50, and the message determination prompt construction unit 55 are realized by the CPU 110 operating according to a program stored in the memory 130.
[0055] Furthermore, although not shown in Figure 4, the computer 100 can incorporate a storage device such as an SSD (Solid State Drive) or HDD (Hard Disk Drive). The message discrimination task training data storage unit 12, the large-scale language model storage unit 15, and the message discrimination model storage unit 16 can be realized using such an internal storage device. In addition, the message discrimination task training data storage unit 12, the large-scale language model storage unit 15, and the message discrimination model storage unit 16 are not limited to these internal storage devices, but can also be realized in an external storage medium such as a USB (Universal Serial Bus) memory, or in an area provided in a storage system such as a database server located in the cloud. Note that in Figure 1, the message discrimination task training data storage unit 12, the large-scale language model storage unit 15, and the message discrimination model storage unit 16 are represented as separate storage units for clarity, but they may be physically constructed within the same storage device.
[0056] Next, an example of the operation of the message validity determination device according to this embodiment, configured as described above, will be explained.
[0057] First, we will explain an example of the operation during model construction performed by the fine-tuning execution unit 11, the message discrimination task training data storage unit 12, the large-scale language model storage unit 15, and the message discrimination model storage unit 16.
[0058] Figure 5 is a flowchart showing an example of the operation during model construction performed by the message validity discrimination device according to this embodiment.
[0059] The fine-tuning execution unit 11 accesses the message discrimination task training data storage unit 12 and obtains the message c, condition d, and validity discrimination result e from the training data b contained in the message discrimination task training data storage unit 12 (S1).
[0060] The fine-tuning execution unit 11 further constructs a text representation that includes the information of message c and condition d (S2).
[0061] The fine-tuning execution unit 11 further fine-tunes the large-scale language model a stored in the large-scale language model storage unit 15 based on the text representation and the corresponding validity discrimination result e, generates a message discrimination model g that enables validity discrimination for the original model, and outputs it to the message discrimination model storage unit 16 (S3).
[0062] The message discrimination model g output from the fine-tuning execution unit 11 is stored in the message discrimination model storage unit 16 and used for message discrimination performed by the message validity discrimination unit 50.
[0063] Next, we will describe an example of the operation during message discrimination performed by the message discrimination model storage unit 16, the message receiving unit 30, the message validity management unit 40, the message validity discrimination unit 50, the message discrimination prompt construction unit 55, and the validity discrimination result return unit 60.
[0064] Figure 6 is a flowchart showing an example of the operation performed by the message validity discrimination device according to this embodiment during message discrimination.
[0065] The client terminal 90 receives input from a user or software that requires message discrimination, consisting of a message h1 to be validated and a condition h2 indicating the recipient of this message h1. The client terminal 90 transmits this input h, including the message h1 and condition h2, to the message discrimination receiving unit 30, and the message discrimination receiving unit 30 receives the input h transmitted from the client terminal 90 (S11).
[0066] The message receiving unit 30 outputs the received input h (including message h1 and condition h2) to the message validity management unit 40 (S12).
[0067] The message validity management unit 40 receives the input h (including message h1 and condition h2) output from the message receiving unit 30 to be judged, and outputs it to the message validity determination unit 50 (S13).
[0068] The message validity determination unit 50 receives the input h (including message h1 and condition h2) output from the message validity management unit 40 and outputs it to the message determination prompt construction unit 55 (S14).
[0069] The message discrimination prompt construction unit 55 constructs a message discrimination prompt i by inputting the message h1 and condition h2 contained in the input h into the template T, and outputs it to the message validity discrimination unit 50 (S15).
[0070] When the message validity determination unit 50 receives prompt i from the message determination prompt construction unit 55, it accesses the message determination model storage unit 16 and inputs prompt i to the message determination model g stored in the message determination model storage unit 16 to generate a response sentence consisting of text information (S16).
[0071] The message validity determination unit 50 further identifies the validity determination result j and the reason for determination k of message h1 from the text information of the response, and outputs the determination result j and the reason for determination k to the message validity management unit 40 (S17).
[0072] The message validity management unit 40 receives the discrimination result j and the reason for discrimination k output from the message validity discrimination unit 50 and outputs them to the validity discrimination result return unit 60 (S18).
[0073] The validity determination result return unit 60 receives the determination result j and the determination reason k output from the message validity management unit 40 and transmits them to the client terminal 90 (S19).
[0074] The client terminal 90 receives the judgment result j and the reason for the judgment k transmitted from the validity judgment result return unit 60. The user who made the input h can see the validity judgment result j and the reason for the judgment k for message h1 sent to the client terminal 90 and determine whether the message h1 they entered was valid or not.
[0075] As described above, the message validity discrimination device 10 according to this embodiment allows for the fine-tuning of the large-scale language model a using training data b whose validity has been evaluated by a human, and enables the modification of a portion of the large-scale language model a so that validity evaluation can be performed.
[0076] Then, when a new message h1 to be validated is input by the user via the client terminal 90, the message h1 and the procedure for validating it are used as a prompt i to a finely tuned large-scale language model a, and the validity judgment result for it is received. This makes it possible to obtain the validity evaluation of the message and the reason for it.
[0077] This makes it possible to determine whether message h1 is likely to cause behavioral change, based on the user's attributes and state, using a relatively small amount of training data b, and to output the determination result j along with the reason k for that determination.
[0078] This allows users to determine whether a message h1, generated by a human or machine, is good or bad, and to utilize the judgment result j and the reason for the judgment k for future message c generation.
[0079] The present invention is not limited to the embodiments described above, and in the implementation stage, the components can be modified and implemented without departing from the gist of the invention. That is, the hardware configuration described in the embodiments above is merely an example, and for example, a GPU can be used for computation. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the embodiments above. For example, some components may be deleted from all the components shown in the embodiments. Moreover, components from different embodiments may be appropriately combined. [Explanation of Symbols]
[0080] 10. Message Validation Discriminant 11 Fine Tuning Execution Unit 12 Message Discrimination Task Training Data Storage Unit 13 Discrimination reason estimation part 15. Large-scale language model memory 16 Message discrimination model storage unit 30 Message receiving unit for identification 40 Message Validity Management Department 50 Message Validity Determination Unit 55 Message Discrimination Prompt Construction Unit 60 Validity Determination Result Return Unit 90 client terminals 100 Computers 120 Input devices 130 memory 140 Output device a Large-scale language model b Training data c message d condition e. Validity determination results f combination g Message Discrimination Model h input h1 Message h2 condition i prompt j Discrimination result k Reason for validity determination T template
Claims
1. A device for determining the validity of messages that promote behavioral change, A tuning execution unit outputs a message discrimination model for determining a message that is appropriate to promote the behavioral change based on a specified prompt, A message discrimination prompt construction unit generates prompts according to the message and conditions by inputting the message to be validated and the conditions indicating the target to which the message should be conveyed into a pre-prepared template that generates prompts according to the message and conditions to be validated. A message validity determination unit inputs the prompt generated by the message determination prompt construction unit to the message determination model and outputs a determination result of whether the message targeted for validity determination has validity to promote the behavior change, and the reason for the determination result. A message validity determination device equipped with [the following features].
2. The message validity discrimination device according to claim 1, wherein the tuning execution unit tunes a language model that generates text based on the specified prompt using specified training data, and outputs the message discrimination model as a result of the tuning.
3. The message validity determination device according to claim 2, wherein the training data includes a combination of a pre-prepared message, conditions indicating the target to whom the message is conveyed, and a result indicating whether or not the message is valid for promoting the behavioral change.
4. The system further includes a message receiving unit that receives input of a message to be used for validity determination and conditions indicating the recipient of this message, The message validity determination device according to claim 1, wherein the message determination prompt construction unit inputs the message and conditions input to the message receiving unit into the pre-prepared template to generate a prompt corresponding to the message and conditions to be determined for validity.
5. A method for determining the validity of a message that promotes behavioral change, The processor, Based on the specified prompt, a message discrimination model is output to identify a message that is appropriate for promoting the aforementioned behavioral change. A pre-prepared template is used to generate prompts based on messages and conditions. The message to be validated and the conditions indicating the recipient of this message are input into the template, and prompts based on the message to be validated and the conditions are generated. The generated prompt is input to the message discrimination model to output a determination result of whether the message subject to validity determination has validity to promote behavioral change, and the reason for the determination result. Message validity determination method.
6. A program for causing a computer to function as one of the components of the message validity determination device according to any one of claims 1 to 4.