Generation apparatus, generation method, and generation program

Abstract

This facilitates the proper analysis of customer feedback. [Solution] The generation device 100 extracts words to be used for aggregating customer opinions from the history of interactions with customers, based on predetermined language analysis. The generation device 100 generates supplementary information that supplements the content of the aggregated results using combinations of subject words and predicate words included in the extracted words used for aggregating customer opinions.

Description

Technical Field

[0001] The present invention relates to a generation device, a generation method, and a generation program.

Background Art

[0002] For the purpose of improving products or services, extraction and analysis of customer dissatisfactions, requests, suggestions, etc. (hereinafter sometimes referred to as "customer opinions") regarding the relevant products or services from customers may be performed. As a technique for that purpose, for example, a technique is known in which customer opinions classified for each predetermined customer need based on morphological analysis of document data in which customer opinions are expressed in natural language are calculated as a score indicating the degree of importance using set keywords and their evaluation values (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technology, it may be difficult to appropriately analyze customer opinions. For example, the conventional technology is a technique for calculating a score for classified customer opinions based on set keywords, but it is difficult for anyone other than a person with knowledge of analysis to appropriately aggregate and analyze the customer opinions.

Means for Solving the Problems

[0005] Therefore, in order to solve the above-mentioned problems and achieve the objective, the present invention is characterized by comprising: an extraction unit that extracts words to be used for aggregating customer opinions from the history of interactions with customers based on predetermined language analysis; and a first generation unit that generates supplementary information to supplement the content of the aggregated results, which are aggregated using combinations of subject words and predicate words included in the words used for aggregating customer opinions extracted by the extraction unit. [Effects of the Invention]

[0006] The present invention has the effect of facilitating the appropriate analysis of customer opinions. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 is a diagram illustrating the overall process of the generating apparatus according to the embodiment. [Figure 2] Figure 2 shows the configuration of the generating apparatus according to this embodiment. [Figure 3] Figure 3 is a table diagram showing an example of customer interaction history information according to the embodiment. [Figure 4] Figure 4 is a table diagram showing an example of word information according to the embodiment. [Figure 5] Figure 5 is a table diagram showing an example of dictionary information according to the embodiment. [Figure 6] Figure 6 shows an example of identification based on a large-scale language model according to the embodiment. [Figure 7] Figure 7 shows an example of generating countermeasures based on a large-scale language model according to the embodiment. [Figure 8] Figure 8 shows an example of generating effectiveness determination results based on a large-scale language model according to the embodiment. [Figure 9] Figure 9 shows an example of the predicted effect of the countermeasures according to this embodiment. [Figure 10] Figure 10 shows an example of generating an effect prediction for countermeasures based on a large-scale language model according to the embodiment. [Figure 11]Figure 11 shows an example of an output screen displaying the aggregated results according to the embodiment. [Figure 12] Figure 12 shows an example of a series of processes performed by the generating apparatus according to the embodiment. [Figure 13] Figure 13 shows an example of the generation of supplementary information based on a large-scale language model according to the embodiment. [Figure 14] Figure 14 shows an example of an output screen displaying supplementary information according to the embodiment. [Figure 15] Figure 15 is a flowchart showing the processing performed by the generation apparatus according to the embodiment. [Figure 16] Figure 16 shows an example of a computer that realizes the generation apparatus according to the embodiment. [Modes for carrying out the invention]

[0008] The embodiments for carrying out the present invention (hereinafter referred to as "embodiments") will be described below with reference to the drawings. However, the embodiments are not limited to those described below.

[0009] <Overview> (background) Marketing analysis may be conducted based on customer feedback for the purpose of improving products or services. In such cases, customer feedback is extracted from the interaction history between operators or other customer service representatives and customers, and this extracted feedback is then analyzed.

[0010] As a reference technique for this purpose, for example, there is a known technique that classifies customer opinions from document data containing customer feedback based on morphological analysis, according to predetermined customer needs, and calculates an importance score using set keywords and their evaluation values.

[0011] However, the above-mentioned reference technology is a technology for calculating a score for customer opinions classified based on set keywords, and it is difficult for a person who does not have knowledge about analysis to appropriately aggregate and analyze the customer opinions. That is, in order to appropriately grasp and respond to customer dissatisfactions, requests, suggestions, etc. with respect to services using the aggregation results of customer opinions, it may be difficult for a person who does not have advanced knowledge or experience in the analysis.

[0012] (Processing by the generation apparatus 100) Accordingly, the generation apparatus 100 according to the present embodiment generates information supplementing the background of the customer opinions with respect to the aggregation results of the customer opinions based on "words used for aggregating customer opinions" extracted from the service interaction history or the like, and outputs the information to a user such as a person in charge of analyzing customer opinions.

[0013] In the present embodiment, "words used for aggregating customer opinions" may be hereinafter referred to as "aggregation words". In addition, "supplementary information that supplements the content of the aggregation results aggregated using a combination of a word indicating a subject and a word indicating a predicate" includes information explaining the background of customer opinions based on the aggregation results of customer opinions, and may be simply referred to as "supplementary information" hereinafter. Further, the "word indicating a subject" may be referred to as "subject", and the "word indicating a predicate" may be referred to as "predicate".

[0014] Here, an overall image of processing by the generation apparatus 100 will be described. Figure 1 is a diagram for explaining the overall image of processing of the generation apparatus 100 according to the embodiment. The generation apparatus 100 shown in Figure 1 is an example of a computer that provides a technology for implementing the information processing described below.

[0015] First, the generation apparatus 100 extracts aggregation words from service interaction history information which is a history of interactions with customers ((1) in Figure 1) (S10). Here, the service interaction history information includes, for example, text sentences obtained by transcribing voice-based interactions, or text sentences described in emails or the like.

[0016] The generation device 100 performs an aggregation process of customer opinions based on the extracted aggregation words (S11). Next, the generation device 100 generates supplementary information based on the aggregation results of customer opinions (Figure 1 (2)) (S12).

[0017] The generation device 100 then outputs the generated supplementary information to the user 10 (S13). The supplementary information includes information that supplements the background of the customer's opinion, which has been compiled based on the subject and predicate, such as "why such an event occurred" and "what the customer is dissatisfied with."

[0018] In this way, the generation device 100 according to this embodiment provides the user 10 with supplementary information about the aggregated results based on subjects and predicates, enabling the user 10 to appropriately analyze the aggregated results even if the user 10 does not have advanced knowledge or experience in analyzing customer opinions. Therefore, the generation device 100 has the effect of facilitating the appropriate analysis of customer opinions.

[0019] <Description of Generator 100> The configuration of the generation device 100 according to this embodiment will now be described. Figure 2 is a diagram showing the configuration of the generation device 100 according to this embodiment. As shown in Figure 2, the generation device 100 has a communication unit 110, a storage unit 120, and a control unit 130.

[0020] Although not shown in Figure 2, the generation device 100 may be equipped with an input unit such as a keyboard or mouse to receive input such as operations from an administrator. Furthermore, the generation device 100 may be equipped with a display or the like to show the extracted aggregated words, generated supplementary information, etc., to the administrator.

[0021] (Communications Department 110) The communication unit 110 performs data communication related to input such as response history, instructions for generating policy information including supplementary information, or instructions for aggregating customer opinions. The communication unit 110 also performs data communication related to output such as aggregation words, word sets, countermeasures, effectiveness judgment results, and effect predictions for countermeasures.

[0022] The communication unit 110 is implemented using a NIC (Network Interface Card) or the like, and controls communication via telecommunication lines such as a LAN (Local Area Network) or the Internet. The communication unit 110 can be connected to the network via wired or wireless connection as needed, and can send and receive information bidirectionally with terminal devices operated by the user.

[0023] (Storage unit 120) The storage unit 120 stores data and programs used for various processes by the control unit 130, as well as various data acquired through the operation of the control unit 130. The storage unit 120 is implemented using semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or storage devices such as hard disks and optical discs. As shown in Figure 2, the storage unit 120 also includes a response history information DB 121, a word information DB 122, a dictionary information DB 123, and a model DB 124.

[0024] (Response history information DB121) The interaction history information DB121 is a database that stores interaction history information, which is information related to the history of interactions between the interaction staff and the customer, such as the conversation history between the interaction staff and the customer, information related to the interaction, and customer attribute information.

[0025] Here, an example of customer service history information stored in the customer service history information DB121 will be explained using Figure 3. Figure 3 is a table diagram showing an example of customer service history information according to the embodiment. The customer service history information DB121 stores the individual data items included in the customer service history information, such as "No," "customer service identification information," "conversation history," "classification," "customer service representative," "date and time," "customer age," "customer gender," and "location," as well as the content related to each item, in a table format such as the one shown in Figure 3.

[0026] The "interaction history identification information" mentioned above refers to information that stores individual interaction histories stored in the interaction history information DB121, and may be unique information based on a combination of numbers, letters, symbols, etc. The "conversation history" is a history of conversation between the interaction staff and the customer, and is a natural language text that includes customer opinions such as, "The staff member's service was poor, and I felt uncomfortable." The "classification" is information that indicates the classification of the interaction history, and includes information that indicates the situation of the interaction, such as "customer service," "telephone," or "email."

[0027] "Responsible Person" refers to the person who assisted the customer, and includes, for example, a "store clerk" or "operator." "Date and Time" includes information about the date and time the interaction took place. "Customer Age" and "Customer Gender" are attribute information of the customer who received the assistance. "Location" is information that identifies the location where the interaction took place; for example, if the interaction situation is "customer service," it includes information such as the store name or the store's address.

[0028] The items listed above are merely examples, and other customer attribute information, such as demographic or psychographic data, may also be included.

[0029] (Word information DB122) The word information DB122 is a database that stores word information, including words for aggregation, candidate words for aggregation, and aggregated words. Here, an example of word information stored in the word information DB122 will be explained using Figure 4. Figure 4 is a table diagram showing an example of word information according to the embodiment.

[0030] As shown in Figure 4, the word information DB122 stores information related to each item, such as "No," "Aggregation word," "Candidate word for aggregation," and "Aggregation word," which are pieces of information that identify individual data contained in the word information, in a table format, for example, as shown in Figure 4. The letters "A to C" written in each item in the table diagram shown in Figure 4 are legends for the information contained in each item.

[0031] The "aggregation words" mentioned above are words extracted by the extraction unit 132 described later, and include a subject or predicate used for aggregating customer opinions. The "aggregation candidate words" are words identified by the identification unit 133 described later as candidates for aggregation processing, and are words from the aggregation words mentioned above that satisfy predetermined conditions, such as the conditions for determining whether a word has a variation in spelling. The "aggregated words" are words that aggregate candidate words are aggregated by the aggregation unit 134 described later.

[0032] (Dictionary Information DB123) The dictionary information DB123 is a database that stores dictionary information used by the identification unit 133 (described later) when identifying candidate words for aggregation.

[0033] Here, an example of dictionary information stored in the dictionary information DB123 will be explained using Figure 5. Figure 5 is a table diagram showing an example of dictionary information according to the embodiment. The dictionary information DB123 stores information related to each item, such as the table format shown in Figure 5, which associates information identifying patterns of variations in spelling (patterns) for words that have different pronunciations but the same or similar meanings (hereinafter sometimes referred to as "words with variations in spelling") with information associating these words with each other (association of words with variations in spelling).

[0034] For example, as shown in Figure 5, the dictionary information DB 123 stores, as a pattern of "Kanji-Hiragana-Katakana", associations of notation fluctuation words such as "okyaku-okyaku" or "yokushitsu-yokusou". The above notation fluctuation words are words that have different notations but the same Hiragana reading. Note that other patterns are as shown in Figure 5, and detailed descriptions thereof are omitted.

[0035] (Model DB 124) The model DB 124 is a database that stores predetermined models used for identification processing by an identification unit 136 to be described later and generation processing by a generation unit 136 to be described later. For example, the model DB 124 can store a large-scale language model, a machine learning model, or the like.

[0036] For example, the model DB 124 can store at least one of "ChatGPT (registered trademark)", which is a large-scale language model having general-purpose knowledge, and "tsuzumi (registered trademark)", which is a predetermined large-scale language model subjected to adapter tuning (see, for example, References 1 and 2) as the large-scale language model.

[0037] (Reference 1): ChatGPT (OpenAI), <URL:https: / / openai.com / chatgpt>, <searched on March 7, Reiwa 7> (Reference 2): NTT Large Language Model "tsuzumi", <URL:https: / / www.rd.ntt / research / LLM_tsuzumi.html>, <searched on March 7, Reiwa 7>

[0038] In addition, the model DB 124 can store a machine learning model constructed based on supervised learning, unsupervised learning, reinforcement learning, or the like. That is, the type of model stored by the model DB 124 is not particularly limited.

[0039] (Control Unit 130) Now, let's return to Figure 2 and continue the explanation. The control unit 130 has an internal memory for temporarily storing programs and processing data that define various processing procedures of the generation device 100, and is realized by electronic circuits such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit), and integrated circuits such as an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array). As shown in Figure 2, the control unit 130 has an acquisition unit 131, an extraction unit 132, a specific unit 133, an aggregation unit 134, an aggregation unit 135, a generation unit 136, and an output unit 137.

[0040] (Acquisition part 131) The acquisition unit 131 acquires customer interaction history information transmitted from an external information processing device or the like. The acquisition unit 131 then stores the acquired customer interaction history information in the customer interaction history information database 121. For example, the acquisition unit 131 can acquire customer interaction history information from a terminal device operated by a customer or customer service representative. The acquisition unit 131 can also acquire target customer interaction history information from an on-premise or cloud-based server device where customer interaction history information is stored.

[0041] (Extraction part 132) The extraction unit 132 extracts words for aggregation from the interaction history information (text sentences included in the item "Conversation History" shown in Figure 3) based on predetermined syntactic analysis. Specifically, the extraction unit 132 extracts words for aggregation, such as words indicating subjects, predicates, or modifiers, from the interaction history information based on predetermined syntactic analysis techniques, such as morphological analysis, which are known techniques. The extraction unit 132 then stores the extracted words for aggregation in the word information DB 122 (the item "Words for Aggregation" shown in Figure 4).

[0042] (Specific Section 133) The identification unit 133, if the aggregated words extracted from the interaction history information include words with variations in spelling, excludes words that meet pre-set exclusion conditions to identify candidate words for aggregation. For example, from aggregated words such as "dog, dog, dog, cat, monkey, bird," the identification unit 133 identifies "dog, dog, dog" as a candidate word for aggregation. The identification unit 133 then stores the identified candidate word for aggregation in the word information DB 122 (item "Candidate word for aggregation" shown in Figure 4).

[0043] As mentioned above, when identifying candidate words for aggregation, the identification unit 133 can exclude words that meet pre-set exclusion conditions. For example, the identification unit 133 can exclude words containing honorifics such as "sama" or "san". The identification unit 133 can also exclude words consisting of a single character, except for pre-registered single-character words. Furthermore, the identification unit 133 can exclude words that contain only hiragana characters. In addition, the identification unit 133 can exclude words where two words partially match, for example, the first word "Tokyo Chiyoda-ku" and the second word "Chiyoda-ku".

[0044] Here, an example of the method for identifying candidate words for aggregation performed by the identification unit 133 will be explained. Specifically, the identification unit 133 identifies candidate words for aggregation by combining one or more of the following methods: an identification method based on hiragana reading, an identification method based on similarity, and an identification method based on a large-scale language model.

[0045] For example, the identification unit 133 performs processing using an identification method based on hiragana readings. Specifically, the identification unit 133 retrieves the hiragana text of multiple words to be compared from the storage unit (dictionary information DB) among the multiple aggregation words extracted from customer opinions. Next, the identification unit 133 compares the hiragana text of the retrieved multiple words and identifies words with matching hiragana spellings as aggregation candidate words.

[0046] As an example, when the words of the first word "kyaku" (written in kanji) and the second word "okyaku" (written in hiragana) are included in the aggregation target words, the identifying unit 133 acquires "okyaku" corresponding to "kyaku" in the pattern "kanji, hiragana, katakana" stored in the dictionary information DB. Then, since the hiragana notations of the first word "kyaku" and the second word "okyaku" are both "okyaku", the identifying unit 133 identifies the first word and the second word as aggregation candidate words.

[0047] Further, the identifying unit 133 executes processing using an identification method based on similarity. Specifically, the identifying unit 133 calculates the similarity for a plurality of words to be compared among the plurality of aggregation target words extracted from customer opinions. For example, the identifying unit 133 executes at least one of calculating the proximity of vectors between words as the similarity, and calculating the number of predetermined character string operations performed until the first word and the second word among the plurality of words become the same character string, as the similarity.

[0048] As an example, the identifying unit 133 converts the first word and the second word into vectors based on the Sentence BERT (Bidirectional Encoder Representations from Transformers) method. Then, the identifying unit 133 can compare the vector of the first word and the vector of the second word, and calculate the proximity (distance) between the vectors as the similarity.

[0049] Further, based on the Levenshtein distance method, the identifying unit 133 can calculate, as the similarity, the number of operations such as character insertion (addition), deletion, and replacement performed between the first word and the second word until both become the same character string.

[0050] The identification unit 133 then identifies words whose calculated similarity levels satisfy pre-set conditions as candidate words for aggregation. For example, the identification unit 133 can identify the first and second words as candidate words for aggregation if the calculated similarity level between them exceeds a pre-set threshold. Note that the "pre-set conditions" are not limited to thresholds and may include other conditions such as weighting of similarity levels.

[0051] The "first word" and "second word" mentioned above are merely examples; two or more words may be identified as candidate words for aggregation, such as "third word," "fourth word," ... "nth word."

[0052] Furthermore, the identification unit 133 executes processing using an identification method based on a large-scale language model. Here, an example of identification based on a large-scale language model will be explained with reference to Figure 6. Figure 6 is a diagram showing an example of identification based on a large-scale language model according to the embodiment.

[0053] First, the identification unit 133 retrieves multiple aggregation words extracted from customer opinions and stored in the word information DB. Next, the identification unit 133 generates a prompt (S20) in which the retrieved multiple aggregation words (Figure 6 (1-1)) and a command to identify identical or similar words from among the multiple aggregation words as aggregation candidate words (Figure 6 (1-2)) are expressed in natural language.

[0054] Next, the identification unit 133 inputs the generated prompt into the large-scale language model (S21). Then, the identification unit 133 obtains the generation result from the large-scale language model (S22). Based on the generation result obtained from the large-scale language model, the identification unit 133 identifies candidate words for aggregation. Then, as shown in (2) of Figure 6, the identification unit 133 lists the candidate words for aggregation.

[0055] (Aggregation section 134) The aggregation unit 134 aggregates identical or similar words based on the comparison result of the identified aggregation candidate words to generate aggregated words. Then, the aggregation unit 134 stores the generated aggregated words in the word information DB 122 (item "aggregated word" shown in FIG. 4).

[0056] Specifically, the aggregation unit 134 aggregates identical or similar words based on the combination of aggregation candidate words designated by the user to generate aggregated words. For example, the aggregation unit 134 presents the combination of aggregation candidate words to the user. Then, the aggregation unit 134 can aggregate the aggregation candidate words based on the word combination designated by the user based on the presentation to generate aggregated words.

[0057] Further, the aggregation unit 134 aggregates words whose comparison result of aggregation candidate words exceeds a predetermined threshold to generate aggregated words. Specifically, when the similarity or the like between aggregation candidate words exceeds a predetermined threshold, the aggregation unit 134 can aggregate the aggregation candidate words to generate aggregated words. For example, in the case of words such as "dog", "DOG", and "dog" (different Japanese notations), the aggregation unit 134 can aggregate the respective words into "dog".

[0058] (Aggregation unit 135) The aggregation unit 135 aggregates customer opinions using at least one of the aggregation target words and the aggregated words. Specifically, the aggregation unit 135 counts the number of customer opinions that include the subject or predicate contained in the aggregation target words or the aggregated words.

[0059] (Generation unit 136) The generation unit 136 generates measure information including supplementary information. The generation unit 136 includes a first generation unit 1361 that generates supplementary information, and a second generation unit 1362 that generates measure information other than supplementary information.

[0060] (First generation unit 1361) The first generation unit 1361 generates supplementary information that supplements the content of the aggregated results, which are aggregated using subject-predicate combinations included in the aggregated words extracted by the extraction unit 132. Furthermore, the first generation unit 1361 generates supplementary information for the aggregated results filtered based on predetermined category designations, as supplementary information that reflects the results of the predetermined category designations.

[0061] (Second generation unit 1362) The second generation unit 1362 generates policy information, including countermeasures, effectiveness assessment results, or predictions of the effects of countermeasures, based on the subject-predicate combinations included in the aggregated words extracted by the extraction unit 132.

[0062] For example, the second generation unit 1362 can generate countermeasures to address complaints and other issues included in customer feedback based on a large-scale language model. Here, an example of the generation of countermeasures based on a large-scale language model will be explained using Figure 7. Figure 7 is a diagram showing an example of the generation of countermeasures based on a large-scale language model according to the embodiment.

[0063] First, the second generation unit 1362 obtains information to be used to generate countermeasures (for example, the information described below) from the storage unit 120, etc. (not shown in Figure 7). The information used to generate the countermeasures described below may be information entered by the user or information collected from an external information processing device, etc. Also, the strings written in {} are variables to be inserted into the corresponding parts of the prompt. Industry information for the target company (e.g., "retail"): {gyoukai} Selected subject (e.g., "raw materials"): {arg} Selected predicate (e.g., "I want to know"): {pred} Attribute classification including customer opinions (e.g., "gender") {category_group} Customer attributes including opinions (e.g., "female") {category}

[0064] Next, the second generation unit 1362 uses the subject and predicate and supplementary information to generate a prompt (Figure 7(1)) in which a command to generate a response to the customer's opinion is expressed in natural language (S30).

[0065] The second generation unit 1362 inputs the generated prompt into the large-scale language model (S31). The second generation unit 1362 then obtains the generation results from the large-scale language model (S32). Based on the generation results obtained from the large-scale language model, the second generation unit 1362 generates countermeasures as policy information (Figure 7(2)).

[0066] Furthermore, the second generation unit 1362 can generate an effectiveness determination result based on a large-scale language model, indicating whether or not the countermeasures to address complaints and other issues included in customer feedback are effective. Here, an example of generating an effectiveness determination result based on a large-scale language model will be explained using Figure 8. Figure 8 is a diagram showing an example of generating an effectiveness determination result based on a large-scale language model according to the embodiment.

[0067] First, the second generation unit 1362 obtains information used to generate the effectiveness determination result from the storage unit 120, etc. (not shown in Figure 8). Note that the information used to generate the effectiveness determination result is the same as the "information used to generate countermeasures" explained using Figure 7, so the explanation is omitted here.

[0068] Next, the second generation unit 1362 generates a prompt (Figure 8(1)) in which a command to generate a determination result of whether or not the countermeasure shows a predetermined effect is expressed in natural language (S40).

[0069] The second generation unit 1362 inputs the generated prompt into the large-scale language model (S41). The second generation unit 1362 then obtains the generation result from the large-scale language model (S42). Based on the generation result obtained from the large-scale language model, the second generation unit 1362 generates the effectiveness determination result as policy information (Figure 8 (2)).

[0070] Furthermore, the second generation unit 1362 can generate predictions of the effectiveness of countermeasures to address complaints and other issues included in customer feedback. The flow of predicting the effectiveness of countermeasures will now be explained using Figure 9. Figure 9 shows an example of the results of predicting the effectiveness of countermeasures according to the embodiment.

[0071] First, the acquisition unit 131 acquires policy information registered by the user, including the name of the countermeasure, associated subject and predicate, and the start date of the countermeasure (S50). Next, the second generation unit 1362 performs "aggregation of the number of customer opinions (S51-1)" and "prediction of the decrease in the number of opinions (S51-2)" for each set unit period (for example, one week, one month, etc.) (S51). Then, the second generation unit 1362 generates the prediction results as shown in (1) of Figure 9 (S52).

[0072] Here, we will explain in detail the "prediction of the decrease in the number of opinions (S51-2)". Specifically, the second generation unit 1362 generates policy information that predicts the effectiveness of countermeasures during the target period, using aggregated results based on subject-predicate combinations.

[0073] For example, the second generation unit 1362 generates a prediction of the effectiveness of countermeasures during the target period based on a predetermined regression model (S51-2a). The regression model mentioned above may be, for example, a known linear regression model or a known nonlinear regression model, and is not particularly limited.

[0074] Furthermore, the second generation unit 1362 can generate predictions of the effectiveness of countermeasures based on a large-scale language model (S51-2b). Here, an example of generating predictions of the effectiveness of countermeasures based on a large-scale language model will be explained using Figure 10. Figure 10 is a diagram showing an example of generating predictions of the effectiveness of countermeasures based on a large-scale language model according to the embodiment.

[0075] First, the second generation unit 1362 acquires information used to generate the prediction of the effectiveness of countermeasures from the storage unit 120, etc. (not shown in Figure 10). The information used to generate the prediction of the effectiveness of countermeasures may be acquired based on, for example, information input by users or aggregated results of past customer opinions.

[0076] The second generation unit 1362 generates a prompt (Figure 10 (1)) in which a command to generate an effect prediction of countermeasures during the target period is expressed in natural language, based on the fluctuation in the number of customer opinions during the unit period and attribute information related to the customer opinions (S60).

[0077] Next, the second generation unit 1362 inputs the generated prompt into the large-scale language model (S61). Then, the second generation unit 1362 obtains the generation results from the large-scale language model (S62). Based on the generation results obtained from the large-scale language model, the second generation unit 1362 generates a prediction of the effectiveness of the countermeasures (Figure 10 (2)).

[0078] (Output section 137) The output unit 137 outputs the aggregated results based on the aggregated words or aggregated words. Here, an example of the output screen will be explained using Figure 11. Figure 11 is a diagram showing an example of the output screen displaying the aggregated results according to the embodiment.

[0079] As shown in Figure 11, the generation device 100 (output unit) outputs the aggregated results to the terminal device 200 operated by the user (S70). The generation device 100 (output unit) then displays the output screen 20 on the terminal device 200 operated by the user.

[0080] For example, the generation device 100 (output unit) can display the period to be aggregated in area (1) of Figure 11 on the output screen 20. In addition, the generation device 100 (output unit) can display selection tabs for display formats such as "word cloud or table," "one level or two levels," and "what (subject) - what happened (predicate), or what happened (predicate) - what (subject)" in area (2) of Figure 11 on the output screen 20.

[0081] Furthermore, the generation device 100 (output unit) can display a list of the aggregated results of customer opinions containing a subject (what) and a predicate (what happened) in area (3) of Figure 11 of the output screen 20. In addition, the generation device 100 (output unit) can display an input window in area (4) of Figure 11 of the output screen 20 for filtering customer opinions to be included in the aggregation based on customer attribute information, etc.

[0082] (An example of processing) From here, an example of processing by the generation device 100 will be explained using Figure 12. Figure 12 is a diagram showing an example of a series of processes by the generation device 100 according to the embodiment.

[0083] The generation device 100 (acquisition unit) acquires the interaction history information from an external information processing device, etc. (S80). For example, the interaction history information includes "text of the telephone conversation as recorded by speech recognition," "text transcribed by the telephone operator," or "text written in an email," as shown in (1) of Figure 12.

[0084] The generation device 100 (extraction unit) performs syntactic analysis on the text sentences included in the interaction history information and extracts words for aggregation (S81). If there are words with variations in spelling among the words for aggregation, the generation device 100 (aggregation unit) identifies candidate words for aggregation and performs aggregation processing (S82). Next, the generation device 100 (aggregation unit) performs aggregation processing of customer opinions based on the subject and predicate specified by the user from among the words for aggregation or aggregation words (S83).

[0085] The generation device 100 (first generation unit) performs a process to generate supplementary information as a generation process (S84-1). Specifically, the generation device 100 (first generation unit) inputs a prompt, in which the command to generate supplementary information is expressed in natural language, into a large-scale language model and generates the supplementary information.

[0086] Here, using Figure 13, we will explain an example of generating supplementary information based on a large-scale language model. Figure 13 is a diagram showing an example of generating supplementary information based on a large-scale language model according to the embodiment.

[0087] First, the generation device 100 (generation unit) obtains information to be used for generating supplementary information (for example, the information described below) from the storage unit 120, etc. (not shown in Figure 13). The information used for generating the supplementary information below may be information entered by the user or information collected from an external information processing device, etc. Also, the strings written in {} are variables to be inserted into the corresponding parts of the prompt. Number of inquiries (e.g., "Number of customer opinions"): {opinions_num} Subject-predicate pairs (e.g., "product - not delivered"): {opinion} Maximum number of characters for an LLM tag: {tag_length} Maximum number of characters for background information {description_length}

[0088] Next, the generation device 100 (first generation unit) generates a prompt (S90) which includes a command to summarize the customer's opinion using at least one of the subject of the customer's opinion and the reason why the customer's opinion arose, a command to assign a predetermined tag to the summary content, and a command to aggregate the summary content to which the predetermined tag has been assigned.

[0089] The "instruction to assign a predetermined tag to the summary content" mentioned above is, for example, the instruction to generate the "LLM tag" shown in area (1-1) of Figure 13. The "instruction to summarize customer opinions using at least one of the subject of the customer opinion and the reason why the customer opinion arose" mentioned above is, for example, the instruction to generate the "LLM supplementary information" shown in area (1-2) of Figure 13. The "instruction to aggregate the summary content to which the predetermined tag has been assigned" mentioned above is, for example, the instruction to aggregate the "number of relevant opinions" shown in area (1-3) of Figure 13.

[0090] The generation device 100 (first generation unit) inputs the generated prompt into the large-scale language model (S91). The generation device 100 (first generation unit) then obtains the generation result from the large-scale language model (S92). Based on the generation result obtained from the large-scale language model, the generation device 100 (first generation unit) generates supplementary information (Figure 13 (2)).

[0091] Now, let's return to Figure 12 and continue explaining an example of a series of processes performed by the generation device 100. Furthermore, the generation device 100 (second generation unit) can perform the generation of countermeasures (S84-2), the determination of effectiveness (S84-3), and the generation of predictions of the effectiveness of the countermeasures (S84-4).

[0092] The generation device 100 (output unit) then outputs policy information to the user 10 that includes at least one of the following: the generated supplementary information (Figure 12 (2-1)), the generated countermeasures (Figure 12 (2-2)), the generated effectiveness judgment result (Figure 12 (2-3)), and the generated effect prediction of the countermeasures (Figure 12 (2-4)) (S85).

[0093] As a result, the generator 100 provides the user with a "word set," making it easy for the user to specify "subject and predicate combinations" when performing analysis. Furthermore, the generator 100 provides the user with "countermeasures," making it easy for the user to formulate appropriate countermeasures when taking action against negative customer feedback. In addition, the generator 100 provides the user with "effectiveness judgment results," making it easy for the user to appropriately evaluate the effectiveness of countermeasures when implementing countermeasures against negative customer feedback or when conducting post-implementation analysis.

[0094] Here, an example of the output of supplementary information will be explained using Figure 14. Figure 14 is a diagram showing an example of an output screen displaying supplementary information according to the embodiment.

[0095] For example, the generation device 100 (output unit) can output supplementary information such as "In-depth analysis" to area (2) of Figure 14 in response to a user pressing a button such as "Analyze with selected opinion" shown in area (1) of Figure 14. The generation device 100 (output unit) can also display the supplementary information in descending order of the total number of items for each LLM tag.

[0096] As an example, the generating device 100 (output unit) can output supplementary information such as the LLM tag "Delivery company misdelivery" and background information "There were many reports of undelivered or damaged goods due to errors by the delivery company, such as incorrect delivery addresses or not arriving on time," to area (2) of Figure 14 on the output screen 30, based on the aggregated results calculated by specifying the subject "product" and the predicate "did not arrive."

[0097] (Processing procedure by the generating device 100) Next, the processing procedure implemented by the generation device 100 according to this embodiment will be explained using Figure 15. Figure 15 is a flowchart showing the processing performed by the generation device 100 according to this embodiment.

[0098] First, the acquisition unit 131 acquires the interaction history information (S101). Next, the extraction unit 132 extracts words for aggregation from the interaction history information (S102).

[0099] If aggregation processing is to be performed (Yes in S103), the aggregation unit 134 generates aggregated words (S104). If aggregation processing is not to be performed (No in S103), the generation device 100 skips step S104.

[0100] The aggregation unit 135 performs aggregation processing using aggregation words or aggregation words (S105). Next, the generation unit 136 generates supplementary information (S106). Subsequently, the output unit 137 outputs the aggregation results and supplementary information (S107). Then, the generation device 100 terminates processing.

[0101] (effect) Next, we will explain the effects of the generation device 100 according to this embodiment. Conventionally, in order to appropriately understand and respond to customer dissatisfaction, requests, suggestions, etc., regarding services using aggregated customer opinions, it was often difficult unless the person had advanced knowledge or experience in such analysis.

[0102] Therefore, the extraction unit 132 of the generation device 100 according to this embodiment extracts words for aggregation from the response history information based on predetermined language analysis. The first generation unit 1361 of the generation device 100 generates supplementary information that supplements the content of the aggregation result aggregated using subject-predicate combinations included in the words for aggregation extracted by the extraction unit 132.

[0103] Therefore, the generation device 100 according to this embodiment generates supplementary information including tags summarizing the aggregated results and background information, thereby facilitating the appropriate analysis of customer opinions even when there is no advanced knowledge or experience in analyzing customer opinions.

[0104] Furthermore, the generation apparatus 100 according to this embodiment achieves predetermined effects by performing the processes described below.

[0105] The first generation unit 1361 inputs a prompt, expressed in natural language, which is a command to generate supplementary information, into a large-scale language model to generate supplementary information. Through the above-described process, the generation device 100 can dynamically generate supplementary information according to various aggregate results by using a large-scale language model. As a result, the generation device 100 has the effect of enabling users to easily analyze customer opinions by referring to the supplementary information, even if the user does not have advanced knowledge or experience in analyzing customer opinions.

[0106] The first generation unit 1361 generates supplementary information using prompts that include a command to summarize the customer's opinion using at least one of the subject of the customer's opinion and the reason why the customer's opinion arose, a command to assign predetermined tags to the summary content, and a command to aggregate the summary content to which the predetermined tags have been assigned.

[0107] Through the process described above, the generation device 100 can provide the user with supplementary information, including a tag that briefly summarizes the aggregated results, background information associated with the tag, and information such as the proportion of the background information to the whole, by using the prompt described above.

[0108] The first generation unit 1361 generates supplementary information for the aggregated results filtered based on predetermined category designations, as supplementary information that reflects the results of the predetermined category designations. Through the above-described process, the generation device 100 can generate supplementary information corresponding to customer opinions extracted according to the category designations. In other words, the generation device 100 can easily generate supplementary information corresponding to desired customer attributes such as gender and age.

[0109] The second generation unit 1362 inputs a prompt, expressed in natural language, which is a command to generate a response to customer feedback using a subject, predicate, and supplementary information, into a large-scale language model to generate a response to customer feedback.

[0110] Through the process described above, the generation device 100 has the effect of enabling users to properly analyze customer opinions, even when it is difficult for them to formulate appropriate countermeasures because they do not have advanced knowledge or experience in analyzing customer opinions, by having the large-scale language model generate countermeasures.

[0111] The second generation unit 1362 inputs a prompt, expressed in natural language, which is a command to generate a determination result on whether or not the countermeasures for customer feedback show a predetermined effect, into a large-scale language model, and generates a determination result on whether or not the countermeasures for customer feedback show a predetermined effect.

[0112] Through the process described above, the generating device 100 has the effect of enabling users to appropriately implement countermeasures to improve customer dissatisfaction, etc., even when it is difficult for the user to judge whether the countermeasures they have devised are appropriate because they do not have advanced knowledge or experience in analyzing customer opinions, by having a large-scale language model determine the effectiveness of the countermeasures.

[0113] <Variation> The following describes modifications that can be implemented by the generation apparatus 100 according to this embodiment.

[0114] (Data, etc.) The correspondence history information, aggregation words, aggregation candidate words, aggregation words, aggregation, aggregation results, dictionary information, identification method based on hiragana reading, identification method based on similarity, identification method based on large-scale language model, word set, countermeasures, effectiveness judgment results, predicted effectiveness of countermeasures, supplementary information, background information, LLM tags, LLM supplementary information, number of relevant opinions, names of functional parts of the generation device 100, steps, processes, names of steps or processes, etc., used in the description of the above embodiment are merely examples and can be changed at will.

[0115] For example, the correspondence history information DB121 was explained to store information by associating each of the following items and their contents, such as "No," which is information that identifies individual data included in the correspondence history information, with "Corresponding Person," "Corresponding History Identification Information," "Conversation History," "Classification," "Corresponding Person," "Date and Time," "Customer Age," "Customer Gender," and "Location," in a table format such as the one shown in Figure 3. However, the items and the information within each item that are stored are not limited to those shown in Figure 3. The word information DB122 was explained to store information by associating each of the following items, such as "No," which is information that identifies individual data included in the word information, with "Aggregation Word," "Candidate Aggregation Word," and "Aggregation Word," in a table format such as the one shown in Figure 4. However, the items and the information within each item that are stored are not limited to those shown in Figure 4. As explained earlier, the dictionary information DB123 stores information related to each item, such as the table format shown in Figure 5, by associating information that identifies the patterns of variations in spelling for words with variations in spelling (patterns) with information that associates the words with variations in spelling (association of words with variations in spelling). However, the items stored and the information within each item are not limited to those shown in Figure 5.

[0116] (Regarding the use of large-scale language models) In this embodiment, the large-scale language model used by the generation device 100 is described as being stored in the model DB 124 of the storage unit 120, but this is not limited to this. For example, the generation device 100 can access an external information processing device (such as a server) and use a predetermined model.

[0117] (Flowcharts, etc.) In flowcharts, each step may be rearranged as long as it does not create inconsistencies, and some steps may be omitted. Furthermore, conjunctions such as "next," "continue," "in addition," "at this time," and "on this occasion" in flowchart descriptions do not limit the order or timing of the processes in the flowchart.

[0118] <Hardware Configuration> Each component of the illustrated device is a functional concept and does not necessarily have to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. Furthermore, each processing function performed by each device can be implemented, all or any part of it, by a CPU and the program that is analyzed and executed by that CPU, or by hardware using wired logic.

[0119] Furthermore, among the processes described in this embodiment, all or part of those described as being performed automatically can be performed manually using known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the drawings can be arbitrarily changed unless otherwise specified.

[0120] <Program> In one embodiment, the various devices constituting the generation device 100 can be implemented by installing the generation program as packaged software or online software on a desired computer. For example, by having the above generation program executed on an information processing device, the various devices constituting the generation device 100 can be made to function. The information processing device referred to here includes desktop or notebook personal computers. In addition, the information processing device also includes mobile communication terminals such as smartphones and mobile phones, and slate terminals such as PDAs (Personal Digital Assistants).

[0121] Figure 16 shows an example of a computer that implements the generation apparatus 100 according to the embodiment. The computer 1000 has, for example, memory 1010 and CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0122] Memory 1010 includes ROM (Read Only Memory) 1011 and RAM 1012. ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0123] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define the various processes of the various devices constituting the generation device 100 are implemented as program modules 1093 in which executable code for a computer is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, a program module 1093 for performing processes similar to the functional configuration of the various devices constituting the generation device 100 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0124] Furthermore, the configuration data used in the processing of the embodiment described above is stored as program data 1094 in, for example, memory 1010 or hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes the processing of the embodiment described above.

[0125] Furthermore, the program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (LAN, WAN (Wide Area Network), etc.). The program module 1093 and program data 1094 may then be read from the other computer by the CPU 1020 via a network interface 1070.

[0126] <Other> Although this embodiment has been described above, this embodiment is not limited by the description and drawings that constitute part of the disclosure. That is, all other embodiments, examples, and operational techniques made by those skilled in the art based on this embodiment are included in the scope of this embodiment. [Explanation of Symbols]

[0127] 100 generator 110 Communications Department 120 Storage section 121 Customer Service History Information Database 122 Word Information Database 123 Dictionary Information Database 124 Model DB 130 Control Unit 131 Acquisition Department 132 Extraction part 133 Specific part 134 Aggregation Section 135 Aggregation Department 136 Generation part 1361 1st generation part 1362 Second generation part 137 Output section

Claims

1. An extraction unit that extracts words to be used for aggregating customer opinions from the history of customer interactions based on predetermined language analysis, A first generation unit generates supplementary information that supplements the content of the aggregated results, using combinations of subject words and predicate words included in the words used to aggregate the customer opinions extracted by the extraction unit. A generating apparatus characterized by having the following features.

2. The first generation unit is, The command to generate the supplementary information involves inputting a prompt expressed in natural language into a large-scale language model to generate the supplementary information. The generating apparatus according to feature 1.

3. The first generation unit is, A command to summarize the customer's opinion using at least one of the subject of the customer's opinion and the reason why the customer's opinion arose, A command to assign a predetermined tag to the summary content, A command to aggregate the summary content to which the predetermined tags have been assigned, The supplementary information is generated using the prompt which includes the following: The generating apparatus according to feature 2.

4. The first generation unit is, Supplementary information for aggregated results filtered based on specified categories, The results of the aforementioned predetermined category specification are generated as supplementary information. The generating apparatus according to any one of claims 1 to 3.

5. Using the subject-indicating word, the preceding descriptive word, and the supplementary information, a prompt expressing a command to generate a response to the customer's opinion in natural language is input to a large-scale language model. The system further includes a second generation unit that generates countermeasures for the aforementioned customer feedback. The generating apparatus according to any one of claims 1 to 3.

6. The second generation unit is, A command to generate a determination result regarding whether the countermeasures for the aforementioned customer opinions show a predetermined effect is input to the large-scale language model as a prompt expressed in natural language. The system generates a determination result indicating whether the countermeasures taken in response to the aforementioned customer feedback demonstrate a predetermined effect. The generating apparatus according to feature 5.

7. A generation method to be executed by a generation device, An extraction unit that extracts words to be used for aggregating customer opinions from the history of customer interactions based on predetermined language analysis, A first generation unit generates supplementary information that supplements the content of the aggregated results, using combinations of subject words and predicate words included in the words used to aggregate the customer opinions extracted by the extraction unit. A method for generating a product, characterized by including the following:

8. An extraction unit that extracts words to be used for aggregating customer opinions from the history of customer interactions based on predetermined language analysis, A first generation unit generates supplementary information that supplements the content of the aggregated results, using combinations of subject words and predicate words included in the words used to aggregate the customer opinions extracted by the extraction unit. A generation program that causes a computer to execute something.

Images