Sentence classification device and sentence classification method

Abstract

The present invention accurately classifies a sentence if sentences are classified by using a generative AI model.　A sentence classification device 10 comprises: an acquisition unit 11 that acquires information indicating a sentence to be classified, a plurality of sentence examples divided into a plurality of groups, and an example of classification of each sentence example into any of a plurality of classification types; a model usage unit 12 that acquires, for each group and from a generative AI model 21, a classification result for the sentence to be classified, by inputting, into the generative AI model 21, the sentence to be classified, a sentence example included in the group, and the classification example for each sentence example, which were acquired by the acquisition unit 11; and a determination unit 13 that determines a final classification of the sentence to be classified from the classification result for each group acquired by the model usage unit 12.

Description

Sentence classification device and sentence classification method

[0001] The present invention relates to a sentence classification device and a sentence classification method for classifying sentences.

[0002] Patent Document 1 discloses a technique for assigning labels indicating classification to sentences through machine learning.

[0003] JP 2018-97468 A

[0004] It is conceivable to classify sentences using a generative AI (artificial intelligence) model such as an LLM (large-scale language model). For example, it is conceivable to input prepared example sentences and classification examples for the example sentences as reference information into the LLM along with the sentences to be classified, and have the LLM classify the sentences to be classified according to the reference information. However, since an LLM typically has a limit on the tokens that can be input, it may not be possible to input all of the prepared reference information into the LLM. If only a portion of the reference information is used, the accuracy of sentence classification may be reduced.

[0005] One embodiment of the present invention has been made in consideration of the above, and aims to provide a sentence classification device and a sentence classification method that can perform accurate classification when classifying sentences using a generative AI model.

[0006] In order to achieve the above object, a sentence classification device according to one embodiment of the present invention comprises an acquisition unit that acquires information indicating a sentence to be classified, a plurality of example sentences divided into a plurality of groups, and an example classification for each example sentence into one of a plurality of classification types; a model utilization unit that inputs, for each group, the sentence to be classified acquired by the acquisition unit, as well as the example sentences included in the groups and the example classification for each example sentence, into a generative AI model and acquires a classification result for the sentence to be classified from the generative AI model; and a judgment unit that determines the final classification of the sentence to be classified from the classification result for each group acquired by the model utilization unit.

[0007] In a sentence classification device according to one embodiment of the present invention, a plurality of example sentences divided into a plurality of groups and classification examples for each example sentence are used, a classification result for the sentence to be classified is obtained for each group from the generative AI model, and a final classification of the sentence to be classified is determined. Therefore, even if all example sentences and classification examples for each example sentence cannot be input to the generative AI model at once due to input limitations on the generative AI model, it is possible to classify sentences taking into account all example sentences and classification examples for each example sentence. Therefore, according to a sentence classification device according to one embodiment of the present invention, accurate classification can be performed when classifying sentences using a generative AI model.

[0008] Incidentally, one embodiment of the present invention can be described not only as an invention of a sentence classification device as described above, but also as an invention of a sentence classification method as described below. These are essentially the same inventions, with the only difference being their categories, and they have similar functions and effects.

[0009] That is, a sentence classification method according to one embodiment of the present invention includes an acquisition step in which the sentence classification device acquires information indicating a sentence to be classified, a plurality of example sentences divided into a plurality of groups, and a classification example for each example sentence into one of a plurality of classification types; a model utilization step in which the sentence classification device inputs, for each group, the sentence to be classified acquired in the acquisition step, as well as the example sentences included in the groups and the classification example for each example sentence, into a generative AI model to acquire a classification result for the sentence to be classified from the generative AI model; and a determination step in which the sentence classification device determines the final classification of the sentence to be classified from the classification result for each group acquired in the model utilization step.

[0010] According to one embodiment of the present invention, when classifying sentences using a generative AI model, classification can be performed with high accuracy.

[0011] FIG. 1 is a diagram showing the configuration of a sentence classification device according to an embodiment of the present invention. FIG. 2 is a diagram showing an overview of processing by the sentence classification device. FIG. 3 is a diagram showing an example of a prompt generated in the sentence classification device. FIG. 4 is a flowchart showing a sentence classification method which is processing executed in the sentence classification device according to an embodiment of the present invention. FIG. 5 is a flowchart showing an example of processing for determining the final classification of sentences to be classified from classification results for each group. FIG. 6 is a diagram showing the hardware configuration of a sentence classification device according to an embodiment of the present invention.

[0012] Hereinafter, an embodiment of a sentence classification device and a sentence classification method according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals and duplicated explanations will be omitted.

[0013] FIG. 1 shows a sentence classification device 10 according to this embodiment. The sentence classification device 10 is a device (system) that classifies a sentence to be classified into one of a plurality of classification types. For example, the sentence classification device 10 is used as follows by a business operator that provides services to customers, such as a telecommunications carrier that provides communication functions to customers.

[0014] A customer or the like asks a business operator a question (inquiry) about a service provided by the business operator. The question may be, for example, asked verbally over the telephone or sent as a text via a communication network. The text to be classified by the text classification device 10 is the question text (question text). The multiple classification types into which the question is classified are set in advance. Hereinafter, the classification type is also referred to as a classification label. The multiple classification labels indicate the type of the question. By classifying the question text into classification types, it is possible to respond appropriately and quickly to the question from the customer or the like.

[0015] The question may be, for example, "Calls tend to be interrupted." The classification labels may be, for example, "Smartphone malfunction," "X company's service," "Call trouble," "Calls cannot be connected," and "Cannot make or receive calls."

[0016] Note that the sentences to be classified and the multiple classification types (classification labels) used in classification by the sentence classification device 10 do not have to be those described above and may be any. Furthermore, classification by the sentence classification device 10 may be performed for purposes other than those described above.

[0017] Classification by the sentence classification device 10 is performed using a generative AI model. The generative AI model is a model that can generate content in response to a prompt input based on any one or a combination of the instructions, context, question, and output format indicated by the prompt, and return the content as response information. The prompt can also include input information, in which case the generative AI model generates response information targeted at the input information. The generative AI model may be, for example, an interactive AI model that includes an LLM and a user interface (UI) for interaction with a user, enabling text-based or voice-based chat with the user. Examples of such generative AI models include ChatGPT, GPT (registered trademark)-3.5, GPT-4V, PaLM2, etc.

[0018] A prompt is a set of instructions or information input to a generative AI model. The prompt may include initial information, parameters, questions, etc., for the generative AI model to execute a specific task. A prompt is information indicating an instruction or question entered by a user in an interactive system, such as a dialogue with a generative AI model or a command line interface (CLI). The prompt may express, through text, for example, the command to be executed by the interactive AI model, the task to be executed by the interactive AI model, the background / context (e.g., role, condition) that the interactive AI model should consider, the question to be answered by the interactive AI model, and the output format of the response information from the interactive AI model. The prompt may also include input information that is the target of the command / task executed by the interactive AI model. Examples of such input information include data files with file names containing a predetermined extension, such as text data, image data, application-related data, audio data, video data, and still image data. Application-related data is data such as document data, table data, and graph data that can be processed by a default application program.

[0019] In this embodiment, the generative AI model 21 is realized by an AI server (server device) 20. The generative AI model 21 may be stored in the AI ​​server 20, or may be stored in another device connected to the AI ​​server 20 via a network and made available by the AI ​​server 20. Also, although the above describes an example of an LLM, other AI models may also be used. Note that the generative AI model 21 and the AI ​​server 20 may be similar to conventional models.

[0020] The generative AI model 21 according to this embodiment has an upper limit (input limit) on the number of tokens or characters that can be input at one time, similar to conventional models. That is, the number of tokens and characters of a prompt input to the generative AI model 21 must be within the above-mentioned input limit. A token is a unit of character string handled by the generative AI model 21.

[0021] The sentence classification device 10 is configured by a computer such as a PC (personal computer) or a server device. The sentence classification device 10 may also be configured by multiple computers. The sentence classification device 10 can exchange information with other devices, including the AI ​​server 20, via a network to acquire information necessary to realize its functions. The sentence classification device 10 may also be a Retrieval-Augmented Generation (RAG) system. In this case, the knowledge database searched by the RAG system may be implemented within the sentence classification device 10 or may be implemented in another device on a network (e.g., the cloud).

[0022] Next, the functions of the sentence classification device 10 according to this embodiment will be described. As shown in Fig. 1, the sentence classification device 10 is configured to include an acquisition unit 11, a model utilization unit 12, and a judgment unit 13. Fig. 2 shows an overview of the processing performed by the sentence classification device 10 according to this embodiment. The functions of the sentence classification device 10 according to this embodiment will be described with reference to this figure as necessary.

[0023] The acquisition unit 11 is a functional unit that acquires information indicating a sentence to be classified, a plurality of example sentences divided into a plurality of groups, and a classification example into one of a plurality of classification types for each example sentence.

[0024] The acquisition unit 11 performs acquisition, for example, as follows. The acquisition unit 11 receives and acquires the sentences to be classified that are transmitted from a terminal of a user of the sentence classification device 10 (for example, the above-mentioned business operator). The acquisition unit 11 may also acquire the sentences to be classified by a method other than the above. Note that one classification target may include multiple sentences.

[0025] The example sentences and the classification examples of the example sentences into one of the multiple classification labels (classification types) are information that serve as criteria for the generative AI model 21 when classifying the sentences to be classified. In other words, the example sentences and classification examples are reference information when classifying sentences. Like the sentences to be classified, the example sentences are sentences that can be classified into one of the multiple classification labels set above in advance. For example, if the sentence to be classified is a question, the example sentence is also a question.

[0026] The information indicating a classification example is information indicating which of the plurality of preset classification labels the example sentence has been classified into. A classification example is paired with each example sentence. The classification of the example sentence (generation of the classification example) may be performed manually by a user of the sentence classification device 10 or may be performed by the sentence classification device 10. The classification example may be one in which one example sentence has been classified into a plurality of classification labels. The information indicating the classification example corresponding to the example sentence is associated with the example sentence and prepared in advance by a user of the sentence classification device 10 or the like.

[0027] Each example sentence is assigned information indicating the classification example corresponding to that example sentence. The number of example sentences is set to a number sufficient to appropriately classify the sentences to be classified. Example sentences are prepared for all of the classification labels that are set in advance. In other words, each of the classification labels is assigned to at least one example sentence.

[0028] Generally, the more example sentences there are, the more appropriate the criteria for classifying the sentences to be classified become, and therefore the accuracy of the classification of the sentences to be classified increases. The total number of tokens and the number of characters of all example sentences (used to classify the sentences to be classified) acquired by the acquisition unit 11 and the information indicating the classification examples corresponding to the example sentences may exceed the above-mentioned input limit of the generative AI model 21.

[0029] The multiple example sentences are grouped into multiple groups. That is, each of the multiple example sentences belongs to one of the multiple groups. The example sentences included in the group are used as input to the generative AI model 21 at one time.

[0030] As described below, when a prompt to be input to the generative AI model 21 is generated using an example sentence belonging to one group, the number of tokens and the number of characters in the prompt are set so as not to exceed the above-mentioned input limits of the generative AI model 21. However, from the viewpoint of performing appropriate classification within one group, one group may contain as many example sentences as possible, and the number of groups may be as small as possible. Furthermore, each group may contain at least one document associated with every classification label.

[0031] Information indicating example sentences for each group and the classification labels (categorization examples) assigned to the example sentences is prepared in advance by, for example, a user of the sentence classification device 10. As shown in FIG. 2 , a database for each group (categorization example set 1, classification example set 2, ..., classification example set N shown in FIG. 2 ) is prepared in advance. Information indicating example sentences included in a group and the classification labels assigned to the example sentences is stored in the database. The database is included in the sentence classification device 10 or is accessible from the sentence classification device 10. The acquisition unit 11 reads and acquires information indicating example sentences for each group and the classification labels assigned to the example sentences from the database.

[0032] Alternatively, the acquiring unit 11 may acquire information indicating a plurality of ungrouped example sentences and classification labels assigned to the example sentences, and group the acquired example sentences. In this case, for example, the acquiring unit 11 stores in advance a grouping rule for grouping example sentences such that each group after grouping satisfies the above, and groups a plurality of example sentences and information indicating classification labels assigned to the example sentences based on the grouping rule.

[0033] The acquiring unit 11 may also acquire information indicating the sentences to be classified, the example sentences grouped into the groups, and the classification labels assigned to the example sentences by a method other than the above. The acquiring unit 11 outputs each piece of acquired information to the model utilizing unit 12.

[0034] The model utilization unit 12 is a functional unit that inputs, for each group, the sentences to be classified acquired by the acquisition unit 11, as well as the example sentences included in the group and the classification examples for each example sentence, into the generative AI model and acquires the classification results for the sentences to be classified from the generative AI model. The model utilization unit 12 may select an example sentence to be used to acquire the classification results again from the example sentences according to the classification results for each group, and input the sentences to be classified, the example sentences, and the classification examples for each example sentence into the generative AI model again to acquire the classification results.

[0035] The model utilization unit 12 acquires the classification results of the sentence to be classified from the generative AI model 21, for example, as follows: The model utilization unit 12 inputs information indicating the sentence to be classified, multiple example sentences divided into multiple groups, and classification labels assigned to the example sentences from the acquisition unit 11. The model utilization unit 12 also acquires classification labels, which are information indicating multiple classification types into which the sentence to be classified is to be classified. For example, as shown in FIG. 2, a database of classification labels (a set of classification labels shown in FIG. 2) is prepared in advance. The model utilization unit 12 reads and acquires the classification labels from the database.

[0036] 2, based on this information, the model utilization unit 12 generates prompts (prompt 1 instructing classification, prompt 2 instructing classification, ..., prompt N instructing classification shown in FIG. 2) for each group to obtain the classification results of the sentences to be classified from the generative AI model 21, and inputs the prompts to the generative AI. The model utilization unit 12 obtains the classification results of the sentences to be classified from the generative AI model 21 for each group.

[0037] The prompt includes information indicating the sentence to be classified, as well as example sentences belonging to the group and classification labels assigned to the example sentences. The information indicating the example sentences and classification labels assigned to the example sentences included in the prompt provides context for the generative AI model 21.

[0038] 3 shows an example of a prompt generated by the model utilization unit 12. The model utilization unit 12 stores rules and information (e.g., prompt templates) for generating prompts in advance, and generates a prompt from the rules and information, sentences to be classified, example sentences belonging to groups, and information indicating example classifications to be assigned to the example sentences.

[0039] The prompt includes information indicating an instruction to classify the sentence to be classified into one of a plurality of classification labels. In the example shown in FIG. 3 , the information is the following section: "#Instruction You are a professional in smartphones and tablets from Company X. Please classify the customer's question into an answer category based on the following #constraints." The "customer's question" in the above section corresponds to the sentence to be classified. The sentence to be classified itself (the content of the sentence to be classified) is shown in the section following the prompt. The answer category in the above section corresponds to the classification label. The classification label itself (the content of the classification label) is shown in the section following the prompt. The section in question is a fixed phrase, and the model utilization unit 12 generates the section using pre-stored information.

[0040] The prompt contains information indicating the conditions for how to classify the sentence to be classified. In the example shown in Figure 3, the information is as follows: "#Constraints - The customer's question must be classified into one of the following answer categories: "Smartphone malfunction, X company's service, call trouble, call not connected, unable to make or receive calls, access restriction service, remote support, ..., manned chat explanation screen, not included in this, after the survey." - The customer's question must be avoided as much as possible to be classified into "not included in this." - The output must be in accordance with the #format, referring to the #classification example."

[0041] In the above section, each of the following items is a classification label: "Smartphone malfunction, X company's service, call trouble, call not connecting, unable to make or receive calls, access restriction service, remote support, ..., manned chat explanation screen, not included in this, after the survey."

[0042] The above section includes information that takes into consideration the example sentences included in the section following the prompt and the information indicating the classification labels to be assigned to those example sentences. The above section also includes information that outputs according to the output format included in the section following the prompt. The information in question is "Output must be done according to the # format, with reference to the # classification examples."

[0043] The model utilization unit 12 generates the part by, for example, embedding information indicating a classification label at a preset position in a template, which is pre-stored information. Alternatively, the model utilization unit 12 may store in advance classification labels that should not be classified (for example, the classification label "not in this" in the above example), and include information in the classification label that indicates that classification should be avoided as much as possible.

[0044] The prompt includes information indicating the format of the output from the generative AI model 21. In the example shown in FIG. 3, this information is the section "#format customer question (input), answer category (output)". In the above example, the format of the output from the generative AI model 21 is a comma-separated list of the sentence to be classified (customer question) and information indicating the classified classification label (answer category). This section is a fixed phrase, and the model utilization unit 12 generates this section using pre-stored information.

[0045] The prompt includes information indicating a sample sentence and a classification label assigned to the sample sentence. In the example shown in FIG. 3, the information is the following portion: "#Classification example: smartphone not working, smartphone malfunction service, X company's service... questionnaire, after the questionnaire." In the above example, each line after "#Classification example" includes a sample sentence and information indicating a classification label, which is a classification example assigned to the sample sentence, separated by a comma. In the above example, the sample sentence "smartphone not working" is assigned the classification label "smartphone malfunction." Furthermore, the sample sentence "service" is assigned the classification label "X company's service." Furthermore, the sample sentence "questionnaire" is assigned the classification label "after the questionnaire."

[0046] The model utilization unit 12 generates the portion by, for example, embedding information indicating a sample sentence and a classification label to be assigned to the sample sentence at a predetermined position in a template sentence, which is pre-stored information. Note that for the same classification of sentences to be classified, the portion will be different for each prompt in the group. Also, for the same classification of sentences to be classified, the rest of the portion will be common to all prompts in the group.

[0047] The prompt includes the sentence to be classified. In the example shown in FIG. 3, the information is the portion "#input: conversations tend to be interrupted,". In the above example, "conversations tend to be interrupted" is the sentence to be classified. The model utilization unit 12 generates the portion by, for example, embedding the sentence to be classified at a predetermined position in a fixed phrase, which is pre-stored information.

[0048] The model utilization unit 12 transmits the generated prompt for each group to the AI ​​server 20 and inputs it to the generative AI model 21. The generative AI model 21 accepts the input of the prompt and classifies the sentence to be classified into one of the classification labels in accordance with the prompt. The generative AI model 21 transmits information indicating the classified classification label, which is the classification result of the sentence to be classified, to the sentence classification device 10 in response to the transmitted prompt. The model utilization unit 12 receives and acquires the information indicating the classification result of the sentence to be classified transmitted from the generative AI model 21. The classification result of the sentence to be classified is acquired for each group. For example, if there are seven groups as shown in FIG. 2, the model utilization unit 12 acquires seven classification results.

[0049] The prompts input to the generative AI model 21 do not necessarily have to be the ones described above, but may be any prompts that can input information indicating the sentences to be classified, as well as example sentences included in the group and the classification labels assigned to the example sentences, to the generative AI model 21, and obtain the classification results for the sentences to be classified from the generative AI model 21. The model utilization unit 12 outputs information indicating the classification results for each group obtained from the generative AI model 21 to the determination unit 13.

[0050] The determination unit 13 is a functional unit that determines the final classification of the sentence to be classified based on the classification results for each group obtained by the model utilization unit 12. The determination unit 13 may determine the final classification of the sentence to be classified based on the number of classification results for each classification type. The determination unit 13 may determine the final classification of the sentence to be classified by majority vote. The determination unit 13 may compare the number of classification results for each classification type with a preset threshold to determine the final classification of the sentence to be classified.

[0051] The judgment unit 13 judges the final classification of the sentence to be classified, for example, as follows. The judgment unit 13 receives information indicating the classification results for each group by the generative AI model 21 from the model utilization unit 12. The judgment unit 13 judges the final classification of the sentence to be classified from the classification results for each group. The final classification is output from the sentence classification device 10 as the classification result for the sentence to be classified. The judgment unit 13 determines a classification label to be assigned to the sentence to be classified as the final classification of the sentence to be classified. The classification label to be assigned to the sentence to be classified may be one classification label (single label) or multiple classification labels (multi-label). Whether the classification label to be assigned to the sentence to be classified is only one classification label or multiple classification labels is set in advance.

[0052] The determination unit 13 counts the number of classification results for each classification label (classification type) from the classification results for each group. The determination unit 13 determines the final classification of the sentence to be classified by majority vote. That is, the determination unit 13 determines the classification label with the largest number among the counted numbers for each classification label as the classification label to be assigned to the sentence to be classified. For example, in the example shown in FIG. 2 , out of the seven classification results, "call trouble," which is the most common classification result with four, is determined as the classification label to be assigned to the sentence to be classified.

[0053] Alternatively, the determination unit 13 may determine a classification label to be assigned to the sentence to be classified as follows, based on the counted number of each classification label. The determination unit 13 compares the count number of the classification label with the largest count number with a preset threshold. If the count number of the classification label is not equal to or greater than the threshold, the determination unit 13 determines that no classification label is to be assigned to the sentence to be classified. That is, in this case, the determination unit 13 determines that the sentence to be classified cannot be classified into the classification label. This is because if the count number is less than a certain number, it is considered that it is not appropriate to assign any classification label to the sentence to be classified. The above threshold is set in advance based on this perspective.

[0054] If the count number of the classification label is equal to or greater than the threshold, the following judgment is subsequently made to determine the classification label to be assigned to the sentence to be classified. If it is set that multiple classification labels may be assigned to the sentence to be classified (in the case of multi-label), the judgment unit 13 determines the classification label whose count number is equal to or greater than the threshold as the classification label to be assigned to the sentence to be classified. Note that in this case, the classification label assigned to the sentence to be classified may be one or more.

[0055] If it is set that only one classification label is assigned to the sentence to be classified (single label case), the determination unit 13 determines whether the classification label with the largest count is 1. If there is 1 classification label, the determination unit 13 determines that this classification label is the classification label to be assigned to the sentence to be classified.

[0056] If the classification label is not one, i.e., if there are multiple classification labels, the model utilization unit 12 again obtains a classification result using the generative AI model 21. In this case, the model utilization unit 12 selects, from the multiple example sentences used to obtain the classification result, one to be used to again obtain a classification result using the generative AI model 21. The model utilization unit 12 selects the example sentence to which the largest number of classification labels is assigned (i.e., the classification example has the largest number of classification labels assigned) as the example to be used to again obtain a classification result using the generative AI model 21. The model utilization unit 12 obtains a classification result using the generative AI model 21 in the same manner as described above, using the selected example sentence and the classification examples assigned to the example sentence.

[0057] When a classification result is obtained again using the generative AI model 21, the group of example sentences (and the classification examples assigned to those example sentences) may be the same as the group obtained the previous time. In this case, the example sentences included in the group will only be those to which the classification label with the largest count is assigned. In other words, the example sentences included in the group will exclude the one to which the classification label with the largest count is assigned.

[0058] Alternatively, a group of example sentences (and example categories assigned to the example sentences) may be newly generated using the selected example sentences (and example categories assigned to the example sentences). The grouping of example sentences and example categories assigned to the example sentences may be performed as described above.

[0059] In this case, the model utilization unit 12 uses only the classification label with the largest count as the classification destination to obtain a classification result again using the generative AI model 21. As described above, the classification label is embedded in the information in the prompt that indicates the conditions for how to classify the sentence to be classified.

[0060] The classification result obtained by using the generative AI model 21 again may be obtained in the same manner as described above, except for the example sentences, the classification examples assigned to the example sentences, and the classification labels to be classified. Once the classification result obtained by using the generative AI model 21 again by the model utilization unit 12 is obtained, the determination unit 13 determines the final classification of the sentence to be classified from the classification result in the same manner as described above.

[0061] The final classification of the sentences to be classified by the determining unit 13 does not necessarily have to be determined as described above, but may be determined based on the classification results for each group.

[0062] The determination unit 13 outputs information indicating the final classification result of the sentence to be classified, for example, information indicating the classification label determined to be assigned to the sentence to be classified. For example, the determination unit 13 transmits the information to the terminal of the user of the sentence classification device 10, which is the sender of the sentence to be classified. The determination unit 13 may also output the information in a manner other than the above. The functions of the sentence classification device 10 according to this embodiment have been described above.

[0063] Next, a sentence classification method, which is a process executed by the sentence classification device 10 according to this embodiment (a method of operation performed by the sentence classification device 10), will be described using the flowchart of FIG. In this process, first, the acquisition unit 11 acquires information indicating a sentence to be classified, a plurality of example sentences divided into a plurality of groups, and a classification example into one of a plurality of classification types for each example sentence (S01, acquisition step). Next, the model utilization unit 12 generates a prompt for acquiring a classification result of the sentence to be classified from the generative AI model 21 for each group (S02, model utilization step). The prompt includes the information acquired by the acquisition unit 11. Next, the model utilization unit 12 inputs the prompt for each group to the generative AI model 21, and acquires a classification result of the sentence to be classified from the generative AI model (S03, model utilization step).

[0064] Next, the determination unit 13 determines the final classification of the sentence to be classified from the classification results for each group acquired by the model utilization unit 12 (S04, determination step). Next, the determination unit 13 outputs information indicating the final classification of the sentence to be classified (S05). The above is the sentence classification method, which is the processing executed by the sentence classification device 10 according to this embodiment.

[0065] Next, an example of the process (S04) by the determination unit 13 for determining the final classification of the sentence to be classified from the classification results for each group obtained by the model utilization unit 12 will be described using the flowchart of Fig. 5. In addition, here, as shown in Fig. 5, a case where seven classification results are obtained by the model utilization unit 12 will be described as an example.

[0066] In this process, the number of classification results for each classification label (classification type) is counted from the classification results for each group. Next, the count for the classification label with the largest count is compared with a preset threshold (S41). If the count for that classification label is less than the threshold ("No" in S41), it is determined that no classification label is assigned to the sentence to be classified (S42).

[0067] If the count number of the classification label is not equal to or greater than the threshold value ("Yes" in S41), the following determination is then made. If it is set that multiple classification labels may be assigned to the sentence to be classified ("Multiple" in S43), the classification label whose count number is equal to or greater than the threshold value is determined as the classification label to be assigned to the sentence to be classified (S44). If there are seven classification results as shown in Figure 5 and the threshold value is 2, the classification labels to be assigned to the sentence to be classified will be "call trouble" and "call not connecting."

[0068] If it is set that only one classification label is to be assigned to the sentence to be classified ("Single" in S43), it is determined whether the classification label with the largest count is one (S45). If there is one classification label ("Yes" in S45), that classification label is determined as the classification label to be assigned to the sentence to be classified (S46). If there are seven classification results as shown in Figure 5, the classification label to be assigned to the sentence to be classified is "Call trouble."

[0069] If the classification label is not one, i.e., if there are multiple classification labels ("No" in S45), the model utilization unit 12 again obtains a classification result using the generative AI model 21 (S47). In this case, from the multiple example sentences used to obtain the classification result, the example sentence to which the classification label with the largest count is assigned is selected and used to obtain the classification result again using the generative AI model 21. For example, if the classification labels "call trouble" and "call not connecting" have the largest and equal counts, only those two classification labels and the example sentences to which they are assigned are used to obtain the classification result again using the generative AI model 21. After obtaining the classification result again using the generative AI model 21, the determination unit 13 makes a determination similar to that described above. The above is an example of processing by the determination unit 13.

[0070] In this embodiment, multiple example sentences divided into multiple groups and classification examples for each example sentence are used, and classification results for the sentences to be classified are obtained for each group from the generative AI model 21, and the final classification of the sentences to be classified is determined. Therefore, even if all example sentences and classification examples for each example sentence cannot be input to the generative AI model 21 at once due to input limitations on the generative AI model 21, it is possible to classify the sentences taking into account all example sentences and classification examples for each example sentence. For example, as described above, by setting the number of tokens and characters in the prompt for each group within the input limitations of the generative AI model 21, it is possible to take into account all example sentences and classification examples for each example sentence, regardless of the input limitations of the generative AI model 21. Therefore, according to this embodiment, accurate classification can be achieved when classifying sentences using a generative AI model.

[0071] As in the present embodiment, the determination unit 13 may determine the final classification of the sentence to be classified according to the number of classification results for each classification type. Specifically, the determination unit 13 may determine the final classification of the sentence to be classified by majority vote. Alternatively, the determination unit 13 may determine the final classification of the sentence to be classified by comparing the number of classification results for each classification type with a preset threshold. This configuration enables reliable and appropriate classification of sentences. For example, by performing majority vote, it is possible to reduce fluctuations in the answers (classifications) of the generative AI model 21.

[0072] However, the final classification of the sentences to be classified by the determination unit 13 only needs to be determined based on the classification results for each group, and does not necessarily have to be determined based on the number of classification results for each classification type as described above. Also, even when the final classification of the sentences to be classified by the determination unit 13 is determined based on the number of classification results for each classification type, the determination may be made by a method other than the above.

[0073] As in the present embodiment, the model utilization unit 12 may select an example sentence to be used to obtain a reclassification result from the example sentences according to the classification result for each group, and may again input the sentences to be classified, the example sentences, and the classification examples for each example sentence into the generative AI model 21 to obtain the classification result. This configuration allows appropriate classification of sentences. However, it is not necessarily required to obtain a reclassification result using the generative AI model 21.

[0074] In the above embodiment, the sentence classification device 10 is a PC or a server device separate from the terminal of the user of the sentence classification device 10, but is not limited to this. For example, the sentence classification device 10 may be part of the terminal. As an example, a RAG system configured by the functional units of the sentence classification device 10 may be implemented in the terminal. In this case, the knowledge database searched by the RAG system may be implemented inside the terminal, or may be implemented in another device on a network (e.g., the cloud).

[0075] Furthermore, the sentence classification device 10 may have the functions of the AI ​​server 20. In this case, the configuration can be realized by installing an application that executes the functions of the AI ​​server 20 in the sentence classification device 10. In this way, the generative AI model 21 may be implemented on a network (e.g., the cloud) other than the AI ​​server 20.

[0076] When the judgment unit 13 determines the final classification of the sentence to be classified by majority vote, the model utilization unit 12 may set multiple people with different roles for each group in the generative AI model 21 and obtain classification results for the sentences for each of the multiple people. The setting of multiple people with different roles in the generative AI model 21 is performed by including information about the setting in a prompt generated by the model utilization unit 12 and input to the generative AI model 21. The information about the setting includes information indicating the role. The information about the role may be information composed of any of the name of the evaluator (the person with the set role), information specifying a person or character, values, attributes (age, gender), job title, occupation, company name, address, hometown, preference information, or information about the persona.

[0077] The model utilization unit 12 sets people with multiple different roles (for example, Mr. A: 27-year-old system engineer (male), Ms. B: 31-year-old manicurist (female), Ms. C: 18-year-old student (male), ..., Ms. J: 27-year-old system engineer (female)), obtains classification results for sentences for each set person, and combines the classification results for each person into a single classification result. In other words, the model utilization unit 12 obtains multiple classification results for each of multiple groups according to the people with multiple roles. The judgment unit 13 determines the final classification of the sentence to be classified from these classification results by majority vote, as described above. Note that information about roles may be stored in a knowledge database. Information about many roles may be stored in a knowledge database, and the model utilization unit 12 may randomly extract the number of roles from there.

[0078] For example, the prompt in this case includes the following information as information for setting people with different roles in the generative AI model and information for causing the generative AI model 21 to classify each of the people: "Roles: Mr. A: 27-year-old system engineer (male), Ms. B: 31-year-old manicurist (female), Ms. C: 18-year-old student (male), ..., Ms. J: 27-year-old system engineer (female) Task: Please become each person and classify the sentence to be classified into one of the answer categories."

[0079] As described above, by obtaining classification results for sentences for each of a plurality of people with different roles, classification that reflects the classification according to the position of each of a plurality of people can be performed.

[0080] The block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of hardware and / or software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are directly or indirectly connected (e.g., wired, wireless, etc.) and these multiple devices. The functional block may also be realized by combining software with the single device or multiple devices.

[0081] Functions include, but are not limited to, judgment, determination, judgment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assignment. For example, a functional block (component) that performs transmission is called a transmitting unit or transmitter. As mentioned above, there are no particular limitations on how these functions are implemented.

[0082] For example, the sentence classification device 10 according to an embodiment of the present disclosure may function as a computer that performs the information processing of the present disclosure. Fig. 6 is a diagram illustrating an example of the hardware configuration of the sentence classification device 10 according to an embodiment of the present disclosure. The above-described sentence classification device 10 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like. The hardware configuration of the AI ​​server 20 may also be as described here.

[0083] In the following description, the term "apparatus" can be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the sentence classification apparatus 10 may be configured to include one or more of the apparatuses shown in the figure, or may be configured to exclude some of the apparatuses.

[0084] Each function of the sentence classification device 10 is realized by loading specific software (programs) onto hardware such as the processor 1001 and memory 1002, causing the processor 1001 to perform calculations, control communication via the communication device 1004, and control at least one of reading and writing data from the memory 1002 and storage 1003.

[0085] The processor 1001, for example, runs an operating system to control the entire computer. The processor 1001 may be configured as a central processing unit (CPU) including an interface with peripheral devices, a control unit, an arithmetic unit, a register, etc. For example, each function of the above-described sentence classification device 10 may be realized by the processor 1001.

[0086] The processor 1001 also reads programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002 and executes various processes in accordance with these. The programs used are those that cause a computer to execute at least some of the operations described in the above-described embodiments. For example, each function of the sentence classification device 10 may be implemented by a control program stored in the memory 1002 and running on the processor 1001. While the above-described various processes have been described as being executed by one processor 1001, they may also be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The programs may also be transmitted from a network via a telecommunications line.

[0087] The memory 1002 is a computer-readable recording medium and may be configured, for example, by at least one of a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The memory 1002 may also be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store executable programs (program codes), software modules, etc. for performing information processing according to an embodiment of the present disclosure.

[0088] The storage 1003 is a computer-readable recording medium and may be composed of at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. The storage 1003 may also be referred to as an auxiliary storage device. The storage medium provided in the sentence classification device 10 may be, for example, a database, a server, or other appropriate medium including at least one of the memory 1002 and the storage 1003.

[0089] The communication device 1004 is hardware (transmission / reception device) for communicating between computers via at least one of a wired network and a wireless network, and is also called, for example, a network device, a network controller, a network card, or a communication module.

[0090] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that outputs to the outside. The input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).

[0091] Furthermore, each device, such as the processor 1001 and the memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

[0092] The sentence classification device 10 may also be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

[0093] The order of the procedures, sequences, flowcharts, etc. of each aspect / embodiment described in this disclosure may be changed unless it is consistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0094] Input and output information may be stored in a specific location (for example, memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be transmitted to another device.

[0095] The determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0096] The aspects / embodiments described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0097] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.

[0098] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0099] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0100] As used in this disclosure, the terms "system" and "network" are used interchangeably.

[0101] Furthermore, the information, parameters, etc. described in this disclosure may be expressed using absolute values, may be expressed using relative values ​​from a predetermined value, or may be expressed using other corresponding information.

[0102] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0103] The terms "connected," "coupled," or any variation thereof, refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using one or more wires, cables, and / or printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0104] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0105] As used in this disclosure, any reference to an element using a designation such as "first," "second," etc. does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must in some way precede the second element.

[0106] When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, when the term "or" is used in this disclosure, it is not intended to be an exclusive or.

[0107] In this disclosure, where articles are added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0108] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different."

[0109] The sentence classification device and sentence classification method of the present disclosure have the following configuration: [1] A sentence classification device comprising: an acquisition unit that acquires information indicating a sentence to be classified, a plurality of example sentences divided into a plurality of groups, and a classification example for each example sentence into one of a plurality of classification types; a model utilization unit that inputs, for each group, the sentence to be classified acquired by the acquisition unit, the example sentences included in the groups, and the classification example for each example sentence, into a generative AI model and acquires a classification result for the sentence to be classified from the generative AI model; and a determination unit that determines a final classification of the sentence to be classified based on the classification result for each group acquired by the model utilization unit. [2] The sentence classification device described in [1], in which the determination unit determines the final classification of the sentence to be classified depending on the number of classification results for each classification type. [3] The sentence classification device described in [2], in which the determination unit determines the final classification of the sentence to be classified by majority vote. [4] The sentence classification device described in [3], in which the model utilization unit sets, for each group, persons with different roles in a generative AI model and acquires a classification result for the sentence for each of the persons. [5] The sentence classification device according to any one of [2] to [4], wherein the determination unit compares the number of classification results for each classification type with a preset threshold to determine the final classification of the sentence to be classified. [6] The sentence classification device according to any one of [1] to [5], wherein the model utilization unit selects an example sentence to be used for obtaining a classification result again from the example sentences according to the classification result for each group, and again inputs the sentence to be classified, the example sentences, and the classification examples for each example sentence into the generative AI model to obtain the classification result. [7] A sentence classification method comprising: an acquisition step in which a sentence classification device acquires information indicating a sentence to be classified, a plurality of example sentences divided into a plurality of groups, and a classification example for each example sentence into one of a plurality of classification types; a model utilization step in which the sentence classification device inputs, for each group, the sentence to be classified acquired in the acquisition step, as well as the example sentences included in the groups and the classification example for each example sentence, to a generative AI model to obtain a classification result for the sentence to be classified from the generative AI model; and a determination step in which the sentence classification device determines a final classification for the sentence to be classified from the classification result for each group acquired in the model utilization step.

[0110] 10...Sentence classification device, 11...Acquisition unit, 12...Model utilization unit, 13...Judgment unit, 20...AI server, 21...Generative AI model, 1001...Processor, 1002...Memory, 1003...Storage, 1004...Communication device, 1005...Input device, 1006...Output device, 1007...Bus.

Claims

1. A sentence classification device comprising: an acquisition unit that acquires information indicating a sentence to be classified, a plurality of example sentences divided into a plurality of groups, and a classification example for each example sentence into one of a plurality of classification types; a model utilization unit that inputs, for each group, the sentence to be classified acquired by the acquisition unit, the example sentences included in the groups, and the classification example for each example sentence, into a generative AI model and acquires a classification result for the sentence to be classified from the generative AI model; and a judgment unit that determines the final classification of the sentence to be classified from the classification result for each group acquired by the model utilization unit.

2. The sentence classification device according to claim 1, wherein the determination unit determines the final classification of the sentence to be classified according to the number of classification results for each classification type.

3. The sentence classification device according to claim 2, wherein the determination unit determines the final classification of the sentence to be classified by majority vote.

4. The sentence classification device according to claim 3, wherein the model utilization unit sets a plurality of people with different roles for each group in the generative AI model and obtains classification results for sentences for each of the plurality of people.

5. The sentence classification device according to claim 2, wherein the determination unit compares the number of classification results for each classification type with a preset threshold value to determine the final classification of the sentence to be classified.

6. The sentence classification device according to claim 1, wherein the model utilization unit selects sentence examples to be used to obtain classification results again from the sentence examples according to the classification results for each group, and again inputs the sentences to be classified, the sentence examples, and the classification examples for each of the sentence examples into the generative AI model to obtain classification results.

7. A sentence classification method comprising: an acquisition step in which a sentence classification device acquires information indicating a sentence to be classified, a plurality of example sentences divided into a plurality of groups, and an example classification for each example sentence into one of a plurality of classification types; a model utilization step in which the sentence classification device inputs, for each group, the sentence to be classified acquired in the acquisition step, as well as the example sentences included in the groups and the example classification for each example sentence, into a generative AI model to obtain a classification result for the sentence to be classified from the generative AI model; and a judgment step in which the sentence classification device determines the final classification of the sentence to be classified from the classification result for each group acquired in the model utilization step.

Images