Interview evaluation device, interview evaluation method and interview evaluation program

Abstract

To solve a problem that a conventional technology is not sufficient to multilaterally evaluate an interview or evaluate a plurality of interviews.SOLUTION: Provided are an interview evaluation device and others, the interview evaluation device including: a check item holding unit that holds a plurality of check items for evaluating an interview; a related utterance extraction unit that extracts a related utterance as an utterance related to each of the check items from recorded utterances in the interview; a match / mismatch value acquisition unit that inputs the recorded utterances in the interview and / or the extracted related utterance to a large-scale language model to acquire a match / mismatch value as a value of match / mismatch for each of the check items in the recorded utterances in the interview; and a summary generation unit that inputs the extracted related utterance to the large-scale language model to generate a summary of communications for each of the check items.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an interview evaluation device that evaluates the contents, etc., of recorded interviews, etc. [Background technology]

[0002] Various types of interviews, such as business negotiations and job interviews, are widely conducted in sales activities. Each interview has a purpose, such as selling the company's products in the case of business negotiations and recruiting talented personnel in the case of job interviews. In other words, the content and quality of the interview determine whether the purpose is achieved or not. Therefore, it is useful to evaluate the content and quality of the interview and be able to utilize it.

[0003] For example, the evaluation system disclosed in Patent Document 1 evaluates the sales negotiating behavior of a sales representative based on whether the order in which digital materials are displayed to a customer is a predetermined exemplary order, and whether the sales representative's speech is exemplary and follows a talk script. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 2020-160425 Summary of the Invention [Problem to be solved by the invention]

[0005] According to the technology disclosed in Patent Document 1, it can be said that the interview is evaluated from one side. However, it cannot be said that the interview is evaluated from many sides. Moreover, it cannot be said that multiple interviews are evaluated. [Means for solving the problem]

[0006] Therefore, in order to solve the above problem, the present invention provides the following interview evaluation device etc. That is, the interview evaluation device has a check item storage unit that stores a plurality of check items for evaluating an interview, a related utterance extraction unit that extracts related utterances that are utterances related to each of the check items from utterances in a recorded interview, a suitability value acquisition unit that inputs the utterances in the recorded interview and / or the extracted related utterances into a large-scale language model to acquire suitability values ​​that are values ​​corresponding to suitability for each of the check items in the utterances in the recorded interview, and a summary generation unit that inputs the extracted related utterances into a large-scale language model to generate a summary of the exchange for each of the check items.

[0007] The present invention also provides an interview evaluation device as described in claim 1, further comprising an answer generation unit that inputs, in addition to the features, instructions for evaluating a plurality of interviews and one or more of the acquired suitability values ​​for the plurality of interviews, the generated summary, and the extracted related speech text into the large-scale language model, and generates an answer to the instructions.

[0008] The present invention also provides an interview evaluation method, which is executed by a computer, comprising: a check item holding step for holding a plurality of check items for evaluating an interview; a related utterance extraction step for extracting related utterances, which are utterances related to each of the check items, from utterances in a recorded interview; a corresponding / non-corresponding value acquisition step for inputting the utterances in the recorded interview and / or the extracted related utterances into a large-scale language model to acquire corresponding / non-corresponding values, which are values ​​corresponding to whether each of the check items is corresponding or not, from the utterances in the recorded interview; and a summary generation step for inputting the extracted related utterances into a large-scale language model to generate a summary of the exchange for each of the check items.

[0009] The present invention also provides an interview evaluation program having a check item holding step for holding a plurality of check items for evaluating an interview, a related utterance extraction step for extracting related utterances that are utterances related to each of the check items from utterances in a recorded interview, a corresponding / non-corresponding value acquisition step for inputting the utterances in the recorded interview and / or the extracted related utterances into a large-scale language model to acquire corresponding / non-corresponding values ​​that are values ​​corresponding to whether each of the check items is corresponding or not from the utterances in the recorded interview, and a summary generation step for inputting the extracted related utterances into the large-scale language model to generate a summary of the exchange for each of the check items. Effect of the Invention

[0010] According to the present invention, a number of check items can be set to evaluate an interview. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing an example of a functional configuration of an interview evaluation device according to a first embodiment. [Diagram 2] A conceptual diagram showing an example of displaying the acquired corresponding value. [Diagram 3] Conceptual diagram showing an example of displaying the appropriateness value for each person in charge [Figure 4] FIG. 13 is a block diagram showing an example of the functional configuration of an interview evaluation device according to another embodiment. [Diagram 5] A conceptual diagram showing an example of the hardware configuration for implementing the interview evaluation device of the first embodiment. [Figure 6] FIG. 1 is a diagram showing an example of a process flow of the interview evaluation device of the first embodiment. [Figure 7] FIG. 11 is a block diagram showing an example of the functional configuration of an interview evaluation device according to a second embodiment. [Figure 8] A conceptual diagram showing the appropriateness and summary for multiple interviews, for each interview. [Figure 9] FIG. 13 is a diagram showing an example of a process flow of the interview evaluation device of the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Note that the present invention should not be limited to these embodiments, and can be embodied in various forms without departing from the spirit of the present invention.

[0013] <Example 1> <Summary> In an interview, participants gather to exchange opinions and discuss a specific purpose, and make decisions such as reaching an agreement or taking measures. Therefore, in order to evaluate what the interview was like, it is effective to confirm whether important opinions were expressed and whether an agreement was reached regarding the purpose of the interview. In the present invention, by setting multiple check items for evaluating the interview and obtaining a value corresponding to whether each check item is applicable or not, it becomes possible to quantitatively evaluate the interview from multiple perspectives.

[0014] The following describes the functions and processing flow of the interview evaluation device, as well as the contents of the hardware. The functional blocks of the system described below can be realized as a combination of hardware and software. Specifically, if a computer is used, the hardware components include a CPU (Central Processing Unit), main memory, bus, or secondary storage device (hard disk drive, non-volatile memory, storage media such as CDs and DVDs, and drives to read these media, etc.), input devices used to input information, printers, display devices, and other external peripheral devices, as well as interfaces for these external peripheral devices, communication interfaces, driver programs and other application programs for controlling these hardware devices, and user interface applications. Then, by the calculation processing of the CPU according to the program deployed on the main memory, data input from input devices and other interfaces and stored in memory and hard disks are processed and stored, and commands for controlling the above hardware and software are generated. Alternatively, the functional blocks of the system may be realized by dedicated hardware.

[0015] This invention can be realized not only as a system, but also as a method. A part of such an invention can be configured as software. Furthermore, a program used to cause a computer to execute such software, and a recording medium on which the program is fixed, are naturally included in the technical scope of this invention (the same applies throughout this specification).

[0016] <Functional configuration> 1 is a block diagram showing an example of the functional configuration of the interview evaluation device of the present embodiment. As shown in FIG. 1, the interview evaluation device 100 includes a check item storage unit 101 and a corresponding / rejected value acquisition unit 102.

[0017] The interview evaluation device 100 is a device that evaluates interviews based on recorded utterances made during interviews. An interview refers to various discussions such as business negotiations, employment interviews, internal interviews, internal meetings, and organizational meetings. In addition, multiple interviews may be treated as a set of interviews. In addition, interviews include not only face-to-face interviews, but also online business negotiations and meetings using the Internet or telephone lines. This device is realized by a personal computer, a smartphone, a tablet terminal, a server device, or the like.

[0018] <Check item storage section> The check item storage unit 101 has a function of storing a plurality of check items for evaluating an interview. As described above, an interview has a purpose, and in order to achieve that purpose, it is necessary to communicate what should be communicated to the person being interviewed and to ask what should be asked. By checking whether these matters have been appropriately communicated with the other party, it is possible to evaluate whether the interview was desirable. Such things that should be confirmed during the exchange of an interview are called check items.

[0019] Various check items are stored depending on the type and purpose of the interview. For example, in a business meeting, multiple check items such as "issues," "budget," "intention to introduce," "concerns," "purchase process," "competition," and "schedule" can be listed.

[0020] Also, check items for evaluating sales negotiations can be set based on various sales methodologies, such as "BANT+C theory," "SPIN sales method," "3C analysis," "4P analysis," "PEST analysis," "VRIO," "TOWS analysis," and "The Model."

[0021] In addition, the check items are defined as higher-level concepts (such as issues and implementation intentions) as described above, and then more specific check items are defined. For example, the check items for "issues" are as follows: ·Ability to ask questions about customer issues ·Ability to ask follow-up questions based on what the customer says about their problem · Has the customer quantified their problem? ·Is customer priority mentioned? Check items such as the following are specified.

[0022] In addition, examples of check items when evaluating a recruitment interview include "confirmation of communication skills," "confirmation of degree of interest and motivation," "attitude toward work," "skill match," "stress tolerance," "cultural match," "orientation," and "settlement." These can each be set and maintained as check items.

[0023] The check item storage unit may store a number and contents of check items that are predetermined in advance, or may store check items according to user settings.

[0024] In addition, specific check items subordinate to these higher-level concepts can be set by the user himself or herself, and some specific check items can be suggested to the user according to the higher-level concepts. Note that such specific check items may be referred to as "individual evaluation items" below.

[0025] <Acquisition of the value> The corresponding / non-corresponding value acquisition unit 102 has a function of acquiring corresponding / non-corresponding values, which are values ​​corresponding to the corresponding / non-corresponding of each check item in the recorded interview utterance. The recorded interview may be recorded in a storage medium such as a storage device provided in the interview evaluation device itself, or may be recorded in an external storage medium such as a database.

[0026] The value according to whether or not the item is true is a value indicating the degree of whether or not the other party has been informed of what should be communicated and listened to what should be listened to regarding the check item. The value used as the value according to whether or not the item is true may be a boolean value such as "1" if the item is true and "0" if the item is not true, but may also be a value according to the degree of whether or not the item is true. For example, the value according to whether or not the item is true may be expressed in three stages such as "Positive, Neutral, Negative". It may also be expressed as a value indicating a percentage such as 0% to 100%.

[0027] The corresponding or non-corresponding value can be obtained by using a large-scale language model. Specifically, for example, the corresponding or non-corresponding value can be obtained by inputting the utterance converted into text based on the audio data or video data of the recorded interview and the conditional expression for calculating the corresponding or non-corresponding value into a large-scale language model (sometimes abbreviated as LLM) and performing inference.

[0028] The textual utterances to be input into the large-scale language model may be configured to convert all utterances from the recorded interview into text and input the entire utterance for inference, or the entire utterance may be divided into several parts and input, and inferences may be made for each part, after which the results of each part are integrated.

[0029] In addition, as a preprocessing step for the recorded interview utterances, a process is performed to extract related utterances that are related to each of the check items (related utterance extraction unit), and the extracted related utterances are input into a large-scale language model to infer and obtain a true / false value for the check item.

[0030] For example, the relevant utterances are extracted by linguistically analyzing the text of the interview speech, extracting keywords related to the check items, and extracting utterances containing the extracted keywords as the relevant utterances. For example, to detect relevant utterances related to the check item "budget," words such as "budget," "cost," "payment," and "fee" are used as keywords, and utterances containing these keywords are detected as related utterance texts.

[0031] Alternatively, a large-scale language model may be used to first extract related utterances, and the extracted related utterances may then be input into the large-scale language model to obtain a corresponding or non-corresponding value for the check item. Extracting related utterances as a preprocessing step for obtaining the corresponding or non-corresponding value may improve the accuracy of obtaining the corresponding or non-corresponding value.

[0032] FIG. 2 is a conceptual diagram showing an example of displaying the acquired suitability value. As shown in the figure, first, a problem is displayed as a "check item" 201, four specific "individual evaluation items" 202 for the check item are displayed, and a "suitability value" 203 for each is displayed. For example, for the item "The current sales system has been confirmed," a suitability value of "Neutral" is displayed, and also, as a "reason" 204 for arriving at the suitability value, "The number of sales staff has been confirmed, but an expression such as 'The number of sales staff is less than 500' is inappropriate." Although not shown in this figure, it can be configured to display the utterance referred to to acquire the suitability value.

[0033] Also, the system can be configured to display the suitability value acquired for each person who has conducted the interview. FIG. 3 is a conceptual diagram showing an example of displaying the suitability value for each person. As shown in the figure, the leftmost column shows "person in charge name" 310 (SHINODA, KOJIMA, TOSHIDA, SATO), and the suitability value for each of the multiple check items set for each person, "issue" 302, "budget" 303, "intention to introduce" 304, "concept" 305, "purchase process" 306, "competition" 307, and "schedule" 308, is shown as 0% to 100%. By showing the suitability value for each check item for each person in this way, it is possible to know which person is good at checking each check item. Also, the suitability value may be acquired for each team or department consisting of multiple people, rather than for each person in charge.

[0034] Also, there are many cases where an interview takes place over multiple sessions. For example, a business negotiation to secure an order for some kind of product can be said to be completed and its purpose achieved through multiple interviews. In such a case, it is also preferable to configure the system to obtain the appropriateness value of the check items at each interview. By expressing the appropriateness value as a range from 0% to 100%, it is possible to know to what extent each check item was performed (or the achievement rate). Therefore, in a business negotiation, it is possible to see at a glance what should be confirmed or agreed upon next from the achievement rate of each check item. Additionally, the appropriateness value for each check item may be displayed in a color with different shading according to its value.

[0035] In addition, by acquiring and displaying the weekly average of the corresponding / failure values ​​for each step, or by displaying the corresponding / failure values ​​for each step in chronological order, it becomes possible to more intuitively understand to what extent each check item has been achieved, as well as the progress of the achievement rate for each check item. In addition, by displaying the weekly average of the corresponding / failure values ​​and the progress over time for each person in charge, it is possible to evaluate the achievement level for each check item for each person in charge. It is also possible to display a ranking of the persons in charge based on the corresponding / failure values.

[0036] Furthermore, by correlating and accumulating the transition of the suitability value with the outcome of the business negotiation (whether or not an order was placed), it is possible to analyze various points for successful business negotiation, for example, which check items need to be confirmed or agreed upon at what point from the start of the business negotiation in order to result in an order being placed. It is also possible to analyze the cause of a business negotiation failure by linking it with the suitability value of the check items. From such a relationship between the success or failure of a business negotiation and the suitability value, it is possible to visualize the desirable way to proceed with the business negotiation, which can also contribute to the education of those in charge of business negotiations.

[0037] In addition, the system can be configured to acquire the result of selection from several options as the appropriate / inappropriate value. For example, if the check item is "job type desired for employment," there are job types (e.g., system engineer, sales, corporate planning, marketing, back office) that are set as options that are candidates for the appropriate / inappropriate value, and from the utterances made during the interview, the option that corresponds to the desired job type (e.g., sales, marketing) is acquired as the appropriate / inappropriate value. When outputting such an appropriate / inappropriate value, the system can be configured to display the exchange that is the basis for the value as a reference utterance.

[0038] By obtaining the corresponding or non-corresponding value, it becomes clear that the check item or individual evaluation item is insufficient or missing. Based on the result, for example, a large-scale language model can be used to generate and present a talk plan for the next interview.

[0039] <Other Examples> The above describes the most basic embodiment of the present invention. Furthermore, the following configuration can also be added to the embodiment. Figure 4 is a block diagram of the functional configuration of an embodiment with further configuration added. As shown in the figure, the interview evaluation device 400 has a check item storage unit 401, a negative value acquisition unit 402, a related utterance extraction unit 403, an evaluation value acquisition unit 404, and a summary generation unit 405. The check item storage unit 401, the negative value acquisition unit 402, and the related utterance extraction unit 403 are as described above, so a detailed explanation will be omitted below.

[0040] <Evaluation value acquisition section> The evaluation value acquisition unit 404 has a function of acquiring an evaluation value of the interview to be evaluated based on the acquired suitability value. This acquires a comprehensive evaluation value for the interview from the suitability value for each check item. There are no particular limitations on the specific method of acquiring the evaluation value, and for example, the evaluation value may be acquired arithmetically by weighting the suitability value for each check item, or it may be acquired by other methods.

[0041] <Summary generation section> The summary generation unit 405 has a function of generating a summary of the exchange for each check item from the recorded utterances in the interview. The summary can be generated by inputting the utterances in the interview converted into text into a large-scale language model and giving an instruction to generate a summary.

[0042] By generating a summary, it is possible to know how the interview actually went with respect to each check item, and by looking at this summary, it is possible to judge whether or not the check item is applicable. It is preferable to display the generated summary together with the applicable / inapplicable value for the check item, for ease of understanding. It is also possible to configure it so that the utterances referenced in generating the summary are also displayed.

[0043] It may also be configured to show whether a summary has been generated. If a summary has been generated for a certain check item, it becomes clear that the check item has been communicated or heard about, while if there is no summary, it is immediately clear that no communication has been made about the check item in the first place. It may also be configured to give priority to displaying only the presence or absence of a summary, and to display the summary in response to a user's display switching, etc.

[0044] <Hardware configuration> Fig. 5 is a conceptual diagram showing an example of the configuration of hardware that realizes an interview evaluation device. As shown in the figure, the interview evaluation device 500 has a CPU 501 that performs various arithmetic processing, a RAM 502 which is a volatile recording medium, a storage 503 such as a flash memory or HDD which is a non-volatile storage medium, a communication interface 504, and an input / output interface 505. The RAM 502 reads out programs that perform various arithmetic processing to cause the CPU 501 to execute them, and also provides a work area (working area) for the programs. In addition, a plurality of addresses are assigned to the RAM 502, and the programs executed by the CPU 501 can exchange data with each other and perform processing by identifying and accessing the addresses (this is the same throughout this specification).

[0045] Here, the function of the check item storage unit 101 of the interview evaluation device 100 in Fig. 1 can be realized mainly by the storage 503 in Fig. 5. Also, the function of the relevant / rejected value acquisition unit 102 is realized mainly by the CPU 501 and RAM 502 in Fig. 5.

[0046] 5. The functions of the check item storage unit 401 and the corresponding / incorrect acquisition unit 402 in the interview evaluation device 400 in Fig. 4 can also be realized in the same way as the configurations with the same names described above. The functions of the related utterance extraction unit 403, the evaluation value acquisition unit 404, and the summary generation unit 405 in the interview evaluation device 400 in Fig. 4 are realized by a CPU 501 and a RAM 502 shown in Fig. 5.

[0047] <Processing flow> 6 is a flow diagram showing an example of the flow of information processing of the interview evaluation device. As shown in the figure, a plurality of check items for evaluating an interview are stored (S601: check item storage step). Then, for the recorded utterances in the interview, a corresponding / non-corresponding value, which is a value corresponding to whether each check item is corresponding or not, is obtained (S602: corresponding / non-corresponding value acquisition step).

[0048] <Effects> According to the interview evaluation device of this embodiment, it is possible to know to what extent important check items in an interview have been completed, and to visualize and evaluate the interview.

[0049] <Example 2> <Summary> This embodiment is based on the first embodiment and is an interview evaluation device suitable for evaluating a plurality of interviews.

[0050] <Functional configuration> 7 is a block diagram showing an example of the functional configuration of the interview evaluation device of this embodiment. As shown in FIG. 7, the interview evaluation device 700 has a check item storage unit 701, a related utterance extraction unit 702, a corresponding or negative value acquisition unit 703, a summary generation unit 704, and an answer generation unit 705.

[0051] <Check item storage section> The check item storage unit 701 has a function of storing a plurality of check items for evaluating an interview. The function of the check item storage unit in this embodiment is similar to that of the check item storage unit in the first embodiment.

[0052] <Related speech extraction unit> The related utterance extraction unit 702 has a function of extracting related utterances that are related to each of the check items from the recorded utterances during the interview. The function of the related utterance extraction unit in this embodiment is basically the same as the function of the related utterance extraction unit in the first embodiment, and in particular extracts text-converted related utterances (related utterance texts).

[0053] <Acquisition of the value> The corresponding / non-corresponding value acquisition unit 703 has a function of inputting the recorded interview utterance and / or the extracted related utterance text into a large-scale language model, and acquiring a corresponding / non-corresponding value which is a value corresponding to the corresponding / non-corresponding of each of the check items in the recorded interview utterance. The function of the corresponding / non-corresponding value acquisition unit in this embodiment is basically the same as the function of the corresponding / non-corresponding value acquisition unit in the first embodiment, but in particular, the related utterance that has been converted into text and extracted is input into a large-scale language model to acquire a corresponding / non-corresponding value.

[0054] In order to obtain suitability values ​​for a large number of interviews, it is effective to input the extracted related speech text, but when obtaining suitability values ​​for a small number of interviews, such as one or two, there is no particular problem in inputting the recorded interview utterances directly into a large-scale language model, and the convenience of being able to obtain the utterances without going through a process of extracting related speech text can be obtained. Also, even for recorded interviews with a relatively short duration, it is possible to input the interview utterances directly into a large-scale language model to obtain suitability values ​​without necessarily extracting related speech text.

[0055] It is also preferable to configure the related utterance text to be compressed by reducing the number of tokens before inputting it to the large-scale language model. For example, the related utterance text can be compressed by reducing the number of tokens by converting it into a bulleted list of a predetermined time (20 to 50 seconds, preferably about 30 seconds) or by converting it into a formal style. By reducing the number of tokens of the related utterance to be input in this way and compressing it, the accuracy of the obtained positive or negative value is improved.

[0056] In this specification, a large-scale language model is a machine learning model that can be pre-trained to handle general-purpose tasks, and may be a small-scale language model or a multi-modal language model that can handle images, audio, and other formats.

[0057] <Summary generation section> The summary generation unit 704 has a function of inputting the extracted related utterances into a large-scale language model and generating a summary of the exchange for each of the check items. The function of the summary generation unit in this embodiment is basically the same as the function of the summary generation unit in embodiment 1. As described in the negative value acquisition unit, it is preferable to input the related utterance text compressed by reducing the number of tokens into the large-scale language model to generate the summary.

[0058] As described above, by obtaining the suitability value for each check item and generating a summary, one interview can be structured, which not only makes it easier to evaluate and analyze one interview, but is also suitable for evaluating multiple interviews together. For example, by extracting the suitability value and summary for a check item called "concern" from each of multiple interviews, it becomes easier to understand the content and tendency of the concern, and it becomes easier to take measures to resolve the concern. In this way, multiple interviews can be analyzed and evaluated across the board, focusing on one or multiple check items.

[0059] Also, by displaying a list of the suitability and disapproval values ​​and summaries for specific check items for each interview, it becomes easier to perform analysis and evaluation focusing on the specific items in many interviews. Fig. 8 is a conceptual diagram showing a list of suitability and disapproval values ​​and summaries for multiple interviews for each interview. As shown in the figure, the table has columns for "Title" 801, "Participants" 802, "Date and Time" 803, "Project Progress" 804, "Purchase Process" 805, and "Summary Creation" 806 from the left in the first row.

[0060] Of these, "project progress" and "purchasing process" are check items that indicate a corresponding or non-corresponding value. In this example, the corresponding or non-corresponding value for "project progress" is shown as a percentage. Also, for "purchasing process", the corresponding or non-corresponding value is shown by facial expression. For example, the corresponding or non-corresponding value for "purchasing process" of "Company F, meeting with advisor" on the fifth line is shown by a crying face 807. The greater the corresponding or non-corresponding value, the more the face smiles, and the highest corresponding or non-corresponding value is shown by a broad smile as in "Company B, planning department, meeting with Suzuki-san" on the third line shown in the figure.

[0061] In the column "Summary generation" 806, a "check mark" 808 indicates whether a summary has been generated for each interview. In this example, a summary is generated for each interview. This summary includes at least summaries for "project progress" and "purchasing process", but may also be a summary for the entire interview including other check items.

[0062] In addition, in Fig. 8, a check mark is used to indicate whether a summary has been generated or not, but it can be configured to display summaries that have already been generated by clicking the check mark, or to display summaries that have already been generated in a list from the beginning. It can also be configured to display summaries with a reduced number of characters.

[0063] It is also possible to configure the system to generate a summary for the entire interview based on the summaries generated for each check item. Since the summary is generated based on the summaries already generated for each check item, it is possible to generate an accurate summary.

[0064] <Answer generator> The answer generation unit 705 has a function of inputting instructions for evaluating multiple interviews and one or more of the acquired suitability values ​​for the multiple interviews, the generated summary, and the extracted related speech text into the large-scale language model, and generating an answer to the instructions.

[0065] An instruction for evaluation is not merely an evaluation but is an instruction that requests some kind of answer through analysis or interpretation. For example, an instruction such as "Analyze the trends in all these interviews regarding the customer's concerns and present the corresponding measures." Such an instruction is input to a large-scale language model together with one or more of the obtained suitability values ​​of the multiple interviews, the generated summary, and the extracted related speech text to generate an answer.

[0066] Here, depending on the content of the instruction, any one or more of the corresponding / non-corresponding value, summary, and related speech text to be input together with the instruction can be arbitrarily input. For example, when analyzing or evaluating an interview broadly and from various angles, inputting all of the corresponding / non-corresponding value, summary, and related speech text is effective for generating a desirable answer. On the other hand, when giving a limited or local instruction such as whether or not a matter that should be explained to the customer about a specific check item has been explained, a sufficient answer can be generated by inputting only the related speech text about the check item. Therefore, depending on the content of the instruction to be input, any one or more of the corresponding / non-corresponding value, summary, and related speech text that are necessary as appropriate may be input. In addition, when the user inputs these, the large-scale language model may be configured to suggest the corresponding / non-corresponding value, summary, and related speech text to be input together based on the instruction input earlier. The user only needs to input the corresponding / non-corresponding value according to this suggestion, which contributes to improving convenience.

[0067] The context size of a large-scale language model (the length of input that can be processed at one time) is limited to roughly 1 million tokens, but for example, a one-hour audio file contains about 80,000 tokens, and even a one-hour transcript contains 20,000 tokens. In other words, it is not easy in terms of context size and cost to input the audio recordings of many interviews and their transcripts directly into a large-scale language model and analyze or evaluate them.

[0068] In addition, when a transcript is directly input into a large-scale language model, if the input prompts are simple and rough, it is likely that only weak answers will be obtained. On the other hand, even if the input prompts are complex and strict, inappropriate answers (irrelevant answers) will be returned.

[0069] As described above, by inputting the suitability value and summary for the interview along with the instructions into a large-scale language model, i.e., by inputting structured data about the interview, the above problem can be solved and an appropriate answer can be obtained for the instructions.

[0070] <Hardware configuration> The interview evaluation device of Example 2 can be realized by the hardware configuration shown in Fig. 5, similar to Example 1. The function of the check item storage unit 701 of the interview evaluation device 700 of Example 2 shown in Fig. 7 can be realized mainly by the storage 503 in Fig. 5. Also, the functions of the related utterance extraction unit 702, the corresponding / non-corresponding value acquisition unit 703, the summary generation unit 704 and the answer generation unit 705 are realized mainly by the CPU 501 and RAM 502 in Fig. 5. Also, when using an external large-scale language model, each function is realized by exchanging signals and information with each other via the communication interface 504 and the input / output interface 505.

[0071] <Processing flow> FIG. 9 is a flow diagram showing an example of the process flow of the interview evaluation device of the second embodiment. As shown in the figure, first, a plurality of check items for evaluating an interview are stored (S901: check item storage step). Then, from the recorded interview utterances, relevant utterance texts obtained by converting utterances related to each of the check items into text are extracted (S902: text extraction step). Then, the recorded interview utterances and / or the extracted relevant utterances are input to an LLM (large-scale language model), and a relevant value, which is a value according to whether each of the check items is relevant or not, is obtained in the recorded interview utterances (S903: relevant value acquisition step). Then, the extracted relevant utterance texts are input to the large-scale language model, and a summary of the exchange for each of the check items is generated (S904: summary generation step). Then, instructions for evaluating multiple interviews and one or more of the acquired suitability values ​​for the multiple interviews, the generated summary, and the extracted related speech text are input into the large-scale language model to generate an answer to the instructions (S905: answer generation step). [Explanation of symbols]

[0072] 100, 400, 700: Interview evaluation device 101, 401, 701: Check item storage section 102, 402, 703: Corresponding value acquisition unit 403, 702: Related utterance extraction unit 404: Evaluation value acquisition unit 405, 704: Summary generation section 705: Answer generation part 501:CPU 502:RAM 503: Storage 504: Communication interface 505: Input / Output Interface

Claims

1. a check item storage unit that stores a plurality of check items for evaluating an interview; a text extraction unit that extracts related utterance texts by converting utterances related to each of the check items from the recorded utterances during the interview; a corresponding / non-corresponding value acquiring unit that inputs the recorded interview speech and / or the extracted related speech text into a large-scale language model to acquire corresponding / non-corresponding values ​​corresponding to corresponding / non-corresponding values ​​of each of the check items in the recorded interview speech; a summary generation unit that inputs the extracted related speech text into a large-scale language model to generate a summary of the exchange for each of the check items; An interview evaluation device having the above structure.

2. The interview evaluation device of claim 1, further comprising an answer generation unit that inputs instructions for evaluating a plurality of interviews and one or more of the acquired suitability values ​​for the plurality of interviews, the generated summary, and the extracted related speech text into the large-scale language model to generate an answer to the instructions.

3. 1. A computer-implemented method comprising: A check item holding step for holding a plurality of check items for evaluating an interview; A text extraction step of extracting related utterance texts obtained by converting utterances related to each of the check items from the recorded utterances during the interview; a corresponding / non-corresponding value acquisition step of inputting the recorded interview speech and / or the extracted related speech text into a large-scale language model to acquire corresponding / non-corresponding values ​​corresponding to corresponding / non-corresponding values ​​for each of the check items in the recorded interview speech; a summary generation step of inputting the extracted related speech text into a large-scale language model to generate a summary of the interaction for each of the check items; An interview evaluation method that has the following features.

4. A check item holding step for holding a plurality of check items for evaluating an interview; A text extraction step of extracting related utterance texts obtained by converting utterances related to each of the check items from the recorded utterances during the interview; a corresponding / non-corresponding value acquisition step of inputting the recorded interview speech and / or the extracted related speech text into a large-scale language model to acquire corresponding / non-corresponding values ​​corresponding to corresponding / non-corresponding values ​​for each of the check items in the recorded interview speech; a summary generation step of inputting the extracted related speech text into a large-scale language model to generate a summary of the interaction for each of the check items; An interview evaluation program that causes a computer to execute the above.

Images