Information processing methods, programs, and information processing systems

Abstract

To provide an information processing method, program, and information processing system that facilitate the implementation of evaluations. [Solution] An information processing method executed by an information processing device, comprising: an evaluation target data acquisition step S100 for acquiring evaluation target data to be evaluated; a division step S110 for dividing the evaluation target data into a plurality of divisions using division setting information obtained by predetermined means based on operational policy information; and an output step S114 for outputting evaluation results obtained by input and acquisition for each divided evaluation target data.

Description

Technical Field

[0001] The present invention relates to an information processing method, a program, and an information processing system.

Background Art

[0002] In Patent Document 1, as a system for assisting in the performance evaluation of a person to be trained, a system is disclosed in which an evaluator selects evaluation item information for content (video or audio data) in which a trainer records lectures, speeches, etc. of the person to be trained and gives evaluations and comments. In this system, the trainer performs evaluations while viewing the content, and the evaluation results are associated with a specific time axis. Then, the person to be trained can confirm the evaluation results on the playback screen and review their own performance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the case of the above prior art, the trainer needs to view the content when performing evaluations, making it difficult to perform evaluations quickly.

[0005] In consideration of the above facts, an object of the present invention is to facilitate the implementation of evaluations.

Means for Solving the Problems

[0006] According to an information processing method according to one embodiment, the information processing method executed by the information processing device includes: an evaluation target acquisition step of acquiring evaluation target data to be evaluated; a division step of dividing the evaluation target data into a plurality of divisions using division setting information obtained by predetermined means based on operational policy information; and an output step of outputting an evaluation result obtained by at least one of input and acquisition for each of the divided evaluation target data.

[0007] According to a program according to one embodiment, the information processing device is made to execute an information processing method which includes an evaluation target acquisition step of acquiring evaluation target data to be evaluated, a division step of dividing the evaluation target data into a plurality of divisions using division setting information obtained by predetermined means based on operational policy information, and an output step of outputting an evaluation result obtained by input and acquisition for each of the divided evaluation target data.

[0008] According to an information processing system according to one embodiment, the information processing system executed by the information processing device includes: an evaluation target acquisition unit that acquires evaluation target data to be evaluated; a division unit that divides the evaluation target data into a plurality of divisions using division setting information obtained by predetermined means based on operational policy information; and an output unit that outputs an evaluation result obtained by at least one of input and acquisition for each of the divided evaluation target data. [Effects of the Invention]

[0009] According to one embodiment, the evaluation can be made easier to carry out. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of the configuration of an information processing system according to the first embodiment. [Figure 2] This figure shows an example of the server hardware configuration according to the first embodiment. [Figure 3] This figure shows an example of the functional configuration of a server according to the first embodiment. [Figure 4] This figure shows an example of the processing flow of the information processing system according to the first embodiment. [Figure 5] This figure shows an example of a screen display in the information processing system according to the first embodiment. [Figure 6] This figure shows an example of a screen display in the information processing system according to the first embodiment. [Figure 7] This figure shows an example of partition setting information in the information processing system according to the first embodiment. [Modes for carrying out the invention]

[0011] (Embodiment) A first embodiment of the information processing system according to the present invention will be described below with reference to Figures 1 to 7. In each figure, identical or equivalent components and parts are given the same reference numerals. Furthermore, the dimensional ratios in the drawings are exaggerated for illustrative purposes and may differ from actual ratios.

[0012] (System Overview) First, an overview of the information processing system 10 according to this embodiment will be described. The information processing system 10 according to this embodiment is a system that primarily acquires conversational audio data from sales calls, evaluates the transcribed text data as evaluation target data, and presents various information to the user U for the evaluator to evaluate the conversational skills of sales representatives and provide feedback for skill improvement. This information processing system 10 is applicable not only to sales calls, but also to various types of voice communication such as call center inquiries, operator conversations at technical support desks, contract confirmation calls for insurance and other services, and inquiry calls for various surveys, as well as text communication such as email and chat. Further details will be described later.

[0013] (System Configuration) FIG. 1 is a diagram showing an example of the configuration of an information processing system 10 according to the present embodiment. As shown in FIG. 1, the information processing system 10 according to the present embodiment includes a server 12 as an information processing device and a plurality of user terminals 14 that are communicably connected to each other via a network N. The network N is, for example, a wired LAN (Local Area Network), a wireless LAN, the Internet, a public switched telephone network, a mobile data communication network, or a combination thereof.

[0014] The user terminal 14 is an example of an information processing device for a user U to perform operations for inputting and displaying various information. The user terminal 14 may be a PC (Personal Computer), a smartphone, a tablet terminal, a server device, a microcomputer, a wearable device, or a combination thereof. Note that the user U may be not only an evaluator but also other persons such as a salesperson, an operator of a call center, and an administrator.

[0015] [[ID=Z]] The server 12 is an example of an information processing device that acquires information input from the user terminal 14, performs processing based on the information, and outputs a result. The server 12 may be a PC (Personal Computer), a smartphone, a tablet terminal, a server device, a microcomputer, or a combination thereof. The specific configuration and operation of the server 12 will be described later.

[0016] (Hardware Configuration) FIG. 2 is a block diagram showing the hardware configuration of the server 12. The server 12 includes a processor 120, a memory 122, a storage 124, a communication I / F 126, an input / output I / F 128, an input device 132, an output device 130, and a drive device 134 that are communicably connected to each other via a bus B.

[0017] Processor 120 controls each component of server 12 and realizes the functions of server 12 by expanding and executing various programs stored in storage 124 in memory 122. Programs executed by processor 120 include, but are not limited to, an OS (Operating System) and program 220 described later. By executing these programs, processor 120 realizes a part of the information processing method according to this embodiment. Processor 120 is, for example, a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), or a combination thereof.

[0018] Memory 122 is, for example, a ROM (Read Only Memory), RAM (Random Access Memory), or a combination thereof. The ROM is, for example, a PROM (Programmable ROM), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), or a combination thereof. The RAM is, for example, a DRAM (Dynamic RAM), SRAM (Static RAM), MRAM (Magnetoresistive RAM), or a combination thereof.

[0019] Storage 124 stores an OS, various programs described later, and various data. Storage 124 is, for example, a flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), SCM (Storage Class Memories), or a combination thereof.

[0020] The communication interface 126 is an interface for connecting the server 12 to external devices, including the user terminal 14 and the imaging device 16, via the network N, and for controlling communication. The communication interface 126 is, for example, an adapter compliant with Bluetooth®, Wi-Fi®, ZigBee®, Ethernet®, or optical communication (e.g., Fibre Channel).

[0021] The input / output interface 128 is an interface for connecting input devices 132 and output devices 130 to the server 12. Input devices 132 include, for example, a mouse, keyboard, touch panel, microphone, scanner, camera, various sensors, operation buttons, or a combination thereof. Output devices 130 include, for example, a display, projector, printer, speaker, vibrator, or a combination thereof.

[0022] The drive device 134 reads and writes data to the disk media 136. The drive device 134 is, for example, a magnetic disk drive, an optical disk drive, a magneto-optical disk drive, or a combination thereof. The disk media 136 is, for example, a CD (Compact Disc), a DVD (Digital Versatile Disc), an FD (Floppy Disk), an MO (Magneto-Optical disk), a BD (Blu-ray® Disc), or a combination thereof.

[0023] In this embodiment, the program may be written to memory 122 or storage 124 during the manufacturing stage of server 12, or it may be provided to server 12 via network N, or it may be provided to server 12 via a non-temporary, computer-readable recording medium such as disk media 136.

[0024] Furthermore, the hardware configuration of the user terminal 14 is substantially the same as that of the server 12 described above, so a detailed explanation will be omitted. The user terminal 14 is equipped with a display as an example of a user interface 34 (see Figure 1).

[0025] (Functional Configuration) Next, the functional configuration of server 12 will be described. Figure 3 shows an example of the functional configuration of server 12. When executing various programs, server 12 uses the above hardware resources to realize various functions. Server 12 has a communication unit 20, a storage unit 22, and a control unit 24 as the functional configuration that server 12 realizes. Each functional configuration is realized when the processor 120 reads and executes a program 220 stored in memory 122 or storage 124.

[0026] The communication unit 20 is implemented by the communication interface 126. The communication unit 20 sends and receives information with the user terminal 14 via the network N. The communication unit 20 receives information input from the user terminal 14. The communication unit 20 also sends information to the user terminal 14 and receives requests from user U from the user terminal 14.

[0027] The memory unit 22 is implemented by memory 122 and storage 124. The memory unit 22 stores the program 220, the trained model 222, the partition setting information DB 224, the evaluation target data DB 226, the evaluation item information DB 228, the evaluation result DB 230, and the user information DB 232.

[0028] The program 220 is read and executed by the processor 120 of the server 12, and is responsible for realizing various procedures of the information processing method according to this embodiment.

[0029] The trained models 222 include cases where machine learning models are used and cases where large language models (LLMs) are used, as well as forms that combine these. Although large language models are a type of machine learning model, they are characterized by the use of large-scale pre-training and inference methods, and in this specification, they will be described separately from conventional machine learning models, taking into account the differences in training methods. In the training phase of the machine learning model, at least one of the audio data of the conversation used for training and the text data of the conversation transcribed (hereinafter simply referred to as "conversation data"), as well as information on products and sales methods as operational policy information, are input, and a training dataset is created by assigning necessary classifications (chunks) and correct answer data (teacher labels) such as evaluation results to this data, and the model parameters are adjusted while comparing the error between the training data and the model's estimation results. For example, information such as which parts of the conversation data correspond to the "opening" in meaning, which positions are treated as "price explanations" in meaning, or what conditions allow for the omission of certain sections, is included in the training data beforehand. The discrepancy (error) between the classification predicted by the model and the actual classification is defined as a loss function, and training proceeds to minimize this loss. By updating the model parameters through this backpropagation mechanism, the trained model can statistically grasp characteristic conversation structures corresponding to products and sales methods (see Figure 7 as an example; the sections circled in the figure correspond to the required sections). Furthermore, by incorporating information on high-performing sales talks (good patterns) accumulated in the evaluation results DB230 into the model's training, the accuracy of segmentation and evaluation support during the inference stage can be improved. Details regarding the classifications will be described later.

[0030] When utilizing a large-scale language model, pre-training has already been performed on a large text dataset. This can be done either by performing additional fine-tuning or by providing instructions to the existing model via prompts without fine-tuning. When fine-tuning is performed, similar to the machine learning model, parameters within the LLM are updated while calculating the error with the correct data (e.g., an example of appropriate chunking) to obtain a model optimized for the desired task. On the other hand, when fine-tuning is not performed, the existing pre-trained model can be directly used, and its suitability for products and sales methods can be improved by adjusting the prompts input to the large-scale language model as output instructions or by using external rules. For example, conditions such as "Explanation of construction is required for product A, and not required for product B" can be incorporated into the prompts. The text output and inference results generated by the LLM can then be compared with other modules of the system (e.g., the control unit 24) as needed, and incorporated as segmentation setting information to divide the conversation data into multiple segments. By combining models that have undergone this training phase, or existing large-scale language models with general-purpose inference capabilities, flexible chunking and evaluation tailored to the characteristics of products and sales methods becomes possible, enabling highly accurate and automated information processing compared to conventional rule-based methods.

[0031] In the inference phase of the trained model 222, the information processing unit 244 inputs operational policy information and conversation data as evaluation target data into the model, and generates and updates segmentation rules as segmentation setting information by utilizing the parameters and knowledge obtained in the training phase. In the case of a machine learning model, it determines, for example, "at what second does the opening of this conversation begin?" or "in product B, does it jump to closing after the price explanation?" according to statistical parameters built on the training data. When using a large-scale language model, it takes into account the results of extensive pre-training to perform more contextual estimation and ranking, and infers the optimal segmentation rules for the product and sales method by combining with external rules and prompt instructions as needed. When combining a conventional machine learning model with LLM, it is also possible to perform multi-stage processing such as the machine learning model re-evaluating summaries and keyword extractions generated by LLM, enabling more accurate analysis of complex sales call flows. These inference results are registered in the segmentation setting information DB 224 and can be applied to actual conversation data in cooperation with other functions of the control unit 24.

[0032] In this embodiment, the segmentation rules registered in the segmentation setting information DB224 include conditional information for dynamically determining whether construction guidance is unnecessary or whether an introduction explanation is mandatory, based on differences in products and sales methods. Specifically, product information as operational policy information (e.g., plans requiring construction, plans not requiring construction) is received by the control unit 24 of the server 12, and the information processing unit 244 of the control unit 24 inputs it into the trained model 222. The trained model 222 treats the product information as a label and analyzes it together with the evaluation results DB230 accumulated in the past to derive segmentation rules such as "construction classification is mandatory for product A" and "construction classification can be omitted for product B". The derived rules are rewritten in the segmentation setting information DB224 and applied to subsequent conversation data processing. As a result, the content of the prompts for inputting items related to the necessity of construction into the large-scale language model and the rule-based settings are updated as needed, eliminating unnecessary steps while ensuring that essential steps are not deleted.

[0033] The segmentation setting information DB224 is a database that records segmentation rules obtained by the aforementioned trained model 222 based on various policy data such as information on products as operational policy information, information on sales methods, internal company regulations, and industry guidelines. It allows for the dynamic definition and updating of conversation segments (chunks) required depending on the product and sales method. Here, information on products as operational policy information includes information on the products (merchandise) and services handled, such as internet line services, mobile communication plans, various insurance products, corporate system implementation support, or personnel placement services, as well as explanations and precautions required based on the characteristics of these products and services. Specific examples include "personnel placement services" as information on products (merchandise) and "construction guidance is essential when selling product A" and "price explanations should be emphasized when providing information on product B," as explanations and precautions. Furthermore, information regarding sales methods includes rules based on sales styles, such as "In sales method B, customer introduction explanations can be omitted." In addition to these, a wide range of policies and regulations that should be considered in actual business operations, such as organization-specific security standards and codes of conduct, and compliance requirements formulated by industry associations, are collectively referred to here as operational policy information. Moreover, "dynamically defining and updating" in this embodiment means that, in response to operational policy information and its changes (such as the addition of new products or revisions to procedures due to changes in sales styles), the classification division rules are automatically or semi-automatically regenerated through processing of trained models 222 and rule-based resetting, and reflected in the division setting information DB 224. For example, if a new product C is added and a case where construction is not required is anticipated, a new classification division rule that allows the omission of construction guidance is created, or existing classification division rules are modified, and the necessary changes are applied each time. This allows the system to flexibly respond to differences in the products handled and changes in diverse sales methods, and to always maintain and update accurate evaluation procedures at the classification level.

[0034] Furthermore, unnecessary and mandatory explanation categories are automatically updated, for example, when new product requirements are added to the operational policy information. For instance, if a service that does not require construction is added as product C, the trained model 222 determines whether or not "construction information" is required and resets it as unnecessary. Even in cases where categories remain based on the existing rule base alone, a prompt stating "Omit construction explanation for product C" is presented to the large-scale language model, and the output inference result is registered in the partitioned configuration information DB 224, so that the construction information category is automatically excluded from subsequent conversation data. The same applies when adding mandatory categories; if a new provision requiring explanation by law is established, the category is registered as "cannot be omitted" in the partitioned configuration information DB 224. These update operations are triggered from the management screen or external system where operational policy information is entered, and the information processing unit 244 dynamically adds and deletes categories using the trained model 222, a pre-registered rule generation module, or a combination of both.

[0035] In this specification, "category" refers to a unit that functionally and logically divides the content of a conversation, such as a sales call or a hearing process. For example, categories that can be organized in stages according to the purpose of the conversation are envisioned, but are not limited to these, such as opening, introductory explanation, price explanation, and closing. The advantages of setting categories are that evaluators can easily follow the flow of the conversation and evaluate each subdivided unit, and sales representatives can clearly identify areas for improvement in specific parts and receive pinpointed feedback. In this embodiment, the processing of the information processing unit 244, described later, allows for flexible switching of which categories are necessary or which can be omitted depending on the product, sales method, industry, etc. In a typical sales call, a series of steps such as opening, proposal, price explanation, confirmation of the other party's response, and closing are categorized, but as mentioned above, in the case of a recruitment service, for example, it is possible to set up unique categories such as hearing the candidate's background and negotiating conditions with the company. Furthermore, it is also possible to set up categories for organizing the flow of the conversation and categories for evaluation separately. By using these variable classification settings, the purpose of each classification and the content of the statements made by the person in charge can be grasped more accurately, and evaluation results and feedback can be presented in detail. These classifications may be explicitly shown to user U as conceptually represented object 42 (see Figures 5 and 6), or they may be used internally without explicit indication.

[0036] The evaluation target data DB226 is a database that stores at least one of the audio data and the transcribed text of the audio data acquired as evaluation target data, as well as additional information such as the date and time of the call, and is used for segmentation and evaluation processing by the information processing unit 244, which will be described later.

[0037] The evaluation item information DB228 is a database that stores evaluation criteria for different categories depending on the product or sales method. In this embodiment, the categories include not only the basic form of dividing into three categories for evaluation: "hearing," "proposal," and "reaction," but also various other forms such as "opening," "introductory explanation," "price explanation," "construction information," "disclaimer explanation," "package discount information," "plan comparison explanation," "questions and answers," and "closing," which are used to organize the flow of the conversation. In addition, as an example, assuming sales for a recruitment service, industry-specific hearing, proposal, and negotiation categories can be considered, such as initiating contact with candidates, hearing about their skills and experience, confirming the proposal content with the company, negotiating contract terms and compensation, and closing through follow-up and contract signing. In this embodiment, by pre-defining evaluation perspectives for each category so that the categories necessary depending on the product or sales method can be freely combined, the content of the divided conversation can be evaluated in a consistent procedure.

[0038] The evaluation item information DB228 includes, as an example for each category, checklist-style evaluation criteria, scoring standards, and standard phrases for comment input. For example, in the category of price explanation, it is possible to set criteria to check whether there are any misunderstandings about price plans or whether sufficient comparison and consideration have been conducted, and to operate the system with a structure that numerically evaluates the accuracy of the explanation, the clarity of the language, and the level of awareness of necessary information. By preparing standard phrases for comment input, it becomes easier to concisely point out cases where discount conditions have not been properly communicated or where there are deficiencies in the explanation of additional options, thus preventing evaluators from overlooking tasks. Furthermore, in the case of products where construction guidance is mandatory, this category can be treated as non-negotiable and managed to ensure that it is always included in the evaluation. For example, in the case of a personnel placement service, it is conceivable that explanations regarding the protection of personal information and the clear disclosure of contract details would be set as mandatory items. In the interview categories shown in Figure 5, "current line," "intended use," "address," "motivation for inquiry," and "key points for implementation" are retrieved from the evaluation item information DB228 and displayed as evaluation item information 40, allowing sales representatives to systematically identify confirmation points that they tend to forget. This ensures that the information to be collected remains clear, while also enabling evaluation based on at least one of the criteria of implementation status, accuracy, or comprehensiveness. By utilizing this evaluation item information 40, when referring to conversation logs divided into multiple chunks in a segmented step, it becomes easier to input scores and add comments using the check points assigned to each category.

[0039] The evaluation results DB230 is a database that stores evaluation results entered by evaluators or obtained from the system. By recording the pass / fail judgment, score, comments, and evaluation time, associated with each segment after division, it can be used for subsequent feedback and analysis. As a specific example, conversation data that has been given a high evaluation as an example of excellent sales is picked out, and the segment labels that should be correct in the training dataset are reassigned based on that conversation data, thereby updating the parameters so that the trained model 222 can estimate more appropriate segment boundaries. By performing such retraining, the system can be configured to continuously incorporate new evaluation examples and improve the accuracy of the segmentation setting information.

[0040] The user information database 232 stores account information and permissions for users U who use this information processing system 10, such as sales representatives and evaluators, enabling access control and permission settings for each user U.

[0041] The control unit 24 is realized by the processor 120 reading and executing a program 220 from memory 122 (see Figure 2) and cooperating with other hardware components. In this embodiment, the control unit 24 includes an information acquisition unit 242 as an evaluation target acquisition unit, an information processing unit 244 as a division unit, and an output unit 246. The information acquisition unit 242 acquires evaluation target data such as voice data and text data. This evaluation target data can be either directly input from the user terminal 14 or read from data already stored in the evaluation target data DB 226. Speech recognition (transcription) of conversation may be handled by the information processing unit 244, which will be described later, but it also includes a form in which the user terminal 14 or an external service receives the text data. Furthermore, the information acquisition unit 242 includes a configuration that checks whether the evaluation target data to be acquired is accessible to the relevant user U by referring to the user information DB 232, and determines the permissions as necessary. Regarding operational policy information, such as information on products and sales methods, this also includes a method in which information entered from external systems or management screens is imported into the server 12 and acquired by the information acquisition unit 242 so that it can be referenced by the information processing unit 244, which will be described later in this specification.

[0042] The information processing unit 244 performs content analysis and segmentation of conversation data by combining information such as the segmentation setting information DB 224 and the output from the trained model 222, while referring to the evaluation target data (voice data or text data) acquired via the information acquisition unit 242, as well as operational policy information regarding products and sales methods. This content analysis and segmentation is performed using the text data of the conversation data. If only the voice data of the conversation is acquired, transcription can be performed not only by calling an external speech recognition service, but also by utilizing the large-scale language model held in advance as the trained model 222 and coordinating with a proprietary speech analysis module or API function. Furthermore, while this embodiment uses the text data of the conversation data for content analysis and segmentation, it is not limited to this, and content analysis and segmentation may also be performed using voice data.

[0043] Furthermore, the information processing unit 244 divides the text data of the conversation data into multiple sections using the section division rules registered in the section division setting information DB 224 as section division setting information, or the section division rules newly generated as section division setting information based on contextual information acquired from a large-scale language model, etc. In this process, the system considers sections defined as essential or optional for each product and sales method, and allows for the pre-exclusion of "unnecessary explanation sections." For example, if product B falls under the category of "plans that do not require construction," the section requiring construction guidance is omitted, or if the sales method is "consultative type," the script for introductory explanation is not made essential. By dynamically reviewing the sections in this way, flexible division according to the actual conversation is possible. "Pre-excluding unnecessary explanation sections" here means that if the explanation content is judged to be redundant or unnecessary under specific conversation conditions, that section is removed from the section division setting information and not processed. The configuration in which the section division rules as section division setting information are obtained using at least one of the data in the section division setting information DB 224, the output of an external system, and the output from a large-scale language model corresponds to the "predetermined means" described in claim 1.

[0044] Furthermore, the information processing unit 244 refers to the evaluation item information DB 228 for each segmented conversation content and performs the process of setting evaluation criteria assigned to each segment (e.g., accuracy of price explanation, likeability of opening, etc.). The segments and evaluation criteria thus determined are presented to the user U through the output unit 246 and stored in the evaluation results DB 230. At this time, summarization functions and keyword extraction using the large-scale language model in the trained model 222 are used in combination to display a digest of only the important points from the conversation (see the content summaries 44 for the "Hearing," "Proposal," and "Partner's Response" items in Figure 5). In addition, if the information processing unit 244 determines that the conversation data contains unnecessary explanation segments, it may perform processes such as automatically assigning a low rating to the entire conversation.

[0045] The output unit 246 is responsible for presenting conversation data categorized by the information processing unit 244, and evaluation results input or acquired for each category, via the user interface 34 as shown in Figure 5. Specifically, it displays evaluation item information 40 for each category and generates a screen (not shown) on the user terminal 14 used by sales representatives and evaluators, where evaluation results can be input and viewed. In addition, it outputs objects 42 that conceptually represent each category arranged in chronological order, as described above, and displays content summaries 44 for each category.

[0046] (Processing performed by the information processing system 10) Next, the operation of the information processing system 10 according to this embodiment will be described. Figure 4 is a flowchart showing an example of the processing flow by the information processing system 10. The processor 120 reads the program 220 stored in the storage 124, loads it into the memory 122, and executes the following processes by running the program 220. Although not shown in the figures, the processor 120 terminates the processing based on the program 220 when it receives termination information or when it determines, based on an internal determination, that no further processing is necessary.

[0047] The processor 120 acquires the data to be evaluated (step S100). Step S100 corresponds to the "data acquisition step" described in claim 1. Specifically, it accepts voice data from telephone sales calls or already acquired text data as input from the user terminal 14, etc. If the data to be evaluated cannot be acquired, it waits for the input of such data and repeats the process until the acquisition is completed.

[0048] The processor 120 acquires operational policy information (step S102). This operational policy information includes, as mentioned above, product information, sales method information, explanation methods and evaluation criteria which differ for each product. If operational policy information is not available, or if cooperation with an external system has not been established, and the operational policy information cannot be acquired, the processor 120 repeats the process in this step until the information has been acquired.

[0049] The processor 120 determines whether the conversation data to be evaluated is text data or not (step S104). If the conversation data is not text data (step S104: NO), the processor 120 performs a transcription process on the conversation data to obtain text data (step S106). Once the transcription process is complete, the process moves on to step S108, which will be described later. On the other hand, if the conversation data is already text data (step S104: YES), the process proceeds directly to step S108.

[0050] The processor 120 performs at least one of generating and referencing segmentation setting information (step S108). Specifically, the following forms are envisioned. First, when dealing with newly handled products or sales methods that differ significantly from conventional ones, the processor 120 may analyze the output from the trained model 222 and the accumulated past conversation logs to select the necessary segments and generate new segmentation setting information. For example, if a segment called "campaign explanation," which did not exist before, becomes necessary, the processor 120 automatically adds this new segment to the standard segments such as opening and price explanation, and generates optimized segmentation setting information. It performs key point extraction and keyword analysis on the transcribed conversation data to accurately identify which parts correspond to the PR portion of the new campaign, defines segments by using the range as a delimiter, and then makes this new segment usable as a rule. On the other hand, if standard scripts or rules for products that have been used for many years already exist, the processor 120 refers to them and applies the previously registered essential segments such as opening and disclaimer explanation as they are. Furthermore, if it is determined that the currently handled products and sales methods can be covered by existing rules, there is no need to generate new segmentation setting information; processing proceeds simply by applying the referenced existing rules and making the necessary minor changes. In addition, in some cases, referencing and generation are performed simultaneously, such as generating categories that correspond to the introduction of new brands or the addition of new functions, while partially referencing existing rules. For example, by adding a "new proposal category" to the conventional category structure, and referencing and continuing with the conventional price explanation and closing, while significantly revising some categories and handling new and old categories in an integrated manner, more flexible and effective segmentation setting information can be obtained. In this way, by freely switching between generation and referencing, or a combination thereof, the processor 120 establishes the optimal segmentation setting information in step S108, making it available for subsequent segmentation and evaluation processing.

[0051] The processor 120 divides the conversation data into multiple sections using the division setting information (step S110). In this embodiment, based on pre-generated or referenced division setting information, it identifies which sections are necessary and divides the conversation into multiple parts according to those sections.

[0052] The processor 120 performs an evaluation for each of the divided sections (step S112). Here, it determines the content of the conversation data and the quality of the response by comparing it with the evaluation criteria (evaluation item information 40 based on operational policy information) corresponding to each section.

[0053] The processor 120 performs processing including outputting the evaluation results to the display of the user terminal 14 or linking them to an external tool (step S114). Step S114 is an example of the final form of the "output step" described in claim 1. Specifically, it includes displaying the evaluation score and presenting improvement comments.

[0054] The processor 120 stores the evaluation results and other data in the storage unit 22 (step S116). The evaluation results, text data, partition setting information, etc., are saved in the storage unit 22 so that they can be used for later analysis and report creation. After step S116, the processing based on this program 220 is completed.

[0055] (Effects of this embodiment) According to the information processing system 10 of this embodiment, the system performs an evaluation target acquisition step to acquire evaluation target data to be evaluated, a division step to divide the evaluation target data into multiple sections using division setting information obtained by predetermined means based on operational policy information, and an output step to output evaluation results obtained by at least one of input and acquisition for each divided evaluation target data. In other words, by managing evaluation target data for each section, evaluators do not need to grasp long audio or text at once throughout the entire recording, and can accurately evaluate by sequentially checking the divided step units. Furthermore, by using division setting information obtained by predetermined means based on operational policy information, it is possible to flexibly accommodate various ways of dividing the data, such as omitting unnecessary steps or adding new explanation steps depending on the product or sales method (see Figures 5 and 6). For example, for products that do not require construction, the construction explanation step can be omitted, and campaign information for new products can be added as a required item, enabling optimal section settings that match the operational policy. As a result, evaluators can more easily focus on the important parts of the conversation, suppressing oversights and judgment errors, and efficiently grasp the quality and areas for improvement of each section. This makes it easier to conduct evaluations.

[0056] Furthermore, in the output step, the information processing system 10 according to this embodiment outputs evaluation item information 40 for each section of the divided evaluation target data. That is, as shown in Figure 5, for example, the evaluation item information 40 can be presented in tag format for each section, including its implementation status (for example, a check mark if completed, and an "×" if not completed), and tags such as "address," "purpose of use," and "reason for inquiry" can be assigned to the required interview items. Based on these tags, the evaluator can easily determine whether the necessary items in each section have been correctly interviewed or whether any confirmation has been missed. As a result, the evaluator does not lose sight of "from what perspective" and "which items" should be checked, and can quickly provide appropriate next actions or correction instructions for each section. This streamlines the overall evaluation work and enables consistent feedback that is less likely to be missed. This makes it easier to conduct the evaluation. In this embodiment, the evaluation item information 40 is in tag format, but it is not limited to this, and other formats that can identify and organize information by section may be used, such as checklists, icons, tabular displays, and text.

[0057] Furthermore, since the information processing system 10 according to this embodiment obtains evaluation item information based on operational policy information, it can automatically prepare evaluation item information that conforms to the regulations of the product, sales method, and organization. This simplifies the process of removing unnecessary criteria from a large number of candidates and adopting only the necessary criteria. This makes it even easier to conduct evaluations.

[0058] Furthermore, in the output step, the information processing system 10 according to this embodiment outputs the multiple sections of the data to be evaluated, which were divided in the division step, in a display arranged chronologically. Therefore, the content of each section can be visualized in accordance with the progress of the call or work, and the evaluator can perform the evaluation of each section without losing sight of the overall flow. This makes it easier to carry out the evaluation.

[0059] In this embodiment, the information processing unit 244 performs the evaluation for each of the divided sections. However, the configuration is not limited to this, and the user U may perform the evaluation, input the results, and acquire and output them, or another system may perform the evaluation, input the results, and acquire and output them.

[0060] Furthermore, although the information processing system 10 according to this embodiment has been described primarily in the context of voice communication via sales calls, the configuration is also applicable to text-based communication such as email and chat. When the body of an email or chat exchange is the data to be evaluated, the transcription process is unnecessary, and the data can be directly obtained as text data. The information processing unit 244 of the control unit 24 divides this text data and extracts introductory sections equivalent to the opening and proposal sections. If necessary, by defining essential sections according to the products and sales methods, it is possible to automatically determine where in the email text or chat log the price explanation and disclaimer explanation are included. Therefore, similar to voice data, the evaluation work can be simplified by omitting unnecessary items and emphasizing essential items. In addition, by registering evaluation criteria related to email-specific formatting rules and greetings to recipients in the evaluation item information DB 228, it can be used as a tool to comprehensively evaluate all types of communication other than call centers.

[0061] Furthermore, when text-based communication such as emails and chats are used as evaluation data, the absence of transcription from audio results in slight differences in the timing of conversation data updates and the method of estimating section boundaries. Specifically, in email text, sections such as opening greetings, proposals, and conclusions may be separated into paragraphs in the source code. Therefore, the information processing unit 244 of the control unit 24 applies section division rules based on paragraph information and keywords at the beginning of sentences. While audio data requires the estimation of divisions according to chronological order, text-only data allows for relatively easy identification of divisions by utilizing existing paragraph structures and line break characters. Consequently, in emails and chats, the same section setting information as in the case of audio can be dynamically applied by determining whether an opening greeting is included at the beginning of the conversation log or whether a signature block corresponding to a closing exists.

[0062] Furthermore, in this specification, the step of describing a program to be recorded on a recording medium includes not only processes that are performed chronologically in that order, but also processes that are not necessarily performed chronologically but are executed in parallel or individually. Also, in this specification, the term "system" refers to an overall system composed of multiple devices, means, etc.

[0063] (Variation 1: Advanced real-time analysis and live support functions) Furthermore, looking at the information processing system described above from a different perspective, the problem (objective) that the information processing system according to this embodiment aims to solve can also be understood as "analyzing conversation content in real time and presenting necessary auxiliary information (such as recommended scripts and alerts) during a call based on the analysis results."

[0064] Given the problem as described above, an invention as a means to solve the problem might look like this: "An information processing method executed by an information processing device, An evaluation target acquisition step that receives audio data during a call and generates input information including transcription and related metadata, A model inference step in which the aforementioned input information is input to a trained model and output information including a recommended script or alert information is obtained, An evaluation step of comparing and evaluating the output information with predetermined information such as predefined operational policy information or product rule information, When the output information is determined to be dissimilar based on the evaluation, an adjustment step is taken to modify a portion of the input information. An information processing method including an output step of presenting the output information obtained again after the adjustment step to the sales representative on the phone.

[0065] According to the above configuration, first, the call audio is transcribed in real time (evaluation target acquisition step), and this transcribed data and product information are input into a trained model (model inference step). Based on this input information, the model generates "output information" such as recommended scripts and alert information. However, if this does not match the predetermined information (e.g., required explanation items or applicable sales policies) (i.e., it is evaluated as dissimilar), the system's control unit readjusts the input information and performs model inference again (adjustment step). By immediately feeding back the output information obtained through such adjustments to the sales representative during the call, it is possible to improve both the quality of the conversation and sales efficiency while gradually enhancing real-time analysis and live support.

[0066] (Variation 2: Enhanced integration with large-scale language models) Furthermore, looking at the information processing system described above from a different perspective, the problem (objective) that the information processing system according to this embodiment aims to solve can also be understood as "achieving flexible chunking and evaluation support by utilizing a large-scale language model (LLM)."

[0067] Given the problem as described above, an invention as a means to solve the problem might look like this: "An information processing method executed by an information processing device, An acquisition step to obtain specified predetermined information and output information which is the result of inputting predetermined input information into a trained model, An evaluation step of comparing and evaluating the predetermined information and the output information, An adjustment step in which the input information is modified if the results of the evaluation are dissimilar, Information processing methods including [this].

[0068] With the above configuration, conversational data can be segmented based on the output information provided by the Large-Scale Language Model (LLM), and simultaneously compared with rule-based operational policy information (predetermined information). If the evaluation results deviate from the intended segmentation settings, the input information (e.g., prompts and rules) can be updated to improve accuracy, enabling flexible chunk segmentation that responds immediately to changes in products or sales methods.

[0069] (Variation 3: Enhancement of semantic search and summarization functions) Furthermore, looking at the information processing system described above from a different perspective, the problem (objective) that the information processing system according to this embodiment aims to solve can also be understood as "to extract semantic summaries and important words from conversation logs and obtain useful insights from large amounts of data."

[0070] Given the problem as described above, an invention as a means to solve the problem might look like this: "An information processing method executed by an information processing device, The evaluation target acquisition step involves acquiring evaluation target data, including conversation logs, and acquiring contextual information and operational policy information contained in said evaluation target data. A model inference step in which the aforementioned data to be evaluated is input into a trained model and summarizes or extracts keywords, An evaluation step involves comparing and evaluating the summary or extraction results obtained in the model inference step with predetermined reference information. If the output information is determined to be dissimilar in the evaluation step, an adjustment step is taken to adjust the input information. An output step which obtains and outputs updated output information by performing model inference again after the adjustment step, Information processing methods including [this].

[0071] According to the above configuration, conversation logs and other data are first collected in the evaluation target acquisition step, and a model inference step focusing on semantic search and summarization is performed. Next, the obtained summaries and key phrases (output information) are compared with predetermined information requested by the user (for example, "expected summary content" or "request to include specific keywords"), and if dissimilar information is found, the input information is modified in the adjustment step (e.g., reviewing the search query or summarization instructions, switching the time range to consider). By repeating this process, the accuracy of summarization and keyword extraction can be improved step by step. As a result, it becomes possible to enhance semantic search and key point extraction so that users can gain useful insights from conversation log data.

[0072] (Variation 4: Multi-channel integrated analysis) Furthermore, looking at the information processing system described above from a different perspective, the problem (objective) that the information processing system according to this embodiment aims to solve can also be understood as "to enable centralized segmentation and evaluation of communication logs spanning multiple channels such as telephone, email, and chat, thereby further improving the efficiency and accuracy of operations."

[0073] Given the problem as described above, an invention as a means to solve the problem might look like this: "An information processing method executed by an information processing device, An evaluation target acquisition step that acquires evaluation target data across multiple channels, An inference step in which evaluation target data obtained from the aforementioned multiple channels is integrated and classification is determined using at least one of a trained model and a predetermined rule while referring to operational policy information, An evaluation step of comparing the aforementioned determination result with predetermined information, If the evaluation results are dissimilar, an adjustment step is made to readjust the input information. An output step that outputs evaluation results for each category in each channel based on the category settings determined after the adjustment step, Information processing methods including [this].

[0074] According to the above configuration, first, in the evaluation target acquisition step, logs from multiple channels (telephone, email, chat, etc.) are acquired. Next, the model inference step and rule application step integrate these logs and determine the classification. Here, settings such as "telephone and email are linked for product A" and "chat interactions are also integrated as the same customer" are referred to as operational policy information, and optimal division and organization are performed using a trained model or a rule generation module that generates predetermined rules. The output information obtained in this way is compared and evaluated with predetermined information (e.g., criteria necessary for integrated evaluation, essential interview items, etc.), and if it is concluded that they are dissimilar, the input information (target channels, analysis scope, etc.) is updated in the adjustment step. Finally, by obtaining and outputting evaluation results for each classification across multiple channels, it becomes possible to analyze conversations and text across channels, thereby improving the efficiency of sales activities and the accuracy of evaluations through multi-channel integrated analysis.

[0075] (Variation 5: Estimation of emotions and psychological state, and collaboration with customer success) Furthermore, looking at the information processing system described above from a different perspective, the problem (objective) that the information processing system according to this embodiment aims to solve can also be understood as "estimating emotions and psychological states from conversation data, enabling early detection and follow-up of customer dissatisfaction."

[0076] Given the problem as described above, an invention as a means to solve the problem might look like this: "An information processing method executed by an information processing device, A step to acquire data to be evaluated and to use it as input information including parameters for estimating sentiment indicators, A model inference step in which the aforementioned input information is input into a trained model to obtain output information including dissatisfaction scores and emotion categories, An evaluation step which involves comparing the output information with predetermined information such as predefined customer satisfaction standards or follow-up response conditions, In the evaluation step, if the output information is determined to be dissimilar, an adjustment step is made to adjust a portion of the input information. After the adjustment step, the model inference is performed again, and an output step is performed to generate a trigger based on the updated output information. Information processing methods including [this].

[0077] According to the above configuration, first, the system acquires the call audio or chat logs to be evaluated (evaluation target acquisition step), and then provides this input information to a trained model to estimate the customer's emotions and level of dissatisfaction (model inference step). The output emotion indicators and scores are then compared against the response criteria defined by the customer success department (for example, "follow up immediately if the level of dissatisfaction is above a certain value") (evaluation step). If the results do not match the criteria (e.g., the level of dissatisfaction is not accurately captured), the system modifies the input information (parameter settings for calculating emotion indicators, partial correction of call data, etc.) in the adjustment step, and then runs model inference again to improve the accuracy of emotion estimation. Finally, in the output step, the system automatically generates a trigger (instruction information) indicating that follow-up procedures should be taken based on the updated output information. In this way, appropriate customer success responses can be provided according to the customer's emotions and psychological state, thereby maintaining and improving customer satisfaction.

[0078] (Variation 6: Enhancement of regulatory and compliance functions) Furthermore, looking at the information processing system described above from a different perspective, the problem (objective) that the information processing system according to this embodiment aims to solve can also be understood as "to strictly consider various laws and regulations and industry-specific compliance requirements when analyzing conversational data, and to enable early detection and management of prohibited words, etc."

[0079] Given the problem as described above, an invention as a means to solve the problem might look like this: "An information processing method executed by an information processing device, An evaluation target acquisition step involves acquiring operational policy information that includes at least one of the regulatory and compliance requirements, and acquiring data to be evaluated. A model inference step or rule application step in which the data to be evaluated is input into a trained model and the detection result of a specific expression is obtained as output information, An evaluation step which involves comparing the output information with predetermined information and evaluating the degree of fit, If a non-conformity is determined in the evaluation step, an adjustment step is performed to change some of the content of the evaluation target data or the input parameters to the model. An output step that presents the verification results of the evaluation step, which is performed again after the adjustment step, Information processing methods including [this].

[0080] According to the above configuration, the system first acquires operational policy information, including at least one of the regulatory and compliance requirements (e.g., prohibited language in industry guidelines, restricted expressions based on the Personal Information Protection Act), and evaluation target data, including conversation logs (evaluation target acquisition step). Subsequently, a model inference step is performed to detect compliance risks using a trained model, and an evaluation step is executed to compare the detection results (output information) with specified information (internal regulations, laws, etc.). If a non-conformity is determined (in the case of dissimilarity), the detection accuracy can be improved by changing the input parameters or the analysis target (adjustment step), or the guidelines can be revised and the NG word check can be performed again to ensure accurate compliance audits. Finally, the output step presents the detected areas and points requiring correction, supporting the consideration of risk reduction and recurrence prevention measures for the conversation. This has the effect of enabling safe business activities that comply with laws and regulations and industry rules.

[0081] (Modification 7: Improving the efficiency of automated learning and model updates) Furthermore, looking at the information processing system described above from a different perspective, the problem (objective) that the information processing system according to this embodiment aims to solve can also be understood as "utilizing evaluation results acquired in the past as retraining data, and updating the trained model as appropriate to always keep up with the latest specifications of products and sales methods."

[0082] Given the problem as described above, an invention as a means to solve the problem might look like this: "An information processing method executed by an information processing device, An evaluation target acquisition step that acquires past evaluation target data and evaluation results assigned to said evaluation target data, Based on the evaluation results, a model update step is performed to retrain or update the trained model, A model inference step in which new evaluation target data is input to the updated trained model and output information related to segmentation and evaluation support is obtained, An evaluation step of comparing the output information with predetermined information, If the evaluation step determines that the models are not similar, the adjustment step involves readjusting the input information and repeating the model update step as necessary. An output step that presents the finalized output information to the user, Information processing methods including [this].

[0083] According to the above configuration, the evaluation target acquisition step first acquires past evaluation target data, such as call logs and text data, along with their evaluation results (good / bad patterns, etc.). Then, in the model update step, the trained model is retrained or its parameters are adjusted. Next, the updated model generates output information (segmentation and evaluation results) for the actual evaluation target data, which is then compared and evaluated against predetermined information (product rules and evaluation criteria). By modifying input information (data range, product labels, etc.) as needed, the optimal segmentation settings and evaluation support can be established step by step. This allows for quick adaptation to new products and changed sales methods, resulting in improved model accuracy and business integration.

[0084] <Note> This embodiment includes the following disclosures.

[0085] (Note 1) An information processing method performed by an information processing device, The evaluation target acquisition step involves acquiring the data to be evaluated, A division step in which the data to be evaluated is divided into multiple sections using division setting information obtained by a predetermined means based on operational policy information, An output step that outputs an evaluation result obtained by input and acquisition for each of the divided evaluation target data, Information processing methods including

[0086] (Note 2) In the output step, evaluation item information is output for each of the divided evaluation target data. The information processing method described in Appendix 1.

[0087] (Note 3) The aforementioned evaluation item information is obtained based on the aforementioned operational policy information. The information processing method described in Appendix 2.

[0088] (Note 4) In the output step, the multiple segments of the data to be evaluated, which were divided in the division step, are output in a display arranged in chronological order. The information processing method described in any one of the items in Appendix 1 to Appendix 3.

[0089] (Note 5) In an information processing device, The evaluation target acquisition step involves acquiring the data to be evaluated, A division step in which the data to be evaluated is divided into multiple sections using division setting information obtained by a predetermined means based on operational policy information, An output step that outputs an evaluation result obtained by input and acquisition for each of the divided evaluation target data, A program for executing an information processing method that includes this method.

[0090] (Note 6) An information processing system executed by an information processing device, An evaluation target acquisition unit that acquires the evaluation target data to be evaluated, A division unit that divides the data to be evaluated into multiple sections using division setting information obtained by predetermined means based on operational policy information, An output unit that outputs an evaluation result obtained by input and acquisition for each of the divided evaluation target data, An information processing system having

[0091] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the present invention is indicated by the claims, not in the sense described above, and is intended to include all modifications in the sense and scope equivalent to the claims. Furthermore, the present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the claims, and embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of Symbols]

[0092] 10. Information Processing Systems 12. Server (Information Processing Device) 40 Evaluation Item Information 42 Evaluation Result Display Section 220 programs 242 Information Acquisition Unit (Evaluation Target Acquisition Unit) 244 Information Processing Unit (Divided Section) 246 Output section

Claims

1. An information processing method performed by an information processing device, The evaluation target acquisition step involves acquiring the data to be evaluated, A division step in which the data to be evaluated is divided into multiple sections using division setting information obtained by a predetermined means based on operational policy information, An output step that outputs an evaluation result obtained by input and acquisition for each of the divided evaluation target data, Information processing methods including

2. In the output step, evaluation item information is output for each of the divided evaluation target data. The information processing method according to claim 1.

3. The aforementioned evaluation item information is obtained based on the aforementioned operational policy information. The information processing method according to claim 2.

4. In the output step, the multiple segments of the data to be evaluated, which were divided in the division step, are output in a display arranged in chronological order. The information processing method according to claim 1.

5. In an information processing device, The evaluation target acquisition step involves acquiring the data to be evaluated, A division step in which the data to be evaluated is divided into multiple sections using division setting information obtained by a predetermined means based on operational policy information, An output step that outputs an evaluation result obtained by input and acquisition for each of the divided evaluation target data, A program for executing an information processing method that includes this method.

6. An information processing system executed by an information processing device, An evaluation target acquisition unit that acquires the evaluation target data to be evaluated, A division unit that divides the data to be evaluated into multiple sections using division setting information obtained by predetermined means based on operational policy information, An output unit that outputs evaluation results obtained by input and acquisition for each of the divided evaluation target data, An information processing system having

Images