Program, information processing system, and information processing method

Abstract

This program provides an inference for appropriate response memos during phone calls between users and customers. [Solution] A program comprising a processor and a memory unit, for causing a computer to manage data related to a call between a user and a customer, wherein the program causes the processor to perform the following steps: receiving voice data related to the call; acquiring call attributes of the call (S302); selecting a learning model based on the call attributes; and inference step (S303) inferring a response memo for the call by applying the voice data to the selected learning model.

Description

Technical Field

[0001] The present disclosure relates to a program, an information processing system, and an information processing method.

Background Art

[0002] Conventionally, in an information processing system that provides a call service, an assistance system for an operator is known. Patent Document 1 discloses a call voice data providing system with metadata that receives and stores call voice data from mobile terminals respectively and reproduces and provides the call voice data desired by each. Patent Document 2 discloses a call handling evaluation system that analyzes the voice information of an operator during a call to evaluate the operator's call handling. Patent Document 3 discloses a response presentation device for assisting an operator of a call center.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the case of attaching a response memo to a call based on the call voice related to the call between a user and a customer, it has not been possible to infer a more suitable response memo.

[0005] Therefore, the present disclosure has been made to solve the above problems, and an object thereof is to provide a technique for inferring a response memo more suitable for a call with respect to a call made between a user and a customer. [Means for solving the problem]

[0006] A program comprising a processor and a memory unit, which causes a computer to manage data related to a call between a user and a customer, the program causing the processor to perform the steps of: receiving voice data related to the call; acquiring call attributes of the call; selecting a learning model based on the call attributes; and inference applying the voice data to the selected learning model to infer a response memo for the call. [Effects of the Invention]

[0007] According to this disclosure, it is possible to infer a response memo that is more appropriate for a call between a user and a customer. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram showing the overall configuration of information processing system 1. [Figure 2] This block diagram shows the functional configuration of Server 10. [Figure 3] This is a block diagram showing the functional configuration of user terminal 20. [Figure 4] This is a diagram showing the functional configuration of CRM system 30. [Figure 5] This is a block diagram showing the functional configuration of customer terminal 50. [Figure 6] This diagram shows the data structure of user table 1012. [Figure 7] This diagram shows the data structure of organization table 1013. [Figure 8] This diagram shows the data structure of the call table 1014. [Figure 9] This diagram shows the data structure of the speech recognition table 1015. [Figure 10] This diagram shows the data structure of summary table 1016. [Figure 11]It is a diagram showing the data structure of the response memo table 1017. [Figure 12] It is a diagram showing the data structure of the customer table 3012. [Figure 13] It is a diagram showing the data structure of the response history table 3013. [Figure 14] It is a flowchart showing the operation of the summarization process (first embodiment). [Figure 15] It is a flowchart showing the operation of the summarization process (second embodiment). [Figure 16] It is a flowchart showing the operation of the response memo assignment process. [Figure 17] It is a flowchart showing the operation of the response memo proposal process. [Figure 18] It is a flowchart showing the operation of the score calculation process. [Figure 19] It is a flowchart showing the operation of the CRM storage process. [Figure 20] It is a flowchart showing the operation of the call display process. [Figure 21] It is a diagram showing the outline of the binding process in the summarization process (first embodiment). [Figure 22] It is a diagram showing the outline of the additional extraction process in the summarization process (second embodiment). [Figure 23] It is a diagram showing the outline of the summary display process. [Figure 24] It is a diagram showing a screen example of the CRM service in the call display process. [Figure 25] It is a block diagram showing the basic hardware configuration of the computer 90.

Modes for Carrying Out the Invention

[0009] The embodiments of this disclosure will be described below with reference to the drawings. In all the drawings illustrating the embodiments, common components are denoted by the same reference numerals, and repeated explanations are omitted. The following embodiments are not intended to unduly limit the content of this disclosure as described in the claims. Not all components shown in the embodiments are necessarily essential components of this disclosure. Also, each drawing is a schematic diagram and is not necessarily a strict illustration.

[0010] <Overview of Information Processing System 1> Figure 1 is a diagram showing the overall configuration of Information Processing System 1. Information Processing System 1 in this disclosure is an information processing system that provides the telephone service related to this disclosure. Information Processing System 1 is an information processing system that provides services related to telephone calls between users and customers, and also stores and manages data related to telephone calls.

[0011] <Basic Configuration of Information Processing System 1> Figure 1 shows the information processing system 1 in this disclosure. The information processing system 1 is configured to include a server 10, multiple user terminals 20A, 20B, 20C, a CRM system 30, a voice server (PBX) 40, and customer terminals 50A, 50B, 50C connected to the voice server (PBX) 40 via a telephone network T, all connected via a network N. Figure 2 is a block diagram showing the functional configuration of server 10. Figure 3 is a block diagram showing the functional configuration of user terminal 20. Figure 4 is a block diagram showing the functional configuration of CRM system 30. Figure 5 is a block diagram showing the functional configuration of customer terminal 50.

[0012] Server 10 is an information processing device that provides services for storing and managing data (call data) related to calls made between users and customers.

[0013] The user terminal 20 is an information processing device operated by a user of the service. The user terminal 20 may be, for example, a stationary PC (Personal Computer), a laptop PC, or a mobile device such as a smartphone or tablet. It may also be a wearable device such as an HMD (Head Mount Display) or a smartwatch.

[0014] A CRM system 30 is an information processing device managed and operated by a provider of CRM (Customer Relationship Management) services (CRM provider). Examples of CRM services include Salesforce, HubSpot, Zoho CRM, and kintone.

[0015] The voice server (PBX) 40 is an information processing device that functions as a switchboard, enabling calls between the user terminal 20 and the customer terminal 50 by connecting the network N and the telephone network T to each other.

[0016] The customer terminal 50 is an information processing device that the customer operates when making a call with the user. The customer terminal 50 may be, for example, a mobile device such as a smartphone or tablet, or a stationary PC (Personal Computer) or laptop PC. It may also be a wearable device such as an HMD (Head Mount Display) or a smartwatch.

[0017] Each information processing device consists of a computer equipped with an arithmetic unit and a memory device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by said hardware configuration will be described later. For each of the server 10, user terminal 20, CRM system 30, voice server (PBX) 40, and customer terminal 50, explanations that overlap with the basic hardware configuration and basic functional configuration of the computer described later will be omitted.

[0018] The configuration and operation of each device are described below.

[0019] <Server 10 Functional Configuration> Figure 2 shows the functional configuration realized by the hardware configuration of server 10. Server 10 includes a storage unit 101 and a control unit 104.

[0020] <Configuration of the storage unit of Server 10> The storage unit 101 of the server 10 includes an application program 1011, a user table 1012, an organization table 1013, a call table 1014, a speech recognition table 1015, a summary table 1016, and a response memo table 1017. Figure 6 shows the data structure of the user table 1012. Figure 7 shows the data structure of the organization table 1013. Figure 8 shows the data structure of the call table 1014. Figure 9 shows the data structure of the speech recognition table 1015. Figure 10 shows the data structure of the summary table 1016. Figure 11 shows the data structure of the response memo table 1017.

[0021] User Table 1012 is a table that stores and manages information about member users (hereinafter referred to as "users") who use the service. When a user registers to use the service, their information is stored in a new record in User Table 1012. This allows the user to use the service related to this disclosure. User Table 1012 has User ID as its primary key and contains columns for User ID, CRM ID, Organization ID, User Name, Integration Mode, User Attributes, and Evaluation Metrics.

[0022] The User ID is an item that stores user identification information used to identify a user. The CRMID is an item in the CRM system 30 that stores identification information for identifying a user. Users can access CRM services by logging into the CRM system 30 using their CRMID. In other words, the user ID on the server 10 and the CRMID on the CRM system 30 are linked. The Organization ID is an item that stores the Organization ID of the organization to which the user belongs. The username field is where the user's name is stored. The integration mode is an item that stores the settings (integration settings) for when data related to calls between users and customers is stored in an external CRM system. In this disclosure, the collaboration mode is stored for each user; however, it may also be stored for each organization and department in the organization table. In that case, the collaboration mode applied to each user will be applied by referring to the collaboration mode for each organization and department stored in the organization table. In this case, a uniform collaboration mode can be applied to users belonging to an organization or department. User attributes are fields that store information about the user's attributes, such as age, gender, place of origin, dialect, and occupation (sales, customer support, etc.). The evaluation metrics are items that store quantitative evaluation indicators of the user's call handling skills. Specifically, the evaluation metrics are numerical values ​​calculated by applying a predetermined algorithm to each of the indicators of the analysis data (Talk:Listen ratio, number of silences, number of overlaps, number of rallies, basic frequency, intonation strength, speaking speed, number of fillers, talk script match rate, etc.) obtained by analyzing the user's past voice data. In fields such as inside sales, evaluation metrics are equivalent to quantitatively representing each user's customer service skills, and users with higher evaluation metrics are expected to achieve higher sales performance.

[0023] Organization Table 1013 is a table that defines information about the organizations to which a user belongs. Organizations include any organization or group, such as companies, corporations, corporate groups, clubs, and various other organizations. Organizations may also be defined for more detailed subgroups, such as departments within a company (sales department, general affairs department, customer support department). Organization Table 1013 is a table with Organization ID as the primary key and has columns for Organization ID, Organization Name, and Organization Attributes.

[0024] The Organization ID is an item that stores organizational identification information used to identify an organization. The "Organization Name" field is used to remember the name of an organization. This field includes any organization or group name, such as company names, legal entity names, corporate group names, club names, or various other group names. Organizational attributes are items that store information about the attributes of an organization, such as the type of organization (company, corporate group, other organization, etc.) and industry (real estate, finance, etc.).

[0025] The call table 1014 is a table that stores and manages call data related to calls made between a user and a customer. The call table 1014 has a call ID as its primary key and includes columns for call ID, user ID, customer ID, call category, call type, voice data, whether or not speech recognition was performed, whether or not summarization was performed, and analysis data.

[0026] The call ID is an item that stores call data identification information used to identify call data. The User ID is an item used to store the user's User ID (user identification information) during phone calls between the user and the customer. The Customer ID is an item used to store the customer's ID (customer identification information) during a call between the user and the customer. The call category is an item that stores the type (category) of call made between the user and the customer. Call data is classified by call category. Depending on the purpose of the call between the user and the customer, the call category may store values ​​such as telephone operator, telemarketing, customer support, or technical support. The "Call Type" field stores information to distinguish whether a call between the user and a customer was initiated by the user (outbound) or received by the user (inbound). The audio data field stores audio data from conversations between the user and the customer. Various audio data formats, such as mp4 and wav, can be used. It may also store reference information (paths) to audio data files located elsewhere. The audio data may be in a format in which identifiers are set that allow the user's voice and the customer's voice to be independently identified. In this case, the control unit 104 of the server 10 can perform independent analysis processing on the user's voice and the customer's voice. In this disclosure, video data containing audio information may be used instead of audio data. Furthermore, in this disclosure, "audio data" is a concept that also includes audio data contained within video data. The "Speech Recognition Status" field stores information to determine whether or not speech recognition processing was performed on the audio data of a call between the user and the customer. If speech recognition processing was performed on the audio data, information indicating that speech recognition processing was performed is stored. If speech recognition processing was not performed on the audio data, information indicating that speech recognition processing was not performed, such as a blank, null, or other information, is stored. The "Summarization Status" field stores information to determine whether or not summarization processing was performed on the audio data of a call between the user and the customer. If summarization processing was performed, information indicating that summarization processing was performed is stored. If summarization processing was not performed, a blank, null, or other information indicating that summarization processing was not performed is stored. The analysis data is an item that stores analysis data obtained by analyzing the audio data of calls made between the user and the customer. The analysis data includes Talk:Listen ratio, number of silences, number of overlaps, number of rallies, base frequency, intonation strength, speaking speed, number of fillers, and talk script match rate.

[0027] The speech recognition table 1015 is a table that stores speech recognition information consisting of utterance time, speaker, and text obtained by performing speech recognition processing on the audio data of a call made between a user and a customer. The speech recognition table 1015 is a table that has columns for call ID, text, utterance time, and speaker.

[0028] The call ID is an item that stores the call ID (call data identification information) of the call data that is the source of the voice recognition information. The text field stores the text information of the text recognition results for each section (utterance section) of the audio data of a call between a user and a customer. Specifically, the content of the sentences spoken by the speaker in each utterance section of the audio data is stored as text data. The speech time is an item that stores the start time within the speech data (segment speech data) of a speech segment. The speech time may also be any time related to each speech segment, such as the start time of each speech segment, the end time of each speech segment, or any time in between each speech segment. The speaker is identified by information stored in the audio data for each section. Specifically, this is information used to identify the user or customer. User identification information, such as a user ID or customer ID, may also be stored to identify the speaker.

[0029] The summary table 1016 is a table that stores summary information consisting of utterance time, speaker, and text obtained by summarizing speech recognition information of a call between a user and a customer. The summary information is information consisting of text that relates to and characterizes the call between the user and the customer. By checking the summary information, the user can quickly understand the content of the call between the user and the customer. The summary table 1016 is a table that has columns for call ID, text, utterance time, and speaker.

[0030] The call ID is an item that stores the call ID (call data identification information) of the call data from which the summary information is generated. The text field is an item that stores the text of the speech recognition information extracted as summary information. The speech time field is an item that stores the time of speech utterance from the speech recognition information extracted as summary information. The speaker is an item that stores the speaker of the speech recognition information extracted as summary information.

[0031] The response memo table 1017 is a table that stores and manages information about response memos that are associated with call data related to calls between users and customers. Users can organize and manage a large amount of call data by setting (assigning) response memos to the call data. In addition, when the server 10 performs various control processing, it can change the processing conditions using the response memos attached to the call data. The response memo table 1017 has a memo ID as its primary key and has columns for memo ID, call ID, assigner ID, memo content, and memo date and time.

[0032] The memo ID is an item that stores the memo identification information used to identify the memo in question. The call ID is an item that stores the call ID (call data identification information) of call data to which a response memo has been attached. The assigner ID is an item that stores the user ID of the user who assigned the response memo to the call data. The memo content field stores the content of the response memo attached to the call data. The memo content is usually text information. The memo date and time field stores the date and time when the user added a response memo to the call data.

[0033] <Configuration of the control unit of server 10> The control unit 104 of the server 10 includes a user registration control unit 1041, a setting unit 1042, a recognition unit 1043, an analysis unit 1044, an importance calculation unit 1045, a summarization unit 1046, a learning unit 1047, a response memo suggestion unit 1048, a response memo assignment unit 1049, a score calculation unit 1050, a CRM storage control unit 1051, and a display control unit 1052. The control unit 104 realizes each functional unit by executing the application program 1011 stored in the storage unit 101.

[0034] The user registration control unit 1041 processes information of users who wish to use the services related to this disclosure and stores it in the user table 1012. Information such as user ID, username, and user attributes, which is stored in the user table 1012, is accessed when a user opens a web page operated by the service provider from any information processing terminal, enters their user ID, username, and user attributes into a designated input form, and sends it to the server 10. The user registration control unit 1041 of the server 10 stores the received user ID, username, and user attributes in a new record in the user table 1012, completing user registration. As a result, users stored in the user table 1012 can use the service. Prior to the registration of user information in the user table 1012 by the user registration control unit 1041, the service provider may perform a prescribed review and restrict whether or not the user can use the service. The user ID can be any string or number that can identify the user, and may be any string or number desired by the user, or the user registration control unit 1041 of the server 10 may automatically set any string or number. The user registration control unit 1041 may, when registering a user, store information such as the organization ID, organization name, and organization attributes of the organization to which the user belongs in the organization table 1013, linked to the user. The information of the organization to which the user belongs may be entered by the user themselves, or it may be registered by the administrator of the organization to which the user belongs, the operator of the service related to this disclosure, etc.

[0035] The setting unit 1042 executes the cooperation mode setting process. Details will be described later. The recognition unit 1043 performs speech recognition processing. Details will be described later. The analysis unit 1044 performs speech analysis processing. Details will be described later. The importance calculation unit 1045 performs the importance calculation process. Details will be described later. The summarization unit 1046 performs summarization processing. Details will be described later. The learning unit 1047 executes the learning process. Details will be described later. The response memo suggestion unit 1048 executes the response memo suggestion process. Details will be described later. The response memo assignment unit 1049 executes the response memo assignment process. Details will be described later. The score calculation unit 1050 performs the score calculation process. Details will be described later. The CRM memory control unit 1051 performs CRM memory processing. Details will be described later. The display control unit 1052 performs call display processing. Details will be described later.

[0036] <Functional configuration of user terminal 20> Figure 3 shows the functional configuration realized by the hardware configuration of the user terminal 20. The user terminal 20 comprises a storage unit 201, a control unit 204, an input device 206 connected to the user terminal 20, and an output device 208. The input device 206 includes a camera 2061, a microphone 2062, a position information sensor 2063, a motion sensor 2064, a keyboard 2065, and a mouse 2066. The output device 208 includes a display 2081 and a speaker 2082.

[0037] <Configuration of the storage unit of user terminal 20> The storage unit 201 of the user terminal 20 stores a user ID 2011, an application program 2012, and a CRM ID 2013 for identifying the user using the user terminal 20. The User ID is the user's account ID for Server 10. The user sends User ID 2011 from User Terminal 20 to Server 10. Server 10 identifies the user based on User ID 2011 and provides the services related to this disclosure to the user. The User ID includes information such as a session ID that is temporarily assigned by Server 10 to identify the user using User Terminal 20. CRMID is the user's account ID for the CRM system 30. The user sends CRMID2013 from the user terminal 20 to the CRM system 30. The CRM system 30 identifies the user based on CRMID2013 and provides CRM services to the user. CRMID2013 also includes information such as a session ID that is temporarily assigned by the CRM system 30 to identify the user using the user terminal 20. The application program 2012 may be pre-stored in the memory unit 201, or it may be configured to be downloaded from a web server operated by the service provider via a communication interface. The application program 2012 includes an interpreted programming language such as JavaScript (registered trademark) that is executed on a web browser application stored in the user terminal 20.

[0038] <Configuration of the control unit of user terminal 20> The control unit 204 of the user terminal 20 comprises an input control unit 2041 and an output control unit 2042. The control unit 204 realizes the functional units of the input control unit 2041 and the output control unit 2042 by executing an application program 2012 stored in the storage unit 201. The input control unit 2041 of the user terminal 20 acquires information output from input devices such as the camera 2061, microphone 2062, position information sensor 2063, motion sensor 2064, keyboard 2065, and mouse 2066 connected to the user terminal 20 and executes various processes. The input control unit 2041 of the user terminal 20 executes a process of transmitting the information acquired from the input device 206 to the server 10 together with the user ID 2011. Similarly, the input control unit 2041 of the user terminal 20 executes a process of transmitting the information acquired from the input device 206 to the CRM system 30 together with the CRM ID 2013. The output control unit 2042 of the user terminal 20 receives operations by the user on the input device 206 and information from the server 10 and the CRM system 30, and executes control processes for the display content of the display 2081 and the audio output content of the speaker 2082 connected to the user terminal 20.

[0039] <Function Configuration of the CRM System 30> The functional configuration realized by the hardware configuration of the CRM system 30 is shown in FIG. 4. The CRM system 30 includes a storage unit 301 and a control unit 304. The user has separately concluded a contract with a CRM operator and can receive the provision of CRM services by accessing (logging in) a website operated by the CRM operator via a web browser or the like using the CRM ID 2013 set for each user.

[0040] <Configuration of the Storage Unit of the CRM System 30> The storage unit 301 of the CRM system 30 includes a customer table 3012 and a response history table 3013. FIG. 12 is a diagram showing the data structure of the customer table 3012. FIG. 13 is a diagram showing the data structure of the response history table 3013.

[0041] The customer table 3012 is a table for storing and managing customer information. The customer table 3012 is a table having columns of customer ID, user ID, name, phone number, customer attribute, customer organization name, and customer organization attribute with the customer ID as the primary key.

[0042] The customer ID is an item that stores customer identification information to identify a customer. The User ID is an item that stores the User ID (user identification information) of a user associated with a customer. Users can view a list of customers associated with their User ID and make calls to those customers. In this disclosure, customers are linked to users, but they may also be linked to organizations (organization IDs in organization table 1013). In that case, users belonging to an organization can view a list of customers linked to their own organization ID and can send messages to those customers. The "Name" field is used to store the customer's name. The phone number field is used to store the customer's phone number. The user can access the website provided by the CRM system, select the customer they wish to call, and perform a predetermined operation such as "Call" to initiate a call to the customer's phone number from the user terminal 20. Customer attributes are items that store information about customer attributes such as age, gender, place of origin, dialect, and occupation (sales, customer support, etc.). The customer organization name is an item that stores the name of the organization to which the customer belongs. The organization name can include any organization or group name, such as company name, corporate name, corporate group name, club name, or various group name. Customer organizational attributes are items that store information about the attributes of the customer's organization, such as the type of organization (company, corporate group, other organization, etc.) and industry (real estate, finance, etc.). Customer attributes, customer organization name, and customer organization attributes may be stored by the user through input, or they may be entered by the customer when they access a designated website.

[0043] The Call History Table 3013 is a table for storing and managing records (call history information) of customer interactions (call history). If the customer interaction is a sales activity, records of past sales activities (date and time, content of sales activities, etc.) are stored. The Call History Table 3013 has a Call History ID as its primary key and contains columns for Call History ID, Call ID, URL, Customer ID, User ID (caller), Dial Date and Time, Call Start Date and Time, Call End Date and Time, and Comments.

[0044] In this disclosure, a new record is created and interaction history information is stored each time a call is made between the user and the customer. The interaction history ID is an item that stores interaction history identification information to identify the interaction history. The call ID is an item that stores the call ID (call data identification information) of the call data related to the interaction history. This links the interaction history in the CRM system with the call data on server 10. The URL is Uniform Resource Locator (URL) information containing a string that uniquely identifies the call ID. The URL may directly contain the call ID, or it may contain a string that can decrypt the call ID, or other specific strings that allow the call ID to be obtained by referring to a table (not shown). By operating the browser on the user terminal 20 and opening the URL, the user can access the server 10 and view an analysis results screen that visualizes the analysis data of the call history. The Customer ID is an item that stores the customer ID (customer identification information) of a customer related to their interaction history. The User ID is an item that stores the user's ID (user identification information) related to the interaction history. The dial date and time field is an item in the call history that records the date and time when the user made a call to a customer. The call start date and time is an item in the call history that records the start date and time of a call between the user and the customer. The call end date and time is an item in the call history that records the end date and time of the call between the user and the customer. Comments are items that store text information such as memos and comments regarding the response history. The user can edit the reasons, such as things noticed or postponed regarding the response history, in the comments. Also, the control unit 104 of the server 10 can freely edit the comments by sending a predetermined request to the API (Application Programming Interface) of the CRM system 30.

[0045] <Configuration of the control unit of the CRM system 30> The control unit 304 of the CRM system 30 includes a user registration control unit 3041. The control unit 304 realizes each functional unit by executing the application program 3011 stored in the storage unit 301.

[0046] The CRM system 30 provides functions called API (Application Programming Interface), SDK (Software Development Kit), and code snippets (hereinafter referred to as "beacons"). The user can perform linking settings such as account information for the server 10 and the CRM system 30 according to the present disclosure in advance, so that the control unit 104 of the server 10 and the control unit 304 of the CRM system 30 can communicate with each other and realize arbitrary information processing.

[0047] <Overview of the voice server (PBX) 40> When there is an outgoing call from the user to the customer, the voice server (PBX) 40 makes an outgoing call (rings) to the customer terminal 50. When there is an incoming call from the customer to the user, the voice server (PBX) 40 sends a message indicating that (hereinafter referred to as "incoming call notification message") to the user terminal 20. Also, the voice server (PBX) 40 can send the incoming call notification message to the beacons, SDK, API, etc. provided by the server 10.

[0048] <Functional configuration of the customer terminal 50> Figure 5 shows the functional configuration realized by the hardware configuration of the customer terminal 50. The customer terminal 50 includes a storage unit 501, a control unit 504, a touch panel 506, a touch-sensitive device 5061, a display 5062, a microphone 5081, a speaker 5082, a location information sensor 5083, a camera 5084, and a motion sensor 5085.

[0049] <Configuration of the memory unit of customer terminal 50> The memory unit 501 of the customer terminal 50 stores the customer's telephone number 5011 and application program 5012. The application program 5012 may be pre-stored in the memory unit 501, or it may be configured to be downloaded from a web server operated by the service provider via a communication interface. The application program 5012 includes an interpreted programming language such as JavaScript (registered trademark) that is executed on a web browser application stored in the customer terminal 50.

[0050] <Configuration of the control unit of customer terminal 50> The control unit 504 of the customer terminal 50 comprises an input control unit 5041 and an output control unit 5042. The control unit 504 realizes the functional units of the input control unit 5041 and the output control unit 5042 by executing an application program 5012 stored in the storage unit 501. The input control unit 5041 of the customer terminal 50 acquires information from the user's operations on the touch-sensitive device 5061 of the touch panel 506, voice input to the microphone 5081, and information output from input devices such as the position information sensor 5083, camera 5084, and motion sensor 5085, and performs various processes. The output control unit 5042 of the customer terminal 50 receives information from the user's operation on the input device and from the server 10, and performs control processing such as the display content of the display 5062 and the audio output content of the speaker 5082.

[0051] <Operation of Information Processing System 1> The following describes each process of Information Processing System 1. Figure 14 is a flowchart showing the operation of the summarization process (first embodiment). Figure 15 is a flowchart showing the operation of the summarization process (second embodiment). Figure 16 is a flowchart showing the operation of the response memo addition process. Figure 17 is a flowchart showing the operation of the response memo suggestion process. Figure 18 is a flowchart showing the operation of the score calculation process. Figure 19 is a flowchart showing the operation of the CRM memory processing. Figure 20 is a flowchart showing the operation of the call display process. Figure 21 is a diagram showing an overview of the binding process in the summarization process (first embodiment). Figure 22 is a diagram showing an overview of the additional extraction process in the summarization process (second embodiment). Figure 23 is a diagram illustrating the overview of the summary display process. Figure 24 shows an example screen of the CRM service in call display processing.

[0052] <Term definition> In explaining each process of Information Processing System 1, the following terms are defined: Call data is data relating to calls between users and customers, and includes data stored in each item of the call table 1014, data stored in each item of the speech recognition table 1015 linked by the call ID, data stored in each item of the summary table 1016 linked by the call ID, etc. It also includes data stored in each item of the response memo table 1017 and the response history table 3013 linked by the call ID. Call attributes are data relating to the attributes of a call between a user and a customer, and include user attributes, the name or attributes of the organization to which the user belongs, customer attributes, the name or attributes of the organization to which the customer belongs, the call category, and the caller / recipient type. In other words, call data is characterized by attribute values ​​such as the user attributes of the user making the call, the customer attributes of the customer making the call, the call category, and the caller / recipient type.

[0053] <Outgoing call processing> Outgoing call processing is the process of a user making an outgoing call to a customer.

[0054] <Overview of outgoing call processing> The outgoing call process is a series of operations in which the user selects a customer they wish to call from among multiple customers displayed on the screen of the user terminal 20, and then makes a call to that customer by performing the call operation.

[0055] <Details of the outgoing call process> This section describes the transmission process of Information Processing System 1 when a user sends a message to a customer.

[0056] When a user makes a call to a customer, the following processes are executed in Information Processing System 1.

[0057] The user operates the user terminal 20 to launch a web browser and access the website of the CRM service provided by the CRM system 30. By opening the customer management screen provided by the CRM service, the user can view a list of their customers on the display 2081 of the user terminal 20. Specifically, the user terminal 20 sends a request to the CRM system 30 to display a list of CRMID2013 and customers. Upon receiving the request, the CRM system 30 searches the customer table 3012 and sends information about the user's customers, such as customer ID, name, phone number, customer attributes, customer organization name, and customer organization attributes, to the user terminal 20. The user terminal 20 displays the received customer information on its display 2081.

[0058] The user selects the customer they wish to call from the list of customers displayed on the user terminal 20's display 2081. With the customer selected, the user sends a request including the phone number to the CRM system 30 by pressing the "Call" button or the phone number button displayed on the user terminal 20's display 2081. The CRM system 30, upon receiving the request, sends the request including the phone number to the server 10. The server 10, upon receiving the request, sends a call request to the voice server (PBX) 40. Upon receiving the call request, the voice server (PBX) 40 makes a call to the customer terminal 50 based on the received phone number.

[0059] Accordingly, the user terminal 20 controls the speaker 2082 and other components to make a sound indicating that a call is being made by the voice server (PBX) 40. In addition, the display 2081 of the user terminal 20 displays information indicating that a call is being made to the customer by the voice server (PBX) 40. For example, the display 2081 of the user terminal 20 may display the words "Calling".

[0060] The customer can make the customer terminal 50 ready for a call by lifting the handset (not shown) on the customer terminal 50 or by pressing the "Receive" button displayed on the customer terminal 50's touch panel 506 when an incoming call is received. In response, the voice server (PBX) 40 sends information indicating that the customer terminal 50 has responded (hereinafter referred to as a "response event") to the user terminal 20 via the server 10, CRM system 30, etc. As a result, the user and the customer become able to communicate using the user terminal 20 and the customer terminal 50, respectively, and can communicate with each other. Specifically, the user's voice, picked up by the microphone 2062 of the user terminal 20, is output from the speaker 5082 of the customer terminal 50. Similarly, the customer's voice, picked up by the microphone 5081 of the customer terminal 50, is output from the speaker 2082 of the user terminal 20.

[0061] When the user terminal 20 becomes ready to make a call, the display 2081 receives an answer event and displays information indicating that a call is in progress. For example, the display 2081 of the user terminal 20 may display the text "Answering".

[0062] When a call becomes possible, the call memory processing and CRM memory processing described later are executed.

[0063] <Incoming Call Processing> Incoming call processing is the process by which a user receives an incoming call from a customer.

[0064] <Overview of incoming call processing> Incoming call processing is a series of processes that occur when a customer makes a call to a user while the user has launched an application on the user terminal 20, and the user receives the call.

[0065] <Details of incoming call processing> This section describes the incoming call processing of Information Processing System 1 when a user receives an incoming call from a customer.

[0066] When a user receives a call from a customer, the following processes are executed in Information Processing System 1.

[0067] The user operates the user terminal 20 to launch a web browser and access the website for the CRM service provided by the CRM system 30. At this time, the user is assumed to be logged into the CRM system 30 with their account in the web browser and waiting. The user only needs to be logged into the CRM system 30 and may be performing other tasks related to the CRM service.

[0068] The customer operates the customer terminal 50, enters a predetermined telephone number assigned to the voice server (PBX) 40, and makes a call to the voice server (PBX) 40. The voice server (PBX) 40 receives the call from the customer terminal 50 as an incoming event.

[0069] The voice server (PBX) 40 sends an incoming call event to the server 10. Specifically, the voice server (PBX) 40 sends an incoming call request to the server 10, including the customer's telephone number 5011. The server 10 then sends the incoming call request to the user terminal 20 via the CRM system 30. Accordingly, the user terminal 20 controls the speaker 2082 and other components to make a sound indicating that an incoming call is being received by the voice server (PBX) 40. The display 2081 of the user terminal 20 displays information indicating that an incoming call is being received from a customer by the voice server (PBX) 40. For example, the display 2081 of the user terminal 20 may display the words "Incoming Call".

[0070] The user terminal 20 accepts responses from the user. Responses are performed, for example, by the user lifting the handset (not shown) on the user terminal 20, or by the user using the mouse 2066 to press a button on the user terminal 20's display 2081 that displays "Answer the call". When the user terminal 20 receives a response request, it sends a response request to the voice server (PBX) 40 via the CRM system 30 and server 10. The voice server (PBX) 40 receives the received response request and establishes voice communication. As a result, the user terminal 20 becomes ready to communicate with the customer terminal 50. The display 2081 of the user terminal 20 displays information indicating that a call is in progress. For example, the display 2081 of the user terminal 20 may display the words "Call in Progress".

[0071] When a call becomes possible, the call memory processing and CRM memory processing described later are executed.

[0072] <Call memory processing> Call memory processing is the process of storing data related to calls made between a user and a customer.

[0073] <Overview of Call Memory Processing> The call memory processing is a series of processes that store data related to a call in the call table 1014 when a call is initiated between a user and a customer.

[0074] <Details of call memory processing> When a call is initiated between a user and a customer, the voice server (PBX) 40 records the voice data related to the call between the user and the customer and sends it to the server 10. Upon receiving the voice data, the control unit 104 of the server 10 creates a new record in the call table 1014 and stores the data related to the call between the user and the customer. Specifically, the control unit 104 of the server 10 stores the user ID, customer ID, call category, call type, and the content of the voice data in the call table 1014.

[0075] The control unit 104 of the server 10 obtains the user's user ID 2011 from the user terminal 20 during outgoing or incoming call processing and stores it in the user ID field of a new record. The control unit 104 of server 10 queries the CRM system 30 based on the telephone number during outgoing or incoming call processing. The CRM system 30 retrieves the customer ID by searching the customer table 3012 using the telephone number and sends it to server 10. The control unit 104 of server 10 stores the retrieved customer ID in the customer ID field of a new record. The control unit 104 of the server 10 stores the call category value, which has been set in advance for each user or customer, in the call category field of the new record. Alternatively, the call category may be stored by the user selecting or entering a value for each call. The control unit 104 of the server 10 identifies whether the call being made was initiated by the user or by the customer, and stores either an outbound (initiated by the user) or inbound (initiated by the customer) value in the call type field of the new record. The control unit 104 of server 10 stores the voice data received from the voice server (PBX) 40 in the voice data field of a new record. Alternatively, the voice data may be stored as a voice data file in another location, and reference information (path) to the voice data file may be stored after the call ends. Furthermore, the control unit 104 of server 10 may be configured to store data after the call ends.

[0076] <Speech Recognition Processing> Speech recognition processing is the process of converting audio data from phone calls between users and customers into text information by performing text recognition, and then storing that information.

[0077] <Overview of speech recognition processing> The speech recognition process involves dividing the speech data stored in the call table 1014 into segments where speech exists (speech segments) and extracting segmental speech data. Speech recognition information is then generated by performing text recognition on each segmental speech data, and this information is stored in the speech recognition table 1015.

[0078] <Details of speech recognition processing> The recognition unit 1043 of server 10 searches the call table 1014 for records in which call storage processing has been performed but speech recognition processing has not been performed. Specifically, the recognition unit 1043 of server 10 searches the call table 1014 for records in which the speech recognition status field contains a blank, null, or other information indicating that speech recognition processing has not been performed. The recognition unit 1043 of server 10 may also determine that records in the call table 1014 in which voice data is stored, but in which the call ID does not exist in the speech recognition table 1015, are records in which speech recognition processing has not been performed.

[0079] The recognition unit 1043 of server 10 acquires (receives) the call ID and voice data of records for which voice recognition processing has not been performed. From the acquired (received) voice data, the recognition unit 1043 of server 10 detects the sections in which voice exists (speech sections) and extracts the voice data for each speech section as section voice data. The section voice data is associated with the speaker and speech time for each speech section. The recognition unit 1043 of server 10 converts the extracted segment audio data into text by performing text recognition on the segment audio data. The specific method of text recognition is not limited. For example, it may be converted using signal processing technology, machine learning using AI (artificial intelligence), or deep learning.

[0080] The recognition unit 1043 of server 10 stores the text for each utterance, a series of data linked to the start time of each utterance and the speaker (user or customer), and the call ID to be processed in the speech recognition table 1015. The recognition unit 1043 of server 10 stores information indicating that the speech recognition process has been completed in the speech recognition status item of the call table 1014.

[0081] The speech recognition table 1015 stores the text of each utterance segment of the speech data, linked to the utterance time and speaker, as continuous time-series data. By checking the data stored in the speech recognition table 1015, the user can confirm the content of the call as text information without listening to the audio data.

[0082] The utterance time may be any time related to each utterance, such as the start time of each utterance, the end time of each utterance, or any time in between each utterance.

[0083] <Torture of speech recognition processing> Furthermore, during speech recognition processing, it is also possible to configure the system to remove information from the text that is meaningless for understanding a conversation between the user and the customer, such as fillers included in the text, and store the speech recognition information in the speech recognition table 1015.

[0084] <Speech analysis processing> Voice analysis processing is the process of analyzing voice data from phone calls between users and customers, generating and storing the analysis data.

[0085] <Overview of speech analysis processing> The voice analysis process is a series of operations that generate analysis data by performing voice analysis on the voice data stored in the call table 1014 and then storing that data in the call table 1014.

[0086] <Details of the speech analysis process> The analysis unit 1044 of server 10 searches for records in the call table 1014 where call storage processing has been performed but voice analysis processing has not. Specifically, the analysis unit 1044 of server 10 searches the call table 1014 for records where voice data is stored but analysis data is not. Not storing analysis data means that the analysis data field is blank, null, or contains other information indicating that it is not stored. In addition, information indicating that a record is subject to speech analysis processing may be stored in a column not shown, and records containing such information may be subjected to speech analysis processing.

[0087] Server 10 acquires the call ID and audio data of records for which audio analysis processing has not been performed. The analysis unit 1044 of Server 10 analyzes the acquired audio data and calculates the Talk:Listen ratio, number of silences, number of overlaps, number of rallies, fundamental frequency, intonation strength, speech rate, speech rate ratio, number of fillers, talk script match, etc. The specific method of analysis is not particularly limited. For example, analysis may be performed using signal processing technology, machine learning using AI (artificial intelligence), or deep learning.

[0088] The Talk:Listen ratio is the ratio of the user's speaking time to the other party's speaking time. The number of silences refers to the number of times silence occurred during a call. The number of overlaps refers to the number of times overlaps occurred during a call. The number of rallies is the number of times an exchange of messages (the number of times the conversation changed) occurred between the user and the customer during a call. The fundamental frequency is defined for each user or customer and represents the fundamental frequency of the user's or customer's voice, i.e., the pitch of their voice. The degree of intonation is defined for each user or customer, and is information that quantitatively evaluates the magnitude of the user's or customer's intonation. Speech speed is defined for each user or customer and is the speed at which the user or customer speaks. Speech speed is indicated, for example, by the number of characters (or words) uttered per second. The speech rate ratio is information regarding the ratio of the speech rates of the user and the customer. Specifically, it is expressed as a numerical value obtained by dividing the user's speech rate by the customer's speech rate. For example, a larger value indicates that the user speaks faster than the customer. The filler count is the number of fillers (e.g., um, uh, etc.) detected in the speech recognition data. The filler count may be defined for each user or customer. The talk script match rate is information regarding the degree of match between the talk script set for each user or organization to which the user belongs and the user's utterances detected from the speech recognition data.

[0089] The analysis unit 1044 of server 10 stores the analysis data in the analysis data field of the record to be processed in the call table 1014.

[0090] <Summary Processing (First Example)> The summarization process (first embodiment) is a process that generates and stores summary information from speech recognition information. The details of the summarization process (first embodiment) will be explained below using the flowchart in Figure 14.

[0091] <Summary of the summary processing (first example)> The summarization process (first embodiment) is a series of processes that involves acquiring speech recognition information stored in the speech recognition table 1015, arranging the text contained in the speech recognition information in order of utterance time, performing a binding process to bind the text before and after the speaker change as a text group, calculating importance for the text and text group after the binding process, extracting text to be used in the summary document based on the importance, generating the summary document, and storing it in the server 10 and CRM system 30.

[0092] <Details of summary processing (first example)> The summarization unit 1046 of server 10 automatically performs summarization processing (first embodiment) periodically. In step S101, the summarization unit 1046 of the server 10 periodically starts the summarization process (first embodiment). Alternatively, the summarization unit 1046 of the server 10 may start the summarization process (first embodiment) when it detects the end of a call between the user and the customer.

[0093] In step S102, the summarization unit 1046 of the server 10 refers to the call table 1014 and searches for a record in the "Speech Recognition Status" column that contains information indicating that speech recognition processing has been completed. When the summarization unit 1046 of the server 10 finds a record that contains information indicating that speech recognition processing has been completed, it retrieves the record and proceeds to step S103 (Yes in step S102). After the call between the user and the customer ends, until the speech recognition process is completed, the summarization unit of the server 10 waits in step S102 because the speech recognition status column of the corresponding record in the call table 1014 does not store information indicating that the speech recognition process has been completed.

[0094] In step S103, the summarization unit 1046 of the server 10 searches for records in which the summarization status field of the records obtained in step S102 contains a blank, null, or other information indicating that summarization processing (first embodiment) has not been performed. The summarization unit 1046 of the server 10 may also determine that records in which voice data is stored in the call table 1014 but the call ID does not exist in the summary table 1016 are records in which summarization processing (first embodiment) has not been performed. If the summarization unit 1046 of server 10 finds a record for which summarization processing (first embodiment) has not been performed, it retrieves the record and proceeds to step S104 (Yes in step S102).

[0095] In step S104, the summarization unit 1046 of the server 10 searches the speech recognition table 1015 based on the call ID of the record for which summarization processing (first embodiment) has not been performed, and obtains speech recognition information consisting of records of text, utterance time, and speaker.

[0096] In step S105, the summarization unit 1046 of the server 10 performs binding processing on the acquired speech recognition information. Specifically, when the acquired speech recognition information is arranged in order of utterance time, it generates a text group that combines multiple texts before and after the speaker information changes. The text group is a data structure such as an array and contains information that includes multiple texts from different speakers. The text group may also contain other speech recognition information such as utterance time and speaker.

[0097] Figure 21 is a diagram illustrating the binding process in the summarization process (first embodiment), where the text for each speaker is arranged downwards in order of utterance time. For each utterance interval for each user and customer, the recognized text is assigned an identification number, such as U1, U2, C1, C2, etc. The summarization unit 1046 of the server 10 generates a text group by combining the texts of C4 and U5, where the speaker information changes from customer to user, for example.

[0098] In this disclosure, a text group was generated by combining one text each from the user and the customer. However, a text group may also be generated by combining two or more texts, one before and one after the speaker information changes. Alternatively, the importance of the texts before and after the speaker information changes (e.g., C4, U5, etc.) may be calculated in advance, and a text group may be generated by combining a number of texts corresponding to the importance value. For example, the higher the importance, the more texts may be combined to generate the text group. Even in conversations involving three or more participants, the system may generate a set of texts that combine multiple texts before and after changes in speaker information.

[0099] In step S106, the importance calculation unit 1045 of the server 10 performs importance calculation processing on the text and text group. Any important sentence extraction algorithm may be applied as the method for calculating importance. In this disclosure, we will explain a method for calculating importance using an algorithm called LexRank as an example. LexRank is an algorithm that takes multiple input sentences, divides each input sentence into individual words using morphological analysis, calculates the similarity (e.g., cosine similarity) of each sentence, calculates a graph structure based on the similarity between sentences, and then calculates the importance of each sentence based on the graph structure. Specifically, the importance calculation unit 1045 of server 10 calculates the importance of each text and text group by applying LexRank to each text and text group, treating each as a single input document.

[0100] <Importance Calculation Process (First Example)> In the importance calculation process (first embodiment), the importance calculation unit 1045 of the server 10 treats a text formed by combining multiple texts included in a text group as a single sentence and applies LexRank to it. For example, it combines C4 "Is Mr. Taguchi here?" and U5 "It's Taguchi, isn't it?" to form "Is Mr. Taguchi here? It's Taguchi, isn't it?" and applies LexRank to it as a single sentence to calculate its importance.

[0101] <Importance Calculation Process (Second Example)> In the importance calculation process (second embodiment), the importance calculation unit 1045 of server 10 applies LexRank to each of the multiple texts included in the text group as an independent document. The importance calculation unit 1045 of server 10 then performs a process in which the sum of the importance calculated for the independent documents becomes the importance of the text group. For example, the importance calculation unit 1045 of server 10 applies LexRank to each of the C4 and U5 sentences included in the text group and calculates the importance for each of C4 and U5. Then, the importance calculation unit 1045 of server 10 executes a process in which the sum of the importance of C4 and the importance of U5 becomes the importance of the text group. Alternatively, statistical values ​​(mean, median, mode, maximum, minimum, etc.) obtained by statistically processing the importance calculated for individual texts may be used as the importance of the text group. Specifically, the mean and maximum of the importance calculated for individual texts are suitable as the importance of the text group.

[0102] In step S107, the summarization unit 1046 of the server 10 extracts texts or groups of texts for which an importance level of a predetermined value or higher has been calculated. The summarization unit 1046 of the server 10 stores the extracted texts and texts included in the text groups in the summarization table 1016, along with the utterance time and speaker of each text in the speech recognition table 1015. As a result, summary information is stored in the summarization table 1016. The summarization unit 1046 of the server 10 stores information indicating that the summarization process (first embodiment) has been completed in the summarization status item of the call table 1014.

[0103] The summarization unit 1046 of server 10 combines the extracted text and the texts included in the text group into a single sentence (summary text) along with the utterance time and speaker of each text in the speech recognition table 1015, generates a request including the call ID and summary text, and sends it to the CRM system 30. When the CRM system 30 receives a request, it searches the call history table 3013 using the call ID included in the request and stores the received summary text in the comment field of the record that has that call ID.

[0104] <Summary Processing (Second Example)> The summarization process (second embodiment) is a second embodiment of the process for generating and storing summary information from speech recognition information. The details of the summarization process (second embodiment) will be explained below using the flowchart in Figure 15.

[0105] <Summary Processing (Second Example) Overview> The summarization process (second embodiment) involves acquiring speech recognition information stored in the speech recognition table 1015, calculating importance for the text contained in the speech recognition information, and extracting text to be used in the summary document based on that importance. When the text contained in the speech recognition information is arranged in order of utterance time, the text before and after the speaker changes is extracted from the extracted text as text to be used in the summary document, and the summary document is generated and stored in the server 10 and the CRM system 30.

[0106] <Details of the summary processing (second example)> The processes from step S201 to step S204 are the same as the processes from step S101 to step S104 of the summarization process (first embodiment), so their explanation is omitted.

[0107] In step S205, the importance calculation unit 1045 of the server 10 performs importance calculation processing on the acquired speech recognition information. Any important sentence extraction algorithm may be applied as the method for calculating importance, but in this disclosure, as with the summarization processing (second embodiment), an importance calculation method using an algorithm called LexRank will be described as an example.

[0108] <Importance Calculation Process (Third Example)> The importance calculation unit 1045 of server 10 treats the text included in the acquired speech recognition information as a single sentence and applies LexRank. Figure 22 is a diagram showing an overview of the additional extraction process in the summarization process (second embodiment), in which the text for each speaker is arranged downwards in order of utterance time. In the example in Figure 22, importance is calculated for all texts from U1 to U8 and from C1 to C6.

[0109] In step S206, the summarization unit 1046 of the server 10 extracts texts for which an importance level of a predetermined value or higher has been calculated. The summarization unit 1046 of the server 10 further extracts texts that are spoken by different speakers and are preceding or following the extracted texts in terms of utterance time. For example, if text U5 is extracted, additional text C4 is extracted, spoken by a different speaker and spoken before or after it.

[0110] In this disclosure, we have stated that we extract one additional text with a different speaker and a different utterance time, but we may extract two or more additional texts. We may also extract a number of additional texts corresponding to the importance value of the extracted texts. For example, the higher the importance, the more additional texts with different speakers and different utterance times may be extracted. Even in conversations involving three or more participants, it is acceptable to extract additional texts before and after changes in speaker information.

[0111] The process in step S207 is the same as the process in step S107 of the summarization process (first embodiment), so its explanation is omitted.

[0112] <Variations of summarization processing> In the summarization process (first embodiment) and the summarization process (second embodiment), the importance calculation unit 1045 of the server 10 may calculate the importance of the text by excluding information that is meaningless for understanding the conversation between the user and the customer, such as fillers contained in the text. Similarly, during the summarization process (first embodiment) and the summarization process (second embodiment), it is also possible to configure the system to remove from the text any information that is meaningless for understanding the conversation between the user and the customer, such as fillers already included in the text, and store the summarized information in the summary table 1016.

[0113] <Summary display processing> The summary display process is the process of displaying summary information to the user as a summary document. The details of the summary display process will be explained below using the screen example in Figure 23.

[0114] <Overview of summary display processing> The summary display process is a series of operations that retrieves summary information stored in the summary table 1016, stores the text contained in the summary information in speech bubbles of different colors for each speaker, arranges them in order of speaking time, and displays them as a summary document on the display 2081 of the user terminal 20.

[0115] <Details of summary display processing> The user logs into server 10 and performs a specified operation to send a request to server 10 to display a list of past call history between themselves and their colleagues in the organization. Upon receiving the request, server 10 queries the call table 1014 and sends the records of past call history to user terminal 20. User terminal 20 displays the received records of past call history as a list on its display 2081.

[0116] The user operates the user terminal 20 to select a call history record for which they want to check summary information, and by pressing the summary display button or similar, sends a request to the server 10 for summary display, including the call ID, for the selected call history record. When server 10 receives a request, it searches the summary table 1016 based on the call ID and sends the summary information related to that call ID to user terminal 20.

[0117] The user terminal 20 displays the received summary information as a summary document on the display 2081. The summary document displayed on the user terminal 20 will be explained based on Figure 23. The display 2081 of the user terminal 20 displays the summary display screen 70. On the summary display screen, the user's utterance (text) is displayed in a speech bubble 702 along with the utterance time 701, and the customer's utterance (text) is displayed in a speech bubble 712 along with the utterance time 711. The speech bubble 702 displaying the user's utterance is oriented to the left, making it distinguishable from the speech bubble 712 displaying the customer's utterance, which is oriented to the right. The display control unit 1052 of the user terminal 20 displays a list of texts spoken by the user in a speech bubble 702 that displays the user's utterance content, and texts spoken by the customer in a speech bubble 712 that displays the customer's utterance content, based on the text, utterance time, and speaker information included in the acquired summary information, starting from the top of the summary display screen 70 in chronological order of utterance time.

[0118] The color of speech bubble 702, which displays the user's utterance, is different from the color of speech bubble 712, which displays the customer's utterance. This allows the user to visually identify the speaker of the utterance when reviewing the summary document. Furthermore, the colors of speech bubbles 702 and 712, which display the utterance, may be displayed by changing at least one of the following: brightness, hue, saturation, or intensity, based on the importance of the text contained in the utterance, as calculated in the summarization process (first embodiment) and the summarization process (second embodiment). For example, the color of speech bubbles 702 and 712 containing text of higher importance may be made darker. This allows the user to visually confirm the importance of each utterance when reviewing the summary document, enabling them to understand the document content in a short amount of time.

[0119] <Speech Recognition Display Processing> The speech recognition display process is the process of displaying speech recognition information to the user as a speech recognition document.

[0120] <Overview of speech recognition display processing> The speech recognition display process is a series of operations that retrieves speech recognition information stored in the speech recognition table 1015, stores the text contained in the speech recognition information in speech bubbles of different colors for each speaker, arranges them in order of utterance time, and displays them as a speech recognition document on the display 2081 of the user terminal 20.

[0121] <Details of speech recognition display processing> The speech recognition display process is identical to the summary display process, except that it uses speech recognition information instead of summary information, i.e., it uses speech recognition table 1015 instead of summary table 1016. Therefore, the explanation is omitted. In the speech recognition display process, the speech recognition information is displayed in a list format in speech bubbles on a speech recognition display screen that is almost identical to Figure 23, in a manner that allows for differentiation by speaker. In the speech recognition display process, the color of the speech bubble displaying the user's utterance is different from the color of the speech bubble displaying the customer's utterance. This allows the user to visually identify the speaker of the utterance when reviewing the speech recognition document. Furthermore, the color of the speech bubble displaying the utterance may be displayed by changing at least one of the following: brightness, hue, saturation, or intensity, based on the importance of the text contained in the utterance calculated in the summarization process (first embodiment) and the summarization process (second embodiment). For example, speech bubbles containing text of higher importance may be made darker in color. This allows the user to visually confirm the importance of each utterance when reviewing the speech recognition document, enabling them to understand the document content in a short time. In other words, even when the user reviews all the speech recognition information of a call between the user and the customer, they can visually and intuitively grasp only the important utterances.

[0122] <Processing for adding response memos> The call memo addition process automatically adds a call memo to calls between the user and the customer. The details of the call memo addition process are explained below using the flowchart in Figure 16.

[0123] <Overview of the process for adding response memos> The call response memo assignment process is a series of operations that involves acquiring audio data related to a call between a user and a customer, applying a learning model to the audio data to infer candidate call response memos, and then storing the inferred candidate call response memos in association with the call data related to that call.

[0124] <Details of the process for adding a memo to the conversation> In step S301, the response memo assignment unit 1049 of the server 10 detects the end of the call between the user and the customer and starts the response memo assignment process.

[0125] In step S302, the response memo assignment unit 1049 of the server 10 acquires the call attributes of the call to be processed. Specifically, the response memo assignment unit 1049 of server 10 searches the call table 1014 based on the call ID of the call to be processed and obtains the call category and call type. The call memo assignment unit 1049 of server 10 searches the user table 1012 based on the user ID of the call to be processed and obtains user attributes. Based on the user ID of the call, the call memo assignment unit 1049 of server 10 searches the organization table 1013 from the organization ID stored in the user table 1012 and obtains the organization name and organization attributes of the organization to which the user belongs. The call memo assignment unit 1049 of server 10 queries the CRM system 30 based on the customer ID of the call to be processed, and retrieves customer attributes, customer organization name, and customer organization attributes from the customer table 3012 of the CRM system 30. The response memo assignment unit 1049 of server 10 does not need to acquire all call attributes; it only needs to acquire at least one call attribute from among multiple call attributes as needed.

[0126] In step S303, the response memo assignment unit 1049 of the server 10 selects a learning model based on the acquired call attributes. A learning model may be prepared for each call attribute, or for each combination of multiple call attributes, for example, a combination of user attributes and customer attributes. The learning model includes various machine learning and deep learning models, and is trained on datasets created according to call attributes. Details of the learning process will be described later. In this disclosure, a deep learning model is described as an example of a learning model. The deep learning model can be any deep learning model that takes arbitrary time-series data as input, such as RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), or GRU (Gated Recurrent Unit). The learning model includes, for example, any deep learning model that includes Attention, Transformer, etc.

[0127] In step S304, the response memo assignment unit 1049 of the server 10 acquires the audio data of the call to be processed and applies the audio data as input data to the selected learning model, thereby outputting (inferring) multiple response memo candidates (a group of response memos) along with their probability distributions as output data. Specifically, for response memos such as "AAA," "BBB," and "CCC," probability distributions such as "0.6," "0.3," and "0.1" will be output. These probability distributions may or may not be normalized using a function such as the softmax function.

[0128] In step S305, the response memo assignment unit 1049 of the server 10 stores the response memo candidate with the highest probability among the outputted response memo candidates in the memo content field of the response memo table 1017, linked to the call ID of the call to be processed. The assignment date and time field may store the current date and time. The assigner ID field may store information indicating that the memo was automatically assigned by a system other than the user, such as the server 10. The response memo assignment unit 1049 of the server 10 may store multiple response memo candidates from the outputted response memo candidates, whose probability is equal to or greater than a predetermined value, in the memo content items of the response memo table 1017.

[0129] <Processing for adding response memos (modified version)> In step S303, the response memo assignment unit 1049 of the server 10 may select multiple learning models based on multiple different call attributes. For example, the response memo assignment unit 1049 of the server 10 may select a first learning model prepared for each user attribute and a second learning model prepared for each customer attribute. In this case, the response memo assignment unit 1049 of the server 10 may select multiple learning models based on any multiple call attributes.

[0130] In step S304, the response memo assignment unit 1049 of the server 10 acquires the audio data of the call to be processed and applies the audio data as input data to a plurality of selected learning models. As a result, for each applied learning model, multiple response memo candidates are output (inferred) along with their probability distributions as output data. The response memo assignment unit 1049 of the server 10 may calculate the probability distribution for each response memo candidate by applying an arbitrary operation to the probability distribution of multiple response memo candidates. For example, the sum or product of the probability distributions of multiple response memo candidates may be used as the probability distribution for each response memo candidate to be output. Specifically, for example, if the probability distribution inference results for the first learning model are "0.6", "0.3", and "0.1" for response memos such as "AAA", "BBB", and "CCC", and the probability distribution inference results for the second learning model are "0.3", "0.4", and "0.1", then the product of these, "0.18", "0.12", and "0.01", can be used as the probability distribution for response memos "AAA", "BBB", and "CCC". The probability distribution may or may not be normalized using a softmax function or similar method.

[0131] <Processing of suggested responses to messages> The call memo suggestion process is a process that suggests call memo options to the user during a call between the user and the customer. The details of the call memo assignment process are explained below using the flowchart in Figure 17.

[0132] <Overview of the response memo suggestion process> The call response memo suggestion process is a series of operations that involves acquiring audio data related to a call between a user and a customer, applying a learning model to the audio data to infer candidate call response memos, suggesting the inferred candidate memos to the user, and storing the candidate memo selected by the user in association with the call data related to that call.

[0133] <Details of the response memo suggestion process> The process from step S401 to step S404 is the same as the process from step S301 to step S304 of the response memo assignment process, so the explanation is omitted.

[0134] In step S405, the response memo suggestion unit 1048 of the server 10 sends the outputted response memo candidates and probability distribution to the user terminal 20. The display 2081 of the user terminal 20 displays the received response memo candidates in a list so that the user can select one. In this case, the user terminal 20 may use the probability of each response memo candidate as the priority of the response memo candidates, and display the response memo candidates with higher priority in a position on the user terminal 20's display 2081 that is easier for the user to select. For example, response memo candidates with higher priority may be displayed higher up in the list, or if multiple response memo candidates are displayed in a carousel or paginated, they may be displayed on the first page. In other words, the user terminal 20 displays the most likely response memo candidates in a position on the user terminal 20's display 2081 that is easy for the user to select. This allows the user to more accurately and easily select the most likely response memo from among multiple response memos.

[0135] In step S406, the user selects one or more response memo candidates from the response memo candidates displayed on the display 2081 of the user terminal 20. The user sends the selected response memo candidates and user ID 2011 to the server 10 by pressing the send button displayed on the display 2081 of the user terminal 20.

[0136] In step S407, the response memo assignment unit 1049 of the server 10 associates the received response memo candidate and user ID 2011 with the call ID of the call to be processed and stores them in the memo content and assigner ID fields of the response memo table 1017. The assignment date and time field may store the current date and time.

[0137] <Learning Process> The learning process involves constructing a learning model that will be used in processes such as adding response memos and suggesting response memos.

[0138] <Overview of the learning process> The learning process involves training the parameters of a learning model by using audio data related to calls between the user and the customer as input data and the memo content associated with that audio data as output data (output vector), for each call attribute related to a call.

[0139] <Details of the learning process> The learning unit 1047 of server 10 searches the call table 1014 and retrieves the audio data and the memo content associated with the audio data by referring to the response memo table 1017 via the call ID. The learning unit 1047 of server 10 divides the audio data by call attribute and creates datasets such as training data, test data, and validation data for each call attribute. The learning unit 1047 of server 10 trains the parameters of the learning model prepared for each call attribute by using the dataset corresponding to each call attribute.

[0140] The learning unit 1047 of server 10 may exclude audio data associated with predetermined response memos. Specifically, when creating a dataset, the learning unit 1047 of server 10 excludes audio data related to calls where the memo content indicates that the call with the customer was not effectively completed, such as "answering machine," "customer (person in charge) unavailable," or "customer reception blocked." By excluding data undesirable for creating a learning model that infers memo content from the dataset, a more accurate learning model can be created.

[0141] For example, the learning unit 1047 of server 10 may create a dataset for each user attribute related to the user making the call, such as their job title, the industry of their organization, or the name of their organization. For example, the learning unit 1047 of server 10 may create a dataset for each customer attribute, such as the job title of the customer making the call, the industry of the customer's organization, or the name of the customer's organization. For example, the learning unit 1047 of server 10 may create a dataset for each call category of the calls made, such as telephone operators, telemarketers, customer support, and technical support. For example, the learning unit 1047 of server 10 may create a dataset for each type of information, either outbound calls originating from a user to a customer, or inbound calls received by a customer from a user.

[0142] When training a learning model for each user attribute of a user making a call, the learning unit 1047 of server 10 may use data that has been processed to extract only the user's voice (excluding the customer's voice) as input data for the training process. In this case, the audio data applied to the learning model in the inference process in steps S304 and S404 of the response memo assignment process and the response memo suggestion process will also be processed to extract only the user's voice (excluding the customer's voice) as input data. By training a user-specific learning model based solely on the user's utterances, it becomes possible to obtain a learning model that can more accurately infer response notes.

[0143] When training a learning model for each customer attribute of a customer making a call, the learning unit 1047 of server 10 may perform the training process using input data that has been processed to extract only the customer's voice (excluding the user's voice). In this case, the audio data applied to the learning model in the inference process in steps S304 and S404 of the response memo assignment process and the response memo suggestion process will also be processed to extract only the customer's voice (excluding the user's voice) and use that as input data. By training a customer-specific learning model based solely on the customer's spoken content, it becomes possible to obtain a learning model that can more accurately infer customer interaction notes.

[0144] <Modified Learning Process> In this disclosure, audio data is used as the input data for the learning model, but data transformed by performing arbitrary information processing on audio data may also be used as input data. For example, instead of audio data, text data obtained by performing text recognition (transcription) on the audio data may be used as input data. Alternatively, data transformed by performing arbitrary information processing on text data may also be used as input data. In this disclosure, applying audio data to a learning model includes applying the learning model to data transformed by performing arbitrary information processing on certain audio data. In this case, when inferring candidate response memos in learning models such as response memo assignment and response memo suggestion, the same information processing can be applied to the input audio data and then the learning model can be applied. For example, when inferring candidate response memos in learning models such as response memo assignment and response memo suggestion, instead of using audio data, text data obtained by performing text recognition (transcription) on the audio data can be used as input data.

[0145] <Score calculation process> The score calculation process is the process of calculating a call score for each user. The details of the score calculation process are explained below using the flowchart in Figure 18.

[0146] <Overview of score calculation process> The score calculation process involves analyzing and statistically processing each user's call data to calculate a call score for each user. This allows, for example, to evaluate each user's customer service skills using quantitative indicators.

[0147] <Details of the score calculation process> In step S501, the user sends a request for a user list to the server 10 by operating the user terminal 20. Based on the received request, the score calculation unit 1050 of the server 10 retrieves a list of users belonging to the same organization (having the same organization ID) as the user who sent the request from the user table 1012 and sends it to the user terminal 20. The display 2081 of the user terminal 20 displays a list of acquired users for the user to select from. The user selects a user for whom they want to calculate a score from the list of users displayed on the display 2081 of the user terminal 20. By pressing the send button displayed on the display 2081 of the user terminal 20, the user sends the user ID of the selected user to the server 10.

[0148] In step S502, the score calculation unit 1050 of the server 10 searches the call table based on the received user ID and obtains analysis data for each call of the target user.

[0149] In this case, the score calculation unit 1050 of server 10 may exclude analysis data associated with predetermined response memos. Specifically, when the score calculation unit 1050 of server 10 acquires analysis data for each call of the target user, it acquires the memo content of the response memo table 1017 associated with the call ID of the call. The score calculation unit 1050 of server 10 excludes analysis data for calls with memo content indicating that the call with the customer was not effectively completed, such as "answering machine," "customer (person in charge) unavailable," or "customer reception blocked." By doing so, when calculating the call score of the target user, a more accurate call score can be calculated by excluding calls that were not effectively completed.

[0150] The score calculation unit 1050 of server 10 calculates a call score by applying a predetermined algorithm to the analysis data for each call. Specifically, the score calculation unit 1050 of server 10 calculates the call score by taking a predetermined weighted sum based on the degree of deviation of various indicator values ​​(number of silences, number of overlaps, number of rallies, etc.) included in the analysis data from reference indicator values. At this time, any processing such as normalization may be applied to the indicator values.

[0151] In step S504, the score calculation unit 1050 of the server 10 calculates statistical values ​​(mean, median, mode, maximum, minimum, etc.) by performing statistical processing on the call score calculated based on the analysis data for each call of the target user, and uses these as the user evaluation index for the target user. Specifically, the average of the call scores calculated based on the analysis data for each call of the target user is suitable as the user evaluation index.

[0152] The score calculation unit 1050 of server 10 stores the calculated user evaluation index in the evaluation index field of the target user's record in user table 1012. The score calculation unit 1050 of server 10 sends the calculated user evaluation index to the user terminal 20 of the user who performed the score calculation process. The display 2081 of user terminal 20 displays the received user evaluation index for the target user to the user.

[0153] <Linkage Mode Setting Process> The linkage mode setting process is a process for setting the storage process of call information to the CRM system 30 in the call storage process.

[0154] <Overview of Linkage Mode Setting Process> The user, or the administrator of the organization or department to which the user belongs, etc., opens a predetermined website provided by the server 10 and displays an edit screen for linkage mode setting. The user can set the storage process of call information to the CRM system 30 in the server 10 by selecting the desired linkage mode on the linkage mode setting screen and performing a save operation or the like. Note that the linkage mode setting may have a predetermined default value without the user setting it. It may also be set when the user first uses the voice call service according to the present disclosure.

[0155] <Details of Linkage Mode Setting Process> The user, or the administrator of the organization or department to which the user belongs (such as the user), operates their own user terminal 20 and opens a website related to the edit screen for linkage mode setting provided by the server 10 using a web browser or the like. The user etc. operates the user terminal 20 and selects either the first mode or the second mode as the desired linkage mode on the linkage mode setting screen. The user etc. operates the user terminal 20 and transmits their user ID 2011 and the selected linkage mode to the server 10. The setting unit 1042 of the server 10 searches the user table 1012 by the received user ID 2011 and stores the received linkage mode in the linkage mode item of the record of the user. Note that the linkage mode may be stored for each organization to which the user belongs, rather than for each user. That is, it may be stored in the linkage mode item provided in the organization table 1013, and the linkage mode of each user may refer to the linkage mode item of the organization table 1013 linked by the organization ID.

[0156] <CRM Storage Process> The CRM memory process is a process for storing information related to a call made between a user and a customer in the CRM system 30 by using the voice call service according to the present disclosure. Hereinafter, the details of the CRM memory process will be described using the flowchart of FIG. 19.

[0157] <Overview of CRM Memory Process> The CRM memory process is a series of processes that, when a call is started between a user and a customer, acquires the linkage mode set for each user or organization, and stores the data related to the call in the CRM system 30 according to the set value of the linkage mode. Thereby, the data related to the call is stored in the CRM system 30 in association with the customer information of the call target.

[0158] <Details of CRM Memory Process> In step S601, after the outgoing call process or the incoming call process (incoming / outgoing call process) is performed, a call is started between the user and the customer. Thereby, the server 10 detects that a call has been started between the user and the customer.

[0159] In step S602, the CRM memory control unit 1051 of the server 10 searches the user table 1012 based on the user ID 2011 received from the user terminal 20, and acquires the linkage mode of the user making the call. When the linkage mode is stored in the organization table 1013 or the like for each organization, the organization table 1013 or the like is searched based on the organization ID of the corresponding user, and the linkage mode associated with the corresponding user is set as the linkage mode of the user.

[0160] In step S603, the CRM memory control unit 1051 of the server 10 determines whether the acquired linkage mode is the first mode or the second mode. If the acquired linkage mode is the first mode, the CRM memory control unit 1051 of the server 10 proceeds to step S604. If the acquired linkage mode is the second mode, the CRM memory control unit 1051 of the server 10 skips step S604 and proceeds to step S605.

[0161] In step S604, the CRM storage control unit 1051 of the server 10 sends a request to the CRM system 30 to store the first call data and the call data, including the call ID in the call table 1014 for that call, in association with the customer ID. The first call data is the data related to the call that can be obtained from the start of the call until the end of the call. Specifically, the CRM system 30 stores the user ID of the user making the call, the customer ID of the customer, the date and time of the call (only if it is an outgoing call), and the date and time the call started, as the first call data, linked to the call ID, in the user ID, customer ID, dial date and time, and call start date and time fields of the response history table 3013, respectively. Note that the first call data may also consist of at least one of the following: user ID, customer ID of the customer, dial date and time, and call start date and time.

[0162] In step S605, the CRM memory control unit 1051 of the server 10 detects the end of the call between the user and the customer.

[0163] In step S606, the CRM storage control unit 1051 of the server 10 determines whether the voice analysis processing for the call has been completed. Specifically, the CRM storage control unit 1051 of the server 10 refers to the call table 1014 and determines whether the analysis data item in the record of the call contains a blank, null, or other information indicating that the voice analysis processing has not been completed. If the voice analysis process is not complete, the CRM storage control unit 1051 of server 10 waits in step S606. If the voice analysis process is complete, the CRM storage control unit 1051 of server 10 proceeds to step S607.

[0164] In step S607, the CRM memory control unit 1051 of the server 10 determines whether the acquired collaboration mode is the first mode or the second mode. If the acquired collaboration mode is the first mode, the CRM memory control unit 1051 of the server 10 proceeds to step S609. If the acquired collaboration mode is the second mode, the CRM memory control unit 1051 of the server 10 proceeds to step S608.

[0165] In step S608, the CRM storage control unit 1051 of the server 10 sends a request to the CRM system 30 to store the second call data and the call data, including the call ID in the call table 1014 for that call, in association with the customer ID. The second call data is data related to the call that can be obtained after the end of the call. The CRM system 30 stores the user ID of the user making the call, the customer ID of the customer, the date and time of the call (only if it is an outgoing call), the start date and time of the call, and the end date and time of the call as second call data, linked to the call ID, in the user ID, customer ID, dial date and time, call start date and time, and call end date and time fields of the response history table 3013, respectively. The CRM system 30 stores the speech recognition information and summary information related to the call, linked to the call ID, in the comment field of the response history table 3013. Specifically, as shown in the speech recognition result and summary result of comment 807 in Figure 24, the information is converted into text format information that combines the speaking time, speaker, and text, and stored in the comment field of the response history table 3013. The CRM system 30 stores the URL generated based on the call ID, linked to the call ID, in the URL field of the response history table 3013. Note that the response history table 3013 may be configured to store only a portion of the information of the second call data.

[0166] In step S609, the CRM storage control unit 1051 of the server 10 sends a request to the CRM system 30 to store the third call data and the call data, including the call ID in the call table 1014 for that call, in association with the customer ID. The third call data is the data related to the call that can be obtained after the end of the call, excluding the data included in the first call data. The CRM system 30 stores the call end date and time as second call data in the call end date and time field of the response history table 3013, linked to the call ID. The CRM system 30 also adds speech recognition information and summary information related to the call to the comment field of the response history table 3013, linked to the call ID. Specifically, as shown in the speech recognition result and summary result of comment 807 in Figure 24, it converts the utterance time, speaker, and text into text format information and adds it to the comment field of the response history table 3013. In this case, if the collaboration mode is the first mode, the CRM memory control unit 1051 of the server 10 appends to the comment of the record in a way that does not overwrite the content already written in the comment of that record, since a record has already been created in the call history table 3013 in step S604. Specifically, if the collaboration mode is the first mode, a new record for the call is created in the call history table 3013, and other users may append to the comment. In such cases as well, the CRM memory control unit 1051 of the server 10 appends to the comment of that record in a way that does not overwrite it. The CRM system 30 associates the call ID with the URL generated based on the call ID and stores it in the URL field of the call history table 3013. The call history table 3013 may also be configured to store only a portion of the third call data.

[0167] <Variation> Server 10 can be configured to store customer information itself by acquiring customer information from the customer table 3012 of the CRM system 30 and storing it in a database (not shown). In this case, the CRM storage control unit 1051 of Server 10 may store call data associated with the customer being called by sending a request to the CRM system 30 that includes a customer ID or the customer's name, customer organization name, etc., instead of a call ID. In this case as well, the call data is stored in the CRM system 30 linked to the customer information of the customer being called.

[0168] In this disclosure, the server 10 also provides services related to calls between the user and the customer, but the call service may be provided by an external service (not shown). In this case, the CRM memory control unit 1051 of the server 10 may detect that a call between the user and the customer has started and start CRM memory processing upon receiving a request for call initiation from the external service. Similarly, it may detect that a call between the user and the customer has ended upon receiving a request for call termination from the external service. In addition, based on the voice data exchanged between the user and the customer, for example, it may detect that a call between the user and the customer has ended when the voice data has stopped.

[0169] <Call display processing> The call display process is the process of displaying call data stored in the CRM system 30 to the user. The details of the call display process will be explained below using the flowchart in Figure 20. Figure 24 shows an example of a screen output by the CRM system 30 during the call display process.

[0170] <Overview of Call Display Processing> After the call with the customer ends, the user operates the user terminal 20 to display the customer interaction history stored in the CRM system 30. The user operates the user terminal 20 to select the customer interaction history for which they want to display analytical data and requests the display of the analytical data. The CRM system 30 then retrieves the analytical data from the server 10 and displays the customer information and analytical data on the display 2081 of the user terminal 20.

[0171] <Details of call display processing> In step S701, the user operates the user terminal 20 to open a designated website provided by the CRM system 30 and displays the interaction history screen. Specifically, the user terminal 20 sends a request to the CRM system 30 for a list of customer information. The CRM system 30 searches the customer table 3012 and sends the records to the user terminal 20. The display 2081 of the user terminal 20 displays the received customer information in a list format that the user can select. The user selects a customer from the list of customer information displayed on the user terminal 20's display 2081 and presses the "Send" button, thereby sending the customer ID of the selected customer to the CRM system 30. Upon receiving the customer ID, the CRM system 30 searches the interaction history table 3013 and sends the interaction history record for the selected customer to the user terminal 20. The user terminal 20's display 2081 displays the received interaction history in a list format for the user to select from.

[0172] In step S702, the user selects the record whose call history they want to view from the call history list displayed on the user terminal 20's display 2081, and sends the call history ID of the selected call to the CRM system 30 by pressing the "Send" button. The CRM system 30 generates a display screen of the selected call history information and sends it to the user terminal 20. The user terminal 20's display 2081 displays the received call history information display screen. The call history information display screen shows the URL, customer information, user information, dial date and time, call start date and time, call end date and time, and speech recognition information and summary information stored in the call history table 3013 through CRM memory processing. Alternatively, without performing steps S701 and S702, the user terminal 20 may, for example, after the call between the user and the customer has ended, select the call history for that call as the selected call history and send the call history ID to the CRM system 30.

[0173] In step S703, the user operates the user terminal 20 and presses the "Analysis Results" button 801 displayed on the call history information display screen. The user terminal 20 sends a request to the CRM system 30 to display the analysis results, including the call history ID related to the call. Based on the received call history ID, the CRM system 30 searches the call history table 3013 to identify the call ID. The CRM system 30 sends a request to the server 10 requesting analysis data, including the identified call ID. Based on the received call ID, the server 10 searches the call table 1014 to check for the existence of analysis data. If no analysis data exists, the system waits in step S703. If analysis data exists, proceed to step S704.

[0174] In step S704, the server 10 searches the call table 1014 based on the received call ID and sends the analysis data to the CRM system 30. Based on the received analysis data, the CRM system 30 generates an analysis results screen 808 that visualizes the analysis data and sends it to the user terminal 20. The analysis results screen 808 includes the voice analysis results 802, response evaluation 804, voice evaluation 805, and speech rate 806 shown in Figure 24. The analysis results screen 808 may also include the text-formatted speech recognition results and summary results, which are stored in the comment column of the response history table 3013, as comment 807. The user can play the call audio by pressing the play / stop button 803. The CRM system 30 stores customer information, such as the customer table 3012. Users can selectively switch between displaying customer information stored in the CRM system 30 and the analysis results screen, which visualizes the analysis data, or display them side-by-side, allowing them to quickly and comprehensively understand their customer interaction history. In other words, by operating the CRM system 30, users can not only manage customer information but also view the visualization results of the analysis data.

[0175] <Variation> In this disclosure, the voice server (PBX) 40 and the customer terminal 50 are described as being connected by a telephone network T, but this is not limited to this. In other words, the voice server (PBX) 40 and the customer terminal 50 may be connected by any means of communication, including, for example, the Internet.

[0176] In this case, the outgoing call process may be configured to make a call to the customer terminal 50 based on arbitrary customer identification information, such as a URL, rather than the customer's phone number. Similarly, in the incoming call process, the incoming call request may include arbitrary customer identification information, such as a URL, and the system may be configured to identify the incoming user terminal 20 based on this customer identification information, such as a URL, and then accept the call.

[0177] <Basic Computer Hardware Configuration> Figure 25 is a block diagram showing the basic hardware configuration of computer 90. Computer 90 comprises at least a processor 901, main memory 902, auxiliary storage 903, and a communication interface IF991. These are electrically connected to each other by a communication bus 921.

[0178] The processor 901 is hardware for executing the instruction set written in a program. The processor 901 consists of an arithmetic unit, registers, peripheral circuits, etc.

[0179] Main memory 902 is used to temporarily store programs and data processed by programs, etc. For example, it is a volatile memory such as DRAM (Dynamic Random Access Memory).

[0180] Auxiliary storage device 903 is a storage device for storing data and programs. Examples include flash memory, HDD (Hard Disc Drive), magneto-optical disk, CD-ROM, DVD-ROM, and semiconductor memory.

[0181] The IF991 communication interface is an interface for inputting and outputting signals for communication with other computers via a network using wired or wireless communication standards. A network consists of various mobile communication systems built on the internet, LANs, wireless base stations, etc. For example, a network includes 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks that can connect to the internet via designated access points (e.g., Wi-Fi®). When connecting wirelessly, communication protocols include, for example, Z-Wave®, ZigBee®, and Bluetooth®. When connecting via a wired connection, the network also includes connections made directly via USB (Universal Serial Bus) cables, etc.

[0182] Furthermore, by distributing all or part of each hardware configuration across multiple computers 90 and connecting them to each other via a network, a computer 90 can be virtually realized. Thus, the concept of computer 90 includes not only a computer 90 housed in a single enclosure or case, but also a virtualized computer system.

[0183] <Basic Functional Configuration of Computer 90> The functional configuration of the computer realized by the basic hardware configuration of computer 90 (Figure 25) will be explained. The computer comprises at least one functional unit: a control unit, a memory unit, and a communication unit.

[0184] Furthermore, the functional units of computer 90 can also be realized by distributing all or part of each functional unit across multiple computers 90 interconnected via a network. The concept of computer 90 includes not only a single computer 90 but also a virtualized computer system.

[0185] The control unit is realized when the processor 901 reads various programs stored in the auxiliary storage device 903, loads them into the main memory device 902, and executes processing according to those programs. The control unit can realize various functional units that perform information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0186] The memory unit is implemented by the main memory 902 and the auxiliary memory 903. The memory unit stores data, various programs, and various databases. The processor 901 can also reserve memory areas corresponding to the memory unit in the main memory 902 or the auxiliary memory 903 according to the program. The control unit can also cause the processor 901 to perform operations such as adding, updating, and deleting data stored in the memory unit according to the various programs.

[0187] A database, specifically a relational database, is used to manage and link together tabular data sets called masters, which are structurally defined by rows and columns. In a database, tables are called tables, masters are called masters, the columns of tables are called columns, and the rows of tables are called records. In a relational database, relationships can be established and linked between tables and masters. Typically, each table and master has a primary key column to uniquely identify records, but setting a primary key column is not mandatory. The control unit can instruct the processor 901 to add, delete, or update records in specific tables and masters stored in the memory unit, according to various programs.

[0188] The communication unit is implemented by the communication IF991. The communication unit provides the functionality to communicate with other computers 90 via the network. The communication unit can receive information transmitted from other computers 90 and input it to the control unit. The control unit can cause the processor 901 to perform information processing on the received information according to various programs. The communication unit can also transmit information output from the control unit to other computers 90.

[0189] <Note> The details described in each of the above embodiments are noted below.

[0190] (Note 1) A program comprising a processor and a memory unit, for causing a computer to manage data related to a call between a user and a customer, the program causing the processor to execute a step of receiving voice data related to the call; an acquisition step (S302, S402) of acquiring call attributes of the call; a selection step (S303, S403) of selecting a learning model based on the call attributes; and an inference step (S304, S404) of inferring a response memo for the call by applying the voice data to the learning model selected in the selection step. This allows a computer that manages data related to calls between users and customers to infer more appropriate response notes for each call. Users can then avoid the cumbersome process of looking up response notes.

[0191] (Note 2) The program described in Appendix 1 includes at least one of the following call attributes: user attributes, customer attributes, call category, and call type. This allows users to infer more appropriate response notes for calls between themselves and customers by using a learning model that is tailored to the call attributes.

[0192] (Note 3) The user attribute includes information on any one of the occupation of the user making the call, the industry type of the organization to which the user belongs, and the name of the organization to which the user belongs. The customer attribute includes information on any one of the occupation of the customer making the call, the industry type of the organization to which the customer belongs, and the name of the organization to which the customer belongs. The call category includes information on the type or purpose of the call made between the user and the customer. The call type includes information on either outbound, where the user initiates a call to the customer, or inbound, where the user receives a call from the customer, as described in Appendix 2. As a result, the user can infer a more suitable response memo for a call made between the user and the customer by using a learning model corresponding to the call attributes.

[0193] (Appendix 4) The call attributes include the user attribute and the customer attribute. The selection step includes steps (S303, S403) of selecting a learning model based on the user attribute and the customer attribute, as described in any one of Appendices 1 to 3. As a result, the user can infer a more suitable response memo by applying a learning model adapted to the call for inferring the response memo based on both the user attribute and the customer attribute.

[0194] (Appendix 5) The call attributes include a first attribute and a second attribute. The selection step includes a step (S303, S403) of selecting a first learning model based on the first attribute and a step (S303, S403) of selecting a second learning model based on the second attribute. The inference step includes a first inference step (S304, S404) of inferring a first response memo group consisting of a plurality of response memos for the voice data by applying the voice data related to the call to the first learning model, a second inference step (S304, S404) of inferring a second response memo group consisting of a plurality of response memos for the voice data by applying the voice data related to the call to the second learning model, and a step (S304, S404) of inferring a response memo based on the first response memo group and the second response memo group, as described in any one of Appendices 1 to 4. Thus, even when there is insufficient voice data (dataset used for learning) for each call attribute, by combining and applying multiple learning models, a more suitable response memo can be inferred.

[0195] (Appendix 6) The program described in Appendix 5, where the first attribute is the user attribute and the second attribute is the customer attribute. Thus, even when there is insufficient voice data for each user attribute and customer attribute, the user can infer a more suitable response memo by combining and applying the respective learning models for the user attribute and customer attribute.

[0196] (Appendix 7) The program described in Appendix 6, where the first learning model is trained based on the voice data related to the user's call part in the voice data and not trained based on the voice data related to the customer's call part, and the second learning model is trained based on the voice data related to the customer's call part in the voice data and not trained based on the voice data related to the user's call part. Thus, the user can infer a more suitable response memo based on both the user attribute and the customer attribute.

[0197] (Appendix 8) The inference step is a step (S405) of inferring a response memo group consisting of multiple response memos, and the program causes the processor to execute a display step (S406) of displaying the multiple response memo groups in a selectable manner to the user, and a storage step (S407) of storing one or more response memos selected by the user from the response memo group in association with the voice data. The program is as described in any of Appendices 1 to 7. Thus, the user can assign a response memo to the voice data without a complicated input operation by selecting the response memo to be assigned to the voice data from the inferred multiple response memos.

[0198] (Appendix 9) The program described in Appendix 8 includes the following steps: the reasoning step includes calculating a priority for each of the response memos included in the group of response memos (S404); the display step includes displaying the first response memo included in the group of multiple response memos in the first position, and displaying the second response memo included in the group of multiple response memos in the second position which is less likely to be selected by the user than the first position, and the first response memo has a higher priority than the second response memo. This allows users to easily select higher-priority response memos by displaying them in a more convenient location, enabling them to assign the most reliable response memo to audio data from multiple memos without cumbersome work.

[0199] (Note 10) The learning model is a program described in any of the appendices 1 to 9, which is trained by excluding audio data that has a predetermined response memo attached to it. This allows for the use of a learning model that excludes calls where a genuine conversation with the customer did not actually take place, such as "answering machine," "customer (or representative) unavailable," or "customer reception blocked," enabling the inference of more appropriate response memos.

[0200] (Note 11) An information processing system comprising a processor and a memory unit for managing data related to a call between a user and a customer, wherein the system causes the processor to perform the following steps: receiving voice data related to the call; acquiring call attributes (S302, S402); selecting a learning model based on the call attributes (S303, S403); and inference steps (S304, S404) for inferring a response memo for the call by applying the voice data to the learning model selected in the selection step. This allows a computer that manages data related to calls between users and customers to infer more appropriate response notes for each call. Users can then avoid the cumbersome process of looking up response notes.

[0201] (Note 12) An information processing method comprising a processor and a memory unit, for causing a computer to manage data related to a call between a user and a customer, wherein the processor is caused to perform the following steps: receiving voice data related to the call; acquiring call attributes (S302, S402); selecting a learning model based on the call attributes (S303, S403); and inference steps (S304, S404) for inferring a response memo for the call by applying the voice data to the learning model selected in the selection step. This allows a computer that manages data related to calls between users and customers to infer more appropriate response notes for each call. Users can then avoid the cumbersome process of looking up response notes. [Explanation of Symbols]

[0202] 1 Information processing system, 10 Server, 101 Storage unit, 103 Control unit, 20A, 20B, 20C User terminals, 201 Storage unit, 204 Control unit, 30 CRM system, 301 Storage unit, 304 Control unit, 50A, 50B, 50C Customer terminals, 501 Storage unit, 504 Control unit

Claims

1. A program comprising a processor and a memory unit, which causes a computer to manage data related to calls between a user and a customer, The program is provided to the processor: A step of receiving voice data related to the aforementioned call, The acquisition step involves obtaining the call attributes of the aforementioned call, A selection step to select a learning model based on the aforementioned call attributes, An inference step in which the voice data is inferred by applying the learning model selected in the selection step to the call response memo, A program that executes the command.

2. The aforementioned call attributes include at least one of user attributes, customer attributes, call category, and call type. The program according to claim 1.

3. The user attribute includes information about one of the following attributes relating to the user making the call: job title, industry of the organization to which the user belongs, or name of the organization to which the user belongs. The aforementioned customer attributes include information relating to one of the following attributes concerning the customer making the call: job title, industry of the organization to which the customer belongs, or name of the organization to which the customer belongs. The aforementioned call category includes information about the type of call or the purpose of the call made between the user and the customer. The aforementioned call type includes information about either outbound calls made by the user to the customer, or inbound calls received by the user from the customer. The program according to claim 2.

4. The aforementioned call attributes include user attributes and customer attributes, The selection step includes selecting the learning model based on the user attributes and the customer attributes. A program according to any one of claims 1 to 3.

5. The aforementioned call attributes include a first attribute and a second attribute, The aforementioned selection step is, The steps include selecting a first learning model based on the first attribute, The steps include selecting a second learning model based on a second attribute, Includes, The aforementioned inference step is, A first inference step involves applying the voice data relating to the call to the first learning model, thereby inferring a first group of response memos consisting of multiple response memos from the voice data; A second inference step involves applying the voice data relating to the call to the second learning model, thereby inferring a second group of response memos consisting of multiple response memos from the voice data; The steps include: inferring the response memo based on the first and second sets of response memos; including, The program according to any one of claims 1 to 4.

6. The first attribute is the user attribute, The second attribute is customer attribute, The program according to claim 5.

7. The first learning model is trained based on the audio data relating to the user's call portion, but not on the audio data relating to the customer's call portion. The second learning model is trained based on the audio data relating to the customer's call portion, but not on the audio data relating to the user's call portion. The program according to claim 6.

8. The inference step is a step of inferring a group of response memos consisting of multiple response memos, The program is provided to the processor: A display step in which multiple sets of aforementioned response memos are displayed to the user for selection, A storage step in which one or more of the aforementioned response memos selected by the user from the aforementioned group of response memos are stored in association with the aforementioned audio data, A program according to any one of claims 1 to 7 that causes to execute.

9. The inference step includes a step of calculating a priority for each of the response memos included in the group of response memos, The aforementioned display step is, At the first position, the first response memo included in the group of multiple response memos is displayed. The second response memo, which is included in the group of multiple response memos, is displayed in a second position that is less likely to be selected by the user than the first position. Includes steps, The first response memo has a higher priority than the second response memo. The program according to claim 8.

10. The aforementioned learning model is trained by excluding the audio data to which a predetermined response memo has been attached. The program according to any one of claims 1 to 9.

11. An information processing system comprising a processor and a memory unit for managing data related to calls between a user and a customer, The aforementioned processor, A step of receiving voice data related to the aforementioned call, The acquisition step involves obtaining the call attributes of the aforementioned call, A selection step to select a learning model based on the aforementioned call attributes, An inference step in which the voice data is inferred by applying the learning model selected in the selection step to the call response memo, An information processing system that executes [this].

12. An information processing method comprising a processor and a memory unit, for causing a computer to manage data related to a call between a user and a customer, The aforementioned processor, A step of receiving voice data related to the aforementioned call, The acquisition step involves obtaining the call attributes of the aforementioned call, A selection step to select a learning model based on the aforementioned call attributes, An inference step in which the voice data is inferred by applying the learning model selected in the selection step to the call response memo, An information processing method that enables execution of [the specified action].

Images