Information processing systems, information processing devices, programs, information processing methods, terminal devices

Abstract

This involves using AI to propose information based on spatial information about a space and audio data including the speech of a person who spoke in that space. [Solution] The present invention relates to an information processing system comprising an information processing device 30, a terminal device, a microphone 301 and a display 60 included in a space, the system comprising: a first receiving unit 31 that receives from the terminal device voice data including speech of a person speaking in the space input to the microphone; a request transmission unit 42 that transmits a prompt to the AI ​​requesting a proposal based on spatial information and the content of the voice data, the spatial information and the content of the voice data to the AI; a response acquisition unit 43 that acquires a response transmitted from the AI; a display data generation unit 38 that generates screen display data including the proposal based on the response; and a first transmitting unit 39 that transmits the display data to the terminal device.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing apparatus, a program, an information processing method, and a terminal device.

Background Art

[0002] When people have a conversation in a space, the progress of the conversation can be supported by proposing information that the people should present. A technique for presenting proposal materials in consideration of the customer's emotions is known (see, for example, Patent Document 1). Patent Document 1 discloses a technology that analyzes the customer's emotions from the customer's voice or video, uses the analysis results to propose materials for solving the customer's problems, compares daily emotion analysis results with emotion analysis results at the time of proposal, and presents proposal materials related to the voice keywords at that time.

Summary of the Invention

Problems to be Solved by the Invention

[0003] However, with conventional techniques, it has not been possible to propose information based on spatial information related to a space and voice data including utterances of people who spoke in the space using AI.

[0004] The present invention provides a technique for proposing information based on spatial information related to a space and voice data including utterances of people who spoke in the space using AI.

Means for Solving the Problems

[0005] In view of the above problems, the present invention is an information processing system including an information processing apparatus, a terminal device, and a microphone and a display included in a space, the information processing system comprising: The information processing device includes: a first receiving unit that receives audio data from the terminal device, including speech of a person speaking in the space, input to the microphone; a request transmission unit that transmits a prompt to the AI, which is a prompt requesting a proposal based on the spatial information and the content of the audio data, to the AI; a response acquisition unit that acquires a response transmitted from the AI; a display data generation unit that generates screen display data including the proposal based on the response; and a first transmission unit that transmits the display data to the terminal device. The terminal device includes a second transmission unit that transmits the audio data to the information processing device, and an information display unit that displays the screen on the display based on the display data. The display installed in the space includes a display control unit that displays the screen, including a suggestion to the person, based on the display data. [Effects of the Invention]

[0006] Information based on spatial information about a space and audio data including the speech of a person who spoke in that space can be proposed using AI. [Brief explanation of the drawing]

[0007] [Figure 1] This is a diagram showing an example configuration of an information processing system arranged in space. [Figure 2] This figure shows an example of the hardware configuration of a server device and a terminal device. [Figure 3] This figure shows an example of the functional configuration of an information processing system. [Figure 4] This figure shows an example of a text data table stored in the text data storage unit. [Figure 5] This figure shows an example of a temporary storage information table registered in the temporary storage information storage unit. [Figure 6] This figure shows an example of a customer information table registered in the customer information storage unit. [Figure 7]This figure shows an example of schedule information registered in the schedule information storage unit. [Figure 8] This figure shows an example of a product information table and a service information table registered in the product information storage unit. [Figure 9] This figure shows an example of a sales negotiation history table registered in the sales negotiation history information storage unit. [Figure 10] This figure shows an example of a daily report information table registered in the daily report information storage unit. [Figure 11] This figure shows an example of a spatial feature information table registered in the spatial feature information storage unit. [Figure 12] This is an example flowchart illustrating the control flow performed by the control unit regarding business negotiations. [Figure 13] This is an example sequence diagram illustrating the opening procedures and actions when a customer enters the store. [Figure 14] This figure shows an example of a welcome message screen displayed on the display. [Figure 15] This diagram shows an example of a sales meeting space settings screen displayed by a terminal device. [Figure 16] This is an example sequence diagram illustrating the opening procedures and actions when a customer enters the store. [Figure 17] This diagram shows an example of a sales meeting space settings screen displayed by a terminal device. [Figure 18] This figure shows an example of a customer list screen displayed by a terminal device. [Figure 19] This is an example sequence diagram illustrating the process by which a server device displays suggestions based on audio data and spatial information acquired from the space. [Figure 20] This figure shows one example of a proposed display screen (1) shown on the server device's display. [Figure 21] This figure shows an example of a proposed display screen (part 2) displayed by the server device on its screen. [Figure 22] This is an example sequence diagram illustrating the process by which a display shows the closing screen. [Figure 23] This is a diagram showing an example of a closing screen displayed on a display. [Figure 24] This is an example sequence diagram for explaining a process in which a server device generates negotiation history and daily reports. [Figure 25] This is a diagram showing an example of a daily report screen displayed by a terminal device. [Figure 26] This is an example sequence diagram for explaining a process in which a server device registers or updates customer information. [Figure 27] This is a diagram showing an example of conversation text data stored in a text data storage unit in solution sales. [Figure 28] This is a diagram showing an example of a proposal display screen (Part 3) displayed by the server device on the display. [Figure 29] This is a diagram showing an example of a proposal display screen (Part 4) displayed by the server device on the display. [Figure 30] This is a diagram showing an example of a proposal display screen (Part 5) displayed by the server device on the display and the terminal device. [Figure 31] This is a diagram showing an example of confidential information. [Figure 32] This is an example sequence diagram for explaining a process in which the server device displays proposals on the terminal device and the display. [Figure 33] This is a diagram showing an example of a proposal display screen (Part 6) displayed by the server device on the display and the terminal device. [Figure 34] This is an example sequence diagram for explaining a process in which the server device displays proposals on the terminal device and the display 60 based on voice data and spatial information in response to an operation by a salesperson. [Figure 35] This is a diagram showing an example of a voice message proposed from an earphone by an AI system based on a conversation. [Figure 36] This is an example sequence diagram for explaining a process in which the server device outputs a voice message from an earphone based on voice data and spatial information. [Figure 37]This figure shows an example of the functional configuration of the information processing system in the second embodiment. [Figure 38] This is an example sequence diagram illustrating the process by which a server device displays a proposal based on audio data acquired from the space and spatial information 1-3, as well as the process of displaying emotional information. [Figure 39] This figure shows an example of a proposed display screen (No. 7) displayed by the server device on the terminal device. [Figure 40] This figure shows an example of the system configuration of an information processing system in the third embodiment. [Figure 41] This figure shows an example of the functional configuration of an information processing system in the third embodiment. [Figure 42] This is an example sequence diagram illustrating the process by which a server device registers information generated during a business negotiation with an external service system. [Figure 43] This is an example sequence diagram illustrating the process and operation of a sales representative downloading information generated during a sales negotiation from an external service system. [Figure 44] This figure shows an example of an information retrieval screen displayed by the terminal device in step S201. [Figure 45] This figure shows an example of a list screen of registration information displayed by the terminal device in step S209. [Figure 46] This figure shows an example of a display arranged in space. [Figure 47] This figure shows an example of the functional configuration of the information processing system in the fourth embodiment. [Figure 48] This diagram shows an example of a settings screen where a sales representative configures the information displayed on the screen. [Figure 49] This figure shows an example of the functional configuration of the information processing system in the fifth embodiment. [Figure 50] This is an example flowchart illustrating the process by which a terminal device displays a suggestion based on audio data acquired from space and spatial information 1-3. [Figure 51]This is a sequence diagram illustrating an example of the process by which a web browser displays optimal or predictive suggestions on a suggestion display screen. [Figure 52] This figure shows a modified example of an information processing system arranged in space. [Figure 53] This figure shows a modified example of an information processing system arranged in space. [Figure 54] This is a perspective view of an example of a wearable microphone. [Modes for carrying out the invention]

[0008] Hereinafter, an example of an embodiment for carrying out the present invention will be described with reference to the drawings, including an information processing system and an information processing method performed by the information processing system.

[0009] <Added value to space> First, referring to Figure 1, we will explain the situations that can arise when people in a space engage in conversations such as business negotiations. Figure 1 shows an example configuration of information processing system 1 placed in a space. In a negotiation room, for example, a sales representative 9 and a customer 8 are having a conversation. It may be difficult for sales representative 9 to elicit the customer 8's true feelings during the conversation and propose a product that leads to a purchase while maintaining an appropriate distance.

[0010] For example, it is preferable for sales representative 9 to understand customer 8's attributes and past purchase history before starting a business negotiation with customer 8. This is to avoid proposing the same product or asking inappropriate questions. However, researching this information before the negotiation may be time-consuming. If customer 8 arrives early, there is a risk of being unprepared.

[0011] Furthermore, even if customer 8 expresses their desired product, if it is vague or the sales representative 9 cannot properly understand its meaning, it is possible that no matter how many products are proposed, they will not match what customer 8 wants. If customer 8's explanation is too long, it may be difficult for the sales representative 9 to grasp the main points and organize the customer's requests.

[0012] Furthermore, even if we can elicit specific product information from customer 8, we often struggle to decide what alternative product to suggest if it's out of stock. Also, if customer 8 has a fixed budget, we can't suggest satisfactory products or services within the same price range. If the same product is also sold online, it's difficult to encourage in-store purchases to avoid price competition.

[0013] Furthermore, sales representative 9 may have used technical jargon in their product explanation, but it may not have been understood. If they realize this, they can rephrase it in language that customer 8 can easily understand, but they may not always notice. Also, even if they do notice, it may be difficult to rephrase it appropriately.

[0014] Furthermore, if sales representative 9 lacks sufficient product knowledge, they may not be able to immediately answer customer 8's questions, potentially creating distrust. They might be unable to immediately answer questions like, "How is this different from other products?" In this case, ideally, they could suggest a superior product, but this may take time if the product falls within a category that sales representative 9 is unfamiliar with.

[0015] Furthermore, even if a sales representative can elicit specific product information from customer 8, some sales representatives 9 struggle to anticipate customer needs and make proactive suggestions. Anticipating needs means proposing products that are likely to be desired in the future, even if they are not directly mentioned in the conversation. Being able to make proactive suggestions allows sales representatives to propose products to customers more quickly, giving them a competitive advantage over other companies.

[0016] Furthermore, if sales representative 9 does not take notes on the details of the negotiation as it nears its end, they will not be able to share the decisions made and the purchased items with the customer. Conversely, if sales representative 9 takes detailed notes on the negotiation, it becomes difficult for them to concentrate on the negotiation. After the negotiation ends, sales representative 9 registers a daily report and negotiation history, but creating these while recalling the details of the negotiation is a significant burden.

[0017] Therefore, in this embodiment, the server device 30 does not simply suggest products from conversation, but rather allows the AI ​​system 50 to understand the spatial situation and then has the AI ​​system 50 suggest products or services. The spatial situation is provided to the AI ​​system 50 by spatial information 1 to 3, which will be described later. However, it is not necessary to include all of spatial information 1 to 3 in the prompt.

[0018] By using not only conversational audio data but also spatial information 1-3, this embodiment provides added value in workspaces such as negotiation rooms and conference rooms. In this embodiment, spaces such as negotiation rooms and conference rooms may be referred to as "space 360". In this embodiment, AI is utilized to realize creative work in space 360.

[0019] As shown in Figure 1, the space 360 ​​is equipped with a microphone 301, a speaker 306, a display 60, a terminal device 10, and the like. The microphone 301 is connected to the terminal device 10 for communication, and the display 60 is also connected to the terminal device 10 for communication. The terminal device 10 is connected to a server device 30 via a network, and the server device 30 is connected to an SFA system 302.

[0020] Microphone 301 acquires audio data of conversations between customers and sales representatives and transmits it to terminal device 10. Microphone 301 can acquire conversations of people in the 360-degree space, not just customers and sales representatives. Terminal device 10 transmits the audio data to server device 30. Server device 30 inputs the audio data acquired by microphone 301, spatial information 1-3, and product information into AI system 50 and generates product suggestions that meet the customer's needs as a response. Server device 30 transmits the generated suggestions to terminal device 10, and terminal device 10 displays them on display 60. The following describes each device.

[0021] The microphone 301 is preferably directional and sensitive to the direction of the customer and sales representative. The microphone 301 is, for example, an array microphone, which combines multiple microphones to emphasize sound from a specific direction and suppress sound from unwanted directions. For example, beamforming technology analyzes the signals from multiple microphones in real time and dynamically adjusts the directivity to match the positions of the customer and sales representative by emphasizing sound from a specific direction. The microphone 301 transmits its direction along with the audio data to the terminal device 10. The microphone 301 may be, for example, a lavalier microphone or a wearable microphone worn by the customer 8 and sales representative 9. An example of a wearable microphone is shown in Figure 54.

[0022] The audio data acquired by microphone 301 is transmitted to terminal device 10. Microphone 301 may also transmit the audio data to server device 30. Terminal device 10 is used, for example, for a user interface, display on its own display, and to send instructions to earphone 305.

[0023] The display 60 displays information received by the terminal device 10 from the server device 30. The display 60 should be large enough to show customers and sales representatives present in the space. The terminal device 10 may mirror the content displayed on its own display to the display 60, or it may display different information on the display 60 than what is displayed on its own display. The display 60 may be, for example, an electronic whiteboard or a tablet device. In this embodiment, the display 60 and microphone 301 may be implemented as an integrated device.

[0024] In Figure 1, the display 60 is connected to the terminal device 10. However, if the server device 30 generates the information that the display 60 displays, the display 60 may be connected to the server device 30 in a way that allows communication.

[0025] The display 60 has a built-in speaker 306. The speaker 306 may be externally attached to the display 60, or it may be located separately from the display 60. The speaker 306 outputs voice messages transmitted from the terminal device 10. The speaker 306 is optional.

[0026] Additionally, sales representatives may wear earphones 305. This is effective when the server device 30 gives instructions that can only be heard by the sales representative. Earphones 305 communicate with the terminal device 10 wirelessly, for example, using Bluetooth® or Wi-Fi®. Earphones 305 output audio based on information generated by the server device 30.

[0027] The terminal device 10 has an application installed that communicates with the server device 30 and performs processing according to the output information from the server device 30. The output information includes, for example, Data to be displayed on display 60, Data to be displayed on terminal device 10, Voice message to speaker 306, Voice message to earphone 305, These are some examples.

[0028] Terminal device 10 receives operations from sales representatives and others and transmits information to server device 30. Terminal device 10 accepts, for example, the start of application execution (startup of information processing system 1), input of information about the customer to be negotiated, and the end of the negotiation. Terminal device 10 can be a general-purpose information processing terminal, such as a PC (Personal Computer), tablet terminal, or smartphone. Alternatively, terminal device 10 can be an electronic whiteboard, AR goggles, MR goggles, etc. Terminal device 10 and display 60 may be integrated. Also, terminal device 10 and microphone 301 may be integrated.

[0029] The terminal device 10 may be a smartphone carried by a sales representative. When a sales representative connects their smartphone to the server device 30 and logs in to the server device 30, they can use the smartphone as a stationary terminal device 10. In this case, it is preferable that the microphone 301 and the display 60 are connected to the server device 30. The server device 30 transmits display data to the display 60 and the smartphone, respectively. Therefore, in this case, the terminal device 10 does not need to be installed in the space beforehand.

[0030] The server device 30 is one or more information processing devices that transmit output information to the space. The server device 30 can communicate with the AI ​​system 50. Also, if the space is a meeting room, the output information from the server device 30 will support the sales representative, so the server device 30 can also be considered part of the SFA system 302. The server device 30 communicates with the SFA system 302 as needed, and retrieves customer information from the CRM system 303 managed by the SFA system 302, or information from various storage units 304. Depending on the type of space, the SFA system 302 may not be necessary.

[0031] SFA System 302 is a system designed to streamline and automate sales activities. SFA System 302 includes functions such as customer information management, sales opportunity management, schedule management, sales forecasting, report generation, and quotation management, supporting increased productivity and sales growth in the sales department.

[0032] The CRM system 303 is responsible for customer information management. CRM system 303 centrally manages customer information (company name, contact person, past interactions, etc.), allowing sales representatives to understand the status of each customer. The sales opportunity management function records the progress of ongoing sales negotiations, visualizing the current negotiation status (pre-negotiation, negotiation in progress, completed). This function allows for the estimation of the likelihood of closing a deal, the determination of negotiation priorities, and the prevention of missed follow-ups. The schedule management function registers customer visits, sales representative visits, and tasks in a calendar system, visualizing the sales representative's schedule. Furthermore, the schedule management function uses a reminder function to notify sales representatives of negotiations and follow-ups at the appropriate time. This enables schedule sharing across the entire team. The sales forecasting function analyzes past sales data to predict future sales for each sales representative and team. The report creation function generates reports of sales performance, such as daily reports, allowing management and managers to review them in real time. The quotation management function automatically creates quotations and invoices.

[0033] While the CRM system 303 shares some functions with the SFA system 302, it is primarily a system for managing customer relationships over the long term. The CRM system 303 supports customer information management, marketing support, and customer support. The CRM system 303 encompasses overall customer management, including departments other than sales, while the SFA system 302 is a system specifically designed to improve the efficiency of the sales team's operations. However, in this embodiment, the SFA system 302 and the CRM system 303 may be integrated.

[0034] The SFA system 302 uses various storage units 304. These storage units 304 contain, for example, product information, but details will be described later.

[0035] AI system 50 is a machine learning model trained to take a prompt as input and generate a response message corresponding to that prompt. A prompt is an inquiry or instruction that specifies the conditions for the response message to be generated by the machine learning model. In order to generate a highly accurate response message from the machine learning model, the prompt needs to be carefully crafted. A prompt may include inquiries, instructions, questions, etc., or it may be a simple conversation without inquiries, instructions, or questions. The prompt input to AI system 50 may include text, images, shapes, audio, etc. Based on the prompt, AI system 50 generates text, images, shapes, audio, etc., as a response message. As an example of such AI system 50, generative AI using a large-scale language model is known. A large-scale language model is a natural language processing model trained using a large amount of text data. The large-scale language model takes in a vast amount of text in advance and learns knowledge from it using deep learning or reinforcement learning. The large-scale language model uses this knowledge to provide a response message to the input prompt. The input messages, which are prompts fed into a large-scale language model, can include not only text such as instructions and questions, but also files, images, shapes, and audio. The AI ​​system 50 that generates this information is sometimes called a generative AI.

[0036] Large-scale language models are used in many NLP tasks, such as generating responses to specific questions, automatically generating text, summarizing text, translation, and sentiment analysis. Furthermore, large-scale language models can be utilized in a variety of applications, including education, entertainment, customer service, and product development. In this embodiment, text data converted from speech data serves as the main input message. The server device 30 may also contain an AI system 50.

[0037] Here, machine learning is a technique for enabling computers to acquire human-like learning abilities. It refers to a technique in which a computer autonomously generates algorithms necessary for judgment, such as data identification, from pre-programmed training data, and applies these algorithms to new data to make predictions. The learning method for machine learning can be supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, or deep learning, or a combination of these methods; the learning method for machine learning is not restricted.

[0038] The server device 30 may generate response messages using RAG (Retrieval Augmented Generation). RAG is a technology that searches a large amount of documents such as business documents, regulations, internal company information, and external information stored in the memory unit to extract information, and then has a large-scale language model respond based on the extracted information. By using RAG, the server device 30 passes information obtained from a database that holds accurate information to the AI ​​system 50, so that the AI ​​system 50 can generate response messages based on facts.

[0039] Furthermore, the configuration shown in Figure 1 is just one example; for example, the server device 30, AI system 50, SFA system 302, CRM system 303, and two or more of the various databases may be integrated into a single unit. The terminal device 10 may also have the functions of the server device 30.

[0040] <About Terminology> A space is a place where multiple people converse. The space can be either enclosed or open. In this embodiment, a meeting room is an example of a space, but any place where conversation can take place, such as a conference room, reception room, lounge, individual booth, or workspace, is acceptable.

[0041] Spatial information can be any information relating to a space. An example of spatial information, Spatial Information 1, is information indicating an event taking place in that space (who is participating, whether it's a business meeting, consultation, or conference, etc.). Another example of spatial information, Spatial Information 2, is information indicating the content or purpose of an event taking place in that space (what is intended to be done in that space, etc.). Spatial Information 3 is information indicating the characteristics of the space, such as the name of the space itself or its intended use.

[0042] A suggestion related to the content of the audio data is a suggestion that supports the purpose of the conversation. If the purpose of the conversation is to sell a product, the suggestion is a product that corresponds to the conversation. A suggestion directed at a person is a suggestion that is presented to that person.

[0043] An example of people conversing in a space is a visitor and a person in charge of the space. In this embodiment, we will explain using a customer and a sales representative as an example, but the visitor may be a consumer, business partner, user, client, etc. The person in charge of the space may be a shop assistant, salesperson, salesman, person in charge, etc.

[0044] An optimal proposal is the best proposal generated based on spatial information 1-3 and audio data. What is optimal depends on the flow of the conversation (context), but for example, it is the best proposal for the customer. The content of the proposal may be the products or services handled by the sales representative. The content of the proposal may also be advice. For example, when customers come in for legal advice, tax advice, illness, counseling, workplace relationships, school relationships, housing and real estate advice, family relationship advice (marital, childcare, elderly care, parent-child relationships, etc.), financial and asset management advice (debt consolidation, investment advice, insurance advice), traffic accident advice, IT and internet advice (security, programming, internet troubles, smartphone selection), travel advice (recommended travel destinations, how to plan), fashion advice, beauty advice (how to choose clothes, makeup, hairstyles), plastic surgery advice, cooking and nutrition advice (recipes, dietary improvement), skill-up career advice, further education and employment career advice, pet advice (how to raise pets, training, health management), self-improvement and motivation advice (how to achieve goals, maintaining motivation), fortune-telling advice, and general advice, the server device 30 uses its specialized knowledge to offer advice to solve their problems.

[0045] A predictive suggestion is a suggestion for a product or service that the customer has not previously mentioned, but which is likely to be purchased based on their past statements. For example, if a customer says "moving," suggesting a moving company is an optimal suggestion, but suggesting furniture suitable for the new floor plan would be a predictive suggestion.

[0046] <Example Hardware Configuration> Figure 2 shows an example of the hardware configuration of a server device 30 and a terminal device 10 in an embodiment of the present invention. As shown in Figure 2, the server device 30 and the terminal device 10 are built by a computer 500. The computer 500 includes a CPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, RAM (Random Access Memory) 103, HD (Hard Disk) 104, HDD (Hard Disk Drive) controller 105, display 106, external device connection I / F (Interface) 108, network I / F 109, data bus 110, keyboard 111, pointing device 112, optical drive 114, and media I / F 116.

[0047] Of these components, the CPU 101 controls the overall operation of the computer 500. The ROM 102 stores programs used to drive the CPU 101, such as the IPL (Initial Program Loader). The RAM 103 is used as the work area for the CPU 101. The HD 104 stores various data, such as programs. The HDD controller 105 controls the reading or writing of various data to the HD 104 according to the control of the CPU 101. The display 106 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 108 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network I / F 109 is an interface for data communication using a communication network. The data bus 110 is an address bus and data bus, etc., for electrically connecting each component, such as the CPU 101 shown in Figure 2.

[0048] The keyboard 111 is a type of input means equipped with multiple keys for inputting characters, numbers, and various instructions. The pointing device 112 is a type of input means for selecting and executing various instructions, selecting processing targets, and moving the cursor. The optical drive 114 controls the reading or writing of various data to the removable optical storage medium 113. The optical storage medium 113 may be a CD, DVD, Blu-ray (registered trademark), etc. The media I / F 116 controls the reading or writing (storage) of data to the recording medium 115, such as flash memory.

[0049] <About the features> Figure 3 shows an example of the functional configuration of the information processing system 1 in this embodiment.

[0050] <<Terminal device 10>> The terminal device 10 includes an audio data acquisition unit 11, a transmission unit 12, an operation reception unit 13, a reception unit 14, a display control unit 15, an information display unit 16, and an audio message output unit 17. Each of these units is realized by a program installed in the terminal device 10 that causes the CPU 501 of the terminal device 10 to execute a process.

[0051] The audio data acquisition unit 11 acquires audio data from the microphone 301. The audio data acquisition unit 11 converts the analog audio data into digital audio data by A / D conversion. The audio data includes the voices of customers and sales representatives present in the space. The A / D conversion may also be performed by the microphone 301.

[0052] The transmitting unit 12 (an example of a second transmitting unit) transmits voice data to the server device 30 in real time or by accumulating it at regular intervals. The transmitting unit 12 also transmits operational information from sales representatives to the server device 30.

[0053] The operation reception unit 13 receives operations from sales representatives on the terminal device 10 (i.e., the application).

[0054] The receiving unit 14 (an example of a second receiving unit) receives output information generated by the server device 30 through analysis of the audio data as a response to the request for output information along with the audio data from the server device 30. There is no need for a one-to-one correspondence between the transmitted audio data and the output information.

[0055] The display control unit 15 displays screens generated by the application on the display 106 of the terminal device 10. The display control unit 15 displays the screen based on the display data received from the server device 30 to the terminal device 10.

[0056] The information display unit 16 displays the screen on the display 60 based on the display data received from the server device 30. If the output information includes an audio message for the speaker 306, the information display unit 16 outputs the audio message to the speaker 306. If the display 60 is an information processing device rather than just a monitor, the display processing is performed by the display 60.

[0057] The voice message output unit 17 outputs the voice message included in the output information for the earphone 305 received from the server device 30 to the earphone 305.

[0058] The program executed by terminal device 10 may be a native application or a web application. A web application is an application in which a program on a web server and a program on a web browser work together to perform processing, and does not require installation on terminal device 10. Web applications are downloaded from server device 30. A native application is an application that is installed on terminal device 10 and used there.

[0059] <<Server device 30>> In Figure 3, the server device 30 has several functional units and several storage units. Note that Figure 3 only shows the main functions used for generating output information, and the server device 30 may have other functions not shown in Figure 3. Also, only the main connections between functional units are shown, and information may be exchanged between functional units that are not connected.

[0060] The server device 30 has the following functional units: a receiving unit 31, a control unit 32, a language processing unit 33, a speech recognition unit 34, a closing detection unit 35, a storage unit 36, an AI control unit 37, a display data generation unit 38, a transmission unit 39, and a system API 41. The server device 30 also has a temporary information storage unit 48 and a text data storage unit 49. Furthermore, the server device 30 can use information stored in various storage units 304 via the SFA system 302. For simplicity, the CRM system 303 is omitted in Figure 3.

[0061] Each functional part of the server device 30 is realized by having one or more programs installed on the server device 30 run on the CPU 101. The temporary information storage unit 48, the text data storage unit 49, and the various storage units 304 can be realized using, for example, the HD 104 or a storage device that can be connected to the server device 30 via a network.

[0062] The receiving unit 31 (an example of the first receiving unit) receives audio data acquired in space. The receiving unit 31 also receives operation information entered by the sales representative to the terminal device 10. More precisely, the sales representative's input is made to the terminal device 10. Therefore, the receiving unit 31 receives the audio data and operation information received by the terminal device 10 from the terminal device 10. The receiving unit 31 outputs the audio data to the language processing unit 33 and the speech recognition unit 34, and outputs the operation information to the control unit 32.

[0063] The language processing unit 33 uses existing speaker dialization to estimate "when and who spoke" from the speech data and uses the results to identify the speaker. Speaker dialization can be performed using methods such as the following: The language processing unit 33 extracts MFCCs from the speech data. Since MFCCs are vectors, they are input into a pre-trained DNN to output speaker features (x-vectors). Speaker features (x-vectors) are the speaker's feature vectors. The speaker identification unit clusters speakers using clustering methods such as k-means, EM algorithm, and Spectral Clustering. Note that this method can distinguish speakers, but it cannot identify who the speaker is. To identify who the speaker is, for example, the AI ​​system 50 can determine from the conversation whether the speaker is a salesperson or a customer. For example, the person who says "How about XX?" is a salesperson, and the person who says "My budget is around XX" is a customer. Also, if the space is a conference room, there is less need to identify who the speaker is, but the person who gives instructions such as "Please do XX" can be identified as the facilitator or supervisor. For example, terminal device 10 authenticates the customer and the sales representative in an existing way when the sales representative and the customer use the space, and by requesting some kind of statement at that time, it is possible to associate speaker features (x-vector) with speaker labels (for example, Person A = Mr. Sato, Person B = Mr. Suzuki). If the microphone 301 is directional, the language processing unit 33 can distinguish the speaker based on the direction from which the audio data was acquired. That is, the microphone 301 transmits the acquired direction along with the audio data to the terminal device 10. This method can also distinguish speakers, but it cannot identify who the speaker is. However, if, for example, the direction in which customers sit and the direction in which sales representatives sit are fixed, then it is possible to identify who the speaker is. The language processing unit 33 may also be provided in the terminal device 10. In that case, the transmission unit 12 of the terminal device 10 transmits to the server device 30 the result of the processing performed by the language processing unit 33 instead of the audio data.

[0064] The speech recognition unit 34 performs speech recognition processing on the voice data. For example, the speech recognition unit 34 converts the spoken voices of the customer and sales representative into text data through speech recognition processing. The speech recognition unit 34 also obtains speaker information from the speaker identification unit and associates the speakers with the text data. In this way, the speech recognition unit 34 adds information such as "Mr. A" and "Mr. B" (speaker labels) to the text data. The speech recognition unit 34 also associates the text data with absolute time or the elapsed time since the start of the business negotiation. The speech recognition unit 34 may also be provided in the terminal device 10. In that case, the transmission unit 12 of the terminal device 10 sends the results of the processing performed by the speech recognition unit 34 to the server device 30 instead of the voice data.

[0065] The control unit 32 performs overall control over the generation of output information. First, the control unit 32 acquires text data associated with the speaker and speaking time from the speech recognition unit 34. The control unit 32 uses the results of the AI ​​system 50's analysis of the text data to cause the display data generation unit 38 to generate output information. In this embodiment, the output information is, for example, the display of information (display data generation), but it may also be the audio output of information from a speaker or earphones.

[0066] The control unit 32 determines the situation based on the results of the AI ​​system 50's analysis of text data and operation information, and performs the following actions. • Opening process • Processing customer proposals • Anticipating customer needs and providing solutions. • Closing process • Customer information registration process • Data registration such as daily reports and business negotiation history. The control unit 32 may also acquire operation information received by the terminal device 10 from the receiving unit 31. Operation information may include operations on the application or input messages. In the case of operations on the application, the control unit 32 executes processing according to the operation. In the case of input messages, the control unit 32 may process them in the same way as text data.

[0067] When the control unit 32 receives text data, it transmits the text data to the storage unit 36. The storage unit 36 ​​stores the text data in chronological order in the text data storage unit 49. In this way, as the conversation that took place in space progresses, the amount of past text data increases. The storage unit 36 ​​may also create a summary when a predetermined amount or time of text data has been accumulated.

[0068] Furthermore, the accumulated text data is also used to register customer information. This allows the control unit 32 to automatically register customer attributes from the content of the conversation. Customer information can include a wide range of details depending on the content of the conversation, such as name, age, gender, nationality, place of residence, occupation, income, family structure, educational background, purchase history, website visit history, etc.

[0069] The closing detection unit 35 detects that the business negotiation is nearing completion based on the text data in the text data storage unit 49. The closing detection unit 35 notifies the control unit 32 that closing has begun. The control unit 32 requests the display data generation unit 38 to display a closing screen that includes the information stored in the temporary information storage unit 48. Note that the control unit 32 may have the AI ​​system 50 perform the detection that the business negotiation is nearing completion. Alternatively, the control unit 32 or the closing detection unit 35 may perform this based on the end time of the schedule information and the current time.

[0070] The AI ​​control unit 37 is the interface with the AI ​​system 50. Request the AI ​​system 50 to extract the requirements. Request the AI ​​system 50 to provide the optimal solution to the request. Request AI system 50 to provide predictive suggestions in response to the request. Request the AI ​​system 50 to extract the decisions. Request the AI ​​system 50 to extract an action list. Request the AI ​​system 50 to extract customer information. Request the AI ​​system 50 to detect closing. For purposes such as those mentioned above, prompts are generated by applying accumulated text data to system prompts. Optimal and predictive suggestions will be discussed in more detail later. Each piece of text data is associated with the speaker and the time of their statement. The AI ​​control unit 37 acquires the response message generated by the AI ​​system 50. The response message contains information generated in response to the request.

[0071] The AI ​​control unit 37 includes a request transmission unit 42 that generates prompts and sends requests to the AI ​​system 50 for these processes, and a response acquisition unit 43 that acquires response messages generated by the AI ​​system 50.

[0072] While details about prompts will be explained in sequence diagrams and other materials, a simple example of a system prompt is as follows:

[0073] Example of a system prompt Please create {......} using the following information. {Accumulated text data} {Data from various storage units 304} {table column name} (This is an example of a system prompt.) Thus, the system prompt defines a section for specific requests, a section for text data, and data stored in the various storage units 304. In this case, the request transmission unit 42 of the AI ​​control unit 37 generates the prompt by writing the specific request in {......} of the system prompt and applying the text data stored by the storage unit 36 ​​to the {stored text data} section. Customer information, product information, schedule information, etc., are set in {data from various storage units 304} as needed. The item names of each table are set in {table item names}. Note that appropriate messages corresponding to requests such as proposals may be pre-set in {......}. When the request transmission unit 42 inputs the generated prompt to the AI ​​system 50, the AI ​​system 50 can generate response messages (requests, optimal proposals, predictive proposals, decisions, action lists, customer information, closing detection, etc.) from the stored text data.

[0074] Furthermore, if the control procedure includes multiple phases, a system prompt may be defined that explicitly indicates where to apply an intermediate response obtained in a phase preceding that phase (for example, an intermediate response obtained in any phase from the 1st phase to the (K-1)th phase to the Kth phase).

[0075] The control unit 32 stores the requests, optimal suggestions, predictive suggestions, decisions, and action lists acquired by the AI ​​control unit 37 from the AI ​​system 50 in the temporary information storage unit 48. The temporary information storage unit 48 is used to display the closing screen (review of the business negotiation).

[0076] Furthermore, the control unit 32 transmits the customer information extracted from the text data by the AI ​​system 50 to the SFA system 302. The SFA system 302 then registers the customer information in the CRM system 303.

[0077] The control unit 32 passes the response message received from the AI ​​control unit 37 to the display data generation unit 38, along with instructions on whether to display it on the terminal device 10 or the display 60. The control unit 32 also passes the response message received from the AI ​​control unit 37 to the display data generation unit 38, specifying the earphone 305 for output. This information that the server device 30 sends to the terminal device 10 is sometimes collectively referred to as "output information".

[0078] System API 41 is an interface with SFA system 302 that requests and retrieves information from SFA system 302 by calling APIs exposed by SFA system 302. However, if the server device 30 has various storage units 304 managed by SFA system 302, or if the server device 30 is on a network that allows direct communication, then system API 41 does not need to be used.

[0079] The display data generation unit 38 generates display data for each display destination if the output information is display data. Although the content may differ depending on the display destination, the generation method can be the same. Display data for web applications can be, for example, HTML, CSS (Cascading Style Sheets), JavaScript (registered trademark), JSON, or XML. Display data for native applications can be, for example, JSON or XML.

[0080] If the output information is an audio message, the display data generation unit 38 may convert the audio message into audio data (WAVE, mp3, etc.). This conversion may be performed by the terminal device 10.

[0081] The transmitting unit 39 (an example of the first transmitting unit) transmits output information to the terminal device 10. The output information includes suggestions for the customer generated by the AI ​​system 50 according to the content of the conversation. The output information also includes voice messages for the sales representative generated by the AI ​​system 50 according to the content of the conversation. The information display unit 16 and display control unit 15 of the terminal device 10 can display suggestions, etc., at a timely pace in response to the content of the conversation. If the output information is a voice message, the voice message output unit 17 of the terminal device 10 can output instructions or warnings to the sales representative from the earphone 305 at a timely pace in response to the content of the conversation.

[0082] ●Text data storage unit 49● Next, with reference to Figure 4 and other figures, the various storage units that the server device 30 has or references will be described. Figure 4 shows an example of a text data table stored in the text data storage unit 49. The text data storage unit 49 stores text data and summaries of business negotiations conducted in space. The records in the text data storage unit 49 increase as the business negotiations progress. Note that the text data table in Figure 4 was generated from a business negotiation at a department store.

[0083] The text data storage unit 49 stores the speaker, the time of the statement, the content of the statement, a summary, and a message vector. • The speaker is the speaker identified by the speaker identification unit. In Figure 4, the speaker's identity is also identified, but they may also be distinguished as Speaker 1, Speaker 2, etc. • The speaking time is the time the customer or sales representative made the statement. The speaking time may be in Japan Standard Time or as the elapsed time since the start of the business negotiation. • The content of the statements is the text data itself, converted from the audio data of the customer and the sales representative. The summary is a concise representation of the main points and arguments of the spoken content, generated by the AI ​​system 50. In Figure 4, a summary is created for each sentence, but the granularity of the text data used for the summary can be adjusted as needed. It is also possible to divide the spoken content into chunks of a certain length. Therefore, it can handle even long original statements. The message vector is a semantic vector obtained by converting each statement or summary into a distributed representation. A distributed representation is a fixed-length vector that represents text data. The conversion to a distributed representation is performed by an existing embedding model. Known embedding models include BERT® and Doc2Vec, but any method can be used to convert to a distributed representation.

[0084] The control unit 32 passes a message vector to the system API 41, which then searches for specific product information in the product information table, allowing the system to retrieve product information relevant to the conversation. This reduces the size of the product information sent to the AI ​​system 50, even if the product information table is large. In this case, the SFA system 302 has the product information table and other data pre-divided into chunks, and chunk vectors are registered. The SFA system 302 calculates the similarity between the message vector generated from real-time text data and the chunk vectors stored in the various storage units 304, and identifies the chunks (such as product information) related to chunk vectors with a similarity of a certain level or higher.

[0085] What sales representatives want to be proposed with are products that are likely to be purchased because they are related to what the customer has said. Therefore, by sending product information related to what the customer has said to the AI ​​system 50, it is expected that the accuracy of the products suggested by the AI ​​system 50 will improve. In this way, by using RAG, it is possible to provide the AI ​​with related product information from the search results and have the AI ​​suggest products from those search results.

[0086] The customer's comments are as follows: "I would like to give my spouse a gift for their birthday." I gave her a bag as a gift the other day, so I'm thinking jewelry would be a good choice this year. I went to an exhibition of [something] and became interested in Italian jewelry. Actually, I recently moved. I've been saying I want a jewelry case to match my new home." ●Temporary retention information storage unit 48● Figure 5 shows an example of a temporary information table registered in the temporary information storage unit 48. The temporary information table stores customer requests, which are special notes extracted from the sales negotiation conversation. The temporary information table is used for reviewing the sales negotiation. By sharing this temporary information between the sales representative and the customer at the end of the negotiation, it becomes easier to conduct a thorough review of the negotiation. • The proposal identification information is the identification information of the proposal that the server device 30 proposed to the customer (displayed on the display 60). The extracted requests are summaries of customer requests extracted by AI system 50 from the content of the conversation. The proposed items are the optimal or predictive suggestions made by AI System 50 in response to the extracted requests. The action list is either an action list extracted by the AI ​​system 50 from the content of the conversation or an action list entered by the sales representative. The AI ​​system 50 recognizes voice data from the customer, such as "Could you email me the details of the exhibition?", and extracts items that need to be addressed later in relation to the request, such as "I am interested in the exhibition," and associates them with the request to create an action list. The decisions are those extracted by the AI ​​system 50 from the content of the conversation. In the example in Figure 5, the AI ​​system 50 extracts items that the customer agreed to or decided on during the conversation (products, location, meal, date and time, gift, etc.) and matches them to the customer's request. For example, the sales representative says, "We will proceed with the reservation and other arrangements using this plan. Is that alright?" and the customer says, "Yes, that's fine." The AI ​​system 50 judges this conversation as a decision regarding the plan and extracts that plan as a decision. In addition, the products that the customer purchased are also recorded as decisions. For example, the customer says, "I'll buy the XXX jewelry case," and the sales representative says, "Thank you." The AI ​​system 50 judges this conversation as a decision regarding the purchase and extracts the product identification information and the number of items purchased as a decision.

[0087] ●Customer information storage section 51● Figure 6 shows an example of a customer information table registered in the customer information storage unit 51. The customer information storage unit is one of the various storage units 304. Figure 6 is just an example, and there may be other customer information items. Also, although all items are registered in Figure 6, the customer information items that are extracted may vary depending on the content of the conversation and the instructions (prompts) given to the AI ​​system 50. Customer identification information is identification information that uniquely identifies a customer. • The name is the customer's name. • Age and gender refer to the customer's age or age group and gender. • Nationality refers to the customer's country of origin or country of affiliation. • The place of residence is the customer's current address, etc. • Occupation refers to information about the customer's occupation or work style. • Income is the customer's approximate annual income (salary and side income). Family structure includes whether the person is married or single, whether they have children, whether they live with relatives, and whether they live separately from their spouse. • Educational background includes the customer's highest level of education (high school graduate, university graduate, postgraduate, etc.), school name, and qualifications. • Purchase history consists of the main products and services that a customer has previously purchased from our company or another company. • Website visit history includes e-commerce sites and information sites that customers frequently use. Purchasing power is the approximate amount of money one can spend on purchasing goods and services. • Payment methods include cash, credit card, bank transfer, and other commonly used payment methods.

[0088] ●Schedule Information Storage Unit 52● Figure 7 shows an example of schedule information registered in the schedule information storage unit 52. The schedule information storage unit 52 is one of the various storage units 304. For example, the schedule information storage unit 52 stores the schedule of a business meeting entered when a meeting room is reserved. Schedule information may also be entered on the meeting space setting screen 210, which will be described later. A meeting room is the name of the room or space where business negotiations take place. • The sales representative is the name of the sales representative in charge of the business negotiation. • The start date and time is the scheduled date and time when the business negotiation is to begin. • The end date and time is the expected date and time for the business negotiation to conclude. • The customer is the name or company name of the business partner. Customer identification information is identification information used to identify a customer. The consultation topic should include the agenda and details of the business negotiation. Additionally, there may be a section indicating the purpose of the conversation (business negotiation, consultation, meeting, etc.). • The opportunity code is a unique identifier assigned to each opportunity.

[0089] ●Product information storage section 53● Figure 8(a) shows an example of product information registered in the product information storage unit 53. The product information storage unit 53 is one of the various storage units 304. Figure 8(a) shows an example where the product is jewelry. Product identification information is unique identification information that identifies a product. • Product names are general names for products, such as brand names or product names. • The category is the type of jewelry (rings, necklaces, etc.). • The materials listed are the main materials used. • The gemstone is the type of gemstone used. • Carat weight refers to the weight (carats), grade, etc., of a gemstone. • Size refers to the dimensions of the jewelry; for rings, it's the ring size, and for necklaces, it's the length. • The price is the retail price. • Purchase price refers to the purchase price of the goods. • The stock quantity shown is the current stock quantity. • A product description is a descriptive text that conveys the features and appeal of a product.

[0090] As shown in Figure 8(a), the server device 30 stores information not only about products but also about services. While a service is an example of a product, service information may be stored separately from tangible products as shown in Figure 8(b). Figure 8(b) shows an example of service information registered in the product information storage unit 53. The server device 30 can also propose services in the same way as products. The product information storage unit 53 stores information for each product and each service as shown in Figures 8(a) and 8(b), and stores multiple products and services. In this embodiment, services and products are not particularly distinguished, and in the following description, they will be described together as products.

[0091] ●Sales negotiation history information storage unit 54● Figure 9 shows an example of a sales negotiation history table registered in the sales negotiation history information storage unit 54. The sales negotiation history table stores past sales negotiation history. • The opportunity history identifier is a unique identification number assigned to each opportunity and is used to manage past opportunities. The opportunity history identifier may also be the opportunity code in the schedule information. • The date and time are those on which the business negotiation took place and will be used for follow-up. Customer identification information is used to identify customers and manage their past purchase history and interaction history. • Purchased product identification information is the identification information of products that a customer has purchased or shown interest in, and is used for inventory management and referencing suggestion history. Action items are specific actions that a sales representative should take after a business negotiation. • The contact person is the name and identification information of the sales representative who handled the deal, and is used to determine responsibility and streamline follow-up.

[0092] As will be described later, the sales negotiation history can be automatically generated by the server device 30 from the text data of the conversation.

[0093] ●Daily report information storage section 55● Figure 10 shows an example of a daily report information table registered in the daily report information storage unit 55. The daily report information table stores the daily reports of sales representatives who have provided customer service records. • Daily report identification information is identification information that uniquely identifies a daily report. • The sales opportunity history identification information is the same as the sales opportunity history identification information in the sales opportunity history table, and it links the daily report with the sales opportunity. • Your requests are used to record customer inquiries and consultations, and to organize response history and improve services. • The purchased items are those whose purchase was decided upon during the business negotiation. • Your next visit will be to schedule your next business meeting. • The staff member's memo is a record of any special notes or points to note during customer service, intended to be used for future interactions and suggestions. It also includes handover information for the next interaction.

[0094] As will be described later, the daily report can be automatically generated by the server device 30 from the text data of the conversation.

[0095] ●Spatial feature information storage unit 56● Figure 11 shows an example of a spatial feature information table registered in the spatial feature information storage unit 56. The spatial feature information table stores a description of the space used for business negotiations. Note that the spatial feature information in Figure 11 corresponds to "Spatial Information 3" described later. As will be explained in detail later, the spatial feature information is transmitted to the AI ​​system 50. Upon transmission to the AI ​​system 50, the AI ​​system 50 can generate proposals and screens based on the spatial feature information and respond accordingly. • Spatial identification information is identification information used to uniquely identify each space and is used during reservations and management. • The space name is an item that records the name of the meeting space, such as the room or meeting booth name. • The space description includes information about the space's intended use and the types of business meetings it is suitable for. The space's characteristics include its size, capacity, soundproofing performance, equipment (monitors, whiteboards, etc.), furniture, and other features.

[0096] <<Display 60>> This section describes the functions of the display 60 when it is not merely a monitor, but an information processing terminal such as a PC, tablet, or electronic whiteboard.

[0097] The display 60 includes a communication unit 61, a display control unit 62, and an operation reception unit 63. Each of these units is realized by a program installed on the display 60 that causes the CPU of the display 60 to execute a process.

[0098] The communication unit 61 controls communication with the server device 30 or the terminal device 10.

[0099] The display control unit 62 controls the display of the screen based on the display data received from the server device 30 or the terminal device 10.

[0100] The operation reception unit 63 receives user operations on the display 60.

[0101] <Overall flow> Referring to Figure 12, the overall control flow performed by the control unit 32 will be explained. Figure 12 is a flowchart illustrating the control flow performed by the control unit 32 in relation to a business negotiation. The process in Figure 12 starts, for example, when a sales representative activates the terminal device 10 and inputs a start command from a predetermined application.

[0102] The control unit 32 determines whether the sales representative has started to pick up the customer (S1001). The start of the pickup is detected by analysis of voice data acquired by the microphone 301 or by the sales representative's operation of the terminal device 10.

[0103] When the start of the pick-up is detected, the control unit 32 performs an opening process (S1002). The opening process involves, for example, identifying the customer and displaying an opening screen on the display 60.

[0104] Once the opening process is complete, the control unit 32 moves on to the proposal process and the predictive proposal process (S1003). Neither the customer nor the sales representative needs to be aware of the transition from the end of the opening process to the proposal process and the predictive proposal process.

[0105] During the proposal processing and predictive proposal processing, the control unit 32 determines from the conversation whether or not the business negotiation is nearing its end (S1004). This is so that the control unit 32 can display the closing screen (summary of the business negotiation) at the appropriate time during the next closing process. The fact that the business negotiation is nearing its end is detected by the time, voice messages, etc.

[0106] When the control unit 32 detects that the business negotiation is nearing its end, it executes a closing process (S1005). The control unit 32 displays a summary of requests received during the negotiation, proposals from the sales representative, decisions made, and an action list.

[0107] The closing process allows the customer and sales representative to review the business negotiation. Afterwards, the customer leaves the negotiation room, or the sales representative sees them off. The sales representative operates the terminal device 10 upon the end of the negotiation, and the control unit 32 determines whether or not the operation to end the negotiation has been received (S1006).

[0108] Upon receiving the operation to end the business negotiation, the control unit 32 executes the customer information registration / update process (S1007). The customer information registration / update process automatically registers customer information based on the text data obtained during the business negotiation, thereby enriching the customer information. Note that the customer information registration / update process may also be performed during the business negotiation (in parallel with S1001 to S1005).

[0109] Furthermore, upon receiving the operation to end the business negotiation, the control unit 32 executes the process of registering the daily report and the business negotiation history (S1008). The information processing system can support the sales representative in creating the daily report and business negotiation history.

[0110] The following describes in detail each process of the control unit 32 as explained in Figure 12.

[0111] <Opening process> First, referring to Figure 13, we will explain the opening process when a sales representative welcomes a customer. During the opening process, a welcome message and other information are displayed on the display 60 when a customer is welcomed. Figure 13 is a sequence diagram illustrating the opening process and actions when a customer enters the store.

[0112] The sales representative operates the terminal device 10 to start the information processing system 1, and the microphone 301 continuously acquires audio. First, the server device 30 performs a process to identify the customer. This is because, in the case of repeat customers, it is effective to understand customer information such as past purchase history and family structure. Methods for identifying customers include the sales representative specifying a meeting from schedule information, obtaining the customer's name from the content of the conversation, or the sales representative directly inputting the information. It is preferable for the display 60 to display a welcome message before the customer enters the room, but in situations where the customer enters before the message is displayed, it is better to display the message when the customer is identified. Figure 13 illustrates a case in which a customer is identified from audio data.

[0113] S1: The sales representative welcomes the customer into the meeting room and begins a conversation, such as a greeting. Microphone 301 acquires audio data of the utterances made by the sales representative and the customer. For example, Microphone 301 acquires audio data of the sales representative's utterance, such as "Welcome, Mr. / Ms. XX." In addition, Microphone 301 may acquire audio data of the customer, such as "I'm XX. Thank you for your time today."

[0114] S2: Microphone 301 transmits audio data to terminal device 10 in real time.

[0115] S3: When the voice data acquisition unit 11 of the terminal device 10 acquires voice data, the transmission unit 12 transmits the voice data and spatial identification information to the server device 30 in real time. The spatial identification information indicates the space where the business negotiation takes place, such as "VIP Room 1," and is used by the server device 30 to identify the facilities and other elements of the space. The spatial identification information is either pre-configured in the terminal device 10 or entered by the sales representative.

[0116] S4: When the receiving unit 31 of the server device 30 receives voice data and spatial identification information, it first requests the voice recognition unit 34 to convert the voice data into text data. The voice recognition unit 34 converts it into text data and passes it to the control unit 32. If the customer has not been identified, the control unit 32 has the language processing unit 33 perform morphological analysis. The language processing unit 33 extracts the name as a proper noun from the text data.

[0117] S5: Since the control unit 32 has not identified the customer, it requests the system API 41 to search for customer information by specifying the name. The system API 41 requests the SFA system 302 to search for customer information by specifying the name. The system API 41 may also request product information, spatial information 1-3, schedule information, etc. from the SFA system 302 for use in subsequent processing.

[0118] S6: The SFA system 302 receives the search results for customers searched by name from the CRM system 303. For example, it is possible that customers of the same gender are registered, so the SFA system 302 may return a list of all customers of the same gender to the server device 30 to ensure no customers are missed. If the voice data includes the name of a sales representative, customers of the same gender as the sales representative will also be included in the search, so the control unit 32 may obtain the schedule information for the meeting room for the day from the SFA system 302 and exclude the sales representative's name from the search. The control unit 32 obtains spatial feature information associated with spatial identification information, and schedule information in which customers matching the name are registered, from the SFA system 302. The control unit 32 also obtains product information from the SFA system 302. If the product information table is large, the size of the product information can be reduced by narrowing down the products based on the past "purchase history" and "content" of the customer information registered in the schedule information.

[0119] S7: When the server device 30 acquires spatial feature information, the control unit 32 stores the spatial feature information. The display data generation unit 38 then sets the spatial feature information on the negotiation space setting screen, which will be described later.

[0120] S8: The control unit 32 stores customer information. The display data generation unit 38 sets the customer information on the negotiation space setting screen.

[0121] S9: The control unit 32 holds the schedule information. The display data generation unit 38 sets the schedule information on the negotiation space setting screen.

[0122] S10: Having obtained the information necessary for the proposal, the control unit 32 instructs the display data generation unit 38 to generate display data for the display 60 and display data for the terminal device 10. First, the control unit 32 instructs the display data generation unit 38 to generate display data for the display 60, which will display a welcome message, etc. The display data generation unit 38 generates the display data for the display 60. If a speaker 306 is installed in the display 60, the display data generation unit 38 may generate audio message output information including music and voice.

[0123] S11: The transmission unit 39 of the server device 30 transmits the display destination (display 60) and the display data to the terminal device 10.

[0124] S12: The receiving unit 14 of the terminal device 10 receives the display data and passes it to the information display unit 16 based on the display destination (display 60). The information display unit 16 transmits the display data to the display 60 so that the display 60 can display a welcome message screen based on the display data (see Figure 14).

[0125] S13: Meanwhile, for the negotiation space setting screen to be displayed by the terminal device 10, the control unit 32 causes the display data generation unit 38 to generate display data for the negotiation space setting screen, which has spatial characteristic information, customer information, and schedule information set. The display data generation unit 38 generates the display data for the negotiation space setting screen.

[0126] In Figure 13, the terminal device 10 and the display 60 are displaying different screens. The display data generation unit 38 may generate the same display data for both the terminal device 10 and the display 60, or it may generate different display data for both. • When generating the same display data Optimal and predictive suggestions (confidential information described later will not be displayed on display 60) Closing process • When generating different display data Opening process Daily report screen (terminal device 10 only) S14: The transmission unit 39 of the server device 30 transmits the display destination (terminal device 10) and the display data to the terminal device 10.

[0127] S15: The receiving unit 14 of the terminal device 10 receives the display data and passes it to the display control unit 15 based on the display destination (terminal device 10). The display control unit 15 displays the negotiation space setting screen on the terminal device 10 based on the display data (see Figure 15). Alternatively, the processing in steps S13 to S15 may be omitted, that is, the negotiation space setting screen 210 described later may not be displayed, and the settings of the spatial feature information, customer information, and schedule information acquired in steps S7 to S9 may be completed, and the system API 41 may pass them to the control unit 32.

[0128] Figure 14 shows an example of a welcome message screen 200 displayed by the display 60. As shown in Figure 14, the welcome message screen 200 displays the customer's name 201 and a welcome message 202. This name 201 may be the customer's name registered in the customer information, the name in the schedule information, or a name obtained from their speech. By seeing their name, the customer can confirm that they are in the correct meeting room and feel reassured that they are welcome. Illustrations 203 and videos may also be displayed on the welcome message screen 200. The content of the illustrations 203 and videos could include products to be proposed in today's meeting, based on the "consultation details" in the schedule information.

[0129] Figure 15 shows the negotiation space setting screen 210 displayed by the terminal device 10. The items on the negotiation space setting screen 210 are explained below. Customer 211 is the name of the customer registered in the customer information. • Person in Charge 212 is the name of the sales representative in charge of the business negotiation, as registered in the schedule information. Here, customer 211 and person in charge 212 are referred to as "spatial information 1". Spatial information 1 is information that indicates the event taking place in that space (who is participating, whether it is a business negotiation, consultation, meeting, etc.). Along with customer 211 and person in charge 212, there may be a field for the purpose of the conversation (business negotiation, consultation, meeting, etc.). • Today's task 213 is the details of a business meeting registered in the schedule information. "Today's task 213" is referred to as "Spatial Information 2". Spatial Information 2 is information that indicates the content or purpose of the event to be held in that space (what you want to do in that space, etc.). • Time 214 represents the start and end times of the business negotiation registered in the schedule information. Location 215 is the name of the meeting room registered in the schedule information, or the spatial name of the space identified by the spatial identification information. Location 215 is referred to as "Spatial Information 3". Spatial Information 3 is information that describes the characteristics of the space (what kind of space it is, for example, if it is a VIP room it is a place to serve customers who have gold cards, what size the display installed in the space is, etc.). • The customer information section 216 displays customer information. The customer information is displayed along with a list of purchase history, and a "View Details" button 217 is also displayed. Pressing the "View Details" button 217 displays previously purchased items, prices, dates, sales representatives, etc. When the setting button 218 is pressed, the information entered on the negotiation space setting screen 210 is sent to the server device 30.

[0130] The AI ​​system 50 generates response messages using spatial information 1-3, etc. This improves the accuracy of the information it proposes. As explained in Figure 16, spatial information 1-3 can also be input by sales representatives. Spatial Information 1: AI System 50 uses information indicating events taking place in that space to understand the assumptions about how that space is being used. Spatial Information 2: The AI ​​system 50 understands the background information of an event by using information that indicates the content or purpose of the event taking place in that space. Spatial Information 3: The AI ​​system 50 uses information that describes the characteristics of the space to generate output suitable for that space. For example, if it is a VIP room, the AI ​​system 50 will suggest high-priced luxury products, or if the display installed in the space is small, it will limit the number of products that can be displayed, and therefore narrow down the number of products it suggests.

[0131] <<Another example of opening processing>> Next, we will explain another example of the opening process with reference to Figure 16. Figure 16 is a sequence diagram illustrating the opening process and operations when a customer enters the store. Figure 16 illustrates a case where the customer is identified through input operations by a sales representative. The server device 30 performs the process of identifying the customer based on the input spatial information 1 to 3. This is because, in the case of repeat customers, it is effective to understand customer information such as past purchase history and family structure.

[0132] S21: The sales representative operates the terminal device 10 to start the information processing system 1. In response to the sales representative's operation, the transmission unit 12 of the terminal device 10 requests the sales negotiation space setting screen from the server device 30.

[0133] S22: The receiving unit 31 of the server device 30 receives the request, and the control unit 32 causes the display data generation unit 38 to display the display data for the negotiation space setting screen. The negotiation space setting screen is displayed only by the terminal device 10. The display data generation unit 38 generates the display data for the negotiation space setting screen, and the transmission unit 39 transmits the display data for the negotiation space setting screen to the terminal device 10 along with the display destination (terminal device 10).

[0134] S22-2: The receiving unit 14 of the terminal device 10 receives the display data and passes it to the display control unit 15 based on the display destination (terminal device 10). The display control unit 15 displays the negotiation space setting screen on the terminal device 10 based on the display data. At this point, only the frame of the screen is displayed (see Figure 17).

[0135] S23: The sales representative enters the name of the customer for the upcoming business meeting into the business meeting space settings screen. The sales representative may also enter other items into the business meeting space settings screen. The operation reception unit 13 receives the input.

[0136] S24: The transmission unit 12 of the terminal device 10 transmits the entered customer name to the server device 30.

[0137] S25: The receiving unit 31 of the server device 30 receives the customer's name as operation information and passes the name to the control unit 32. Since the customer has not yet been identified, the control unit 32 requests the system API 41 to search for customer information by specifying the name. The system API 41 requests the SFA system 302 to search for customer information by specifying the name.

[0138] S26: The SFA system 302 receives the search results for customers searched by name from the CRM system 303. For example, it is possible that customers of the same gender are registered, so the SFA system 302 may return a list of all customers of the same gender to the server device 30 to ensure no customers are missed.

[0139] S27: When the system API 41 of the server device 30 obtains a list of customers that match the search, the control unit 32 decides to display it on the terminal device 10 for selection by the sales representative. The control unit 32 designates the terminal device 10 as the display destination and has the display data generation unit 38 display the display data of the customer list. The customer list is displayed only on the terminal device 10. The display data generation unit 38 generates the display data of the customer list, and the transmission unit 39 transmits the display data of the customer list to the terminal device 10 along with the display destination (terminal device 10). If the server device 30 has obtained schedule information, the schedule information can be integrated and displayed with the customer list 221.

[0140] S27-2: The receiving unit 14 of the terminal device 10 receives the display data and passes it to the display control unit 15 based on the display destination (terminal device 10). The display control unit 15 displays the customer list on the terminal device 10 based on the display data (see Figure 18).

[0141] S28: The sales representative selects a customer from the customer list for the upcoming business negotiation. The operation reception unit 13 receives the selection.

[0142] S29: The transmission unit 12 of the terminal device 10 transmits the selected customer identification information to the server device 30.

[0143] S30: The receiving unit 31 of the server device 30 receives customer identification information as operation information and passes it to the control unit 32. The control unit 32 has identified the customer and requests the system API 41 to retrieve customer information by specifying the customer identification information. The system API 41 requests the SFA system 302 to retrieve customer information by specifying the customer identification information. The system API 41 may also request product information, spatial feature information, schedule information, etc., from the SFA system 302. This is because, as will be described later, these will be used for optimal suggestion and predictive suggestion processing.

[0144] S31: The SFA system 302 receives customer information identified by customer identification information from the CRM system 303. The SFA system 302 returns the customer information to the server device 30. The control unit 32 may also obtain product information from the SFA system 302.

[0145] S32: System API 41 retrieves customer information and passes it to control unit 32. Control unit 32 retains the customer information once it has retrieved it. Control unit 32 also retains product information.

[0146] The control unit 32 designates the terminal device 10 as the display destination and causes the display data generation unit 38 to generate display data for customer information. The display data generation unit 38 may also generate display data for the business negotiation setting screen where customer information has been set.

[0147] S33: The transmission unit 39 of the server device 30 transmits the display destination (terminal device 10) and the display data to the terminal device 10.

[0148] S33-2: The receiving unit 14 of the terminal device 10 receives the display data and passes it to the display control unit 15 based on the display destination (terminal device 10). The display control unit 15 displays customer information on the negotiation space setting screen based on the display data.

[0149] S34: The sales representative enters spatial characteristic information and schedule information for the space where the business negotiation will take place into the business negotiation space setting screen (see Figure 15). The operation reception unit 13 receives the input. Note that the schedule information may be set automatically by the customer's selection. The spatial characteristic information may also be set automatically by identifying the spatial identification information or schedule information set in the terminal device 10.

[0150] S35: The transmission unit 12 of the terminal device 10 transmits the input spatial feature information and schedule information to the server device 30.

[0151] S36: The receiving unit 31 of the server device 30 receives spatial feature information and schedule information as operation information and passes it to the control unit 32. The control unit 32 stores the spatial feature information and schedule information.

[0152] S37: The control unit 32, having identified the customer, designates the display 60 as the display destination and instructs the display data generation unit 38 to generate display data for the welcome message screen. The control unit 32 instructs the display data generation unit 38 to generate display data for the display 60 to display a welcome message, etc. The display data generation unit 38 generates display data for the welcome message screen to be displayed on the display 60.

[0153] S38: The transmission unit 39 of the server device 30 transmits the display destination (display 60) and the display data to the terminal device 10.

[0154] S39, S40: The receiving unit 14 of the terminal device 10 receives the display data and passes it to the information display unit 16 based on the display destination (display 60). The information display unit 16 displays a welcome message screen on the display 60 based on the display data (see Figure 14).

[0155] Figure 17 shows the negotiation space setting screen 210 displayed by the terminal device 10 in step S22-2. The negotiation space setting screen 210 in Figure 17 is the same as in Figure 15, but in step S22-2, the input fields on the negotiation space setting screen 210 are empty. The sales representative enters the customer's name, etc., via the negotiation space setting screen 210.

[0156] Figure 18 shows the customer list screen 220 displayed in step S27-2. A button to display a customer search popup is displayed near the customer item 211 on the negotiation space setting screen. For example, if the name entered by the sales representative is "Tanaka-san", the customer list screen 220 is displayed showing the customer list 221 where "Tanaka" is included in the name. The customer list 221 in Figure 18 has schedule information integrated into it, so the sales representative, location, and consultation details registered in the schedule information are also displayed.

[0157] The sales representative confirms the date and time of the meeting, the customer, the location, and the details of the consultation, confirms that the meeting is about to begin in VIP Room 1, and presses the start button 222. When the start button 222 is pressed, the customer is selected. Based on the press of the start button 222, the meeting space setting screen 210 may be redisplayed, and the meeting space setting screen may be displayed in a state similar to Figure 15, where the customer information and schedule information of the selected customer are reflected and each input field is filled in.

[0158] Note that the negotiation space setting screen 210 after the spatial feature information and schedule information are entered in step S34 may be the same as in Figure 15. The welcome message screen displayed in step S40 may be the same as in Figure 14.

[0159] As described above, the server device 30 of this embodiment can display the welcome message and the business negotiation space setting screen 210 in several ways.

[0160] <Display of suggestions based on audio data> In parallel with identifying the customer, the customer and sales representative begin negotiations. The process from when the customer and sales representative talk until the server device 30 displays a proposal is described below.

[0161] Figure 19 is a sequence diagram illustrating the process by which the server device 30 displays a proposal based on audio data acquired from space and spatial information 1-3.

[0162] S51: The customer and the sales representative begin a conversation. Microphone 301 acquires audio data of what the sales representative and the customer say.

[0163] S52: Microphone 301 transmits audio data to terminal device 10 in real time or at regular intervals.

[0164] S53: When the audio data acquisition unit 11 of the terminal device 10 acquires audio data, the transmission unit 12 transmits the audio data to the server device 30 in real time or at regular intervals.

[0165] S54: When the receiving unit 31 of the server device 30 receives audio data, it passes it to the language processing unit 33 and the speech recognition unit 34. The language processing unit 33 performs speaker dialization to identify the speaker at an appropriate level of granularity, such as for each piece of audio data, each sentence, or each word.

[0166] S55: The speech recognition unit 34 also converts the speech data into text data. The speech recognition unit 34 obtains the speaker from the speaker identification unit and associates the speaker with the text data. The speech recognition unit 34 passes the text data to the control unit 32. Steps S54 and S55 can be performed in any order.

[0167] S56: The control unit 32 requests the storage unit 36 ​​to store text data, and the storage unit 36 ​​stores the text data in the text data storage unit 49.

[0168] S57: The control unit 32 requests the AI ​​control unit 37 to generate a prompt using the accumulated text data each time a certain amount of text data is accumulated or at regular intervals (such as the time required for a conversation to progress, e.g., 60 seconds). This prompt requests the generation of an optimal suggestion (an example of a first suggestion) and a predictive suggestion (an example of a second suggestion), etc.

[0169] The request transmission unit 42 of the AI ​​control unit 37 generates prompts to instruct the AI ​​system 50 to generate optimal suggestions, predictive suggestions, decisions, and action lists, based on the accumulated text data and pre-prepared system prompts. The prompt for optimal suggestions requests that the AI ​​system 50 propose the best product based on the customer's past statements. In other words, it is a prompt that instructs the AI ​​system 50 to make suggestions based on the customer's statements. The prompt for predictive suggestions requests that the AI ​​system 50 propose products that the customer has not mentioned before but that are predicted to be likely to be purchased based on past statements. In other words, it is a prompt that instructs the AI ​​system 50 to make suggestions that are suitable for the customer other than those that directly address the customer's stated requests. The prompt for decisions requests that the AI ​​system 50 extract decisions that can be inferred from the customer's and sales representative's past statements. The prompt for action lists requests that the AI ​​system 50 extract action lists that can be inferred from the customer's and sales representative's past statements. These prompts may be sent to the AI ​​system 50 in multiple parts.

[0170] A simple example of the system prompt is as follows. This prompt may be sent to the AI ​​system 50 in multiple parts. [Example of a system prompt follows] The participants in the business meeting are as follows: {Customer name, Sales representative name} The customer requests that have been registered in advance are as follows: {Consultation details} The business negotiations will take place at the following locations: {meeting room} The customer information is as follows: {Customer information} The following is a list of products we can offer. {Product information} Based on the subsequent conversation, please select and propose the most suitable product or service for the customer. Based on the subsequent conversation, anticipate the customer's likely future needs and select and propose products or services that they are most likely to desire. Also, if any decisions were made during this business negotiation, please let us know what those decisions were. Also, if this business negotiation conversation included an action list, please provide it to us.

[0171] The following dialogue takes place in a specific space. This space has the following characteristics: {Customer Name, Sales Representative Name}, {Consultation Details}. Based on the circumstances of this space, please propose the most suitable product for Mr. / Ms. XX (the customer). This space also has the characteristics of a {Meeting Room}. Please propose a product that is best suited to be displayed in this type of space. {Conversation text data} Furthermore, the decisions made during the business negotiations so far are as follows: {Decisions} Furthermore, the action list for the business negotiations to date is as follows: {Action List} (This is an example of a system prompt.) In this case, the AI ​​control unit 37 fills in the {***} portion of the system prompt with the information entered by the sales representative and the information obtained from each memory unit to create the prompt. {Customer name, Sales representative name} will be as follows: {Customer Identification Number C0001: Mr. Ichiro Tanaka, Sales Representative: Suzuki} These are entered by sales representatives or registered in customer information or schedule information. These are spatial information 1. In addition to the above, other information associated with Ichiro Tanaka in the customer information storage unit 51 may also be included. The details of your inquiry are as follows: I'd like to discuss gift ideas. This information is either entered by the sales representative or registered in the schedule information. Note that this is spatial information 2. The {meeting room} will look like this: {VIP Room 1} This information is either entered by the sales representative or stored as schedule information or spatial feature information. This is spatial information 3. In addition to the spatial name "VIP Room 1" mentioned above, it may also include a description of the space and its characteristics, which are stored as spatial feature information.

[0172] {Customer Information}, {Product Information}, and {Conversation Text Data} are the contents of the customer information storage unit 51, product information storage unit 53, and text data storage unit 49, respectively. For example, customer information includes at least one of age, gender, or purchase history. {Conversation Text Data} increases in length as the conversation time increases. {Decisions} and {Action List} are the contents of the decisions and action lists in the temporary information storage unit 48, respectively.

[0173] S58: The AI ​​control unit 37 inputs the prompt generated as described above to the AI ​​system 50.

[0174] This section provides additional information regarding prompt generation using RAG. The control unit 32 may perform the necessary processing for RAG. When using RAG, the control unit 32 converts the acquired text data into a vector (hereinafter referred to as "message vector") that represents the meaning of the text data using multidimensional numerical values. Such a semantic vector is called a distributed representation. By converting the original text into a distributed representation, the meaning and content of the original text can be represented numerically, and comparison using vector data becomes possible. Conventional text data searches and comparisons had drawbacks, such as failing to match a search if even one character was different, or matching a search if even one character was the same. In contrast, by comparing using vector data, information can be searched based on the degree of similarity in meaning and content of the original text. Since the distributed representation is vector data, the original text can be represented as a point in the vector space. The similarity of vector data can be calculated using existing methods, such as cosine similarity.

[0175] The control unit 32 uses the generated message vector to extract product information from the product information stored in the product information storage unit 53 that has a relatively high correlation with the message vector. The product information stored in the product information storage unit 53 is pre-divided into chunks, and chunk vectors obtained by converting these chunks into distributed representations are also stored. The chunks that are the source of chunk vectors similar to the message vector are called "similar chunks". The control unit 32 obtains the product information that is the source of the similar chunks from the product information storage unit 53 as a search result.

[0176] The AI ​​control unit 37 generates prompts by applying the product information identified through the search to the system prompts. This allows the product information to be narrowed down before being sent to the AI ​​system 50. The prompts may also contain instructions to suggest only products that are in stock. The prompts may also contain instructions to suggest only products that fit the budget. The prompts may also contain instructions to suggest only products that do not exist online (cannot be found through search). The prompts may also contain instructions to suggest the advantages of a product over similar products that exist online. The prompts may also contain instructions to rephrase technical terms into language that is easy for the customer 8 to understand.

[0177] S59: The AI ​​system 50 generates a response message that includes optimal suggestions, predicted suggestions, decisions, and an action list based on the learned parameters. Note that the optimal suggestions, predicted suggestions, decisions, and action list are not always all included. The AI ​​system 50 sends the response message to the AI ​​control unit 37. The response acquisition unit 43 of the AI ​​control unit 37 acquires the response from the AI ​​system 50. Note that depending on the control procedure, the first response is not the final response, and the final response message is obtained by repeating steps S58 and S59 multiple times. The AI ​​control unit 37 sends the generated response message to the control unit 32. The response message is sent in a format that the server device 30 can recognize, including the optimal suggestions, predicted suggestions, decisions, and action list. In addition, topics extracted from the conversation are associated with the optimal suggestions and predicted suggestions and sent accordingly.

[0178] S60: If the response message includes optimal suggestions, predictive suggestions, decisions, and action lists, the control unit 32 registers these in the temporary information storage unit 48. Specifically, the control unit 32 identifies the product information of the products included in the optimal suggestion and predictive suggestion of the response message from the product information included in the prompt, and registers the product information of the identified products in the temporary information storage unit 48. At this time, the control unit 32 may refer to the product information table and register the product information obtained from the product information table in the temporary information storage unit 48. The response message may also include product identification information to identify the product. Furthermore, if the response message includes decisions and action items, the control unit 32 identifies the decisions and action items and registers them in the temporary information storage unit 48.

[0179] S61: The control unit 32 displays the same proposal on the terminal device 10 and the display 60 (mirroring). Alternatively, the proposal may be displayed only on the display 60 or only on the terminal device 10. The sales representative may specify the display destination. First, the control unit 32 designates the display 60 as the display destination and requests the display data generation unit 38 to generate display data including the optimal proposal and the predicted proposal. The display data generation unit 38 generates display data for the proposal display screen that displays the optimal proposal and the predicted proposal on the display 60. When the terminal device 10 and the display 60 display the same proposal display screen 250 (mirroring), the control of the display 60 may be controlled by the information display unit 16 of the terminal device 10. In that case, the control unit 32 requests the display data generation unit 38 to generate one display data for displaying the proposal display screen, which is the screen to be displayed on the terminal device 10 and the screen to be displayed on the display 60, and the display data generation unit 38 generates one display data. When the terminal device 10 and the display 60 display different proposal display screens 250, the display of the display 60 may be controlled by the information display unit 16 of the terminal device 10 to display a different screen on the display 60 than the screen displayed on the terminal device 10.

[0180] S62: The transmission unit 39 of the server device 30 transmits the generated display data and the display destination (display 60) to the terminal device 10. The display destination information does not need to be included. In step S61, it was stated that the display data generation unit 38 of the server device 30 generates the display data, but this is not the only way. Alternatively, the transmission unit 39 of the server device 30 may transmit the optimal suggestion and prediction suggestion received from the AI ​​system 50 to the terminal device 10, and the information display unit 16 of the terminal device 10 may generate the display data.

[0181] S63, S64: The receiving unit 14 of the terminal device 10 receives display data and passes it to the information display unit 16 based on the display destination (display 60). Alternatively, the terminal device 10 may perform the process of displaying on the display 60 based on its settings without using the information of the display destination. For example, the information display unit 16 of the terminal device 10 can open a web browser window and control the display to show that window on the display 60 based on its settings. Or, an application installed on the terminal device 10 can perform the same control. In this case, the terminal device 10 and the display 60 are connected with an HDMI® cable or a VGA cable, causing the terminal device 10 to display on the display 60. This technology is one in which the GPU (graphics processing unit) of the terminal device 10 generates display data, converts it into a signal via HDMI or VGA, and outputs it to another display, and is equipped in most general terminals. The information display unit 16 displays a suggested display screen on the display 60 based on the display data (see Figure 20).

[0182] S65: Next, the control unit 32 designates the terminal device 10 as the display destination and requests the display data generation unit 38 to generate display data including the optimal suggestion and the predictive suggestion. The display data generation unit 38 generates display data for a suggestion display screen that displays the optimal suggestion and the predictive suggestion on the terminal device 10. An audio message to the speaker 306 may be sent to the terminal device 10 along with the display data. For example, the audio message may output a warning melody from the speaker 306 in synchronization with the display of the suggestion.

[0183] S66: The transmission unit 39 of the server device 30 transmits the generated display data and the display destination (terminal device 10) to the terminal device 10.

[0184] S67: The receiving unit 14 of the terminal device 10 receives the display data and passes it to the display control unit 15 based on the display destination (terminal device 10). The display control unit 15 causes the terminal device 10 to display the suggested display screen based on the display data. The suggested display screen may be the same as in Figure 20. The server device 30 may also generate a voice message and send it to the earphone 305. The audio of the transmitted voice message will be played back through the speaker of the earphone 305.

[0185] <Example of the display screen shown by Display 60> Figure 20 shows the proposal display screen (part 1) 250 displayed on the display 60 by the server device 30. On the proposal display screen (part 1) 250, six proposals generated by the AI ​​system 50 are displayed in multiple regions 251 to 256, respectively. The display 60 can display proposals region by region, display multiple proposals on one screen, and display the optimal proposal and predictive proposal on a single screen.

[0186] The screen configuration in Figure 20 is just one example, but the proposal display screen (part 1) 250 always displays six areas 251-256. The proposal display screen (part 1) 250 is a screen that makes proposals for each topic of conversation. In business negotiations, various topics are discussed, such as topics of conversation, requests, customer issues, and products. This invention allows for the visual summarization of these topics and the display of product proposals corresponding to those topics. Even if the topic of conversation shifts during a business negotiation, proposals for each topic ensure that customer requests are not missed. Furthermore, by automatically making proposals for each topic, unexpected products can be introduced. In addition, displaying proposals for each topic makes the business negotiation more visible, and allows for further proposals beyond those made by the sales representative. The number of areas is just an example. In the initial stages of the conversation, it is not necessary for all of areas 251-256 to display proposals. As the conversation progresses, more information is obtained from the conversation, and areas 251-256 are gradually filled in. Alternatively, as will be described later in Figure 21, if there are fewer than six suggestions based on the conversation, the server device 30 may request the AI ​​system 50 to suggest products based solely on customer information, and keep some suggestions displayed in the six areas 251-256 at all times.

[0187] When a new proposal is generated while some proposal is displayed in the six areas 251-256, the oldest proposal is overwritten. The AI ​​system 50 determines the similarity between the existing proposal and the newly generated proposal, and if there are similar proposals, it specifies the existing proposal and links it to the newly generated proposal. The server device 30 sends the new proposal linked to the existing proposal to the terminal device 10. In this way, the terminal device 10 can display similar proposals in the same area. Customers can see similar proposals in the same area.

[0188] As an example, the upper part of Figure 20 shows the optimal suggestion generated by the AI ​​system 50 from conversation information and customer information. In a sales negotiation use case, the optimal suggestion is a recommended product. An optimal suggestion is a suggestion of products or services that fulfill the customer's explicit requests. A suggestion of products or services that the customer has requested in their statements is also an optimal suggestion. As another example, the lower part of Figure 20 shows a predictive suggestion that the AI ​​system 50 predicted (predicted) from the text data accumulated through the conversation, even though the customer did not directly request it in their statements. A predictive suggestion is a suggestion of products that are not directly included in the content of the conversation but are likely to be desired in the future. It can also be said that a predictive suggestion is a suggestion of products or services that the customer has not requested in their statements. It can also be said that it is a suggestion that is suitable for the customer, other than a suggestion that corresponds to the customer's stated requests. The number of optimal suggestions and predictive suggestions may be determined by the AI ​​system 50's suggestions, or the sales representative may decide on the number of each and set it in the terminal device 10.

[0189] (1) Area 251 displays the topic "Jewelry that Mr. / Ms. △△ likes" and a photo of the suggested jewelry item. This suggestion is the optimal suggestion proposed by the AI ​​system 50 after it analyzes the customer's statement, "I think jewelry would be good this year," which is the customer's request (topic). The server device 30 retrieves the photo of the optimal suggested product from the product information and sends it to the terminal device 10. It should be noted that, including the following suggestions, it is an estimation of which customer statements the AI ​​system 50 based the optimal and predictive suggestions on.

[0190] (2) Area 252 displays the topic "○○ Exhibition" and, as a suggestion, the date, time, and location of the exhibition. This suggestion is the optimal suggestion proposed by the AI ​​system 50 after it analyzes the customer's statement, "I went to the ○○ exhibition and was interested in Italian jewelry." The search was performed using the keyword ○○. In this way, the optimal and predictive suggestions may present not only products and services stored in the memory unit, but also information obtained by searching for information on the internet. In that case, a prompt containing instructions to search for information suitable for the customer's needs will be sent to the AI ​​system 50. The AI ​​system 50 communicates with the search server to search for information and sends the response to the server device 30.

[0191] (3) Area 253 displays the topic "Jewelry Case" and a photo of the suggested product, a jewelry case. This suggestion is the optimal suggestion proposed by the AI ​​system 50 after it analyzes the customer's statement, "They said they wanted a jewelry case to match their new house." The server device 30 retrieves the photo of the optimally suggested product from the product information and sends it to the terminal device 10.

[0192] Thus, the server device 30 of this embodiment does not simply suggest products based on conversation, but rather allows the AI ​​system 50 to understand the spatial situation and then has the AI ​​system 50 suggest products or services, thereby enabling it to suggest appropriate products.

[0193] (4) Area 254 displays the topic "Custom-made furniture to fit the floor plan" and photos of furniture related to the suggestion. This suggestion is a predictive suggestion that AI system 50 analyzed based on the customer's statement, "Actually, I recently moved," and predicted that the customer would likely want products in the future. In other words, AI system 50 predicted that since the customer had moved, they would need furniture that suited the floor plan in the near future. Since AI system 50 does not know the floor plan, it assumes several floor plans such as 1R, 2DK, and 3LDK, and suggests photos of furniture suitable for each. For example, for a 3LDK, AI system 50 would suggest the following furniture. "Queen bed or double bed (master bedroom), semi-double or single bed (children's room / guest room), dining table & chairs (for 4-6 people), large sofa (L-shaped or seating 3 or more people), coffee table or low table, large TV stand & storage shelves, bookshelf or living room storage, work desk & chair (for study), china cabinet or cupboard, kitchen counter or island table, entryway storage (shoe rack, umbrella stand), dresser." The predictive suggestions do not need to be accurate. Customers can look at the display 60 and, if they are interested in furniture, they may say something like, "By the way, it would be nice to have a functional storage shelf." The microphone 301 acquires this voice data, and the AI ​​system 50 further analyzes the text data to identify that the customer's statement includes "storage shelf," so the AI ​​system 50 can suggest recommended storage shelves. The server device 30 can preferably display photos of the storage shelves in the same area 254.

[0194] (5) Area 255 displays a topic and suggestions titled "Recommended Plans for Your Spouse's Birthday." These suggestions are predictive suggestions generated by AI System 50, which analyzes the customer's statement "Spouse's Birthday" and anticipates services that are likely to be desired in the future. Specifically, AI System 50 predicted that because the customer's spouse's birthday is approaching, a plan to spend the birthday together will be needed in the near future.

[0195] (6) Area 256 displays a topic and suggestion titled "Accommodation in the XX area." This suggestion is a predictive suggestion generated by AI system 50, which analyzed the customer's statement, "I plan to travel on my birthday," and anticipated services that are likely to be desired in the future. Specifically, AI system 50 predicted that the customer would be traveling with their spouse and would need recommendations for hotels and other services in their destination in the near future. Since AI system 50 does not know the destination, it picks a destination that will be in the tourist season on the birthday and suggests recommended hotels in that destination.

[0196] The optimal suggestion is a product or service included in the conversation's text data, so the server device 30 can suggest all products or services included in the customer's statements without fail. The predictive suggestion is a product or service that is not included in the customer's statements but is likely to be desired in the future, so it can uncover and suggest demand. Furthermore, through predictive suggestions, the server device 30 can suggest products or services to address latent needs that the customer is unaware of.

[0197] Furthermore, the statements that formed the basis of the proposal may be displayed in each area 251-256. The statements that formed the basis of the proposal may be displayed when a sales representative operates the system. The classification of each proposal as an optimal proposal or a predicted proposal may be displayed. In this way, the AI ​​system 50 can include statement data, including statements made in a space, and spatial information of that space (what kind of event is taking place in that space, what people want to do in that space, what characteristics does that space have), and generate proposals from the AI ​​system 50 that take into account the spatial information and the content of the statements.

[0198] Referring to Figure 21, the suggestion display screen in the initial stages of the conversation will be explained. Figure 21 is the suggestion display screen (part 2) 260 displayed on the display 60 by the server device 30. Note that the explanation of Figure 21 will mainly explain the differences from Figure 20. In the suggestion display screen (part 2) 260 of Figure 21, if there are fewer than six suggestions based on the conversation, the AI ​​system 50 displays suggestions generated solely from customer information. "Solely from customer information" means that the conversation text data is not included, but information from various storage units 304, such as spatial information 1-3, is used.

[0199] The upper part of Figure 21 is the same as Figure 20. The lower part of Figure 21 shows the optimal suggestion proposed by the AI ​​system 50 using only customer information, without using text data accumulated through conversation. In the early stages of a conversation, there is little information that can be obtained from the conversation, but suggestions can be made based on the customer's past purchase history and other information.

[0200] Furthermore, suggestions like the one shown in the lower part of Figure 21 can be made by the AI ​​system 50 when the AI ​​control unit 37 generates prompts such as, "Until there are a total of six suggestions based on the subsequent conversation, please suggest products that are most likely to be desired based solely on customer information."

[0201] (7) Area 254 displays a suggestion titled "Latest information on brands you have previously purchased," as well as a collection of products from that brand. This suggestion is an optimal suggestion based on the AI ​​system 50's analysis of the customer's purchase history, suggesting products from the same brand. In other words, the AI ​​system 50 determined that this customer is likely to like the same brand and retrieved products from that brand's product information. The server device 30 retrieves photos of the suggested optimal products from the product information and transmits them to the terminal device 10.

[0202] (8) In area 255, the topic "the latest menu of the same restaurant as the dinner on last year's birthday" and the proposed menu are displayed. This proposal is an optimal proposal made by the AI system 50 by analyzing the purchase history in the customer information and proposing the menu of the same restaurant that was used in the past. The purchase history in the customer information also records the purchase history of services proposed by the sales representative or the server device 30. The AI system 50 determines that this customer may prefer meals at the same restaurant, and displays the latest menu of the restaurant. The latest menu of the restaurant may be searched by the AI system 50 via a network, or may be searched by the server device 30. The server device 30 transmits the latest menu of the restaurant to the terminal device 10.

[0203] (9) In area 256, the topic "brands popular among people of the same generation" and the proposed collection of the brands are displayed. This proposal is an optimal proposal made by the AI system 50 based on the age and gender in the customer information, proposing brands that are popular among men of the same generation. The AI system 50 searches for brands of products that are popular for each generation. The server device 30 transmits the proposed brand to the terminal device 10.

[0204] <Closing Process> When the remaining time until the end time of the business negotiation registered in the schedule information is less than a certain value, or when it is detected from the conversation that the business negotiation is heading toward completion, the display 60 displays a closing screen summarizing the content of the business negotiation. The closing screen may be displayed according to an operation by the sales representative or a voice instruction.

[0205] Figure 22 is a sequence diagram for describing the process in which the display 60 displays the closing screen.

[0206] S71: The closing detection unit 35 detects that the business negotiation is nearing its end based on phrases in the conversation, such as "Thank you for your time today," "How about your next visit?", and "We will send you a document summarizing today's discussion later." The closing detection unit 35 also compares the current time with the end time in the schedule information and detects that the business negotiation is nearing its end when the remaining time falls below a certain amount. Alternatively, the AI ​​control unit 37 may generate a prompt from the conversation text data that indicates the business negotiation is nearing its end and repeatedly send it to the AI ​​system 50. This prompt may be, for example, "If the conversation text data contains expressions indicating the business negotiation is nearing its end, please generate a closing detection flag." When the AI ​​control unit 37 obtains the closing detection flag from the AI ​​system 50, the control unit 32 determines that it is time to close the deal. The closing detection unit 35 may also determine that the deal is closed if closing is detected by both the time and the closing detection flag. Alternatively, the sales representative may instruct the terminal device 10 to start the closing process.

[0207] S72: When the control unit 32 of the server device 30 receives a notification that a closing has been detected from the closing detection unit 35 or the AI ​​control unit 37, it reads the temporary information recorded for this business negotiation from the temporary information storage unit 48.

[0208] S73: The control unit 32 sets the display destination to the display 60 and causes the display data generation unit 38 to generate display data for the closing screen. The display data generation unit 38 generates display data for the closing screen using the temporarily held information.

[0209] S74: The transmitting unit 39 of the server device 30 transmits the closing screen display data along with the display destination (display 60) to the terminal device 10. The receiving unit 14 of the terminal device 10 receives the display data and passes it to the information display unit 16 based on the display destination (display 60).

[0210] S75,76: The information display unit 16 displays a closing screen on the display 60 based on the display data (see Figure 23).

[0211] The control unit 32 may also display the closing screen on the terminal device 10. In this case, the display data generation unit 38 also generates display data for the closing screen for the terminal device 10. This allows the terminal device 10 and the display 60 to display the same closing screen. Furthermore, the terminal device 10 that the sales representative checks may be different from the display 60. For example, only the terminal device 10 may display the purchase probability estimated by the AI ​​system 50.

[0212] Alternatively, the server device 30 may not use the information in the temporary information storage unit 48 to display the closing screen, but may send a prompt containing the information in the temporary information storage unit 48 to the AI ​​system 50 to generate the closing screen information. Or, the server device 30 may send a prompt containing all the accumulated text data, spatial information 1 to 3, and the instruction "Create information summarizing the entire business negotiation" to the AI ​​system 50 to generate the closing screen information.

[0213] Figure 23 shows an example of a closing screen 270 displayed by the display 60. The closing screen 270 includes a request section 271, a proposal section 272, a decision section 273, and an action list section 274.

[0214] All of the information displayed on the closing screen 270 is registered in the temporary information table. The Requests field 271 displays a list of the contents of the "Extracted Requests" column in the temporary information table. The Proposals field 272 displays a list of the contents of the "Proposals" column in the temporary information table. The Decisions field 273 displays a list of the contents of the "Decisions" column in the temporary information table. The Action List field 274 displays a list of the contents of the "Action List" column in the temporary information table. In this way, the closing screen 270 displays requests, proposals, decisions, and the action list all on one screen.

[0215] The closing screen 270 is automatically displayed on display 60, allowing the customer and sales representative to review the meeting and share decisions and actions. It also allows the customer to review the proposal again, which can be beneficial for re-proposing the product.

[0216] Additionally, illustration 275 may be displayed on the closing screen 270. The AI ​​system 50 may generate illustration 275 based on one or more main conversational points, or it may be obtained from the network.

[0217] <After the business meeting> When a business meeting ends, such as when a customer leaves the meeting room, the sales representative generates a meeting history and a daily report. In this embodiment, the server device 30 generates a meeting history summarizing the details of the meeting and a daily report, and registers them in the SFA system 302. This helps to prevent sales representatives from forgetting to create meeting histories and daily reports, or from spending time creating them.

[0218] Figure 24 is a sequence diagram illustrating the process by which the server device 30 generates sales history and daily reports.

[0219] S81: A sales representative inputs the end of a business negotiation to the terminal device 10 (application). The sales representative may press some button, or may issue an instruction by voice such as "Create a business negotiation history and a daily report as the business negotiation has ended". The operation accepting unit 13 of the terminal device 10 accepts the end of the business negotiation. Furthermore, when a certain period of time elapses from the end time of the schedule information, the server device 30 may automatically start the business negotiation end process.

[0220] S82: The transmitting unit 12 of the terminal device 10 transmits a business negotiation end processing request to the server device 30.

[0221] S83: The receiving unit 31 of the server device 30 accepts the business negotiation end processing request as operation information, and notifies the control unit 32. The control unit 32 determines to create a business negotiation history and a daily report as the business negotiation end process. The control unit 32 requests the AI control unit 37 to generate a prompt using accumulated text data and the like.

[0222] The AI control unit 37 generates a prompt for instructing the AI system 50 to generate the business negotiation history and the daily report, based on the accumulated text data, spatial information 1 to 3, temporary holding information, various storage units 304, and a pre-prepared system prompt.

[0223] A simple example of the system prompt is as follows. 《Example of system prompt starting here》 The participants of the business negotiation are as follows. {Customer name, Sales representative name} The pre-registered customer request is as follows. {Consultation content} The location where the business negotiation is held is as follows. {Business negotiation room} The customer information is as follows. {Customer information} The temporary holding information is as follows. {Temporary holding information} The content of the conversation is as follows. {Text data of the conversation} The items to be included in the daily report are as follows: {Daily report items} Sales negotiation history items {Items in the sales negotiation history} Please use the above information to generate the sales history and daily report. (This is an example of a system prompt.) In this case, the AI ​​control unit 37 fills in the {***} portion of the system prompt with the information entered by the sales representative and the information obtained from each memory unit to create the prompt. {Customer name, Sales representative name} will be as follows: {Customer Identification Number C0001: Mr. Ichiro Tanaka, Sales Representative: Suzuki} These are either entered by sales representatives or registered in customer information or schedule information. This is spatial information 1. The details of your inquiry are as follows: I want to give a gift. This information is either entered by the sales representative or registered in the schedule information. This is spatial information 2. The {meeting room} will look like this: {VIP Room 1} This information is either entered by the sales representative or registered in the schedule information. This is spatial information 3. {Customer Information}, {Temporarily Stored Information}, and {Conversation Text Data} are the contents of the customer information storage unit 51, the temporary stored information storage unit 48, and the text data storage unit 49, respectively. The items in the daily report will be as follows: {Daily report identification information, sales negotiation history identification information, purpose of visit, proposal, purchased items, next visit date, staff member's memo} This information is an item in the daily report information table. The {sales negotiation history items} will be as follows: {Sales history identifier, date and time, customer identifier, purchased product identifier, action item, person in charge} This information is an item in the sales history table.

[0224] S84: The AI ​​control unit 37 inputs the prompt generated as described above to the AI ​​system 50.

[0225] S85: The AI ​​system 50 generates a response message including the sales history and daily report based on the learned parameters. The AI ​​system 50 sends this response message to the AI ​​control unit 37. The response acquisition unit 43 of the AI ​​control unit 37 acquires the response from the AI ​​system 50. Depending on the control procedure, the first response may not be the final response, and the final response message may be obtained by repeating steps S84 and S85 multiple times. The AI ​​control unit 37 sends the generated response message to the control unit 32.

[0226] S86: The control unit 32 designates the terminal device 10 as the display destination and requests the display data generation unit 38 to generate display data for the daily report screen. The display data generation unit 38 generates display data for display on the daily report screen on the terminal device 10. If the sales representative does not need to check the sales negotiation history, it will not be displayed on the terminal device 10. However, if the sales representative needs to check it, the display data generation unit 38 generates display data for display on the sales negotiation history screen on the terminal device 10.

[0227] S87: The transmission unit 39 of the server device 30 transmits the generated display data and the display destination (terminal device 10) to the terminal device 10.

[0228] S88: The receiving unit 14 of the terminal device 10 receives the display data and passes it to the display control unit 15 based on the display destination (terminal device 10). The display control unit 15 displays the daily report screen on the terminal device 10 based on the display data (see Figure 25).

[0229] S89: The sales representative inputs the operation to edit and register the daily report as needed (registration button 286 in Figure 25) to the terminal device 10. The sales representative may also input the operation to edit and register the daily report by voice. The terminal device 10 accepts the editing and registration operation.

[0230] S90: The transmission unit 12 of the terminal device 10 transmits the daily report information to the server device 30 along with the registration request.

[0231] S91: The receiving unit 31 of the server device 30 receives a request to register daily report information as operation information and notifies the control unit 32. The control unit 32 requests registration from the system API 41, specifying the daily report information and the sales negotiation history. The system API 41 requests registration of the daily report information and the sales negotiation history from the SFA system 302.

[0232] Figure 25 shows an example of the daily report screen 280 displayed by the terminal device 10. The daily report screen 280 is a sales report for today's business negotiations. In Figure 25, items such as bibliographic information 281, requests 282, proposals 283, purchased items 284, and next visit 285 are displayed. Bibliographic information 281 is recorded in customer information and schedule information, while the others are registered in temporary storage information. Purchased items 284 are included in the decided items. In addition, the daily report screen 280 can also display conversation text data.

[0233] The sales representative can edit bibliographic information 281, requests 282, proposals 283, purchased items 284, and next visit date 285. This is because the AI ​​system 50 may generate information that is not intended or may be missing. When the sales representative presses the registration button 286, the terminal device 10 sends the daily report information to the server device 30.

[0234] <Customer information registration / update> Audio data recorded during business negotiations may contain customer information, but it is time-consuming and prone to errors for sales representatives to take notes and then register or update the customer information later. In this embodiment, the server device 30 registers customer information in the SFA system 302 based on the text data acquired during the business negotiation, either during or after the negotiation.

[0235] Figure 26 is a sequence diagram illustrating the process by which the server device 30 registers or updates customer information. The process in Figure 26 may be performed during a business negotiation or after the negotiation has concluded.

[0236] S101: The control unit 32 determines that customer information should be registered or updated by acquiring text data during the negotiation, and by performing a negotiation completion operation after the negotiation is completed. The control unit 32 requests the AI ​​control unit 37 to generate a prompt using the accumulated text data, etc.

[0237] The AI ​​control unit 37 generates prompts to instruct the AI ​​system 50 to register or update customer information based on the accumulated text data, spatial information 1 to 3, temporary storage information, various storage units 304, and pre-prepared system prompts.

[0238] A simple example of the system prompt is as follows: [Example of a system prompt follows] The current customer information is as follows: {Customer information} The customer information items are as follows: {Customer Information Items} Based on the subsequent conversation, please extract customer information from the text data that has a field for customer information but is not registered. Even if there is no field for customer information, please extract information that describes the customer's characteristics along with the field name. Based on the subsequent conversation, identify customer information that has already been registered in the customer information fields but has changed, and extract the customer information that needs to be updated. {Conversation text data} (This is an example of a system prompt.) In this case, the AI ​​control unit 37 creates a prompt by substituting the information obtained from each storage unit into the {***} portion of the system prompt. {Customer Information} and {Conversation Text Data} are the contents of the customer information storage unit 51 and the text data storage unit 49, respectively. {Customer Information Fields} can be any field in the customer information table.

[0239] S102: The AI ​​control unit 37 inputs the prompt generated as described above to the AI ​​system 50.

[0240] S103: The AI ​​system 50 generates a response message containing customer information to be registered or updated based on learned parameters. If the conversation text data does not contain customer information, it generates a response message to that effect. The AI ​​system 50 sends this response message to the AI ​​control unit 37. The response acquisition unit 43 of the AI ​​control unit 37 acquires the response from the AI ​​system 50. Depending on the control procedure, the first response may not be the final response, and the final response message may be obtained by repeating steps S102 and S103 multiple times. The AI ​​control unit 37 sends the generated response message to the control unit 32.

[0241] S104: The control unit 32 determines whether or not the customer information to be registered or updated is included in the response message.

[0242] S105: If the response message contains customer information to be registered or updated, the control unit 32 requests the system API 41 to register the customer information and item name. The system API 41 requests the SFA system 302 to register or update the customer information.

[0243] This allows customer information to be registered and updated without sales representatives having to perform registration or update tasks themselves. The server device 30 can also update customer information based on conversations. For example, if the conversation includes the statement that a customer has "moved," the AI ​​system 50 determines that the customer has moved. Next, if the AI ​​system 50 detects that an address is included in the customer's conversation, it decides to update the address in the customer's information. Alternatively, the AI ​​system 50 may decide to update the phone number based on the conversation statement, "I changed my phone number."

[0244] <Use cases for solution sales> The information processing system of this embodiment can be used in a wide range of business negotiations, not just product negotiations. As an example, we will describe a proposal displayed on the display 60 in a solution sales scenario.

[0245] First, Figure 27 shows an example of conversation text data stored in the text data storage unit 49 during solution sales. The content of the conversation is different from that in Figure 4, but the items are the same. In this way, the storage unit 36 ​​can store text data of business negotiations conducted in space in the text data storage unit 49. The sequence diagram for the process of displaying the optimal proposal and the predicted proposal can be the same as in Figure 19.

[0246] The customer's comments are as follows: "We're promoting DX (Digital Transformation) within the company, but we don't have much of a budget. It's about ¥XX." The decision-maker is the department head, so I want to move forward as quickly as possible. I have the following problem, and I would like to solve it. Previously, we implemented the following solution, but it wasn't quite satisfactory. The part-time workers weren't able to use it properly. Figure 28 shows the proposal display screen (part 3) 290 displayed on the display 60 by the server device 30. The proposal display screen (part 3) 290 displays six proposals generated by the AI ​​system 50. The configuration of the proposal display screen (part 3) 290 is the same as that of Figure 20, and the main differences will be explained.

[0247] As an example, the upper part of Figure 28 shows the optimal suggestion generated by the AI ​​system 50 from conversational information and customer information. In solution sales, the optimal suggestion is the recommended solution. Also as an example, the lower part of Figure 28 shows a predictive suggestion that the AI ​​system 50 predicted from text data accumulated through conversation, even though the customer did not directly request it in their statements. Predictive suggestions in solution sales are suggestions that anticipate services or products that are likely to be desired in the future, even if they are not directly included in the content of the conversation.

[0248] (1) Area 291 displays a topic titled "Solutions under XX yen" and a list of proposed solutions. These proposals are the optimal solutions proposed by AI system 50 after analyzing customer statements such as "We are promoting DX within the company" and "We don't have much of a budget, around XX yen." Server device 30 retrieves a list of the optimally proposed products from the product information and displays it on terminal device 10.

[0249] (2) Area 292 displays a topic titled "Plans with support for usage after implementation" and a list of proposed solutions. This proposal is the optimal suggestion that the AI ​​system 50 proposed based on the customer's statement, "We implemented it, but it wasn't quite satisfactory," and determined that the customer was requesting a solution with support. The AI ​​system 50 identified a solution with support from the list of solutions in area 291. The server device 30 displays this optimal suggestion on the terminal device 10.

[0250] (3) Area 293 displays a topic titled "Schedule until quotation" and a proposed schedule until quotation. This proposal is the optimal proposal generated by the AI ​​system 50 based on the customer's statement "I want it urgently," which determined that the customer is in a hurry to implement the solution. The AI ​​system 50 generates a general schedule until quotation. The server device 30 displays this optimal proposal on the terminal device 10.

[0251] (4) Area 294 displays the topic "Part-time Employment Support Solution" and a list of proposed solutions. This proposal is a predictive suggestion that the AI ​​system 50 predicted is likely to be desired in the future, based on the customer's statement, "We weren't able to make good use of our part-time employees." In other words, the AI ​​system 50 predicted that this customer would need a part-time employment support solution in the near future because they wanted to make better use of their part-time employees. The server device 30 displays this predictive suggestion on the terminal device 10.

[0252] (5) Area 295 displays the topic "Equipment setup due to relocation" and a suggestion. This suggestion is a predictive suggestion that the AI ​​system 50 predicted based on the customer's statement, "I'm thinking of moving my office to Shibuya," and anticipated the services that are likely to be needed in the future. In other words, the AI ​​system 50 predicted that this customer would need equipment setup services due to relocation in the near future. The server device 30 displays this predictive suggestion on the terminal device 10.

[0253] (6) Area 296 displays a topic titled "Workplace Design Proposal" and a proposal. This proposal is a predictive proposal generated by the AI ​​system 50 after analyzing the customer's statement, "I'm thinking of moving my office to Shibuya," and anticipating a service that is likely to be desired in the future. In other words, the AI ​​system 50 foresaw that this customer would need a service to design the workplace at their new office in the near future because they are moving. The server device 30 acquires this predictive proposal and displays it on the terminal device 10.

[0254] Furthermore, the statements that formed the basis of the proposal may be displayed in each area 291-296. The statements that formed the basis of the proposal may be displayed when a sales representative operates the system. The classification of each proposal as either an optimal proposal or a predicted proposal may be displayed.

[0255] <Action playback on the suggestion display screen> The server device 30 may display demo videos of the relevant products and services on the suggestion display screen, in addition to simply displaying the products and services. In this embodiment, a large screen display 60 is installed in the space, so this can be utilized.

[0256] Figure 29 shows the proposal display screen (part 4) 300 displayed on the display 60 by the server device 30. The explanation of Figure 29 mainly describes the differences from Figure 28. In the proposal display screen (part 4) 300 of Figure 29, videos 287 and 288 are played in areas 292 and 296. Videos 287 and 288 are, for example, demo videos of a solution. The AI ​​system 50 obtains the demo video or URL from the solution's Web and passes it to the control unit 32. The demo video or URL may be stored in the product information, and the control unit 32 may obtain the demo video or URL from the product information database based on the product identification information transmitted from the AI ​​system. The control unit 32 obtains the demo video or the URL of the demo video and causes the display data generation unit 38 to generate display data to be displayed in areas 292 and 296 of the proposal display screen. The demo video may be embedded in areas 292 and 296, or the URL may be embedded. The display 60 connects to the URL and downloads the demo video.

[0257] In Figure 29, videos 287 and 288 are displayed only in regions 292 and 296, but videos may be displayed in all regions 291 to 296. When generating an optimal suggestion or predictive suggestion, the AI ​​system 50 determines whether there are any relevant videos on the web, and if there are, it associates them with the regions and passes them to the control unit 32.

[0258] Furthermore, the server device 30 may not only play existing videos 287 and 288, but may also send prompts to the AI ​​system 50 instructing it to generate videos suitable for products and solutions, causing the AI ​​system 50 to generate videos. The server device 30 causes the AI ​​system 50 to generate a concept movie that combines customer statements with key points of the solutions that can be provided.

[0259] The prompts for generating the video might look like this: [Example of a system prompt follows] The participants in the business meeting are as follows: {Customer name, Sales representative name} The customer requests that have been registered in advance are as follows: {Consultation details} The business negotiations will take place at the following locations: {meeting room} The customer information is as follows: {Customer information} The following is a list of the solutions we can propose. {Solution Product Information} Based on the subsequent conversation, please select and propose the most suitable solution for the customer. Based on the subsequent conversation, anticipate and select products or services that the customer is likely to desire in the future, and propose them to them. Also, if any decisions were made during this business negotiation, please let us know what those decisions were. Also, if this business negotiation conversation included an action list, please provide it to us. Additionally, based on the subsequent conversation, select the most suitable solution for the customer and generate one or more videos to promote the sale of that selected solution. {Conversation text data} Furthermore, the decisions made during the business negotiations so far are as follows: {Decisions} Furthermore, the action list for the business negotiations to date is as follows: {Action List} (This is an example of a system prompt.) In this way, the display 60 can improve the appeal of products and services by playing videos.

[0260] <Content changes depending on the display destination> Even when the display 60 and the terminal device 10 are showing the same screen, there are cases where it is desirable to display different information on the display 60 and the sales representative's terminal device 10. For example, costs, information about other customers, or internal company information may be displayed on the terminal device 10 but not on the display 60. Hereinafter, this information will be referred to as confidential information. The control unit 32 of the server device 30 maintains a list of confidential information that should not be displayed on the display 60, and controls the system to display confidential information only on the terminal device 10.

[0261] Figure 30 shows the proposal display screens (5) 310 and 320 displayed by the server device 30 on the display 60 and the terminal device 10, respectively. The explanation of Figure 30 mainly focuses on the differences from Figure 28. Figure 30(a) is the proposal display screen (5) 310 displayed by the terminal device 10, and Figure 30(b) is the proposal display screen (5) 320 displayed by the display 60. The proposal display screen (5) 310 shows a cost of 297, but the proposal display screen (5) 320 does not show a cost. In this way, the server device 30 can display confidential information only on the terminal device 10.

[0262] Figure 31 shows an example of confidential information. The AI ​​control unit 37 transmits the confidential information to the AI ​​system 50, and if the proposal contains confidential information, it generates a prompt to notify the system of this. In this way, the control unit 32 can obtain from the AI ​​system 50 which information is confidential if the proposal contains confidential information. The control unit 32 controls the system to display the confidential information on the terminal device 10, but not on the display 60. It is also possible that the product information table contains information that should be kept confidential. In that case, the control unit 32 retrieves product information for the proposed product from the product information table, displays the product information excluding the confidential information on the display 60, and displays the product information including the confidential information on the terminal device 10.

[0263] Figure 32 is a sequence diagram illustrating the process by which the server device 30 displays a proposal on the terminal device 10 and the display 60. The explanation of Figure 32 mainly describes the differences from Figure 19. Steps S51 to S60 can be the same as in Figure 19. However, the response message now also includes whether or not the proposal contains confidential information.

[0264] S60-2: When the control unit 32 receives a response message from the AI ​​control unit 37, it determines whether or not it contains confidential information. In this case, since it contains confidential information, the control unit 32 determines that it will generate different display data for the display 60 and the terminal device 10.

[0265] S61: The control unit 32 first designates the display 60 as the display destination, removes confidential information, and requests the display data generation unit 38 to generate display data including the optimal suggestion and the prediction suggestion. The display data generation unit 38 generates display data (an example of the first display data) for the suggestion display screen that displays the optimal suggestion and the prediction suggestion on the display 60.

[0266] S62: The transmission unit 39 of the server device 30 transmits the generated display data and the display destination (display 60) to the terminal device 10.

[0267] S63, S64: The receiving unit 14 of the terminal device 10 receives the display data and passes it to the information display unit 16 based on the display destination (display 60). The information display unit 16 displays a suggested display screen on the display 60 based on the display data (Figure 30(b)).

[0268] S65: The control unit 32 then designates the terminal device 10 as the display destination and requests the display data generation unit 38 to generate display data including the optimal suggestion and prediction suggestion, while retaining the confidential information. The display data generation unit 38 generates display data (an example of second display data) for the suggestion display screen that displays the optimal suggestion, prediction suggestion, and confidential information on the terminal device 10.

[0269] S66: The transmission unit 39 of the server device 30 transmits the generated display data and the display destination (terminal device 10) to the terminal device 10.

[0270] S67: The receiving unit 14 of the terminal device 10 receives the display data and passes it to the display control unit 15 based on the display destination (terminal device 10). The display control unit 15 causes the terminal device 10 to display the suggested display screen based on the display data (Figure 30(a)).

[0271] In this way, the server device 30 can be controlled to display confidential information only on the terminal device 10, even if it is included in the proposal. The display control method is not limited to the above method, and the information display unit 16 of the terminal device 10 may be controlled to display a different screen on the display 60 than the terminal device 10.

[0272] <Display of proposals as operated by the sales representative> The same proposal is displayed on the sales representative's terminal device 10 and the display 60. Alternatively, the proposal may be displayed only on the display 60. In this way, the display 60 displays proposals, but it may also display only the proposals selected by the sales representative. That is, initially, the server device 30 displays proposals only on the terminal device 10, and when the sales representative selects a proposal, the server device 30 displays the proposal on the display 60. Therefore, the display on the sales representative's terminal device 10 and the display 60 can be made different. In addition, the sales representative can select and display to the customer the products they want to sell.

[0273] Figure 33 shows the proposal display screen (6) 330 and 340 displayed by the server device 30 on the display 60 and the terminal device 10. The explanation of Figure 33 mainly describes the differences from Figure 28. Figure 33(a) is the proposal display screen (6) 330 displayed by the terminal device 10, and Figures 33(b) and (c) are the proposal display screen (6) 340 displayed by the display 60. In the proposal display screen (6) 330, display display buttons 298 are displayed in each area 291 to 296. When a display display button 298 is pressed by the mouse cursor 299 or by tapping, the proposal for that area is also displayed on the display 60.

[0274] In the proposal display screen (part 6) 340 in Figure 33(b), proposals are displayed in areas 291 to 293, but not in areas 294 to 296. In this state, by pressing the display button 298 in area 294, the same proposal can be displayed in area 294 of the display 60, as shown in Figure 33(c).

[0275] Figure 34 is a sequence diagram illustrating the process by which the server device 30 displays a proposal on the terminal device 10 and the display 60 based on voice data and spatial information 1-3 in response to the sales representative's operation. The explanation of Figure 34 mainly focuses on the differences from Figure 19. Steps S51-S60 may be the same as in Figure 19.

[0276] S65: First, the control unit 32 designates the terminal device 10 as the display destination and requests the display data generation unit 38 to generate display data including optimal suggestions and predictive suggestions. The display data generation unit 38 generates display data for a suggestion display screen that displays the optimal suggestions and predictive suggestions on the terminal device 10. The display data generation unit 38 places display buttons 298 in each area.

[0277] S66: The transmission unit 39 of the server device 30 transmits the generated display data and the display destination (terminal device 10) to the terminal device 10.

[0278] S67: The receiving unit 14 of the terminal device 10 receives the display data and passes it to the display control unit 15 based on the display destination (terminal device 10). The display control unit 15 causes the terminal device 10 to display the suggested display screen based on the display data (Figure 33(a)).

[0279] S121: The sales representative reviews the proposal and presses the display button 298 to select the proposal (product or service) they want to present to the customer. The operation reception unit 13 receives the press.

[0280] S122: The transmission unit 12 of the terminal device 10 sends a request to the server device 30 to display a proposal, specifying the area name.

[0281] S123: The receiving unit 31 of the server device 30 receives a display request as operation information and notifies the control unit 32. The control unit 32 specifies the area name and display destination (display 60) in response to the display request and requests the display data generation unit 38 to generate display data for that area. The display data generation unit 38 generates display data to display the optimal suggestion or prediction suggestion displayed in the area on the display 60. This display data may be display data only for the relevant area.

[0282] S124: The transmission unit 39 of the server device 30 transmits the generated display data and the display destination (display 60) to the terminal device 10. The display data has a specified area.

[0283] S125, S126: The receiving unit 14 of the terminal device 10 receives the display data and passes it to the information display unit 16 based on the display destination (display 60). Based on the display data, the information display unit 16 displays the optimal suggestion or predictive suggestion in the designated area of ​​the suggestion display screen displayed on the display 60 (Figure 33(c)).

[0284] In this way, the proposal is displayed first on the terminal device 10, and the customer selects the proposal that the sales representative presents, so the sales representative can pick out the products they want to sell and display them to the customer. The display control method is not limited to the above method, and the information display unit 16 of the terminal device 10 may be controlled to display a different screen on the display 60 than the terminal device 10.

[0285] <Audio message output by the earphones> If a supervisor were to overhear a conversation between a customer and a sales representative, they could whisper instructions that would be effective only for the sales representative. In this embodiment, the AI ​​system 50 can perform this action.

[0286] Figure 35 shows an example of a voice message suggested by the AI ​​system 50 via the earphones based on the conversation. "The customer showed interest in XX. A more detailed explanation would be effective." This indicates a keyword of high customer interest. Since AI System 50 detected a keyword of high customer interest, it instructs the sales representative to provide a more detailed explanation of the features that the customer indicated interest in purchasing. "Customers are interested in features, not price. Emphasize the benefits." This is a helpful tip for making effective product proposals. The AI ​​system 50 has detected what the customer is interested in, and instructs the sales representative to provide a detailed explanation of the features the customer has shown interest in. "Product △△ is close to the customer's needs. Presenting other options will make comparison easier." is a helpful hint for making appropriate product suggestions. Since AI System 50 has detected a product that closely matches the customer's needs, it instructs the sales representative to suggest a more appropriate product. "The customer is comparing our product to another store's product. Let's explain our advantages." This is a key point in competitive comparison. AI System 50 detected that the customer was comparing our product to those of other companies, so it instructed the sales representative to explain our advantages. "The product you requested is out of stock, but we can suggest □□, which has similar characteristics." This indicates the stock status and provides an alternative suggestion. AI System 50 detected that the product was out of stock, so it instructs the sales representative to describe a similar product. "The customer's reaction is weakening. Ask more questions to capture their interest." This is a detection of a change in customer attitude. AI system 50 has detected a change in the customer's reaction and instructs the salesperson to change the questions or topics. "That explanation seems difficult. Let's rephrase it in simpler terms." is a simplified explanation of technical jargon. AI System 50 determined that the salesperson's explanation was difficult, so it instructed the salesperson to rephrase or explain the technical terms. "Customers are concerned about price. Offering package discounts and special offers is effective," is price negotiation advice. AI System 50 detects that the customer has a problem with the price, so it instructs the sales representative to explain why the price is reasonable. "The customer is hesitant to buy. Offering a 'special offer available only today' might encourage them to make a decision." This is the moment when a deal is about to be closed. AI System 50 detects that the customer is close to making a purchase, so it instructs the salesperson to provide information to encourage the customer to decide. "The customer is not making an immediate decision, but is interested. Following up later would be effective." This indicates the need for follow-up. AI System 50 detected that the customer is not purchasing but may purchase later, so it instructs the sales representative to follow up.

[0287] The AI ​​control unit 37 generates a prompt, as follows, to request the AI ​​system 50 to generate instructions for the earphones. The rest of the prompt configuration may be the same as in the case of optimal suggestions and predictive suggestions. "Based on the subsequent conversation, please generate effective advice for the sales representative to help them close the deal. For example, this could be information a supervisor who overheard the conversation between the customer and the sales representative might want to share with the sales representative privately." Figure 36 is a sequence diagram illustrating the process by which the server device 30 outputs a voice message from the earphones based on voice data and spatial information 1-3. The explanation of Figure 36 mainly focuses on the differences from Figure 19. Steps S51-S67 can be the same as in Figure 19.

[0288] S131: If the response message from the AI ​​system 50 includes instructions for the sales representative, the control unit 32 designates the earphone 305 as the output destination and requests the display data generation unit 38 to generate an audio message. The display data generation unit 38 generates an audio message that outputs the instructions to the earphone 205. Other examples of the generated audio message include audio data that reads aloud the text of the display data generated in step S65, or audio data that reads aloud a text that summarizes the content of the display data.

[0289] S132: The transmitting unit 39 of the server device 30 transmits the generated voice message and output destination (earphone 305) to the terminal device 10.

[0290] S133, S134: The receiving unit 14 of the terminal device 10 receives the voice message and passes it to the voice message output unit 17 based on the output destination (earphone 305). The voice message output unit 17 outputs sound to the earphone 305 based on the voice message.

[0291] In this way, the AI ​​system 50 can provide effective instructions only to sales representatives based on conversations.

[0292] <Main effects> The server device 30 of this embodiment does not simply suggest products based on conversation, but rather allows the AI ​​system 50 to understand the spatial situation and then has the AI ​​system 50 suggest products or services, thereby enabling the suggestion of appropriate products or services.

[0293] [Second Embodiment] In this embodiment, we will describe a server device 30 that detects the customer's emotions during a conversation using an emotion sensor, visualizes those emotions, and provides advice on appropriate responses to those emotions.

[0294] <About the features> Figure 37 shows an example of the functional configuration of the information processing system 1 in this embodiment. In the description of Figure 37, the same reference numerals are used for the same functional parts as in Figure 3, and their descriptions are omitted.

[0295] The terminal device 10 newly includes an emotion information acquisition unit 18. The emotion information acquisition unit 18 acquires customer emotion information from the emotion sensor 307. An example of the function of the emotion sensor 307 will be described. The emotion sensor 307 consists of a part that measures pulse waves using the Doppler method and a microcontroller part that analyzes pulse wave data using a library, and these are integrated as a single sensor. As a result, the emotion sensor 307 can perform all necessary information processing on the edge side, from acquiring biological information (vital data) to analyzing and visualizing emotions. Specifically, the emotion sensor 307 uses a 24GHz Doppler sensor to measure pulse waves non-contact. The emotion sensor 307 analyzes the measured pulse wave shape and the fluctuations in pulse intervals with a microcontroller to understand the state of brain activity (central nervous system) and autonomic nervous system. Based on this, a predetermined algorithm estimates emotions such as "concentration (immersion)," "relaxation," "drowsiness (boredom, laziness)," "fatigue," "activity level (tension, energy)," and "exhaustion (degree of fatigue)" in real time and "visualizes" them numerically. The emotion sensor 307 can measure the emotional information of a person from close range to several meters away without contact. The emotion sensor 307 detects emotional information at predetermined time intervals.

[0296] The emotion sensor 307 may be either a contact type or a non-contact type. Customers may wear the emotion sensor 307 by hanging it around their neck, wearing it like a wristwatch, or wearing it on a hat. Alternatively, the emotion sensor 307 may be installed separately from the customer and acquire emotional information remotely. Furthermore, the emotion sensor 307 may estimate emotions using not only a person's vital data but also their tone of voice, speaking speed, etc.

[0297] The transmission unit 12 transmits emotional information along with audio data to the server device 30. In the server device 30, the display data generation unit 38 generates display data that visualizes and displays the customer's emotions during the business negotiation.

[0298] Alternatively, instead of the emotion sensor 307, the estimation unit 44 of the server device 30 may estimate emotions using one or more of the vital data, voice tone, and speaking speed. In this case, the vital data is transmitted to the server device 30. Furthermore, the emotion estimation may be performed by an external server instead of the server device 30, or at least one of the vital data or voice data (for the analysis of voice tone and speaking speed) may be transmitted to the AI ​​system 50, allowing the AI ​​system 50 to estimate the emotions.

[0299] <Display of suggestions based on audio data> Figure 38 is a sequence diagram illustrating the process by which the server device 30 displays a proposal and emotional information based on audio data acquired from space and spatial information 1-3. The explanation of Figure 38 mainly focuses on the differences from Figure 19.

[0300] S52-2: The emotion information acquisition unit 18 of the terminal device 10 acquires emotion information from the emotion sensor 307.

[0301] S53-2: The transmitting unit 12 transmits emotion information to the server device 30 in real time or at regular intervals. The timing of transmission of voice data and emotion information does not need to be the same.

[0302] The subsequent processing is the same as in Figure 19, but in step S65, the control unit 32 designates the terminal device 10 as the display destination and requests the display data generation unit 38 to generate display data including optimal suggestions and predictive suggestions. Furthermore, in this embodiment, the control unit 32 designates the terminal device 10 as the display destination and requests the display data generation unit 38 to generate display data including sentiment information. The display data generation unit 38 generates display data for a suggestion display screen that displays optimal suggestions, predictive suggestions, and sentiment information on the terminal device 10.

[0303] Furthermore, the control unit 32 may request information from the AI ​​system 50 to improve the customer's emotional information. An example of a system prompt is shown. This system prompt is an instruction to generate suggestions based on the emotional information. [Example of a system prompt follows] We will send sentiment information of the participants in the business negotiation. {emotional information} If the customer's emotional information is negative (e.g., low concentration, high drowsiness, low activity level, high exhaustion), and therefore unsuitable for the business negotiation, please generate information to improve that emotional information. (This is an example of a system prompt.) Such system prompts allow the AI ​​system 50 to suggest information that, for example, can improve negative emotions into positive ones. The AI ​​system 50 may suggest, for example, "How about taking a break?" or "It seems your concentration is decreasing." These suggestions are also included in the display data and sent to the terminal device 10. The transmission unit 39 may also transmit information that improves emotional information as voice data to the earphone 305.

[0304] The server device 30 may obtain information to improve emotional information from another server without using the AI ​​system 50, or the server device 30 may generate information to improve emotional information.

[0305] S66: The transmission unit 39 of the server device 30 transmits the generated display data (including meeting information) and the display destination (terminal device 10) to the terminal device 10.

[0306] S67: The receiving unit 14 of the terminal device 10 receives the display data and passes it to the display control unit 15 based on the display destination (terminal device 10). The display control unit 15 causes the terminal device 10 to display a suggestion display screen based on the display data. This suggestion display screen displays emotion information.

[0307] <Screen example> Figure 39 shows the suggestion display screen (7) 350 displayed on the terminal device 10 by the server device 30. On the suggestion display screen (7) 350, six suggestions generated by the AI ​​system 50 are displayed in multiple areas 251 to 256, and emotion information is displayed in emotion information areas 321 to 323.

[0308] The emotional information area 321 displays the customer's emotional changes in real time over time during a business negotiation. The customer's emotional information (concentration, relaxation, drowsiness, fatigue, and activity level) is color-coded, and the changes in each intensity over time are displayed as a graph. Note that emotional information for the sales representative may also be displayed.

[0309] The emotion information area 322 displays the current emotions of each customer participating in the business negotiation as icons. In Figure 39, there are three customers, and their current emotions are displayed. The emotion name is displayed above the icon, and the customer name is displayed below it. Each customer's current emotion is updated, for example, at regular intervals or whenever their emotion changes.

[0310] The emotion information area 323 displays emotion information, which summarizes the emotions of multiple customers, using icons. This summarized emotion information can include the average emotion of multiple customers, the most frequently occurring emotion, or the emotion least favorable for business negotiations. Sales representatives can grasp customer reactions at a glance. This summarized emotion information is updated at regular intervals or whenever emotions change.

[0311] Furthermore, the advice display area 324 shows information generated by the AI ​​system 50 that can be used to improve the customer's emotional information. The person in charge can refer to this information as needed to proceed with the business negotiation.

[0312] <Main effects> According to this embodiment, in addition to the effects of the first embodiment, sales representatives can conduct business negotiations while checking customer emotional information, allowing them to proceed with the negotiations appropriately according to the customer's emotions. Information that improves the customer's emotional information is displayed, enabling sales representatives to improve the customer's emotions to be more suitable for the negotiation.

[0313] [Third Embodiment] In this embodiment, a server device 30 that registers information generated during a business negotiation with an external service system 308 after the negotiation has taken place will be described. Conventionally, meeting minutes and other information were registered with the external service system 308 manually, but this will enable automatic linking.

[0314] Figure 40 shows an example of the system configuration of information processing system 1. The explanation of Figure 40 mainly describes the differences from Figure 1. In this embodiment of information processing system 1, the server device 30 can cooperate with the external service system 308. The external service system 308 provides not only storage services but also services that manage information in a format suitable for that information. The external service system 308 could be a cloud service that allows users to create applications that streamline and automate business processes without coding. This external service system 308 can centrally manage customer information, project management, task progress, etc. It allows for flexible customization of items and integration with other services. Users without programming knowledge can build internal business applications tailored to their company's needs. It can also be used as storage for saving and sharing files and data. The external service system 308 could be, for example, a customer relationship management (CRM) system. This external service system 308 centrally manages customer information, sales history, activity logs, etc., optimizing sales activities. The external service system 308 supports decision-making with its rich analytical functions and dashboards, can be integrated with various extension applications, and can also automate tasks. It can also store files and documents as a storage function. The external service system 308 could be, for example, an online whiteboard tool. This external service system 308 is an online whiteboard tool with an infinite canvas. External service system 308 allows users to freely place sticky notes, shapes, images, etc., making it suitable for brainstorming and meetings. External service system 308 allows multiple users to edit simultaneously, enabling remote collaboration. External service system 308 also has a wide range of templates, supporting the creation of business flows and mind maps. Created boards are saved on the cloud and function as storage. The external service system 308 could be, for example, a note-taking application. This external service system 308 incorporates AI into the note-taking process and is suitable for information organization and learning support. Based on the entered documents and notes, the AI ​​in the external service system 308 summarizes and answers questions. This external service system 308 allows for efficient knowledge accumulation and retrieval, making it suitable for supporting research and learning. Users can upload documents and interact with the AI. Saved documents are stored in the cloud.

[0315] <About the features> Figure 41 is a diagram showing an example of the functional configuration of the information processing system 1 in this embodiment. In the description of Figure 41, the same reference numerals are used for the same functional parts as in Figure 3, and their descriptions are omitted.

[0316] The system API 41 of the server device 30 can communicate with the external service system 308. The system API 41 converts information generated during a business negotiation into a data format suitable for the external service system 308 and exports (registers) it. The system API 41 also converts the information generated during a business negotiation stored in the external service system 308 into a data format for the server device 30 and imports it.

[0317] <Registration of information generated during business negotiations> Figure 42 is a sequence diagram illustrating the process by which the server device 30 registers information generated during a business negotiation with the external service system 308. The explanation of Figure 42 mainly focuses on the differences from Figure 24. Steps S81 to S91 can be the same as in Figure 24.

[0318] S92: The sales representative who has registered the daily report and sales negotiation history then inputs an operation to register the information generated during the sales negotiation to the external service system 308 to the terminal device 10. The terminal device 10 accepts the registration operation. The information generated during the sales negotiation includes temporary storage information, daily report information, sales negotiation history, text data, and voice data. Note that the sales representative may register the information manually, or, following step S91, the server device 30 may automatically start registering the information generated during the sales negotiation to the external service system 308. This enables automatic linking without manual operation.

[0319] S93: The transmission unit 12 of the terminal device 10 sends a registration request for information generated during the business negotiation to the server device 30.

[0320] S94: The receiving unit 31 of the server device 30 receives the registration request and notifies the control unit 32. The control unit 32 requests registration from the system API 41, specifying the temporary storage information, daily report information, negotiation history, text data, and voice data. In addition, summaries of text data for each utterance and summaries of text data for each speaker may also be registered. The system API 41 converts these into the data format of the external service system 308 to be registered, associates them with the negotiation identification information, and requests registration from the external service system 308. The data format of the external service system 308 to be registered is known. For example, in the case of an external service system 308 that creates business applications without coding, information is stored in a tabular format, so the information to be stored in each column of the table is associated and converted into a tabular data format. At this time, the system API 41 creates a correspondence table that shows which column the temporary storage information, daily report information, negotiation history, text data, and voice data are stored in. If the temporary storage information itself has multiple items, a correspondence table is created for each item and each column. This allows the system API 41 to import temporary information, daily report information, sales negotiation history, text data, and audio data from external service systems.

[0321] <Download information generated during business negotiations> Figure 43 is a sequence diagram illustrating the process and operation of a sales representative downloading information generated during a sales negotiation from the external service system 308. For convenience, information generated during a sales negotiation and registered in the external service system 308 is referred to as "registered information."

[0322] S201: The sales representative operates the terminal device 10 to start the information processing system 1. In response to the sales representative's operation, the transmission unit 12 of the terminal device 10 requests the information search screen from the server device 30. The information search screen is a screen for searching the registered information registered in the external service system 308.

[0323] S202: The receiving unit 31 of the server device 30 receives the request, and the control unit 32 causes the display data generation unit 38 to display the display data for the information search screen. The information search screen is displayed only by the terminal device 10. The display data generation unit 38 generates the display data for the information search screen, and the transmission unit 39 transmits the display data for the information search screen to the terminal device 10 along with the destination (terminal device 10).

[0324] S203: The receiving unit 14 of the terminal device 10 receives the display data and passes it to the display control unit 15 based on the display destination (terminal device 10). The display control unit 15 displays the information retrieval screen on the terminal device 10 based on the display data. At this point, only the frame of the screen is displayed (see Figure 44).

[0325] S204: To identify the registration information they want to download, the sales representative enters information that identifies the business deal into the search screen. This information may include the sales representative's name, the customer's name, and the date of the business deal. The operation reception unit 13 receives the input.

[0326] S205: The transmission unit 12 of the terminal device 10 sends a search request to the server device 30 along with information that can identify the entered business opportunity.

[0327] S206: The receiving unit 31 of the server device 30 receives a search request and passes information that can identify the business opportunity to the control unit 32. The control unit 32 requests the system API 41 to search for registered information, specifying the information that can identify the business opportunity. The system API 41 requests the external service system 308 to search for registered information, specifying the information that can identify the business opportunity.

[0328] S207: The external service system 308 searches the registered information using information that can identify a business opportunity and sends the search results to the server device 30. The system API 41 of the server device 30 receives the search results (list of registered information).

[0329] S208: When the system API 41 of the server device 30 obtains a list of registered information that matches the search, the control unit 32 determines that it is a request from a sales representative and should be displayed on the terminal device 10. The control unit 32 designates the terminal device 10 as the display destination and has the display data generation unit 38 display the display data for the list of registered information. The list of registered information is displayed only on the terminal device 10. The display data generation unit 38 generates the display data for the list of registered information, and the transmission unit 39 transmits the display data for the list of registered information to the terminal device 10 along with the display destination (terminal device 10).

[0330] S209: The receiving unit 14 of the terminal device 10 receives the display data and passes it to the display control unit 15 based on the display destination (terminal device 10). The display control unit 15 displays a list of registered information on the terminal device 10 based on the display data (see Figure 45).

[0331] S210: The sales representative selects the registered information of the business opportunity they want to display from the list. The operation reception unit 13 accepts the selection.

[0332] S211: The transmission unit 12 of the terminal device 10 transmits the identification information of the selected registration information to the server device 30.

[0333] S212: The receiving unit 31 of the server device 30 receives the registration information identification information as operation information and passes it to the control unit 32. The control unit 32 has identified the registration information identification information and requests the system API 41 to retrieve the registration information by specifying the registration information identification information. The system API 41 requests the external service system 308 to retrieve the registration information by specifying the registration information identification information.

[0334] S213: The external service system 308 receives the registration information identified by the registration information identification information. The external service system 308 returns the registration information to the server device 30.

[0335] S214: System API 41 retrieves registration information from the external service system 308, which is the registration destination, and converts it to the data format of the server device 30 using the above correspondence table. Once System API 41 retrieves the registration information, it passes it to the control unit 32. Once the control unit 32 retrieves the registration information, it specifies the terminal device 10 as the display destination and causes the display data generation unit 38 to generate display data for the registration information. The display data generation unit 38 generates, for example, display data for the business negotiation setting screen in which the registration information has been set.

[0336] S215: The transmission unit 39 of the server device 30 transmits the display destination (terminal device 10) and the display data to the terminal device 10.

[0337] S216: The receiving unit 14 of the terminal device 10 receives the display data and passes it to the display control unit 15 based on the display destination (terminal device 10). The display control unit 15 displays the registered information on the suggested display screen, etc., based on the display data.

[0338] <Screen example> Figure 44 shows the information search screen 330 displayed by the terminal device 10 in step S201. The information search screen 330 in Figure 44 has a customer field 331, a person in charge field 332, a date field 333, etc. The sales representative presses the search button 334 to specify one or more of the customer, person in charge, and date fields for the business deal for which they want to display the registered information.

[0339] Figure 45 shows the registration information list screen 340 displayed by the terminal device 10 in step S209. The registration information list screen 340 has the customer name, sales representative name, and date (when the business negotiation took place) associated with the identification information of the registration information. In this case, since the sales representative was "Mr. Suzuki," a list of registration information associated with Mr. Suzuki is displayed. The sales representative selects the registration information they want to display using the select button 341 and presses the OK button 342. This allows the terminal device 10 to display the selected registration information.

[0340] The registered information includes temporary storage information, daily report information, negotiation history, text data, and audio data, and will be displayed in a format suitable for the information, such as in Figure 20.

[0341] <Main effects> According to this embodiment, information generated during business negotiations can be registered in and downloaded from the external service system 308. This eliminates the need to manually register meeting minutes and other documents in the external service system 308, thereby reducing the burden on the user.

[0342] [Fourth Embodiment] In this embodiment, a terminal device 10 that displays optimal suggestions and predictive suggestions on multiple displays 60 when multiple displays 60 exist in space 360 ​​will be described.

[0343] Figure 46 shows displays 60 arranged in space 360. Figure 46 displays three displays 60: 60A, 60B, and 60C. Any of these displays 60A, 60B, and 60C is referred to as "display 60". The number of displays 60 in space 360 ​​is just an example. Also, displays 60A, 60B, and 60C do not have to be the same size.

[0344] In Figure 46, as an example, the welcome message screen 200 is displayed on display 60A. Areas 291 and 294 of Figure 28 are displayed on display 60B. Areas 292, 293, 295, and 296 of Figure 28 are displayed on display 60C. In this way, multiple displays 60 can divide and display optimal suggestions, predicted suggestions, decisions, action lists, etc., by area. Therefore, each area can be displayed at a large size, or if not displayed at a large size, many optimal suggestions, predicted suggestions, decisions, action lists, etc. can be displayed at once, improving the overview of information.

[0345] <About the features> Figure 47 shows an example of the functional configuration of the information processing system 1 in this embodiment. In the description of Figure 47, the same reference numerals are used for the same functional parts as in Figure 3, and their descriptions are omitted. According to Figure 47, multiple displays 60A to 60C are connected to the information display unit 16. Various algorithms may be applied to how the information display unit 16 displays optimal suggestions and predictive suggestions on the displays 60A to 60C. Furthermore, this can be pre-configured by the sales representative.

[0346] Figure 48 shows the settings screen 350, where a sales representative configures the information to be displayed on displays 60A to 60C. Figure 48 uses a multi-display configuration as an example, where multiple displays 60 are connected to the terminal device 10. Figure 48 explains the case where there are six displays 60. For example, a sales representative assigns a priority to cell 351 on the settings screen 350. The information display unit 16 displays optimal proposals, predicted proposals, decisions, action lists, etc., according to this priority. For example, a sales representative can set which cell 351 on the settings screen displays the welcome message screen 200, or which cells display optimal proposals, predictive proposals, decisions, action lists, etc.

[0347] With these settings, the information display unit 16 can perform the following display control. (1) Display 60A will always display the welcome message screen 200. Alternatively, after displaying the welcome message screen 200 for a certain period of time, it will be used to display optimal suggestions, predicted suggestions, decisions, action lists, etc. (2) Display 60B displays a certain amount of data at a time, and when it is full, it then displays data on display 60C. When display 60C is full, it then overwrites the oldest data on display 60B. (3) In addition, the direction of the customer's gaze may be detected, and after all displays 60B and 60C are filled, the areas of interest may be retained while areas of low interest are replaced. Areas containing information highly relevant to the text data included in the conversation between the customer and the salesperson may also be retained. Furthermore, information regarding the product that the customer has decided to purchase may always be displayed. Areas of low interest or areas containing information less relevant to the text data may be reduced in size and displayed on a predetermined display 60. In this case, each area is reduced in size as the area displayed on the predetermined display 60 increases.

[0348] The display processing flow may be the same as the sequence diagram in Figure 19. In step S63, the information display unit 16 of the terminal device 10 determines which of the displays 60A to 60C to display the optimal suggestion, predicted suggestion, decision, or action list, etc., based on the settings in Figure 48 and the rules in (1) to (3). The information display unit 16 also maintains the size (resolution) of the displays 60A to 60C and determines the number of areas that can be displayed on each display 60B based on the size of the area and the size of the displays 60A to 60C.

[0349] Furthermore, sales representatives may use voice commands to specify the information to be displayed on displays 60A to 60C. For example, a sales representative can instruct the server device 30 to "display the best proposal on display 60A." The sales representative can also specify which area to display on which display 60, for example, "move this area (or information name) from display 60B to display 60C."

[0350] Furthermore, instead of placing one or more areas such as optimal suggestions, predictive suggestions, decisions, or action lists within a single display 60, one large area may be displayed across multiple displays 60. Such a display can be achieved by setting the screen in Figure 48 to treat multiple cells as a single display 60.

[0351] <Main effects> According to this embodiment, each area displaying optimal suggestions, predicted suggestions, decisions, action lists, etc., can be displayed in a larger size, and if not displayed in a larger size, the ability to see the information at a glance is improved.

[0352] [Fifth Embodiment] In this embodiment, a terminal device 10 having an AI system within a web browser is described. Web browsers are becoming more sophisticated and AI-driven, making it possible for a web browser to have all or some of the functions of an AI system. In this case, the terminal device 10 can minimize communication with the server device 30 and display optimal suggestions or predictive suggestions with a fast response.

[0353] <About the features> Figure 49 is a diagram showing an example of the functional configuration of the information processing system 1 in this embodiment. The terminal device 10 has the same functions as the server device 30 shown in Figure 4, which was described in the first embodiment. That is, the server device 30 has a receiving unit 31, a system API 41, and a transmitting unit 39, but the other functional units are in the form of the terminal device 10. The receiving unit 31 receives a request for a web application from the terminal device 10, and the transmitting unit 39 sends the web application to the terminal device 10. From there, the terminal device 10 executes the web application and, in cooperation with the AI ​​system 50, generates and displays optimal suggestions or predictive suggestions, etc. Therefore, the functions of each functional unit of the terminal device 10 may be the same as those of the server device 30.

[0354] In Figure 49, the terminal device 10 has an AI system 50, while the server device 30 does not. However, the server device 30 may also have an AI system 50. In this case, the terminal device 10 can perform simple AI processing such as chat, while the server device 30 performs more demanding AI processing such as data analysis and image generation. The terminal device 10 can then use its own AI system 50 and the server device 30's AI system 50 depending on the processing content. One way to use them is for the sales representative to explicitly request processing by the server device 30, or the AI ​​system 50 of the terminal device 10 can decide whether to process the data using the terminal device 10 or the server device 30 based on instructions from the control unit 32.

[0355] <About operation> Figure 50 is a flowchart illustrating the process by which the terminal device 10 displays a proposal based on audio data acquired from the space and spatial information 1-3. Note that the processing of the terminal device 10 in Figure 50 is the same as the processing of the server device 30 in Figure 19, so please refer to Figure 19 for details of each step.

[0356] In step S301, the customer and the sales representative begin a conversation. Microphone 301 acquires audio data spoken by the sales representative and the customer. Microphone 301 transmits the audio data to the terminal device 10 in real time or at regular intervals. The audio data acquisition unit 11 of the terminal device 10 acquires the audio data.

[0357] In step S302, the language processing unit 33 performs speaker dialization to identify the speaker at an appropriate level of granularity, such as for each audio data, each sentence, or each word.

[0358] In step S303, the speech recognition unit 34 converts the speech data into text data.

[0359] In step S304, the control unit 32 requests the storage unit 36 ​​to store text data, and the storage unit 36 ​​stores the text data in the text data storage unit 49.

[0360] In step S305, the AI ​​control unit 37 generates prompts to instruct the AI ​​system 50 to generate optimal suggestions, predictive suggestions, decisions, and action lists.

[0361] In step S306, the AI ​​system 50 generates a response message that includes optimal suggestions, predictive suggestions, decisions, and an action list in response to prompt input. The response acquisition unit 43 acquires the response message from the AI ​​system 50 and passes it to the control unit 32.

[0362] In step S307, if the response message includes an optimal suggestion, a predictive suggestion, a decision, and an action list, the control unit 32 registers these in the temporary information storage unit 48.

[0363] In step S308, the control unit 32 designates the display 60 as the display destination and requests the display data generation unit 38 to generate display data including the optimal suggestion and the predictive suggestion. The display data generation unit 38 generates display data for a suggestion display screen that displays the optimal suggestion and the predictive suggestion on the display 60. The display data generation unit 38 also generates display data for a suggestion display screen that displays the optimal suggestion and the predictive suggestion on the terminal device 10.

[0364] In step S309, the information display unit 16 displays a suggested display screen on the display 60 based on the display data.

[0365] In step S310, the display control unit 15 displays a suggested display screen on the display 506 based on the display data.

[0366] <Main effects> According to this embodiment, in addition to the effects of the first embodiment, the terminal device 10 can generate optimal suggestions or predictive suggestions, thus shortening the time from conversation to displaying optimal suggestions or predictive suggestions.

[0367] [Seventh Embodiment] In this embodiment, the processing performed by the web browser executed on the terminal device 10 in the first to fifth embodiments will be described in detail. The web browser may be a browser application or an application that uses a web view function. A web view is a mechanism or component for displaying a web page within a native application. A web view can display web content (HTML / CSS / JavaScript) as part of the application screen without opening a web browser.

[0368] Here, a web browser has a browser engine, a scripting engine, and a network engine. A browser engine is the function that interprets the structure (HTML) and style (CSS) of a web page and displays it on the screen. Its main functions include HTML parsing, which constructs the DOM (Document Object Model) tree; CSS parsing, which applies style rules; layout calculation, which determines the position and size of elements; painting, which performs pixel-level rendering (reflecting it in the UI); and recalculation and redrawing, which re-renders the page in response to changes in the DOM / CSS. The network engine is the function that retrieves and sends web page resources (HTML, CSS, scripts, images, etc.) using communication protocols such as HTTP(S). Its main functions include generating and sending URL requests to the server (such as GET / POST), protocol management that communicates according to the protocol (HTTP / HTTPS, HTTP2, etc.), receiving responses to retrieve HTML documents, images, JSON, etc., cache control to manage the reuse of the same resource, and security measures such as TLS communication and certificate checking. The script engine is an execution engine that interprets and executes JavaScript, and is responsible for the dynamic behavior of web pages (event handling, DOM manipulation, etc.). Its main functions include parsing and syntax analysis, which converts JavaScript code into an AST (Abstract Syntax Tree); JIT compilation, which converts it into bytecode for faster execution; execution context management, which holds variables, functions, and scopes; garbage collection, which manages and automatically frees memory; and interaction with the DOM.

[0369] Figure 51 is a sequence diagram illustrating the process by which a web browser displays optimal suggestions or predictive suggestions on a suggestion display screen.

[0370] First, steps S401 to S409 describe the display flow of the screen that is initially shown on the display. This screen corresponds to, for example, the proposed display screen (part 1) 250 in Figure 20. S401: The user (person in charge) inputs an operation to connect to the server device 30 to the terminal device 10. The operation reception unit 13 of the terminal device 10 receives the operation.

[0371] S402: First, this operation causes the web browser's browser engine to notify the network engine of the URL of the server device 30.

[0372] S403: The network engine generates and sends an HTTP request destined for the URL. An HTTP request using the so-called GET method signifies a request to refresh the entire screen.

[0373] S404: The receiving unit 31 of the server device 30 receives an HTTP request, and as described in the above embodiment, the control unit 32 generates a response message. The control unit 32 determines that it is a normal HTTP request and therefore returns information for the entire screen. The display data generation unit 38 generates an HTTP response. The content of the HTTP response is display data that displays the screen including the response message. The HTTP response includes HTML, CSS, scripts, etc.

[0374] S405: The transmitting unit 39 of the server device 30 sends an HTTP response to the terminal device 10.

[0375] S406: The network engine receives the HTTP response and notifies the browser engine.

[0376] S407: The browser engine passes the written script to the script engine.

[0377] S408: The script engine loads the script into memory.

[0378] S409: The script engine generates content by executing scripts and passes it to the browser engine. The browser engine parses the HTML and CSS to generate a screen, places the content generated by the script engine, and displays the screen. Because the HTML contains ~, the browser engine discards the current DOM, parses the new HTML, and redraws.

[0379] Thus, when a web browser displays a screen for the first time, it uses HTML, CSS, and scripts to display the screen (see, for example, Figure 20).

[0380] Next, steps S410 to S421 describe the process of updating and displaying messages and images on the overview screen while a screen is already displayed on the display. These screens correspond to, for example, the proposed display screen (part 2) 270 in Figure 23, the proposed display screen (part 3) 290 in Figure 28, and the proposed display screen (part 4) 300 in Figure 29. S410: Next, the user inputs voice data into the terminal device 10 for business negotiations. The voice data acquisition unit 11 acquires the voice data.

[0381] S411: The browser engine notifies the script engine of the audio data.

[0382] S412: The script engine executes the processing described in the script for when audio data is input. When the script engine receives audio data, it includes / api / in the destination URL of the HTTP request. The inclusion of / api / in the URL is a convention on web servers to indicate a data-only request. The path containing / api / is a route that returns program data, not HTML for screen updates. Therefore, the server device 30 returns only data in a format such as JSON or XML. The script engine decides to include the audio data after / api / and send it.

[0383] S413: The scripting engine notifies the network engine of the voice data.

[0384] S414: The network engine sends the server device 30 as the destination, including the voice data in the destination URL of the HTTP request.

[0385] S415: The receiving unit 31 of the server device 30 receives an HTTP request and generates a response message by sending a prompt to the AI ​​system 50 as in the embodiment described above.

[0386] S416: The display data generation unit 38 of the server device 30 receives audio data along with / api / , and includes only the differences from the current screen in the JSON. In this embodiment, the JSON includes the response message.

[0387] S417: The display data generation unit 38 of the server device 30 generates an HTTP response containing JSON. The transmission unit 39 of the server device 30 sends the HTTP response to the terminal device 10.

[0388] S418: The network engine receives the HTTP response and notifies the script engine.

[0389] S419: The script engine executes the processing described in the script when display data is received. For example, the script engine parses "document.getElementById("chat-area").innerHTML = renderMessages(data)" and decides to display an additional response message (such as an optimal suggestion or a predictive suggestion).

[0390] S420: The script engine sends JSON to the browser engine to request a screen update.

[0391] S421: The browser engine updates the page by displaying the response message contained in the JSON on the suggested display screen. This allows the terminal device 10 to display only the updated portion of the web page instead of redisplaying the entire web page. Furthermore, single-page applications load faster.

[0392] <Main effects> The server device 30 of this embodiment can reduce the processing load for displaying optimal suggestions or predictive suggestions on the suggestion display screen, etc., and shorten the time until display.

[0393] <Other application examples> Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited in any way to these examples, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0394] For example, the information processing system may have the configuration shown in Figure 52. Figure 52 is a diagram showing a modified example of the information processing system 1 arranged in space. A microphone 301 and a display 60 are connected to the server device 30. In this configuration, voice data from customers and sales representatives is sent to the server device 30, and the server device 30 sends proposals to the display 60. The server device 30 also sends display data such as the negotiation space setting screen and the daily report screen to the terminal device 10. Therefore, the processing of the server device 30 is the same as in Figure 19, and the server device 30 only needs to switch the destination of the display data transmission.

[0395] Furthermore, as shown in Figure 53, the terminal device 10 does not necessarily have to be in space 360. In this configuration, the terminal device 10 does not display the negotiation space setting screen, so it is possible to identify the customer from voice data, or the server device 30 may display the negotiation space setting screen on the display 60. Also, instructions from the sales representative can be given via voice data, or the display 60 can accept the operation. In addition, screens that the terminal device 10 displays, such as the daily report screen, are displayed on the PC used by the sales representative in the office or at home. The display 60 can send voice messages to the earphone 305, or the server device 30 can send them directly to the earphone 305.

[0396] Figure 54 is a perspective view of an example of a wearable microphone 309. The wearable microphone 309 is worn, for example, by wrapping it around the wrist or hanging it around the neck like a necklace. The wearable microphone 309 acquires all sounds it hears. The user can also manually mute it. The wearable microphone 309 may also be a smartphone. This wearable microphone 309 may also be an emotion sensor 307 for emotion estimation or a sensor for acquiring vital data. Furthermore, the configuration examples shown in Figure 3 and other figures are divided according to their main functions to facilitate understanding of the processing performed by the server device 30 and the terminal device 10. The present invention is not limited by the way the processing units are divided or the names of those units. The processing of the server device 30 and the terminal device 10 can be further divided into many more processing units depending on the processing content. Alternatively, each processing unit can be divided to include even more processing.

[0397] Furthermore, the apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, the server apparatus 30 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including a network or shared memory, and perform the processing disclosed herein. Furthermore, as a variation of the display example in Figure 20, the content of speech spoken in space, acquired by a microphone (either built-in or wearable), may also be displayed on the display 60. Specifically, the content of speech spoken in space is converted into text by the terminal device 10, the server device 30, or the AI ​​system 50 and displayed on the screen of the display 60. In this case, the speech content may be summarized. The displayed speech content may be divided into units of time, meaning, or speech. The divided speech content may be enclosed in a rectangle. It may be displayed in correspondence with time. Alternatively, the speech content may be written on sticky notes that can be moved on the screen according to user operation, and displayed for each sticky note. The user can freely move and display the speech in divided units. In the example of Figure 20, the image in area 251 can be moved in units of sticky notes, or the date, time, and location in area 252 can be moved in units of sticky notes. Movement may be possible across areas 251 to 256. Furthermore, the workflow defining the processing of the server device 30 and the AI ​​system 50 described above may be created using a workflow creation tool. This allows the workflow creation tool to create and edit a series of processes (workflows), including the type and content of prompts to be sent to the AI ​​system 50, the processing before sending the prompts, and the processing of the server device 30 after sending the prompts. Specifically, the workflow creation tool can be used to create and edit the series of processes shown in Figure 19. This improves the degree of freedom in the spatial experience. Note that a workflow defines a series of work procedures and flows for a task, and is a definition for determining the order of processing based on rules for which processes to execute and for automatically executing those processes.

[0398] Furthermore, the server device 30 can be configured to share the disclosed processing steps, such as those shown in Figures 13, 16, and 19, in various combinations. For example, a process executed by a predetermined unit can be executed by multiple information processing devices within the server device 30. Also, the server device 30 may be a single server device or it may be divided into multiple devices.

[0399] Each of the embodiments described above can be implemented by one or more processing circuits. Here, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules designed to execute each of the functions described above.

[0400] <Mode> [Aspect 1] An information processing system including an information processing device, a terminal device, and a microphone and display included in space, The aforementioned information processing device is A first receiving unit receives audio data from the terminal device, which includes speech data input to the microphone, including speech spoken by a person in the space. A prompt for transmitting spatial information, which is information relating to the space, and the content of the audio data to the AI, comprising a request transmission unit that transmits a prompt to the AI ​​requesting a proposal based on the spatial information and the content of the audio data, A response acquisition unit that acquires the response transmitted from the AI, A display data generation unit generates screen display data including the proposal based on the above response, It has a first transmission unit that transmits the display data to the terminal device, The aforementioned terminal device is A second transmission unit that transmits the aforementioned audio data to the information processing device, An information display unit that displays the screen on the display based on the display data, The display installed in the aforementioned space is An information processing system characterized by having a display control unit that displays the screen including a suggestion to the person based on the display data. [Aspect 2] The request transmission unit further includes information about the person in the prompt, The aforementioned prompt is a prompt that instructs the person to make a suggestion. The information processing system described in Embodiment 1. [Aspect 3] The information relating to the person includes at least one of the following: the person's age, gender, or purchase history. The aforementioned prompt is a prompt that instructs the person to propose a product or service. The information processing system described in Embodiment 2. [Aspect 4] The prompt is a prompt that indicates a first suggestion of a product or service suitable for the request contained in the voice data. The information processing system according to embodiment 2 or 3. [Aspect 5] The prompt is a prompt that, based on the content of the voice data, instructs a second suggestion of a product or service that is presumed to be requested based on the voice data. The information processing system described in Embodiment 4. [Aspect 6] The display causes the first proposal and the second proposal to be displayed on a single screen. The information processing system described in aspect 5. [Aspect 7] The display displays the suggestions for each topic of the utterances extracted from the audio data. An information processing system as described in any one of the descriptions in 1 to 6. [Aspect 8] The aforementioned terminal device is The display control unit causes the terminal device to display a screen for setting the spatial information. It has an operation reception unit that receives the spatial information entered by the user on the screen, The second transmission unit transmits the spatial information to the information processing device. An information processing system as described in any one of the descriptions 1 to 7. [Aspect 9] The spatial information includes at least one of the following: information about an event taking place in the space, information about a person who spoke in the space, or information that describes the characteristics of the space. An information processing system as described in any one of the eight descriptions (1 to 8). [Aspect 10] The aforementioned display data generation unit generates display data to display multiple proposals on a single screen. The first transmission unit transmits display data to the space that displays the multiple proposals on a single screen. The display installed in the aforementioned space displays the multiple proposals on a single screen based on the display data. An information processing system as described in any one of the descriptions 1 to 9. [Aspect 11] The request transmission unit, Along with the content of the aforementioned audio data and the aforementioned spatial information, a prompt requesting the generation of a daily report is sent to the AI. The response acquisition unit acquires the daily report transmitted from the AI, The aforementioned display data generation unit generates display data to display the acquired daily report, The first transmitting unit transmits the display data to the space in order to display the display data on the display, The display installed in the aforementioned space displays the daily report based on the display data. An information processing system as described in any one of the descriptions 1 to 10. [Aspect 12] The request transmission unit, Along with the content of the audio data and the spatial information, a prompt is sent to the AI ​​requesting the generation of a video related to the proposal that relates to the content of the audio data. The response acquisition unit acquires the video or the URL of the video transmitted from the AI, The aforementioned display data generation unit generates display data to display the acquired video, The first transmitting unit transmits the display data to the space in order to display the display data on the display, The display installed in the aforementioned space plays the video based on the display data. An information processing system as described in any one of the descriptions 1 to 11. [Aspect 13] The display data generation unit generates second display data which is different from the first display data to be displayed on the display and includes information that is not to be displayed on the display. The first transmitting unit transmits the first display data to the space for display on the display, and transmits the second display data to the terminal device for display on the terminal device. The display installed in the space displays the screen including the suggestion to the person based on the first display data, The terminal device displays suggestions for the person and information that is not to be displayed on the display, based on the second display data. The information processing system described in Embodiment 8. [Aspect 14] The display data generation unit generates display data to be displayed on the terminal device, The first transmission unit transmits the display data to be displayed on the terminal device to the terminal device. When the first receiving unit receives from the terminal device user operation information to display the suggestions included in the display data on the display and information specifying the suggestions, The aforementioned display data generation unit generates display data for the specified proposal, The first transmitting unit transmits the specified proposed display data to the space for display on the display, The display installed in the aforementioned space displays the proposal specified based on the display data. The information processing system described in Embodiment 8. [Aspect 15] In the aforementioned space, a sales representative and a customer are conversing, and the sales representative is wearing earphones. The request transmission unit, Along with the content of the voice data and the spatial information, a prompt requesting the generation of instructions for the sales representative, which is a proposal related to the content of the voice data, is sent to the AI. The response acquisition unit acquires the instruction transmitted from the AI, The display data generation unit generates an audio message to output the acquired instruction. The first transmitting unit transmits the voice message to the space in order to output the voice message from the earphone. The earphone outputs the instruction based on the voice message. An information processing system as described in any one of the descriptions in 1 to 12. [Claim 16] It has an emotion sensor or estimation unit that estimates the emotional information of the person, The display data generation unit generates display data to display the estimated emotion information, The first transmitting unit transmits the display data to the space in order to display the display data on the terminal device. The terminal device displays the emotion information based on the display data. The information processing system according to claim 1. [Claim 17] The aforementioned information processing system can communicate with an external service system via a system API. The system API converts the proposal to the person into the data format of the external service system and registers it with the external service system. The information processing system according to claim 1. [Claim 18] The aforementioned information processing system can communicate with an external service system via a system API. The system API transmits data summarizing the content of the voice data, including the utterance, to the external service system in order to register it with the external service system. The information processing system according to claim 1. [Claim 19] The system API receives data from the external service system that summarizes the contents of past voice data, which is registered in the external service system. The information processing system according to claim 18. [Explanation of Symbols]

[0401] 1. Information Processing System 10 Terminal devices 30 Server devices 60 displays [Prior art documents] [Patent Documents]

[0402] [Patent Document 1] Japanese Patent Publication No. 2020-184216

Claims

1. An information processing system including an information processing device, a terminal device, and a microphone and display included in space, The aforementioned information processing device is A first receiving unit receives audio data from the terminal device, which includes speech data input to the microphone, including speech spoken by a person in the space. A prompt that transmits spatial information, which is information relating to the space, and the content of the audio data to the AI, the prompt that transmits a prompt to the AI ​​requesting a proposal based on the spatial information and the content of the audio data, A response acquisition unit that acquires the response transmitted from the AI, A display data generation unit generates screen display data including the proposal based on the above response, It has a first transmission unit that transmits the display data to the terminal device, The aforementioned terminal device is A second transmission unit that transmits the aforementioned audio data to the information processing device, An information display unit that displays the screen on the display based on the display data, The display installed in the aforementioned space is A display control unit that displays the screen including a suggestion to the person based on the display data, An information processing system characterized by having the following features.

2. The request transmission unit further includes information about the person in the prompt, The aforementioned prompt is a prompt that instructs the person to make a suggestion. The information processing system according to claim 1.

3. The information relating to the person includes at least one of the following: the person's age, gender, or purchase history. The aforementioned prompt is a prompt that instructs the person to propose a product or service. The information processing system according to claim 2.

4. The prompt is a prompt that indicates a first suggestion of a product or service suitable for the request contained in the voice data. The information processing system according to claim 2 or 3.

5. The prompt is a prompt that, based on the content of the voice data, instructs a second suggestion of a product or service that is presumed to be requested based on the voice data. The information processing system according to claim 4.

6. The display displays the first proposal and the second proposal on a single screen. The information processing system according to claim 5.

7. The display displays the suggestions for each topic of the utterances extracted from the audio data. The information processing system according to claim 1.

8. The aforementioned terminal device is The display control unit causes the terminal device to display a screen for setting the spatial information. It has an operation reception unit that receives the spatial information entered by the user on the screen, The second transmission unit transmits the spatial information to the information processing device. The information processing system according to claim 1.

9. The spatial information includes at least one of the following: information about an event taking place in the space, information about a person who spoke in the space, or information that describes the characteristics of the space. The information processing system according to claim 1.

10. The aforementioned display data generation unit generates display data to display multiple proposals on a single screen. The first transmission unit transmits display data to the space that displays the plurality of proposals on a single screen. The display installed in the aforementioned space displays the multiple proposals on a single screen based on the display data. The information processing system according to claim 1.

11. The request transmission unit, Along with the content of the aforementioned audio data and the aforementioned spatial information, a prompt requesting the generation of a daily report is sent to the AI. The response acquisition unit acquires the daily report transmitted from the AI, The aforementioned display data generation unit generates display data to display the acquired daily report, The first transmission unit transmits the display data to the space in order to display the display data on the display, The display installed in the aforementioned space displays the daily report based on the display data. The information processing system according to claim 1.

12. The request transmission unit, Along with the content of the audio data and the spatial information, a prompt is sent to the AI ​​requesting the generation of a video related to a proposal that relates to the content of the audio data. The response acquisition unit acquires the video or the URL of the video transmitted from the AI. The aforementioned display data generation unit generates display data to display the acquired video, The first transmission unit transmits the display data to the space in order to display the display data on the display, The display installed in the aforementioned space plays the video based on the display data. The information processing system according to claim 1.

13. The display data generation unit generates second display data which is different from the first display data to be displayed on the display and includes information that is not to be displayed on the display. The first transmitting unit transmits the first display data to the space for display on the display, and transmits the second display data to the terminal device for display on the terminal device. The display installed in the space displays the screen including the suggestion to the person based on the first display data, The terminal device displays suggestions for the person and information that is not to be displayed on the display, based on the second display data. The information processing system according to claim 1.

14. The display data generation unit generates display data to be displayed on the terminal device, The first transmission unit transmits the display data to be displayed on the terminal device to the terminal device. When the first receiving unit receives from the terminal device user operation information to display the suggestions included in the display data on the display and information specifying the suggestions, The aforementioned display data generation unit generates display data for the specified proposal, The first transmitting unit transmits the specified proposed display data to the space for display on the display. The display installed in the aforementioned space displays the proposal specified based on the display data. The information processing system according to claim 1.

15. In the aforementioned space, a sales representative and a customer are conversing, and the sales representative is wearing earphones. The request transmission unit, Along with the content of the voice data and the spatial information, a prompt requesting the generation of a proposal related to the content of the voice data and instructions for the sales representative is sent to the AI. The response acquisition unit acquires the instruction transmitted from the AI, The display data generation unit generates an audio message to output the acquired instruction. The first transmitting unit transmits the voice message to the space in order to output the voice message from the earphone. The earphone outputs the instruction based on the voice message. The information processing system according to claim 1.

16. It has an emotion sensor or estimation unit that estimates the emotional information of the person, The display data generation unit generates display data to display the estimated emotion information, The first transmission unit transmits the display data to the space in order to display the display data on the terminal device. The terminal device displays the emotion information based on the display data. The information processing system according to claim 1.

17. The aforementioned information processing system can communicate with an external service system via a system API. The system API converts the proposal to the person into the data format of the external service system and registers it in the external service system. The information processing system according to claim 1.

18. The aforementioned information processing system can communicate with an external service system via a system API. The system API transmits data summarizing the content of the voice data, including the utterance, to the external service system in order to register it with the external service system. The information processing system according to claim 1.

19. The system API receives data from the external service system that summarizes the contents of past voice data, which is registered in the external service system. The information processing system according to claim 18.

20. A microphone and display included in a space, a terminal device, and an information processing device that can communicate via a network, A first receiving unit receives audio data from the terminal device, which includes speech data input to the microphone, including speech spoken by a person in the space. A prompt that transmits spatial information, which is information relating to the space, and the content of the audio data to the AI, the prompt that transmits a prompt to the AI ​​requesting a proposal based on the spatial information and the content of the audio data, A response acquisition unit that acquires the response transmitted from the AI, A display data generation unit generates screen display data including the proposal based on the above response, A first transmission unit that transmits the display data to the terminal device, An information processing device characterized by having the following features.

21. A microphone and display, terminal device, and information processing device that can communicate via a network are included in the space. A first receiving unit receives audio data from the terminal device, which includes speech data input to the microphone, including speech spoken by a person in the space. A prompt that transmits spatial information, which is information relating to the space, and the content of the audio data to the AI, the prompt that transmits a prompt to the AI ​​requesting a proposal based on the spatial information and the content of the audio data, A response acquisition unit that acquires the response transmitted from the AI, A display data generation unit generates screen display data including the proposal based on the above response, A first transmission unit that transmits the display data to the terminal device, A program designed to function as such.

22. An information processing method performed by an information processing system including an information processing device, a terminal device, and a microphone and display contained in a space, The aforementioned information processing device is The process of receiving audio data, including the speech of a person who spoke in the space, input to the microphone, from the terminal device, A prompt that transmits spatial information, which is information relating to the space, and the content of the audio data to the AI, the process of transmitting a prompt to the AI ​​requesting a proposal based on the spatial information and the content of the audio data, The process of obtaining the response transmitted from the AI, Based on the above response, a process is performed to generate screen display data including the above proposal, The process involves transmitting the display data to the terminal device, The aforementioned terminal device is A process of transmitting the aforementioned audio data to the information processing device, The process of displaying the screen on the display based on the display data is performed, The display installed in the aforementioned space displays the screen, including a suggestion to the person based on the display data. An information processing method characterized by the following:

23. An information processing system including an information processing device, a microphone and a display located in space, The aforementioned information processing device is A first receiving unit that receives audio data including speech from a person speaking in the space input to the microphone, A prompt that transmits spatial information, which is information relating to the space, and the content of the audio data to the AI, the prompt that transmits a prompt to the AI ​​requesting a proposal based on the spatial information and the content of the audio data, A response acquisition unit that acquires the response transmitted from the AI, A display data generation unit generates screen display data including the proposal based on the above response, It has a first transmission unit that transmits the display data to the display, The display installed in the aforementioned space is A display unit that displays the screen including a suggestion to the person based on the display data, An information processing system characterized by having the following features.

24. An information processing device, and a terminal device capable of communicating with a microphone and a display contained within a space, A voice data acquisition unit that acquires voice data including speech of a person speaking in the space input to the microphone, A prompt that transmits spatial information, which is information relating to the space, and the content of the audio data to the AI, the prompt that transmits a prompt to the AI ​​requesting a proposal based on the spatial information and the content of the audio data, A response acquisition unit that acquires the response transmitted from the AI, A display data generation unit generates screen display data including the proposal based on the above response, An information display unit that displays the screen on the display based on the display data, A display control unit that displays the screen including a suggestion to the person based on the display data, A terminal device characterized by having the following features.

Images