system

Abstract

Provide a system. 【Solution means】 Means for analyzing customer information, Means for summarizing and presenting the analysis results of the customer information, Means for generating proposal contents related to business negotiations, Means for real-time voice recognition and conversion of the conversation content during business negotiations into text, Means for analyzing the conversation content and summarizing relevant information, Means for generating and presenting a meeting record after the business negotiation ends, Means for generating the next proposal schedule and extracting and presenting information on similar business negotiations, A system including the above.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance that responds to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional sales system, since there were not sufficient means for efficiently collecting and analyzing customer information before business negotiations, users had to spend a great deal of time on information collection. Also, it was difficult to analyze the content of a customer's speech in real time during business negotiations and make appropriate proposals based on it. Furthermore, follow-up work after business negotiations was often done manually, and for this reason, users were required to spend a lot of time and effort. The present invention is provided for the purpose of solving these problems and enabling users to conduct sales activities more efficiently.

Means for Solving the Problems

[0005] The present invention is a sales support system that includes the following means: means for analyzing customer information, and means for summarizing and presenting the results of the customer information analysis, thereby enabling efficient collection and analysis of customer information before a business negotiation. Furthermore, by including means for generating proposal content related to the business negotiation, the user can create an optimal proposal in a short amount of time.

[0006] Furthermore, by including means for real-time speech recognition and text conversion of conversation content during business negotiations, and means for analyzing the conversation content and summarizing relevant information, real-time analysis and immediate proposals during business negotiations become possible.

[0007] Furthermore, by including means for generating and presenting meeting minutes after the negotiation, generating a schedule for the next proposal, and extracting and presenting information on similar negotiations, follow-up work after negotiations can also be carried out efficiently. This allows users to concentrate on communication in their sales activities.

[0008] "Customer information" refers to data such as past sales negotiation history, news articles, company website information, and internal records related to the customer being considered for business.

[0009] "Analysis" refers to the process of classifying and organizing collected customer information, and then extracting and analyzing important information.

[0010] "Summarization" refers to concisely compiling information extracted through analysis and presenting it to the user in an easy-to-understand manner.

[0011] "Proposal content" refers to the specific proposals presented during business negotiations based on the results of customer information analysis.

[0012] "Speech recognition" is a technology that converts the content of a business conversation into text data in real time.

[0013] "Text conversion" is a method of representing audio data acquired through speech recognition as text data.

[0014] "Meeting minutes" are documents that record the content of conversations during and after business negotiations, summarizing the important points and leaving them as written records.

[0015] A "proposal schedule" outlines the plan and timeline for what proposals will be made in the next business meeting.

[0016] A "similar deal" refers to a past deal in the history of negotiations that is similar to the current deal.

[0017] "Extraction" refers to the process of extracting information that meets specific criteria from a large amount of data.

[0018] "Generation" refers to the manual process of creating new documents such as proposals and meeting minutes based on the necessary information. [Brief explanation of the drawing]

[0019] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0020] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described according to the accompanying drawings.

[0021] First, the language used in the following description will be explained.

[0022] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0023] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0024] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0025] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0027] [First Embodiment]

[0028] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0029] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0032] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0035] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0039] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0040] This invention relates to a sales support system for efficiently collecting and analyzing customer information, processing sales negotiations in real time, and following up after sales negotiations. Detailed embodiments of this system are described below.

[0041] Customer information collection and analysis

[0042] User actions:

[0043] The user logs into the system and sends a request to retrieve information about a specific customer. The user's device sends this request to the server.

[0044] Server processing:

[0045] After receiving a customer information retrieval request, the server accesses the customer database to retrieve past sales negotiation history. Next, it gathers the latest information from external sources (news API, company website, etc.). Furthermore, it accesses the internal information database to retrieve relevant internal memos and past project information. The collected information is summarized by a natural language processing engine, and the server sends the summarized information to the user's terminal.

[0046] Specific example:

[0047] If a user has an upcoming business meeting with "Company A," the server collects and summarizes Company A's past business meeting history, latest news, and relevant internal records, and provides this information to the user's terminal. The user then inputs their proposal based on this information, and the server generates a proposal document based on that input.

[0048] Information processing during business negotiations

[0049] User actions:

[0050] The user initiates a business meeting and activates the recording function on their device. The audio data of the meeting is transmitted to the server in real time.

[0051] Server processing:

[0052] The server converts received audio data into text using a real-time speech recognition engine. It extracts important keywords and phrases from the converted text and embeds them into a meeting minutes template. The server further analyzes the sales meeting content and suggests relevant products to the user in real time.

[0053] Specific example:

[0054] When a user conducts an online business meeting with a representative from "Company A," the terminal sends the audio of the meeting to the server. The server converts the audio to text and detects statements such as "I am interested in cost reduction." In this case, the server immediately proposes products related to cost reduction to the user.

[0055] Follow-up after business negotiations

[0056] User actions:

[0057] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[0058] Server processing:

[0059] The server finalizes the meeting minutes at the end of the meeting and sends them to the user's terminal. Next, it re-analyzes the meeting minutes and sales negotiation details to generate the next proposed content and schedule. It also extracts relevant sales opportunities from a database of similar past sales opportunities, summarizes that information, and sends it to the user's terminal.

[0060] Specific example:

[0061] After the user concludes a business negotiation with "Company A," the server reviews the meeting minutes and presents the content and timeline for the next proposal. Furthermore, if there have been similar negotiations with "Company B" in the past, the server summarizes that information and provides it to the user. When the user enters proposal details again, the server generates a new proposal document.

[0062] In this way, this system enables users to efficiently utilize information at each stage—before, during, and after a business negotiation—and to conduct sales activities smoothly.

[0063] The following describes the processing flow.

[0064] Customer information collection and analysis

[0065] Step 1:

[0066] A user logs into the system and sends a request from their terminal to the server to retrieve information about a specific customer.

[0067] Step 2:

[0068] The server receives a request to retrieve customer information.

[0069] Step 3:

[0070] The server accesses the customer database and retrieves the customer's past sales history.

[0071] Step 4:

[0072] The server collects the latest information from external sources (news APIs, company websites, etc.).

[0073] Step 5:

[0074] The server accesses the company's internal information database to retrieve relevant internal memos and past project information.

[0075] Step 6:

[0076] The server collects information and then uses a natural language processing engine to summarize it.

[0077] Step 7:

[0078] The server sends the summarized information to the user's terminal.

[0079] Step 8:

[0080] The user reviews the summary information and enters the suggested content into their device.

[0081] Step 9:

[0082] The terminal sends the user's inputted suggestions to the server.

[0083] Step 10:

[0084] The server analyzes the proposal and generates a proposal document based on that information.

[0085] Step 11:

[0086] The server sends the generated proposal to the user's terminal.

[0087] Information processing during business negotiations

[0088] Step 1:

[0089] The user initiates a business meeting and activates the recording function on their device.

[0090] Step 2:

[0091] The terminal transmits the audio data of the business negotiation to the server in real time.

[0092] Step 3:

[0093] The server converts the received audio data into text using a real-time speech recognition engine.

[0094] Step 4:

[0095] The server extracts important keywords and phrases from the converted text.

[0096] Step 5:

[0097] The server extracts the information and embeds it into the meeting minutes template.

[0098] Step 6:

[0099] The server analyzes the details of the business negotiation and suggests relevant products to the user in real time.

[0100] Step 7:

[0101] Users can view real-time suggestions from the server and use them immediately.

[0102] Follow-up after business negotiations

[0103] Step 1:

[0104] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[0105] Step 2:

[0106] The server finalizes the meeting minutes at the time of shutdown and sends them to the user's terminal.

[0107] Step 3:

[0108] The server re-analyzes the meeting minutes and negotiation details and generates a schedule of what should be proposed next.

[0109] Step 4:

[0110] The server extracts relevant sales case examples from a database of similar past sales opportunities.

[0111] Step 5:

[0112] The server extracts similar business opportunity information, summarizes it, and sends it to the user's terminal.

[0113] Step 6:

[0114] The user enters a new proposal into the terminal for the next business meeting.

[0115] Step 7:

[0116] The terminal sends the new suggestions entered by the user to the server.

[0117] Step 8:

[0118] The server analyzes the new proposal and generates a new proposal document based on that information.

[0119] Step 9:

[0120] The server sends the new proposal to the user's terminal.

[0121] (Example 1)

[0122] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0123] In conventional sales support systems, customer information collection and analysis, real-time processing during negotiations, and post-negotiation follow-up were all handled independently, making it difficult to utilize information consistently and conduct efficient sales activities. Furthermore, there was a need for improved accuracy in real-time proposals during negotiations and in the content of follow-up proposals after negotiations concluded.

[0124] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0125] In this invention, the server includes means for analyzing customer information, means for summarizing and presenting the results of the customer information analysis, means for receiving information collection requests and collecting information from a customer database and external information sources, means for summarizing the collected information using a natural language processing engine, means for generating proposals related to business negotiations, means for real-time speech recognition and text conversion of conversation content during business negotiations, means for analyzing conversation content, extracting important keywords, and proposing related products, means for generating and presenting meeting minutes after the conclusion of business negotiations, means for generating a schedule for the next proposal, and means for extracting and presenting information on similar business negotiations. This makes it possible to efficiently utilize information at each stage before, during, and after business negotiations and to smoothly carry out sales activities.

[0126] "Customer data analysis" is the process of collecting data about customers and analyzing patterns and trends.

[0127] A "summary of customer information analysis results" is a concise compilation of the most important points extracted from the analyzed information.

[0128] An "information gathering request" is the process of requesting detailed information or up-to-date data about a specific customer.

[0129] A "customer database" is a data storage system that aggregates and stores detailed information about customers.

[0130] "External information sources" refer to information providers other than internal databases, such as company websites and news APIs.

[0131] A "natural language processing engine" is software that analyzes text data and executes algorithms to understand and generate human language.

[0132] "Generating proposals related to business negotiations" is the process of creating the optimal proposal based on the objectives of the negotiation and the customer's needs.

[0133] "Real-time speech recognition" is a technology that instantly converts audio generated during business negotiations into text data.

[0134] "Keyword extraction" is the process of selecting particularly important words and phrases from conversations or texts.

[0135] "Proposing products or services" refers to suggesting suitable products or services based on the needs identified during a business negotiation.

[0136] "Generating meeting minutes after a business negotiation" is the process of organizing the content and conclusions of a business negotiation and documenting them so that they can be referenced later.

[0137] "Generating the next proposal schedule" is the process of planning what should be proposed next and when.

[0138] "Extracting information on similar business opportunities" means detecting cases similar to the current business opportunity from past business opportunity history and reusing that information.

[0139] The present invention relates to a sales support system for efficiently collecting and analyzing customer information, processing sales negotiations in real time, and following up after sales negotiations. This system includes means for analyzing customer information, means for summarizing and presenting the analysis results, means for generating proposals related to sales negotiations, means for real-time speech recognition and text conversion of conversations during sales negotiations, means for analyzing conversations and summarizing related information, means for generating and presenting meeting minutes after the sales negotiation is completed, means for generating a schedule for the next proposal, and means for extracting and presenting information on similar sales negotiations.

[0140] Customer information collection and analysis

[0141] User actions:

[0142] The user logs into the sales support system and sends a request to retrieve information about a specific customer. The user's device sends this request to the server.

[0143] Server processing:

[0144] After receiving a customer information retrieval request, the server accesses the customer database to retrieve past sales history. Next, it gathers the latest information from external sources such as news APIs and company websites. Furthermore, it accesses internal information databases to retrieve relevant internal memos and past project information. The collected information is summarized by a natural language processing engine (e.g., Google® Cloud Natural Language API), and the server sends the summarized information to the user's terminal.

[0145] Specific example:

[0146] For example, if a user has a business meeting scheduled with "Company A," the server collects past business meeting history, latest news, and relevant internal records of "Company A," and provides summarized information to the user's terminal. The user then inputs their proposal based on this information, and the server generates a proposal document based on that input.

[0147] Examples of prompts for a generative AI model:

[0148] "Please provide a summary of past business negotiations with a certain company, the latest news, and relevant internal memos."

[0149] Information processing during business negotiations

[0150] User actions:

[0151] The user initiates the business negotiation and activates the recording function on their device.

[0152] Server processing:

[0153] The terminal transmits audio data of the business negotiation to the server in real time. The server passes the received audio data to a real-time speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the audio to text. It extracts important keywords and phrases from the converted text and suggests relevant products to the user in real time based on that.

[0154] Specific example:

[0155] When a user conducts an online business meeting with a representative from "a certain company," the terminal transmits the audio of the meeting to the server. The server converts the audio to text and detects statements such as "I am interested in cost reduction." In this case, the server immediately proposes products related to cost reduction to the user.

[0156] Examples of prompts for a generative AI model:

[0157] "Extract key keywords from the following text and propose corresponding products / services: 'A representative from a certain company has stated they are interested in cost reduction.'"

[0158] Follow-up after business negotiations

[0159] User actions:

[0160] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[0161] Server processing:

[0162] The server finalizes the meeting minutes at the end of the meeting and sends them to the user's terminal. Next, it re-analyzes the minutes and negotiation details to generate the next proposal and schedule. It also extracts relevant deals from a database of similar past deals, summarizes that information, and sends it to the user's terminal. The user can then create a proposal based on the next proposal.

[0163] Specific example:

[0164] After the user concludes a business negotiation with "a certain company," the server reviews the meeting minutes and presents the content and timeline for the next proposal. Furthermore, if there have been similar negotiations with "other companies" in the past, the server summarizes that information and provides it to the user. Based on this, the user creates a new proposal.

[0165] Examples of prompts for a generative AI model:

[0166] "Please generate the next proposal and timeline based on the following meeting minutes: 'We need a proposal regarding cost reduction in negotiations with a certain company. Please provide a schedule for creating the next proposal.'"

[0167] In this way, this system enables users to efficiently utilize information at each stage—before, during, and after a business negotiation—and to conduct sales activities smoothly.

[0168] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0169] Step 1: User Login

[0170] The user accesses the login screen of the sales support system and enters their authentication information (user ID and password). The terminal sends this information to the server, which verifies it against the database to perform authentication. If authentication is successful, the user's session begins.

[0171] Input: User ID, Password

[0172] Output: Authentication result (success or failure)

[0173] Specific operation: When a user enters their user ID and password and presses the login button, the terminal sends the authentication information to the server, which then authenticates the user by comparing it with the information stored in the database.

[0174] Step 2: Customer Information Request

[0175] The user enters the customer's name and customer ID to retrieve information about a specific customer. The terminal sends this request to the server, and the server receives the request.

[0176] Input: Customer name, Customer ID

[0177] Output: Confirmation of receipt of customer information retrieval request

[0178] Specific operation: When the user enters the customer name and customer ID and presses the information retrieval button, the terminal sends that information to the server. The server confirms the request and proceeds to the next processing step.

[0179] Step 3: Information Gathering

[0180] Based on the received request, the server first accesses the company's internal customer database to retrieve the customer's past sales history. Next, it sends queries to external sources such as news APIs and company websites to gather the latest information. It also accesses the internal information database to retrieve internal memos and past deal information related to that customer.

[0181] Input: Customer name, Customer ID

[0182] Output: Collected customer information data

[0183] Specific operation: The server generates a database search query based on the customer name and customer ID, and searches the customer database. Furthermore, it accesses external APIs to collect the latest information.

[0184] Step 4: Information Summary

[0185] The server passes the collected information to a natural language processing engine (e.g., Google Cloud Natural Language API) to generate a summary. This summary includes the latest customer trends and key points from past deals.

[0186] Input: Collected customer information data

[0187] Output: Summarized customer information

[0188] Specific operation: The server inputs text data collected from the database into a natural language processing engine and generates summarized data.

[0189] Step 5: Providing information to users

[0190] The server sends summarized information to the user's terminal. The user can then use the provided information to prepare for the business negotiation.

[0191] Input: Summarized customer information

[0192] Output: Providing information to users

[0193] Specific operation: The server sends summary data to the user's terminal, and the user receives and displays it.

[0194] Step 6: Start the negotiation

[0195] The user starts the business negotiation and enables the recording function on their device.

[0196] Input: Instruction to start the business negotiation

[0197] Output: Enable recording function

[0198] Specific operation: When the user presses the record start button, the device starts recording and prepares to send the audio data to the server.

[0199] Step 7: Real-time transmission of audio data

[0200] The terminal transmits audio data of the business negotiation to the server in real time.

[0201] Input: Sales negotiation audio data

[0202] Output: Transmission of real-time audio data

[0203] Specific operation: The terminal divides the recorded audio and sends it to the server sequentially.

[0204] Step 8: Speech to Text

[0205] The server passes the received audio data to a real-time speech recognition engine (for example, Google Cloud Speech-to-Text) to convert the audio into text.

[0206] Input: Sales negotiation audio data

[0207] Output: Text-converted deal details

[0208] Specific operation: The server passes the audio data to the speech recognition engine and receives the output as text data.

[0209] Step 9: Keyword Extraction

[0210] The server extracts important keywords and phrases from the converted text.

[0211] Input: Text-converted sales opportunity details

[0212] Output: Extracted keywords

[0213] Specific operation: The server uses an extraction algorithm to identify and extract specific keywords or phrases.

[0214] Step 10: Real-time product proposal

[0215] The server suggests relevant products to the user in real time based on the extracted keywords. The suggestions are displayed as pop-up notifications on the device.

[0216] Input: Extracted keywords

[0217] Output: Product proposal notification

[0218] Specific operation: The server searches the product database corresponding to the extracted keywords and notifies the user's terminal of the relevant products.

[0219] Step 11: Ending the Deal

[0220] The user ends the deal and clicks the "End Deal" button.

[0221] Input: Instruction to end the business negotiation

[0222] Output: Start of deal closing process

[0223] Specific operation: When the user presses the "End Deal" button, the device stops recording and sends a notification to the server that the deal has ended.

[0224] Step 12: Send a follow-up request

[0225] The device sends a follow-up request to the server.

[0226] Input: Follow-up request

[0227] Output: Confirmation of the start of follow-up processing.

[0228] Specific operation: When the user presses the follow-up request button, the device sends a request to the server.

[0229] Step 13: Finalize the meeting minutes

[0230] The server finalizes the meeting minutes at the end of the meeting and sends them to the user's terminal. The minutes include key points and agreements from the business negotiation.

[0231] Input: Completed meeting minutes data

[0232] Output: Providing meeting minutes to users

[0233] Specific operation: The server generates meeting minutes from the text data of the business negotiation and sends them to the user's terminal.

[0234] Step 14: Re-analyzing the details of the business negotiation

[0235] The server re-analyzes the meeting minutes and sales negotiation details to generate the next proposals and schedules. It also extracts relevant information from a database of similar past sales negotiations and sends that information to the user's terminal.

[0236] Input: Meeting minutes data, sales negotiation text data

[0237] Output: Next proposal and schedule

[0238] Specific operation: The server uses meeting minutes and sales negotiation text data to generate the content and schedule for the next proposal, and collects and summarizes information on similar sales negotiations.

[0239] Step 15: Generating the next proposal and schedule

[0240] The server summarizes relevant information from a database of similar past business opportunities and generates the content and schedule for the next proposal. The user can then create a proposal based on this information.

[0241] Input: Past similar deal data

[0242] Output: Next proposal content, proposal schedule

[0243] Specific operation: The server searches the database for information on similar business opportunities and provides the user with the next proposal and its schedule.

[0244] In this way, this system enables users to efficiently utilize information at each stage—before, during, and after a business negotiation—through a series of processing steps, thereby facilitating smooth sales activities.

[0245] (Application Example 1)

[0246] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0247] During maintenance and troubleshooting on production lines within factories, there are problems with efficiently gathering and analyzing necessary information. Furthermore, the lack of means to provide useful information in real time during maintenance work can potentially reduce the accuracy and speed of the work.

[0248] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0249] In this invention, the server includes means for analyzing customer information, means for summarizing and presenting the results of the analysis of the customer information, means for generating proposal content related to the business negotiation, means for real-time speech recognition of the conversation content during the business negotiation and converting it into text, means for analyzing the conversation content and summarizing related information, means for generating and presenting meeting minutes after the business negotiation is completed, means for generating the next proposal schedule and means for extracting and presenting information on similar business negotiations, means for acquiring maintenance information based on identification information of the work target, and means for analyzing voice commands and presenting related maintenance information. This makes it possible to perform maintenance and troubleshooting work in the factory efficiently and accurately.

[0250] "Customer information" is a general term for data such as basic customer information, past transaction history, and inquiry history.

[0251] "Analysis means" refers to software or hardware configurations used to analyze collected information and generate useful insights.

[0252] A "summarization tool" is a function for concisely summarizing and presenting the analysis results.

[0253] "Proposal content generation means" refers to a system function that automatically creates proposal items related to business negotiations.

[0254] "Speech recognition means" refers to technology that converts speech data into text data.

[0255] "Text conversion means" refers to a function that converts speech into text information.

[0256] An "information summarization tool" is a function that displays collected or analyzed data in a shortened form.

[0257] A "meeting minutes generation system" is a system for recording and documenting the content of business negotiations and meetings.

[0258] The "proposal schedule generation method" is a function that plans the date for the next proposal or meeting.

[0259] The "similar business opportunity information extraction method" is a function that searches for and extracts information on similar business opportunities from a database of past business opportunities.

[0260] "Identification information" refers to information used to uniquely identify a specific work object or piece of equipment.

[0261] A "maintenance information acquisition method" is a function that acquires maintenance history and procedures related to equipment and systems based on identification information.

[0262] A "voice command analysis means" is a function that analyzes voice input and presents specific instructions or information based on its content.

[0263] This invention provides a system to support maintenance and troubleshooting within a factory. Specific embodiments are described below.

[0264] This system includes means for analyzing customer information, means for summarizing and presenting the results of the analysis of the customer information, means for generating proposals related to business negotiations, means for real-time speech recognition of conversations during business negotiations and converting them into text, means for analyzing the conversations and summarizing related information, means for generating and presenting meeting minutes after the business negotiation is completed, means for generating a schedule for the next proposal and means for extracting and presenting information on similar business negotiations, means for acquiring maintenance information based on identification information of the work target, and means for analyzing voice commands and presenting related maintenance information.

[0265] Hardware and software configuration

[0266] The hardware configuration of this system includes a microphone mounted on the robot body used in the factory, a server, and an internet connection. The software configuration includes a Python program, a speech recognition library (speech_recognition), and an HTTP request library (requests).

[0267] Process Overview

[0268] 1. Voice input recognition

[0269] The user inputs voice commands to the robot to obtain maintenance information for the object being worked on. This voice is collected via a microphone.

[0270] 2. Text conversion of audio data

[0271] The server converts the collected audio data into text data using a speech recognition library. At this stage, the sensitivity is adjusted to account for ambient noise.

[0272] 3. Information acquisition based on identification information

[0273] The server retrieves relevant maintenance information based on identification information (e.g., equipment ID) in the text data. The retrieved information is parsed in JSON format, and the necessary information is extracted.

[0274] 4. Display of maintenance information

[0275] The server provides users with acquired maintenance information in real time. This information is displayed on the robot's display and on a separate terminal, making it immediately available to the worker.

[0276] Specific example

[0277] For example, when a sudden trouble occurs in "Device 123" on a production line in a certain factory, the operator inputs a voice command "Tell me the maintenance information of Device 123" to the robot. The robot recognizes this command, obtains the latest maintenance information and logs related to "Device 123" from the server through the Internet, and provides them to the operator.

[0278] Example of prompt sentence

[0279] "Please enter questions for providing appropriate maintenance and troubleshooting information for the problems of the devices that occurred on the factory production line. For example, 'Tell me the latest maintenance information of Device 123' 'Tell me the past trouble history of Device 456' and so on."

[0280] The flow of the specific process in Application Example 1 will be described with reference to FIG. 12.

[0281] Step 1:

[0282] The user inputs a voice command to the robot to obtain the maintenance information of the work target. The input voice is collected through the microphone of the robot body.

[0283] Step 2:

[0284] The terminal (robot) transmits the collected voice data to the server. The server converts this voice data into text data using a speech recognition library (speech_recognition). The input is voice data, and the output is the corresponding text data.

[0285] Step 3:

[0286] The server analyzes the converted text data and extracts identification information (e.g., device ID). This identification information is identified as a specific keyword from the input voice command, and the output is the identification information.

[0287] Step 4:

[0288] The server sends a request to the maintenance information acquisition API based on the identification information. The input is the identification information, and the output is the response data of the maintenance information. This response data is returned in JSON format.

[0289] Step 5:

[0290] The server parses the acquired maintenance information, extracts and summarizes the relevant information. The input is the maintenance information in JSON format, and the output is the summarized maintenance information.

[0291] Step 6:

[0292] The terminal (robot) displays the summarized maintenance information sent from the server to the user. The input is the summarized maintenance information, and the output is the information in text format presented to the user's vision. This information is displayed on the robot's display and is immediately available to the operator.

[0293] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform specific processing using the user's emotion.

[0294] The present invention relates to a sales support system that combines the collection and analysis of customer information, real-time processing during negotiations, follow-up after negotiations, and an emotion engine that recognizes the user's emotion and optimizes the proposed content. The detailed embodiments of this system will be described below.

[0295] Collection and Analysis of Customer Information

[0296] User Operations:

[0297] The user logs into the system and sends a request to retrieve information about a specific customer. The user's device sends this request to the server.

[0298] Server processing:

[0299] After receiving a customer information retrieval request, the server accesses the customer database to retrieve past sales negotiation history. Next, it gathers the latest information from external sources (news API, company website, etc.). Furthermore, it accesses the internal information database to retrieve relevant internal memos and past project information. The collected information is summarized by a natural language processing engine, and the server sends the summarized information to the user's terminal.

[0300] Specific example:

[0301] If a user has a business meeting scheduled with "Company A," the server collects Company A's past business meeting history, latest news, and relevant internal records, and provides this summarized information to the user's terminal. The user then inputs their proposal based on this information, and the server generates a proposal document based on that input.

[0302] Information processing and emotional engine during business negotiations

[0303] User actions:

[0304] The user initiates a business negotiation and activates the recording and emotion recognition functions on their device. The audio data of the negotiation and emotion data such as the user's facial expressions are transmitted to the server in real time.

[0305] Server processing:

[0306] The server converts the received voice data into text using a real-time voice recognition engine. It extracts important keywords and phrases from the converted text and embeds them into the minutes template. It also has a function to analyze the user's emotional state by an emotion engine and adjust the proposed content in real time based on it. The server further analyzes the negotiation content and proposes relevant commercial materials to the user in real time.

[0307] Specific example:

[0308] When the user conducts an online negotiation with the person in charge of "Company A", the terminal sends the negotiation voice and the user's facial expression data to the server. The server converts the voice into text and detects a statement such as "interested in cost reduction". In this case, if the emotion engine detects that the user's emotional state is "attractive", the server immediately presents commercial materials related to cost reduction to the user.

[0309] Follow-up after negotiation and utilization of emotion data

[0310] User's operation:

[0311] The user ends the negotiation and sends a follow-up request from the terminal to the server.

[0312] Server's processing:

[0313] The server finalizes the minutes at the end point and sends them to the user's terminal. Next, it re-analyzes the minutes and the negotiation content to generate the content and schedule to be proposed next. It also extracts relevant negotiations from the past similar negotiation database, summarizes the information, and sends it to the user's terminal. Furthermore, it optimizes the proposed content based on the emotion data obtained by the emotion engine.

[0314] Specific example:

[0315] After the user concludes a business meeting with "Company A," the server reviews the meeting minutes and presents a proposal and timeline for the next meeting. Furthermore, if there have been similar business meetings with "Company B" in the past, the server summarizes that information and provides it to the user. If the emotion engine predicts that the representative from "Company A" will have a positive sentiment towards the next meeting, the server adjusts the proposal based on that prediction. Once the user enters the proposal again, the server generates a new proposal.

[0316] In this way, this system enables users to efficiently utilize information at each stage—before, during, and after a business negotiation—and also takes into account the user's emotional state, allowing sales activities to proceed smoothly.

[0317] The following describes the processing flow.

[0318] Customer information collection and analysis

[0319] Step 1:

[0320] A user logs into the system and sends a request from their terminal to the server to retrieve information about a specific customer.

[0321] Step 2:

[0322] The server receives a request to retrieve customer information.

[0323] Step 3:

[0324] The server accesses the customer database and retrieves the customer's past sales history.

[0325] Step 4:

[0326] The server collects the latest information from external sources (news APIs, company websites, etc.).

[0327] Step 5:

[0328] The server accesses the company's internal information database to retrieve relevant internal memos and past project information.

[0329] Step 6:

[0330] The server collects information and then uses a natural language processing engine to summarize it.

[0331] Step 7:

[0332] The server sends the summarized information to the user's terminal.

[0333] Step 8:

[0334] The user reviews the summary information and enters the suggested content into their device.

[0335] Step 9:

[0336] The terminal sends the user's inputted suggestions to the server.

[0337] Step 10:

[0338] The server analyzes the proposal and generates a proposal document based on that information.

[0339] Step 11:

[0340] The server sends the generated proposal to the user's terminal.

[0341] Information processing and emotional engine during business negotiations

[0342] Step 1:

[0343] The user initiates a business negotiation and activates the recording and emotion recognition functions on their device.

[0344] Step 2:

[0345] The terminal transmits audio data of the business negotiation and emotional data such as the user's facial expressions to the server in real time.

[0346] Step 3:

[0347] The server converts the received audio data into text using a real-time speech recognition engine.

[0348] Step 4:

[0349] The server extracts important keywords and phrases from the converted text.

[0350] Step 5:

[0351] The server extracts the information and embeds it into the meeting minutes template.

[0352] Step 6:

[0353] The emotion engine analyzes the user's emotional state and adjusts the suggested content in real time based on that analysis.

[0354] Step 7:

[0355] The server analyzes the details of the business negotiation and suggests relevant products to the user in real time.

[0356] Step 8:

[0357] Users can view real-time suggestions from the server and use them immediately.

[0358] Post-sales follow-up and utilization of emotional data

[0359] Step 1:

[0360] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[0361] Step 2:

[0362] The server finalizes the meeting minutes at the time of shutdown and sends them to the user's terminal.

[0363] Step 3:

[0364] The server re-analyzes the meeting minutes and negotiation details and generates a schedule of what should be proposed next.

[0365] Step 4:

[0366] The server extracts relevant sales case examples from a database of similar past sales opportunities.

[0367] Step 5:

[0368] The server extracts similar business opportunity information, summarizes it, and sends it to the user's terminal.

[0369] Step 6:

[0370] The suggested content is optimized based on emotional data acquired by the emotion engine.

[0371] Step 7:

[0372] Based on the information from Step 6, the server generates a new proposal for the next business meeting.

[0373] Step 8:

[0374] The user enters a new proposal into the terminal for the next business meeting.

[0375] Step 9:

[0376] The terminal sends the new suggestions entered by the user to the server.

[0377] Step 10:

[0378] The server analyzes the new proposal and generates a new proposal document based on that information.

[0379] Step 11:

[0380] The server sends the new proposal to the user's terminal.

[0381] (Example 2)

[0382] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0383] Traditional sales support systems lacked the ability to integrate customer information collection, real-time information processing during negotiations, emotional data analysis, and post-sales follow-up. As a result, sales representatives had to manage each process individually, leading to decreased efficiency. Furthermore, there was no effective way to utilize emotional data during negotiations, making it difficult to respond quickly to changes in customer emotions. This resulted in insufficient optimization of proposals and a lower success rate for sales negotiations.

[0384] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0385] In this invention, the server includes means for analyzing customer information, means for summarizing and presenting the results of the analysis of the customer information, means for real-time speech recognition of conversation content during negotiations and conversion to text, and means for analyzing emotional data acquired during negotiations and adjusting the content of proposals. This makes it possible to effectively collect and analyze customer information at each stage before, during, and after negotiations, and to further optimize the content of proposals in real time by utilizing emotional data.

[0386] "Customer information" refers to data, history, and related notes about customers.

[0387] "Analysis" refers to the process of processing data and extracting meaningful information or patterns.

[0388] "Summary" refers to providing a concise overview of detailed information.

[0389] "Presentation" refers to displaying information in a way that is easy for users to understand.

[0390] "Business negotiation" refers to discussions or meetings regarding sales or purchases.

[0391] "Proposed content" refers to the products, services, or their terms and conditions presented during business negotiations.

[0392] "Speech recognition" refers to the technology that converts speech into text.

[0393] "Converting to text" refers to replacing audio data with text data.

[0394] "Conversation content" refers to the words and statements exchanged during a business negotiation.

[0395] "Emotional data" refers to information about a user's emotions obtained during a business negotiation, such as their facial expressions and voice.

[0396] "Meeting minutes" refers to a document that records and summarizes the key points of a business negotiation.

[0397] "Next proposal schedule" refers to the planned visits and proposals for the next business meeting.

[0398] "Information on similar deals" refers to data and history related to similar deals that have taken place in the past.

[0399] "Real-time" refers to processing being performed at the very moment an event occurs.

[0400] "Adjustment" refers to changing the content or method depending on the situation.

[0401] This invention relates to a sales support system that combines customer information collection and analysis, real-time processing during sales negotiations, post-sales follow-up, and an emotion engine that recognizes user emotions and optimizes proposals. Specific embodiments for carrying out the invention are described below.

[0402] Customer information collection and analysis

[0403] User actions:

[0404] The user logs into the system and sends a request to retrieve information about a specific customer. The user's device sends this request to the server.

[0405] Server processing:

[0406] After receiving a customer information retrieval request, the server accesses the customer database to retrieve past sales history. Next, it gathers the latest information from external sources (e.g., news APIs or company websites). Furthermore, it accesses the internal information database to retrieve relevant internal memos and past project information. This information is summarized by a natural language processing engine (NLP engine), and the server sends the summarized information to the user's terminal.

[0407] Specific example:

[0408] If a user has an upcoming business meeting with "Company A," they log into the system and request customer information. The server retrieves "Company A's" past business meeting history, latest news, and related internal records, and summarizes the information using an NLP engine. The summarized information is then provided to the user's device. For example, the Google News API could be used as the news API, and a CRM system could be used for internal information management.

[0409] Information processing and emotional engine during business negotiations

[0410] User actions:

[0411] The user initiates a business negotiation and activates the recording and emotion recognition functions on their device. The audio data of the negotiation and the user's emotion data (facial expression data, etc.) are transmitted to the server in real time.

[0412] Server processing:

[0413] The server converts received audio data into text using a real-time speech recognition engine. It extracts important keywords and phrases from the converted text and embeds them into a meeting minutes template. The server also has the capability to analyze the user's emotional state using an emotion recognition engine and adjust the proposed content in real time. Furthermore, it analyzes the content of business negotiations and proposes relevant products and services to the user in real time.

[0414] Specific example:

[0415] When a user conducts an online business meeting with a representative from "Company A," the user enables recording and emotion recognition features in advance. The device sends the audio of the meeting and the user's facial expression data to the server. The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the audio to text and extract keywords. For example, if the representative says, "I'm interested in cost reduction," the server uses an emotion recognition algorithm to determine the emotional state is "attractive" and quickly presents the user with product information related to cost reduction.

[0416] Post-sales follow-up and utilization of emotional data

[0417] User actions:

[0418] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[0419] Server processing:

[0420] The server finalizes the meeting minutes at the end of the business negotiation and sends them to the user's terminal. Next, it re-analyzes the minutes and negotiation details to generate the next proposed content and schedule. Furthermore, it extracts relevant information from a database of similar past negotiations and provides a summary to the user's terminal. Finally, it optimizes the proposed content based on sentiment data obtained by the sentiment recognition engine.

[0421] Specific example:

[0422] After the user concludes a business meeting with "Company A," the server finalizes the meeting minutes and sends them to the user. The server searches for similar past business meeting data and provides a summary of successful cases, such as those with "Company B." It also generates and proposes a schedule for the next visit and proposals to the user. The emotion engine determines that the emotional state of the "Company A" representative is positive and adjusts the next proposal based on that prediction.

[0423] In this way, this system enables users to efficiently utilize information at each stage—before, during, and after a business negotiation—and also takes into account the user's emotional state, allowing sales activities to proceed smoothly.

[0424] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0425] Customer information collection and analysis

[0426] Step 1:

[0427] The user logs into the system and sends a request to retrieve information about a specific customer. Specifically, the user clicks the "Retrieve Customer Information" button on the terminal's operation screen and enters information such as the company name and customer name. The terminal sends this request data to the server. The input is "Company Name: Company A" and the output is "Customer Information Retrieval Request".

[0428] Step 2:

[0429] The server receives a customer information retrieval request. The server first accesses the customer database to retrieve past sales negotiation history for "Company A". At this stage, the input is the "customer information retrieval request" and the output is the "past sales negotiation history".

[0430] Step 3:

[0431] The server accesses external sources (such as news APIs and company websites) to collect the latest information related to "Company A". At this stage, the input is "Company Name: Company A" and the output is "Latest News". Specifically, the server calls APIs to collect data and stores it internally.

[0432] Step 4:

[0433] The server accesses the company's internal information database to retrieve internal memos and past project information related to "Company A". At this stage, the input is "Company Name: Company A", and the output is "Internal Memos and Project Information". Specifically, it executes database queries to extract the necessary information.

[0434] Step 5:

[0435] The collected information is summarized by a natural language processing engine (NLP engine). The server uses past sales history, latest news, and internal memos as input data to perform summarization and generate the results. At this stage, the input is "all collected information," and the output is "summarized customer information." Specifically, the NLP engine's algorithm is executed to generate the summary.

[0436] Step 6:

[0437] The server sends summarized information to the user's terminal. At this stage, the input is "summarized customer information," and the output is "customer information displayed on the user's terminal." Specifically, the server sends an HTTP response, and the terminal displays that information.

[0438] ---

[0439] Information processing and emotional engine during business negotiations

[0440] Step 1:

[0441] The user initiates a business meeting and activates the recording and emotion recognition functions on their device. The user clicks the "Start Recording" and "Start Emotion Recognition" buttons on the device. The input is "Notification of Business Meeting Start," and the output is "Start of Recording and Emotion Recognition."

[0442] Step 2:

[0443] The terminal transmits audio and facial expression data from the business negotiation to the server in real time. At this stage, the input is "audio and facial expression data," and the output is "data sent to the server." Specifically, the terminal transmits the data in real time using streaming technology.

[0444] Step 3:

[0445] The server converts the received audio data into text using a real-time speech recognition engine. At this stage, the input is "audio data" and the output is "converted text". Specifically, it calls the Google Cloud Speech-to-Text API to convert audio to text.

[0446] Step 4:

[0447] The server extracts important keywords and phrases from the text and embeds them into a meeting minutes template. At this stage, the input is the "converted text," and the output is the "keywords embedded in the meeting minutes." Specifically, a text analysis algorithm is executed to extract keywords.

[0448] Step 5:

[0449] The server uses an emotion recognition engine to analyze the user's emotional state and adjusts the suggestions in real time. At this stage, the input is "facial expression data and converted text," and the output is "adjusted suggestions." Specifically, it uses an emotion recognition algorithm to analyze the emotional state and adjusts the suggestions based on that data.

[0450] Step 6:

[0451] The server analyzes the details of the business negotiation and proposes relevant products and services to the user's terminal in real time. The input at this stage is "text data of the adjusted proposal and business negotiation," and the output is "product information presented to the user." Specifically, it queries a related product database to generate appropriate proposals.

[0452] ---

[0453] Post-sales follow-up and utilization of emotional data

[0454] Step 1:

[0455] The user concludes the business negotiation and sends a follow-up request from their device to the server. The input is "Follow-up notification," and the output is "Follow-up request sent."

[0456] Step 2:

[0457] The server finalizes the meeting minutes and sends them to the user's terminal. At this stage, the input is "meeting minutes created in real time," and the output is "sending the finalized meeting minutes to the user." Specifically, it completes the meeting minutes template and provides it to the user.

[0458] Step 3:

[0459] The server re-analyzes the meeting minutes and negotiation details to generate the next proposal and schedule. At this stage, the input is the "finalized meeting minutes and negotiation details," and the output is the "next proposal and schedule." Specifically, it runs a data analysis algorithm to plan the next proposal.

[0460] Step 4:

[0461] The server checks a database of similar past deals, extracts relevant information, and provides it to the user. At this stage, the input is "finalized meeting minutes and deal details," and the output is "provision of information on similar deals." Specifically, it performs a database search to identify similar past cases, summarizes that information, and provides it.

[0462] Step 5:

[0463] The server optimizes the next proposal based on sentiment data. At this stage, the input is "sentiment data and finalized meeting minutes," and the output is "optimized next proposal." Specifically, the proposal is readjusted based on sentiment analysis data.

[0464] Step 6:

[0465] The server sends the optimized suggestions to the user's terminal. At this stage, the input is the "optimized next suggestion," and the output is the "optimized suggestion presented to the user." Specifically, the server sends an HTTP response, and the terminal displays that information.

[0466] As described above, by clearly defining the specific operation and input / output of the system at each step, and by clearly indicating the roles of the user, terminal, and server, the present invention can be effectively implemented.

[0467] (Application Example 2)

[0468] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0469] In modern brick-and-mortar stores, customer service often relies heavily on the experience and skills of store staff, leading to inconsistencies in customer satisfaction. Furthermore, accurately understanding customer needs and emotions in real time and providing optimal suggestions based on that understanding is challenging. Traditional systems fail to adequately optimize suggestions based on customer emotions, making efficient sales support difficult. Moreover, real-time information processing during sales negotiations and effective follow-up after negotiations are often lacking.

[0470] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0471] In this invention, the server includes means for analyzing customer information, means for summarizing and presenting the results of the customer information analysis, means for generating proposal content related to the business negotiation, means for real-time speech recognition and text conversion of conversation content during the business negotiation, means for analyzing the conversation content and summarizing related information, means for analyzing video data during the business negotiation to recognize the user's emotions, means for optimizing the proposal content based on the user's emotion data, means for generating and presenting meeting minutes after the business negotiation is completed, means for generating a schedule for the next proposal, and means for extracting and presenting information on similar business negotiations. This makes it possible to grasp customer needs and emotions in real time at physical stores and make optimal proposals. Furthermore, it is expected that follow-up during and after business negotiations can be efficiently carried out, leading to improved customer satisfaction and sales efficiency.

[0472] "Customer information" refers to data and historical information about customers, including, for example, past purchase history, inquiry history, and personal profile information.

[0473] "Means of analysis" refers to methods and techniques for analyzing collected data and extracting useful information from it. Specifically, this includes machine learning algorithms and statistical analysis techniques.

[0474] "Speech recognition" is a technology that converts speech data into text data, and includes the process of automatically transcribing the user's speech into text.

[0475] "Means of converting to text" refers to technologies and systems for converting collected audio data into text, such as speech recognition engines.

[0476] "Methods of summarization" refer to methods and techniques for organizing analyzed information into a concise and easily understandable form. Natural language processing and summarization algorithms are used in this process.

[0477] "Means of recognizing emotions" refers to technologies that determine a user's emotional state at a given time from their video and audio data. This includes emotion recognition engines and image processing technologies.

[0478] "Methods for optimizing suggestions" refer to technologies that generate the most suitable suggestions by considering user sentiment data and other factors. This includes recommendation algorithms and personalization engines.

[0479] "Means for generating and presenting meeting minutes" refers to technologies and systems that automatically create meeting minutes based on text data recorded during business negotiations and provide them to users.

[0480] "Methods for generating proposal schedules" refer to technologies that generate the content and schedule of the next proposal based on the details of the business negotiation and past history. This includes scheduling management systems and task management tools.

[0481] "Methods for extracting and presenting information on similar business opportunities" refers to technologies that extract cases similar to the current business opportunity from a database of past business opportunities and provide them as reference information. Data mining and clustering algorithms are used in this process.

[0482] In this invention, to specifically realize a sales support system for physical stores, smart glasses, a server, a speech recognition engine, a natural language processing engine, an emotion recognition engine, and a recommendation algorithm work together as a single unit. The operation of this system will be described below in order.

[0483] System configuration and operation

[0484] 1. Data Collection

[0485] The user (staff member) wears smart glasses.

[0486] When a conversation with a customer begins, the smart glasses collect audio and video data in real time and send it to the server.

[0487] 2. Data Processing

[0488] The server converts the received audio data into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[0489] The converted text data is sent to a natural language processing engine (e.g., a BERT model) where important keywords and phrases are extracted.

[0490] The video data is used to analyze the emotional state of customers using an emotion recognition engine (e.g., Microsoft® Azure® Face API).

[0491] Based on extracted keywords and sentiment data, a recommendation algorithm (e.g., collaborative filtering algorithm) is used to select the most suitable product suggestions.

[0492] 3. Data Output

[0493] The selected product information is displayed on the smart glasses' screen and presented to the user (staff).

[0494] The user suggests selected products to customers who visit the store.

[0495] Specific example

[0496] 1. Examples of customer service

[0497] When a user asks, "Hello, what can I help you with?", the customer replies, "I've come to look at winter coats."

[0498] The smart glasses send the audio data to a server, where speech recognition and natural language processing extract the keyword "winter coat."

[0499] The emotion recognition engine detects emotions such as "excitement" and "interest" from video data of customers visiting the store.

[0500] A recommendation algorithm selects multiple coats based on inventory information and popular product data, and displays them on smart glasses.

[0501] A user who has checked the display of smart glasses suggests to a customer, "These two coats are very popular."

[0502] Example of a prompt

[0503] "Enter the following user voice data into the speech recognition engine: Hello, I'm looking for a winter coat."

[0504] "The converted text data is passed to a natural language processing engine to extract keywords."

[0505] "Video data is input into an emotion recognition engine to detect the user's emotional state."

[0506] "Next, the extracted keywords and sentiment data are passed to a product recommendation algorithm to select the most suitable product."

[0507] In this way, it becomes possible to provide optimal suggestions tailored to customer needs in physical stores in real time, thereby improving customer satisfaction.

[0508] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0509] Step 1:

[0510] The user (staff member) wears smart glasses and begins a conversation with a customer. The smart glasses collect audio and video data in real time and send it to the server. The input is the audio and video data of the conversation with the customer, and the output is sending this data to the server.

[0511] Step 2:

[0512] The server converts the received audio data into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text). Audio data is the input, and text data is the output. Specifically, the audio data is analyzed, and the spoken content is extracted as a string of characters.

[0513] Step 3:

[0514] The server passes the converted text data to a natural language processing engine (e.g., a BERT model) to extract important keywords and phrases. The input is text data, and the output is important keywords and phrases. Specifically, it analyzes the context of the text and extracts information relevant to the business deal.

[0515] Step 4:

[0516] Video data is sent to a server, which uses an emotion recognition engine (e.g., Microsoft Azure Face API) to analyze the user's emotional state. The input is video data, and the output is recognized emotion data. Specifically, emotions are determined from facial expressions, and levels of excitement and interest are quantified.

[0517] Step 5:

[0518] The server uses a recommendation algorithm (e.g., collaborative filtering algorithm) to select the most suitable product based on extracted keywords and sentiment data. The input is keywords and sentiment data, and the output is a list of recommended products. Specifically, it refers to past sales history and inventory information to select the most relevant products.

[0519] Step 6:

[0520] The server displays selected product information on the smart glasses' screen and suggests it to the user (staff). The input is a list of recommended products, and the output is the product information displayed on the smart glasses' screen. Specifically, it provides the user with optimized suggestions immediately.

[0521] Step 7:

[0522] The user (staff) checks the information displayed on the smart glasses and makes suggestions to customers. The input is product information displayed on the screen, and the output is suggestions to customers. Specifically, the staff makes verbal suggestions based on the information they visually confirm.

[0523] In this way, the system of the present invention can efficiently and effectively handle customer interactions in physical stores.

[0524] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0525] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0526] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0527] [Second Embodiment]

[0528] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0529] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0530] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0531] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0532] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0533] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0534] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0535] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0536] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0537] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0538] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0539] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0540] This invention relates to a sales support system for efficiently collecting and analyzing customer information, processing sales negotiations in real time, and following up after sales negotiations. Detailed embodiments of this system are described below.

[0541] Customer information collection and analysis

[0542] User actions:

[0543] The user logs into the system and sends a request to retrieve information about a specific customer. The user's device sends this request to the server.

[0544] Server processing:

[0545] After receiving a customer information retrieval request, the server accesses the customer database to retrieve past sales negotiation history. Next, it gathers the latest information from external sources (news API, company website, etc.). Furthermore, it accesses the internal information database to retrieve relevant internal memos and past project information. The collected information is summarized by a natural language processing engine, and the server sends the summarized information to the user's terminal.

[0546] Specific example:

[0547] If a user has an upcoming business meeting with "Company A," the server collects and summarizes Company A's past business meeting history, latest news, and relevant internal records, and provides this information to the user's terminal. The user then inputs their proposal based on this information, and the server generates a proposal document based on that input.

[0548] Information processing during business negotiations

[0549] User actions:

[0550] The user initiates a business meeting and activates the recording function on their device. The audio data of the meeting is transmitted to the server in real time.

[0551] Server processing:

[0552] The server converts received audio data into text using a real-time speech recognition engine. It extracts important keywords and phrases from the converted text and embeds them into a meeting minutes template. The server further analyzes the sales meeting content and suggests relevant products to the user in real time.

[0553] Specific example:

[0554] When a user conducts an online business meeting with a representative from "Company A," the terminal sends the audio of the meeting to the server. The server converts the audio to text and detects statements such as "I am interested in cost reduction." In this case, the server immediately proposes products related to cost reduction to the user.

[0555] Follow-up after business negotiations

[0556] User actions:

[0557] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[0558] Server processing:

[0559] The server finalizes the meeting minutes at the end of the meeting and sends them to the user's terminal. Next, it re-analyzes the meeting minutes and sales negotiation details to generate the next proposed content and schedule. It also extracts relevant sales opportunities from a database of similar past sales opportunities, summarizes that information, and sends it to the user's terminal.

[0560] Specific example:

[0561] After the user concludes a business negotiation with "Company A," the server reviews the meeting minutes and presents the content and timeline for the next proposal. Furthermore, if there have been similar negotiations with "Company B" in the past, the server summarizes that information and provides it to the user. When the user enters proposal details again, the server generates a new proposal document.

[0562] In this way, this system enables users to efficiently utilize information at each stage—before, during, and after a business negotiation—and to conduct sales activities smoothly.

[0563] The following describes the processing flow.

[0564] Customer information collection and analysis

[0565] Step 1:

[0566] A user logs into the system and sends a request from their terminal to the server to retrieve information about a specific customer.

[0567] Step 2:

[0568] The server receives a request to retrieve customer information.

[0569] Step 3:

[0570] The server accesses the customer database and retrieves the customer's past sales history.

[0571] Step 4:

[0572] The server collects the latest information from external sources (news APIs, company websites, etc.).

[0573] Step 5:

[0574] The server accesses the company's internal information database to retrieve relevant internal memos and past project information.

[0575] Step 6:

[0576] The server collects information and then uses a natural language processing engine to summarize it.

[0577] Step 7:

[0578] The server sends the summarized information to the user's terminal.

[0579] Step 8:

[0580] The user reviews the summary information and enters the suggested content into their device.

[0581] Step 9:

[0582] The terminal sends the user's inputted suggestions to the server.

[0583] Step 10:

[0584] The server analyzes the proposal and generates a proposal document based on that information.

[0585] Step 11:

[0586] The server sends the generated proposal to the user's terminal.

[0587] Information processing during business negotiations

[0588] Step 1:

[0589] The user initiates a business meeting and activates the recording function on their device.

[0590] Step 2:

[0591] The terminal transmits the audio data of the business negotiation to the server in real time.

[0592] Step 3:

[0593] The server converts the received audio data into text using a real-time speech recognition engine.

[0594] Step 4:

[0595] The server extracts important keywords and phrases from the converted text.

[0596] Step 5:

[0597] The server extracts the information and embeds it into the meeting minutes template.

[0598] Step 6:

[0599] The server analyzes the details of the business negotiation and suggests relevant products to the user in real time.

[0600] Step 7:

[0601] Users can view real-time suggestions from the server and use them immediately.

[0602] Follow-up after business negotiations

[0603] Step 1:

[0604] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[0605] Step 2:

[0606] The server finalizes the meeting minutes at the time of shutdown and sends them to the user's terminal.

[0607] Step 3:

[0608] The server re-analyzes the meeting minutes and negotiation details and generates a schedule of what should be proposed next.

[0609] Step 4:

[0610] The server extracts relevant sales case examples from a database of similar past sales opportunities.

[0611] Step 5:

[0612] The server extracts similar business opportunity information, summarizes it, and sends it to the user's terminal.

[0613] Step 6:

[0614] The user enters a new proposal into the terminal for the next business meeting.

[0615] Step 7:

[0616] The terminal sends the new suggestions entered by the user to the server.

[0617] Step 8:

[0618] The server analyzes the new proposal and generates a new proposal document based on that information.

[0619] Step 9:

[0620] The server sends the new proposal to the user's terminal.

[0621] (Example 1)

[0622] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0623] In conventional sales support systems, customer information collection and analysis, real-time processing during negotiations, and post-negotiation follow-up were all handled independently, making it difficult to utilize information consistently and conduct efficient sales activities. Furthermore, there was a need for improved accuracy in real-time proposals during negotiations and in the content of follow-up proposals after negotiations concluded.

[0624] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0625] In this invention, the server includes means for analyzing customer information, means for summarizing and presenting the results of the customer information analysis, means for receiving information collection requests and collecting information from a customer database and external information sources, means for summarizing the collected information using a natural language processing engine, means for generating proposals related to business negotiations, means for real-time speech recognition and text conversion of conversation content during business negotiations, means for analyzing conversation content, extracting important keywords, and proposing related products, means for generating and presenting meeting minutes after the conclusion of business negotiations, means for generating a schedule for the next proposal, and means for extracting and presenting information on similar business negotiations. This makes it possible to efficiently utilize information at each stage before, during, and after business negotiations and to smoothly carry out sales activities.

[0626] "Customer data analysis" is the process of collecting data about customers and analyzing patterns and trends.

[0627] A "summary of customer information analysis results" is a concise compilation of the most important points extracted from the analyzed information.

[0628] An "information gathering request" is the process of requesting detailed information or up-to-date data about a specific customer.

[0629] A "customer database" is a data storage system that aggregates and stores detailed information about customers.

[0630] "External information sources" refer to information providers other than internal databases, such as company websites and news APIs.

[0631] A "natural language processing engine" is software that analyzes text data and executes algorithms to understand and generate human language.

[0632] "Generating proposals related to business negotiations" is the process of creating the optimal proposal based on the objectives of the negotiation and the customer's needs.

[0633] "Real-time speech recognition" is a technology that instantly converts audio generated during business negotiations into text data.

[0634] "Keyword extraction" is the process of selecting particularly important words and phrases from conversations or texts.

[0635] "Proposing products or services" refers to suggesting suitable products or services based on the needs identified during a business negotiation.

[0636] "Generating meeting minutes after a business negotiation" is the process of organizing the content and conclusions of a business negotiation and documenting them so that they can be referenced later.

[0637] "Generating the next proposal schedule" is the process of planning what should be proposed next and when.

[0638] "Extracting information on similar business opportunities" means detecting cases similar to the current business opportunity from past business opportunity history and reusing that information.

[0639] The present invention relates to a sales support system for efficiently collecting and analyzing customer information, processing sales negotiations in real time, and following up after sales negotiations. This system includes means for analyzing customer information, means for summarizing and presenting the analysis results, means for generating proposals related to sales negotiations, means for real-time speech recognition and text conversion of conversations during sales negotiations, means for analyzing conversations and summarizing related information, means for generating and presenting meeting minutes after the sales negotiation is completed, means for generating a schedule for the next proposal, and means for extracting and presenting information on similar sales negotiations.

[0640] Customer information collection and analysis

[0641] User actions:

[0642] The user logs into the sales support system and sends a request to retrieve information about a specific customer. The user's device sends this request to the server.

[0643] Server processing:

[0644] After receiving a customer information retrieval request, the server accesses the customer database to retrieve past sales history. Next, it gathers the latest information from external sources such as news APIs and company websites. Furthermore, it accesses internal information databases to retrieve relevant internal memos and past project information. The collected information is summarized by a natural language processing engine (e.g., Google Cloud Natural Language API), and the server sends the summarized information to the user's terminal.

[0645] Specific example:

[0646] For example, if a user has a business meeting scheduled with "Company A," the server collects past business meeting history, latest news, and relevant internal records of "Company A," and provides summarized information to the user's terminal. The user then inputs their proposal based on this information, and the server generates a proposal document based on that input.

[0647] Examples of prompts for a generative AI model:

[0648] "Please provide a summary of past business negotiations with a certain company, the latest news, and relevant internal memos."

[0649] Information processing during business negotiations

[0650] User actions:

[0651] The user initiates the business negotiation and activates the recording function on their device.

[0652] Server processing:

[0653] The terminal transmits audio data of the business negotiation to the server in real time. The server passes the received audio data to a real-time speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the audio to text. It extracts important keywords and phrases from the converted text and suggests relevant products to the user in real time based on that.

[0654] Specific example:

[0655] When a user conducts an online business meeting with a representative from "a certain company," the terminal transmits the audio of the meeting to the server. The server converts the audio to text and detects statements such as "I am interested in cost reduction." In this case, the server immediately proposes products related to cost reduction to the user.

[0656] Examples of prompts for a generative AI model:

[0657] "Extract key keywords from the following text and propose corresponding products / services: 'A representative from a certain company has stated they are interested in cost reduction.'"

[0658] Follow-up after business negotiations

[0659] User actions:

[0660] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[0661] Server processing:

[0662] The server finalizes the meeting minutes at the end of the meeting and sends them to the user's terminal. Next, it re-analyzes the minutes and negotiation details to generate the next proposal and schedule. It also extracts relevant deals from a database of similar past deals, summarizes that information, and sends it to the user's terminal. The user can then create a proposal based on the next proposal.

[0663] Specific example:

[0664] After the user concludes a business negotiation with "a certain company," the server reviews the meeting minutes and presents the content and timeline for the next proposal. Furthermore, if there have been similar negotiations with "other companies" in the past, the server summarizes that information and provides it to the user. Based on this, the user creates a new proposal.

[0665] Examples of prompts for a generative AI model:

[0666] "Please generate the next proposal and timeline based on the following meeting minutes: 'We need a proposal regarding cost reduction in negotiations with a certain company. Please provide a schedule for creating the next proposal.'"

[0667] In this way, this system enables users to efficiently utilize information at each stage—before, during, and after a business negotiation—and to conduct sales activities smoothly.

[0668] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0669] Step 1: User Login

[0670] The user accesses the login screen of the sales support system and enters their authentication information (user ID and password). The terminal sends this information to the server, which verifies it against the database to perform authentication. If authentication is successful, the user's session begins.

[0671] Input: User ID, Password

[0672] Output: Authentication result (success or failure)

[0673] Specific operation: When a user enters their user ID and password and presses the login button, the terminal sends the authentication information to the server, which then authenticates the user by comparing it with the information stored in the database.

[0674] Step 2: Customer Information Request

[0675] The user enters the customer's name and customer ID to retrieve information about a specific customer. The terminal sends this request to the server, and the server receives the request.

[0676] Input: Customer name, Customer ID

[0677] Output: Confirmation of receipt of customer information retrieval request

[0678] Specific operation: When the user enters the customer name and customer ID and presses the information retrieval button, the terminal sends that information to the server. The server confirms the request and proceeds to the next processing step.

[0679] Step 3: Information Gathering

[0680] Based on the received request, the server first accesses the company's internal customer database to retrieve the customer's past sales history. Next, it sends queries to external sources such as news APIs and company websites to gather the latest information. It also accesses the internal information database to retrieve internal memos and past deal information related to that customer.

[0681] Input: Customer name, Customer ID

[0682] Output: Collected customer information data

[0683] Specific operation: The server generates a database search query based on the customer name and customer ID, and searches the customer database. Furthermore, it accesses external APIs to collect the latest information.

[0684] Step 4: Information Summary

[0685] The server passes the collected information to a natural language processing engine (e.g., Google Cloud Natural Language API) to generate a summary. This summary includes the latest customer trends and key points from past deals.

[0686] Input: Collected customer information data

[0687] Output: Summarized customer information

[0688] Specific operation: The server inputs text data collected from the database into a natural language processing engine and generates summarized data.

[0689] Step 5: Providing information to users

[0690] The server sends summarized information to the user's terminal. The user can then use the provided information to prepare for the business negotiation.

[0691] Input: Summarized customer information

[0692] Output: Providing information to users

[0693] Specific operation: The server sends summary data to the user's terminal, and the user receives and displays it.

[0694] Step 6: Start the negotiation

[0695] The user starts the business negotiation and enables the recording function on their device.

[0696] Input: Instruction to start the business negotiation

[0697] Output: Enable recording function

[0698] Specific operation: When the user presses the record start button, the device starts recording and prepares to send the audio data to the server.

[0699] Step 7: Real-time transmission of audio data

[0700] The terminal transmits audio data of the business negotiation to the server in real time.

[0701] Input: Sales negotiation audio data

[0702] Output: Transmission of real-time audio data

[0703] Specific operation: The terminal divides the recorded audio and sends it to the server sequentially.

[0704] Step 8: Speech to Text

[0705] The server passes the received audio data to a real-time speech recognition engine (for example, Google Cloud Speech-to-Text) to convert the audio into text.

[0706] Input: Sales negotiation audio data

[0707] Output: Text-converted deal details

[0708] Specific operation: The server passes the audio data to the speech recognition engine and receives the output as text data.

[0709] Step 9: Keyword Extraction

[0710] The server extracts important keywords and phrases from the converted text.

[0711] Input: Text-converted sales opportunity details

[0712] Output: Extracted keywords

[0713] Specific operation: The server uses an extraction algorithm to identify and extract specific keywords or phrases.

[0714] Step 10: Real-time product proposal

[0715] The server suggests relevant products to the user in real time based on the extracted keywords. The suggestions are displayed as pop-up notifications on the device.

[0716] Input: Extracted keywords

[0717] Output: Product proposal notification

[0718] Specific operation: The server searches the product database corresponding to the extracted keywords and notifies the user's terminal of the relevant products.

[0719] Step 11: Ending the Deal

[0720] The user ends the deal and clicks the "End Deal" button.

[0721] Input: Instruction to end the business negotiation

[0722] Output: Start of deal closing process

[0723] Specific operation: When the user presses the "End Deal" button, the device stops recording and sends a notification to the server that the deal has ended.

[0724] Step 12: Send a follow-up request

[0725] The device sends a follow-up request to the server.

[0726] Input: Follow-up request

[0727] Output: Confirmation of the start of follow-up processing.

[0728] Specific operation: When the user presses the follow-up request button, the device sends a request to the server.

[0729] Step 13: Finalize the meeting minutes

[0730] The server finalizes the meeting minutes at the end of the meeting and sends them to the user's terminal. The minutes include key points and agreements from the business negotiation.

[0731] Input: Completed meeting minutes data

[0732] Output: Providing meeting minutes to users

[0733] Specific operation: The server generates meeting minutes from the text data of the business negotiation and sends them to the user's terminal.

[0734] Step 14: Re-analyzing the details of the business negotiation

[0735] The server re-analyzes the meeting minutes and sales negotiation details to generate the next proposals and schedules. It also extracts relevant information from a database of similar past sales negotiations and sends that information to the user's terminal.

[0736] Input: Meeting minutes data, sales negotiation text data

[0737] Output: Next proposal and schedule

[0738] Specific operation: The server uses meeting minutes and sales negotiation text data to generate the content and schedule for the next proposal, and collects and summarizes information on similar sales negotiations.

[0739] Step 15: Generating the next proposal and schedule

[0740] The server summarizes relevant information from a database of similar past business opportunities and generates the content and schedule for the next proposal. The user can then create a proposal based on this information.

[0741] Input: Past similar deal data

[0742] Output: Next proposal content, proposal schedule

[0743] Specific operation: The server searches the database for information on similar business opportunities and provides the user with the next proposal and its schedule.

[0744] In this way, this system enables users to efficiently utilize information at each stage—before, during, and after a business negotiation—through a series of processing steps, thereby facilitating smooth sales activities.

[0745] (Application Example 1)

[0746] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0747] During maintenance and troubleshooting on production lines within factories, there are problems with efficiently gathering and analyzing necessary information. Furthermore, the lack of means to provide useful information in real time during maintenance work can potentially reduce the accuracy and speed of the work.

[0748] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0749] In this invention, the server includes means for analyzing customer information, means for summarizing and presenting the results of the analysis of the customer information, means for generating proposal content related to the business negotiation, means for real-time speech recognition of the conversation content during the business negotiation and converting it into text, means for analyzing the conversation content and summarizing related information, means for generating and presenting meeting minutes after the business negotiation is completed, means for generating the next proposal schedule and means for extracting and presenting information on similar business negotiations, means for acquiring maintenance information based on identification information of the work target, and means for analyzing voice commands and presenting related maintenance information. This makes it possible to perform maintenance and troubleshooting work in the factory efficiently and accurately.

[0750] "Customer information" is a general term for data such as basic customer information, past transaction history, and inquiry history.

[0751] "Analysis means" refers to software or hardware configurations used to analyze collected information and generate useful insights.

[0752] A "summarization tool" is a function for concisely summarizing and presenting the analysis results.

[0753] "Proposal content generation means" refers to a system function that automatically creates proposal items related to business negotiations.

[0754] "Speech recognition means" refers to technology that converts speech data into text data.

[0755] "Text conversion means" refers to a function that converts speech into text information.

[0756] An "information summarization tool" is a function that displays collected or analyzed data in a shortened form.

[0757] A "meeting minutes generation system" is a system for recording and documenting the content of business negotiations and meetings.

[0758] The "proposal schedule generation method" is a function that plans the date for the next proposal or meeting.

[0759] The "similar business opportunity information extraction method" is a function that searches for and extracts information on similar business opportunities from a database of past business opportunities.

[0760] "Identification information" refers to information used to uniquely identify a specific work object or piece of equipment.

[0761] A "maintenance information acquisition method" is a function that acquires maintenance history and procedures related to equipment and systems based on identification information.

[0762] A "voice command analysis means" is a function that analyzes voice input and presents specific instructions or information based on its content.

[0763] This invention provides a system to support maintenance and troubleshooting within a factory. Specific embodiments are described below.

[0764] This system includes means for analyzing customer information, means for summarizing and presenting the results of the analysis of the customer information, means for generating proposals related to business negotiations, means for real-time speech recognition of conversations during business negotiations and converting them into text, means for analyzing the conversations and summarizing related information, means for generating and presenting meeting minutes after the business negotiation is completed, means for generating a schedule for the next proposal and means for extracting and presenting information on similar business negotiations, means for acquiring maintenance information based on identification information of the work target, and means for analyzing voice commands and presenting related maintenance information.

[0765] Hardware and software configuration

[0766] The hardware configuration of this system includes a microphone mounted on the robot body used in the factory, a server, and an internet connection. The software configuration includes a Python program, a speech recognition library (speech_recognition), and an HTTP request library (requests).

[0767] Process Overview

[0768] 1. Voice input recognition

[0769] The user inputs voice commands to the robot to obtain maintenance information for the object being worked on. This voice is collected via a microphone.

[0770] 2. Text conversion of audio data

[0771] The server converts the collected audio data into text data using a speech recognition library. At this stage, the sensitivity is adjusted to account for ambient noise.

[0772] 3. Information acquisition based on identification information

[0773] The server retrieves relevant maintenance information based on identification information (e.g., equipment ID) in the text data. The retrieved information is parsed in JSON format, and the necessary information is extracted.

[0774] 4. Display of maintenance information

[0775] The server provides users with acquired maintenance information in real time. This information is displayed on the robot's display and on a separate terminal, making it immediately available to the worker.

[0776] Specific example

[0777] For example, if a sudden malfunction occurs in "device 123" on a production line in a factory, the worker would input a voice command to the robot saying, "Tell me the maintenance information for device 123." The robot would recognize this command, retrieve the latest maintenance information and logs related to "device 123" from a server via the internet, and provide them to the worker.

[0778] Example of a prompt

[0779] "Please enter questions to provide appropriate maintenance and troubleshooting information for equipment problems that have occurred on the factory production line. For example, 'Please provide the latest maintenance information for equipment 123,' or 'Please provide the past trouble history for equipment 456.'"

[0780] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0781] Step 1:

[0782] The user inputs voice commands to the robot to obtain maintenance information for the object being worked on. The input voice is collected through the microphone on the robot itself.

[0783] Step 2:

[0784] The terminal (robot) sends the collected audio data to the server. The server converts this audio data into text data using a speech recognition library (speech_recognition). The input is audio data, and the output is the corresponding text data.

[0785] Step 3:

[0786] The server analyzes the converted text data and extracts identification information (e.g., device ID). This identification information is identified as a specific keyword from the input voice command, and the output is this identification information.

[0787] Step 4:

[0788] The server sends a request to the maintenance information retrieval API based on the identification information. The input is the identification information, and the output is the maintenance information response data. This response data is returned in JSON format.

[0789] Step 5:

[0790] The server parses the acquired maintenance information, extracts relevant information, and summarizes it. The input is maintenance information in JSON format, and the output is the summarized maintenance information.

[0791] Step 6:

[0792] The terminal (robot) displays summarized maintenance information sent from the server to the user. The input is summarized maintenance information, and the output is text-based information presented to the user's visual input. This information is displayed on the robot's screen and is immediately available to the worker.

[0793] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0794] This invention relates to a sales support system that combines customer information collection and analysis, real-time processing during sales negotiations, post-sales follow-up, and an emotion engine that recognizes user emotions and optimizes proposals. Detailed embodiments of this system are described below.

[0795] Customer information collection and analysis

[0796] User actions:

[0797] The user logs into the system and sends a request to retrieve information about a specific customer. The user's device sends this request to the server.

[0798] Server processing:

[0799] After receiving a customer information retrieval request, the server accesses the customer database to retrieve past sales negotiation history. Next, it gathers the latest information from external sources (news API, company website, etc.). Furthermore, it accesses the internal information database to retrieve relevant internal memos and past project information. The collected information is summarized by a natural language processing engine, and the server sends the summarized information to the user's terminal.

[0800] Specific example:

[0801] If a user has a business meeting scheduled with "Company A," the server collects Company A's past business meeting history, latest news, and relevant internal records, and provides this summarized information to the user's terminal. The user then inputs their proposal based on this information, and the server generates a proposal document based on that input.

[0802] Information processing and emotional engine during business negotiations

[0803] User actions:

[0804] The user initiates a business negotiation and activates the recording and emotion recognition functions on their device. The audio data of the negotiation and emotion data such as the user's facial expressions are transmitted to the server in real time.

[0805] Server processing:

[0806] The server converts received audio data into text using a real-time speech recognition engine. It extracts important keywords and phrases from the converted text and embeds them into a meeting minutes template. It also features an emotion engine that analyzes the user's emotional state and adjusts the proposed content in real time based on that analysis. Furthermore, the server analyzes the sales negotiation content and proposes relevant products to the user in real time.

[0807] Specific example:

[0808] When a user conducts an online business meeting with a representative from "Company A," the terminal sends audio of the meeting and the user's facial expression data to the server. The server converts the audio to text and detects statements such as "I am interested in cost reduction." In this case, if the emotion engine detects that the user's emotional state is "attractive," the server immediately presents the user with products related to cost reduction.

[0809] Post-sales follow-up and utilization of emotional data

[0810] User actions:

[0811] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[0812] Server processing:

[0813] The server finalizes the meeting minutes at the end of the meeting and sends them to the user's terminal. Next, it re-analyzes the meeting minutes and sales negotiation content to generate the next proposal and schedule. It also extracts relevant deals from a database of similar past deals, summarizes that information, and sends it to the user's terminal. Furthermore, it optimizes the proposal content based on sentiment data obtained by the sentiment engine.

[0814] Specific example:

[0815] After the user concludes a business meeting with "Company A," the server reviews the meeting minutes and presents a proposal and timeline for the next meeting. Furthermore, if there have been similar business meetings with "Company B" in the past, the server summarizes that information and provides it to the user. If the emotion engine predicts that the representative from "Company A" will have a positive sentiment towards the next meeting, the server adjusts the proposal based on that prediction. Once the user enters the proposal again, the server generates a new proposal.

[0816] In this way, this system enables users to efficiently utilize information at each stage—before, during, and after a business negotiation—and also takes into account the user's emotional state, allowing sales activities to proceed smoothly.

[0817] The following describes the processing flow.

[0818] Customer information collection and analysis

[0819] Step 1:

[0820] A user logs into the system and sends a request from their terminal to the server to retrieve information about a specific customer.

[0821] Step 2:

[0822] The server receives a request to retrieve customer information.

[0823] Step 3:

[0824] The server accesses the customer database and retrieves the customer's past sales history.

[0825] Step 4:

[0826] The server collects the latest information from external sources (news APIs, company websites, etc.).

[0827] Step 5:

[0828] The server accesses the company's internal information database to retrieve relevant internal memos and past project information.

[0829] Step 6:

[0830] The server collects information and then uses a natural language processing engine to summarize it.

[0831] Step 7:

[0832] The server sends the summarized information to the user's terminal.

[0833] Step 8:

[0834] The user reviews the summary information and enters the suggested content into their device.

[0835] Step 9:

[0836] The terminal sends the user's inputted suggestions to the server.

[0837] Step 10:

[0838] The server analyzes the proposal and generates a proposal document based on that information.

[0839] Step 11:

[0840] The server sends the generated proposal to the user's terminal.

[0841] Information processing and emotional engine during business negotiations

[0842] Step 1:

[0843] The user initiates a business negotiation and activates the recording and emotion recognition functions on their device.

[0844] Step 2:

[0845] The terminal transmits audio data of the business negotiation and emotional data such as the user's facial expressions to the server in real time.

[0846] Step 3:

[0847] The server converts the received audio data into text using a real-time speech recognition engine.

[0848] Step 4:

[0849] The server extracts important keywords and phrases from the converted text.

[0850] Step 5:

[0851] The server extracts the information and embeds it into the meeting minutes template.

[0852] Step 6:

[0853] The emotion engine analyzes the user's emotional state and adjusts the suggested content in real time based on that analysis.

[0854] Step 7:

[0855] The server analyzes the details of the business negotiation and suggests relevant products to the user in real time.

[0856] Step 8:

[0857] Users can view real-time suggestions from the server and use them immediately.

[0858] Post-sales follow-up and utilization of emotional data

[0859] Step 1:

[0860] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[0861] Step 2:

[0862] The server finalizes the meeting minutes at the time of shutdown and sends them to the user's terminal.

[0863] Step 3:

[0864] The server re-analyzes the meeting minutes and negotiation details and generates a schedule of what should be proposed next.

[0865] Step 4:

[0866] The server extracts relevant sales case examples from a database of similar past sales opportunities.

[0867] Step 5:

[0868] The server extracts similar business opportunity information, summarizes it, and sends it to the user's terminal.

[0869] Step 6:

[0870] The suggested content is optimized based on emotional data acquired by the emotion engine.

[0871] Step 7:

[0872] Based on the information from Step 6, the server generates a new proposal for the next business meeting.

[0873] Step 8:

[0874] The user enters a new proposal into the terminal for the next business meeting.

[0875] Step 9:

[0876] The terminal sends the new suggestions entered by the user to the server.

[0877] Step 10:

[0878] The server analyzes the new proposal and generates a new proposal document based on that information.

[0879] Step 11:

[0880] The server sends the new proposal to the user's terminal.

[0881] (Example 2)

[0882] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0883] Traditional sales support systems lacked the ability to integrate customer information collection, real-time information processing during negotiations, emotional data analysis, and post-sales follow-up. As a result, sales representatives had to manage each process individually, leading to decreased efficiency. Furthermore, there was no effective way to utilize emotional data during negotiations, making it difficult to respond quickly to changes in customer emotions. This resulted in insufficient optimization of proposals and a lower success rate for sales negotiations.

[0884] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0885] In this invention, the server includes means for analyzing customer information, means for summarizing and presenting the results of the analysis of the customer information, means for real-time speech recognition of conversation content during negotiations and conversion to text, and means for analyzing emotional data acquired during negotiations and adjusting the content of proposals. This makes it possible to effectively collect and analyze customer information at each stage before, during, and after negotiations, and to further optimize the content of proposals in real time by utilizing emotional data.

[0886] "Customer information" refers to data, history, and related notes about customers.

[0887] "Analysis" refers to the process of processing data and extracting meaningful information or patterns.

[0888] "Summary" refers to providing a concise overview of detailed information.

[0889] "Presentation" refers to displaying information in a way that is easy for users to understand.

[0890] "Business negotiation" refers to discussions or meetings regarding sales or purchases.

[0891] "Proposed content" refers to the products, services, or their terms and conditions presented during business negotiations.

[0892] "Speech recognition" refers to the technology that converts speech into text.

[0893] "Converting to text" refers to replacing audio data with text data.

[0894] "Conversation content" refers to the words and statements exchanged during a business negotiation.

[0895] "Emotional data" refers to information about a user's emotions obtained during a business negotiation, such as their facial expressions and voice.

[0896] "Meeting minutes" refers to a document that records and summarizes the key points of a business negotiation.

[0897] "Next proposal schedule" refers to the planned visits and proposals for the next business meeting.

[0898] "Information on similar deals" refers to data and history related to similar deals that have taken place in the past.

[0899] "Real-time" refers to processing being performed at the very moment an event occurs.

[0900] "Adjustment" refers to changing the content or method depending on the situation.

[0901] This invention relates to a sales support system that combines customer information collection and analysis, real-time processing during sales negotiations, post-sales follow-up, and an emotion engine that recognizes user emotions and optimizes proposals. Specific embodiments for carrying out the invention are described below.

[0902] Customer information collection and analysis

[0903] User actions:

[0904] The user logs into the system and sends a request to retrieve information about a specific customer. The user's device sends this request to the server.

[0905] Server processing:

[0906] After receiving a customer information retrieval request, the server accesses the customer database to retrieve past sales history. Next, it gathers the latest information from external sources (e.g., news APIs or company websites). Furthermore, it accesses the internal information database to retrieve relevant internal memos and past project information. This information is summarized by a natural language processing engine (NLP engine), and the server sends the summarized information to the user's terminal.

[0907] Specific example:

[0908] If a user has an upcoming business meeting with "Company A," they log into the system and request customer information. The server retrieves "Company A's" past business meeting history, latest news, and related internal records, and summarizes the information using an NLP engine. The summarized information is then provided to the user's device. For example, the Google News API could be used as the news API, and a CRM system could be used for internal information management.

[0909] Information processing and emotional engine during business negotiations

[0910] User actions:

[0911] The user initiates a business negotiation and activates the recording and emotion recognition functions on their device. The audio data of the negotiation and the user's emotion data (facial expression data, etc.) are transmitted to the server in real time.

[0912] Server processing:

[0913] The server converts received audio data into text using a real-time speech recognition engine. It extracts important keywords and phrases from the converted text and embeds them into a meeting minutes template. The server also has the capability to analyze the user's emotional state using an emotion recognition engine and adjust the proposed content in real time. Furthermore, it analyzes the content of business negotiations and proposes relevant products and services to the user in real time.

[0914] Specific example:

[0915] When a user conducts an online business meeting with a representative from "Company A," the user enables recording and emotion recognition features in advance. The device sends the audio of the meeting and the user's facial expression data to the server. The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the audio to text and extract keywords. For example, if the representative says, "I'm interested in cost reduction," the server uses an emotion recognition algorithm to determine the emotional state is "attractive" and quickly presents the user with product information related to cost reduction.

[0916] Post-sales follow-up and utilization of emotional data

[0917] User actions:

[0918] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[0919] Server processing:

[0920] The server finalizes the meeting minutes at the end of the business negotiation and sends them to the user's terminal. Next, it re-analyzes the minutes and negotiation details to generate the next proposed content and schedule. Furthermore, it extracts relevant information from a database of similar past negotiations and provides a summary to the user's terminal. Finally, it optimizes the proposed content based on sentiment data obtained by the sentiment recognition engine.

[0921] Specific example:

[0922] After the user concludes a business meeting with "Company A," the server finalizes the meeting minutes and sends them to the user. The server searches for similar past business meeting data and provides a summary of successful cases, such as those with "Company B." It also generates and proposes a schedule for the next visit and proposals to the user. The emotion engine determines that the emotional state of the "Company A" representative is positive and adjusts the next proposal based on that prediction.

[0923] In this way, this system enables users to efficiently utilize information at each stage—before, during, and after a business negotiation—and also takes into account the user's emotional state, allowing sales activities to proceed smoothly.

[0924] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0925] Customer information collection and analysis

[0926] Step 1:

[0927] The user logs into the system and sends a request to retrieve information about a specific customer. Specifically, the user clicks the "Retrieve Customer Information" button on the terminal's operation screen and enters information such as the company name and customer name. The terminal sends this request data to the server. The input is "Company Name: Company A" and the output is "Customer Information Retrieval Request".

[0928] Step 2:

[0929] The server receives a customer information retrieval request. The server first accesses the customer database to retrieve past sales negotiation history for "Company A". At this stage, the input is the "customer information retrieval request" and the output is the "past sales negotiation history".

[0930] Step 3:

[0931] The server accesses external sources (such as news APIs and company websites) to collect the latest information related to "Company A". At this stage, the input is "Company Name: Company A" and the output is "Latest News". Specifically, the server calls APIs to collect data and stores it internally.

[0932] Step 4:

[0933] The server accesses the company's internal information database to retrieve internal memos and past project information related to "Company A". At this stage, the input is "Company Name: Company A", and the output is "Internal Memos and Project Information". Specifically, it executes database queries to extract the necessary information.

[0934] Step 5:

[0935] The collected information is summarized by a natural language processing engine (NLP engine). The server uses past sales history, latest news, and internal memos as input data to perform summarization and generate the results. At this stage, the input is "all collected information," and the output is "summarized customer information." Specifically, the NLP engine's algorithm is executed to generate the summary.

[0936] Step 6:

[0937] The server sends summarized information to the user's terminal. At this stage, the input is "summarized customer information," and the output is "customer information displayed on the user's terminal." Specifically, the server sends an HTTP response, and the terminal displays that information.

[0938] ---

[0939] Information processing and emotional engine during business negotiations

[0940] Step 1:

[0941] The user initiates a business meeting and activates the recording and emotion recognition functions on their device. The user clicks the "Start Recording" and "Start Emotion Recognition" buttons on the device. The input is "Notification of Business Meeting Start," and the output is "Start of Recording and Emotion Recognition."

[0942] Step 2:

[0943] The terminal transmits audio and facial expression data from the business negotiation to the server in real time. At this stage, the input is "audio and facial expression data," and the output is "data sent to the server." Specifically, the terminal transmits the data in real time using streaming technology.

[0944] Step 3:

[0945] The server converts the received audio data into text using a real-time speech recognition engine. At this stage, the input is "audio data" and the output is "converted text". Specifically, it calls the Google Cloud Speech-to-Text API to convert audio to text.

[0946] Step 4:

[0947] The server extracts important keywords and phrases from the text and embeds them into a meeting minutes template. At this stage, the input is the "converted text," and the output is the "keywords embedded in the meeting minutes." Specifically, a text analysis algorithm is executed to extract keywords.

[0948] Step 5:

[0949] The server uses an emotion recognition engine to analyze the user's emotional state and adjusts the suggestions in real time. At this stage, the input is "facial expression data and converted text," and the output is "adjusted suggestions." Specifically, it uses an emotion recognition algorithm to analyze the emotional state and adjusts the suggestions based on that data.

[0950] Step 6:

[0951] The server analyzes the details of the business negotiation and proposes relevant products and services to the user's terminal in real time. The input at this stage is "text data of the adjusted proposal and business negotiation," and the output is "product information presented to the user." Specifically, it queries a related product database to generate appropriate proposals.

[0952] ---

[0953] Post-sales follow-up and utilization of emotional data

[0954] Step 1:

[0955] The user concludes the business negotiation and sends a follow-up request from their device to the server. The input is "Follow-up notification," and the output is "Follow-up request sent."

[0956] Step 2:

[0957] The server finalizes the meeting minutes and sends them to the user's terminal. At this stage, the input is "meeting minutes created in real time," and the output is "sending the finalized meeting minutes to the user." Specifically, it completes the meeting minutes template and provides it to the user.

[0958] Step 3:

[0959] The server re-analyzes the meeting minutes and negotiation details to generate the next proposal and schedule. At this stage, the input is the "finalized meeting minutes and negotiation details," and the output is the "next proposal and schedule." Specifically, it runs a data analysis algorithm to plan the next proposal.

[0960] Step 4:

[0961] The server checks a database of similar past deals, extracts relevant information, and provides it to the user. At this stage, the input is "finalized meeting minutes and deal details," and the output is "provision of information on similar deals." Specifically, it performs a database search to identify similar past cases, summarizes that information, and provides it.

[0962] Step 5:

[0963] The server optimizes the next proposal based on sentiment data. At this stage, the input is "sentiment data and finalized meeting minutes," and the output is "optimized next proposal." Specifically, the proposal is readjusted based on sentiment analysis data.

[0964] Step 6:

[0965] The server sends the optimized suggestions to the user's terminal. At this stage, the input is the "optimized next suggestion," and the output is the "optimized suggestion presented to the user." Specifically, the server sends an HTTP response, and the terminal displays that information.

[0966] As described above, by clearly defining the specific operation and input / output of the system at each step, and by clearly indicating the roles of the user, terminal, and server, the present invention can be effectively implemented.

[0967] (Application Example 2)

[0968] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0969] In modern brick-and-mortar stores, customer service often relies heavily on the experience and skills of store staff, leading to inconsistencies in customer satisfaction. Furthermore, accurately understanding customer needs and emotions in real time and providing optimal suggestions based on that understanding is challenging. Traditional systems fail to adequately optimize suggestions based on customer emotions, making efficient sales support difficult. Moreover, real-time information processing during sales negotiations and effective follow-up after negotiations are often lacking.

[0970] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0971] In this invention, the server includes means for analyzing customer information, means for summarizing and presenting the results of the customer information analysis, means for generating proposal content related to the business negotiation, means for real-time speech recognition and text conversion of conversation content during the business negotiation, means for analyzing the conversation content and summarizing related information, means for analyzing video data during the business negotiation to recognize the user's emotions, means for optimizing the proposal content based on the user's emotion data, means for generating and presenting meeting minutes after the business negotiation is completed, means for generating a schedule for the next proposal, and means for extracting and presenting information on similar business negotiations. This makes it possible to grasp customer needs and emotions in real time at physical stores and make optimal proposals. Furthermore, it is expected that follow-up during and after business negotiations can be efficiently carried out, leading to improved customer satisfaction and sales efficiency.

[0972] "Customer information" refers to data and historical information about customers, including, for example, past purchase history, inquiry history, and personal profile information.

[0973] "Means of analysis" refers to methods and techniques for analyzing collected data and extracting useful information from it. Specifically, this includes machine learning algorithms and statistical analysis techniques.

[0974] "Speech recognition" is a technology that converts speech data into text data, and includes the process of automatically transcribing the user's speech into text.

[0975] "Means of converting to text" refers to technologies and systems for converting collected audio data into text, such as speech recognition engines.

[0976] "Methods of summarization" refer to methods and techniques for organizing analyzed information into a concise and easily understandable form. Natural language processing and summarization algorithms are used in this process.

[0977] "Means of recognizing emotions" refers to technologies that determine a user's emotional state at a given time from their video and audio data. This includes emotion recognition engines and image processing technologies.

[0978] "Methods for optimizing suggestions" refer to technologies that generate the most suitable suggestions by considering user sentiment data and other factors. This includes recommendation algorithms and personalization engines.

[0979] "Means for generating and presenting meeting minutes" refers to technologies and systems that automatically create meeting minutes based on text data recorded during business negotiations and provide them to users.

[0980] "Methods for generating proposal schedules" refer to technologies that generate the content and schedule of the next proposal based on the details of the business negotiation and past history. This includes scheduling management systems and task management tools.

[0981] "Methods for extracting and presenting information on similar business opportunities" refers to technologies that extract cases similar to the current business opportunity from a database of past business opportunities and provide them as reference information. Data mining and clustering algorithms are used in this process.

[0982] In this invention, to specifically realize a sales support system for physical stores, smart glasses, a server, a speech recognition engine, a natural language processing engine, an emotion recognition engine, and a recommendation algorithm work together as a single unit. The operation of this system will be described below in order.

[0983] System configuration and operation

[0984] 1. Data Collection

[0985] The user (staff member) wears smart glasses.

[0986] When a conversation with a customer begins, the smart glasses collect audio and video data in real time and send it to the server.

[0987] 2. Data Processing

[0988] The server converts the received audio data into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[0989] The converted text data is sent to a natural language processing engine (e.g., a BERT model) where important keywords and phrases are extracted.

[0990] The video data is used to analyze the emotional state of customers using an emotion recognition engine (e.g., Microsoft Azure Face API).

[0991] Based on extracted keywords and sentiment data, a recommendation algorithm (e.g., collaborative filtering algorithm) is used to select the most suitable product suggestions.

[0992] 3. Data Output

[0993] The selected product information is displayed on the smart glasses' screen and presented to the user (staff).

[0994] The user suggests selected products to customers who visit the store.

[0995] Specific example

[0996] 1. Examples of customer service

[0997] When a user asks, "Hello, what can I help you with?", the customer replies, "I've come to look at winter coats."

[0998] The smart glasses send the audio data to a server, where speech recognition and natural language processing extract the keyword "winter coat."

[0999] The emotion recognition engine detects emotions such as "excitement" and "interest" from video data of customers visiting the store.

[1000] A recommendation algorithm selects multiple coats based on inventory information and popular product data, and displays them on smart glasses.

[1001] A user who has checked the display of smart glasses suggests to a customer, "These two coats are very popular."

[1002] Example of a prompt

[1003] "Enter the following user voice data into the speech recognition engine: Hello, I'm looking for a winter coat."

[1004] "The converted text data is passed to a natural language processing engine to extract keywords."

[1005] "Video data is input into an emotion recognition engine to detect the user's emotional state."

[1006] "Next, the extracted keywords and sentiment data are passed to a product recommendation algorithm to select the most suitable product."

[1007] In this way, it becomes possible to provide optimal suggestions tailored to customer needs in physical stores in real time, thereby improving customer satisfaction.

[1008] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1009] Step 1:

[1010] The user (staff member) wears smart glasses and begins a conversation with a customer. The smart glasses collect audio and video data in real time and send it to the server. The input is the audio and video data of the conversation with the customer, and the output is sending this data to the server.

[1011] Step 2:

[1012] The server converts the received audio data into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text). Audio data is the input, and text data is the output. Specifically, the audio data is analyzed, and the spoken content is extracted as a string of characters.

[1013] Step 3:

[1014] The server passes the converted text data to a natural language processing engine (e.g., a BERT model) to extract important keywords and phrases. The input is text data, and the output is important keywords and phrases. Specifically, it analyzes the context of the text and extracts information relevant to the business deal.

[1015] Step 4:

[1016] Video data is sent to a server, which uses an emotion recognition engine (e.g., Microsoft Azure Face API) to analyze the user's emotional state. The input is video data, and the output is recognized emotion data. Specifically, emotions are determined from facial expressions, and levels of excitement and interest are quantified.

[1017] Step 5:

[1018] The server uses a recommendation algorithm (e.g., collaborative filtering algorithm) to select the most suitable product based on extracted keywords and sentiment data. The input is keywords and sentiment data, and the output is a list of recommended products. Specifically, it refers to past sales history and inventory information to select the most relevant products.

[1019] Step 6:

[1020] The server displays selected product information on the smart glasses' screen and suggests it to the user (staff). The input is a list of recommended products, and the output is the product information displayed on the smart glasses' screen. Specifically, it provides the user with optimized suggestions immediately.

[1021] Step 7:

[1022] The user (staff) checks the information displayed on the smart glasses and makes suggestions to customers. The input is product information displayed on the screen, and the output is suggestions to customers. Specifically, the staff makes verbal suggestions based on the information they visually confirm.

[1023] In this way, the system of the present invention can efficiently and effectively handle customer interactions in physical stores.

[1024] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1025] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1026] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1027] [Third Embodiment]

[1028] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1029] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1030] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1031] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1032] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1033] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1034] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1035] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1036] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1037] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1038] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1039] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1040] This invention relates to a sales support system for efficiently collecting and analyzing customer information, processing sales negotiations in real time, and following up after sales negotiations. Detailed embodiments of this system are described below.

[1041] Customer information collection and analysis

[1042] User actions:

[1043] The user logs into the system and sends a request to retrieve information about a specific customer. The user's device sends this request to the server.

[1044] Server processing:

[1045] After receiving a customer information retrieval request, the server accesses the customer database to retrieve past sales negotiation history. Next, it gathers the latest information from external sources (news API, company website, etc.). Furthermore, it accesses the internal information database to retrieve relevant internal memos and past project information. The collected information is summarized by a natural language processing engine, and the server sends the summarized information to the user's terminal.

[1046] Specific example:

[1047] If a user has an upcoming business meeting with "Company A," the server collects and summarizes Company A's past business meeting history, latest news, and relevant internal records, and provides this information to the user's terminal. The user then inputs their proposal based on this information, and the server generates a proposal document based on that input.

[1048] Information processing during business negotiations

[1049] User actions:

[1050] The user initiates a business meeting and activates the recording function on their device. The audio data of the meeting is transmitted to the server in real time.

[1051] Server processing:

[1052] The server converts received audio data into text using a real-time speech recognition engine. It extracts important keywords and phrases from the converted text and embeds them into a meeting minutes template. The server further analyzes the sales meeting content and suggests relevant products to the user in real time.

[1053] Specific example:

[1054] When a user conducts an online business meeting with a representative from "Company A," the terminal sends the audio of the meeting to the server. The server converts the audio to text and detects statements such as "I am interested in cost reduction." In this case, the server immediately proposes products related to cost reduction to the user.

[1055] Follow-up after business negotiations

[1056] User actions:

[1057] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[1058] Server processing:

[1059] The server finalizes the meeting minutes at the end of the meeting and sends them to the user's terminal. Next, it re-analyzes the meeting minutes and sales negotiation details to generate the next proposed content and schedule. It also extracts relevant sales opportunities from a database of similar past sales opportunities, summarizes that information, and sends it to the user's terminal.

[1060] Specific example:

[1061] After the user concludes a business negotiation with "Company A," the server reviews the meeting minutes and presents the content and timeline for the next proposal. Furthermore, if there have been similar negotiations with "Company B" in the past, the server summarizes that information and provides it to the user. When the user enters proposal details again, the server generates a new proposal document.

[1062] In this way, this system enables users to efficiently utilize information at each stage—before, during, and after a business negotiation—and to conduct sales activities smoothly.

[1063] The following describes the processing flow.

[1064] Customer information collection and analysis

[1065] Step 1:

[1066] A user logs into the system and sends a request from their terminal to the server to retrieve information about a specific customer.

[1067] Step 2:

[1068] The server receives a request to retrieve customer information.

[1069] Step 3:

[1070] The server accesses the customer database and retrieves the customer's past sales history.

[1071] Step 4:

[1072] The server collects the latest information from external sources (news APIs, company websites, etc.).

[1073] Step 5:

[1074] The server accesses the company's internal information database to retrieve relevant internal memos and past project information.

[1075] Step 6:

[1076] The server collects information and then uses a natural language processing engine to summarize it.

[1077] Step 7:

[1078] The server sends the summarized information to the user's terminal.

[1079] Step 8:

[1080] The user reviews the summary information and enters the suggested content into their device.

[1081] Step 9:

[1082] The terminal sends the user's inputted suggestions to the server.

[1083] Step 10:

[1084] The server analyzes the proposal and generates a proposal document based on that information.

[1085] Step 11:

[1086] The server sends the generated proposal to the user's terminal.

[1087] Information processing during business negotiations

[1088] Step 1:

[1089] The user initiates a business meeting and activates the recording function on their device.

[1090] Step 2:

[1091] The terminal transmits the audio data of the business negotiation to the server in real time.

[1092] Step 3:

[1093] The server converts the received audio data into text using a real-time speech recognition engine.

[1094] Step 4:

[1095] The server extracts important keywords and phrases from the converted text.

[1096] Step 5:

[1097] The server extracts the information and embeds it into the meeting minutes template.

[1098] Step 6:

[1099] The server analyzes the details of the business negotiation and suggests relevant products to the user in real time.

[1100] Step 7:

[1101] Users can view real-time suggestions from the server and use them immediately.

[1102] Follow-up after business negotiations

[1103] Step 1:

[1104] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[1105] Step 2:

[1106] The server finalizes the meeting minutes at the time of shutdown and sends them to the user's terminal.

[1107] Step 3:

[1108] The server re-analyzes the meeting minutes and negotiation details and generates a schedule of what should be proposed next.

[1109] Step 4:

[1110] The server extracts relevant sales case examples from a database of similar past sales opportunities.

[1111] Step 5:

[1112] The server extracts similar business opportunity information, summarizes it, and sends it to the user's terminal.

[1113] Step 6:

[1114] The user enters a new proposal into the terminal for the next business meeting.

[1115] Step 7:

[1116] The terminal sends the new suggestions entered by the user to the server.

[1117] Step 8:

[1118] The server analyzes the new proposal and generates a new proposal document based on that information.

[1119] Step 9:

[1120] The server sends the new proposal to the user's terminal.

[1121] (Example 1)

[1122] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1123] In conventional sales support systems, customer information collection and analysis, real-time processing during negotiations, and post-negotiation follow-up were all handled independently, making it difficult to utilize information consistently and conduct efficient sales activities. Furthermore, there was a need for improved accuracy in real-time proposals during negotiations and in the content of follow-up proposals after negotiations concluded.

[1124] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1125] In this invention, the server includes means for analyzing customer information, means for summarizing and presenting the results of the customer information analysis, means for receiving information collection requests and collecting information from a customer database and external information sources, means for summarizing the collected information using a natural language processing engine, means for generating proposals related to business negotiations, means for real-time speech recognition and text conversion of conversation content during business negotiations, means for analyzing conversation content, extracting important keywords, and proposing related products, means for generating and presenting meeting minutes after the conclusion of business negotiations, means for generating a schedule for the next proposal, and means for extracting and presenting information on similar business negotiations. This makes it possible to efficiently utilize information at each stage before, during, and after business negotiations and to smoothly carry out sales activities.

[1126] "Customer data analysis" is the process of collecting data about customers and analyzing patterns and trends.

[1127] A "summary of customer information analysis results" is a concise compilation of the most important points extracted from the analyzed information.

[1128] An "information gathering request" is the process of requesting detailed information or up-to-date data about a specific customer.

[1129] A "customer database" is a data storage system that aggregates and stores detailed information about customers.

[1130] "External information sources" refer to information providers other than internal databases, such as company websites and news APIs.

[1131] A "natural language processing engine" is software that analyzes text data and executes algorithms to understand and generate human language.

[1132] "Generating proposals related to business negotiations" is the process of creating the optimal proposal based on the objectives of the negotiation and the customer's needs.

[1133] "Real-time speech recognition" is a technology that instantly converts audio generated during business negotiations into text data.

[1134] "Keyword extraction" is the process of selecting particularly important words and phrases from conversations or texts.

[1135] "Proposing products or services" refers to suggesting suitable products or services based on the needs identified during a business negotiation.

[1136] "Generating meeting minutes after a business negotiation" is the process of organizing the content and conclusions of a business negotiation and documenting them so that they can be referenced later.

[1137] "Generating the next proposal schedule" is the process of planning what should be proposed next and when.

[1138] "Extracting information on similar business opportunities" means detecting cases similar to the current business opportunity from past business opportunity history and reusing that information.

[1139] The present invention relates to a sales support system for efficiently collecting and analyzing customer information, processing sales negotiations in real time, and following up after sales negotiations. This system includes means for analyzing customer information, means for summarizing and presenting the analysis results, means for generating proposals related to sales negotiations, means for real-time speech recognition and text conversion of conversations during sales negotiations, means for analyzing conversations and summarizing related information, means for generating and presenting meeting minutes after the sales negotiation is completed, means for generating a schedule for the next proposal, and means for extracting and presenting information on similar sales negotiations.

[1140] Customer information collection and analysis

[1141] User actions:

[1142] The user logs into the sales support system and sends a request to retrieve information about a specific customer. The user's device sends this request to the server.

[1143] Server processing:

[1144] After receiving a customer information retrieval request, the server accesses the customer database to retrieve past sales history. Next, it gathers the latest information from external sources such as news APIs and company websites. Furthermore, it accesses internal information databases to retrieve relevant internal memos and past project information. The collected information is summarized by a natural language processing engine (e.g., Google Cloud Natural Language API), and the server sends the summarized information to the user's terminal.

[1145] Specific example:

[1146] For example, if a user has a business meeting scheduled with "Company A," the server collects past business meeting history, latest news, and relevant internal records of "Company A," and provides summarized information to the user's terminal. The user then inputs their proposal based on this information, and the server generates a proposal document based on that input.

[1147] Examples of prompts for a generative AI model:

[1148] "Please provide a summary of past business negotiations with a certain company, the latest news, and relevant internal memos."

[1149] Information processing during business negotiations

[1150] User actions:

[1151] The user initiates the business negotiation and activates the recording function on their device.

[1152] Server processing:

[1153] The terminal transmits audio data of the business negotiation to the server in real time. The server passes the received audio data to a real-time speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the audio to text. It extracts important keywords and phrases from the converted text and suggests relevant products to the user in real time based on that.

[1154] Specific example:

[1155] When a user conducts an online business meeting with a representative from "a certain company," the terminal transmits the audio of the meeting to the server. The server converts the audio to text and detects statements such as "I am interested in cost reduction." In this case, the server immediately proposes products related to cost reduction to the user.

[1156] Examples of prompts for a generative AI model:

[1157] "Extract key keywords from the following text and propose corresponding products / services: 'A representative from a certain company has stated they are interested in cost reduction.'"

[1158] Follow-up after business negotiations

[1159] User actions:

[1160] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[1161] Server processing:

[1162] The server finalizes the meeting minutes at the end of the meeting and sends them to the user's terminal. Next, it re-analyzes the minutes and negotiation details to generate the next proposal and schedule. It also extracts relevant deals from a database of similar past deals, summarizes that information, and sends it to the user's terminal. The user can then create a proposal based on the next proposal.

[1163] Specific example:

[1164] After the user concludes a business negotiation with "a certain company," the server reviews the meeting minutes and presents the content and timeline for the next proposal. Furthermore, if there have been similar negotiations with "other companies" in the past, the server summarizes that information and provides it to the user. Based on this, the user creates a new proposal.

[1165] Examples of prompts for a generative AI model:

[1166] "Please generate the next proposal and timeline based on the following meeting minutes: 'We need a proposal regarding cost reduction in negotiations with a certain company. Please provide a schedule for creating the next proposal.'"

[1167] In this way, this system enables users to efficiently utilize information at each stage—before, during, and after a business negotiation—and to conduct sales activities smoothly.

[1168] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1169] Step 1: User Login

[1170] The user accesses the login screen of the sales support system and enters their authentication information (user ID and password). The terminal sends this information to the server, which verifies it against the database to perform authentication. If authentication is successful, the user's session begins.

[1171] Input: User ID, Password

[1172] Output: Authentication result (success or failure)

[1173] Specific operation: When a user enters their user ID and password and presses the login button, the terminal sends the authentication information to the server, which then authenticates the user by comparing it with the information stored in the database.

[1174] Step 2: Customer Information Request

[1175] The user enters the customer's name and customer ID to retrieve information about a specific customer. The terminal sends this request to the server, and the server receives the request.

[1176] Input: Customer name, Customer ID

[1177] Output: Confirmation of receipt of customer information retrieval request

[1178] Specific operation: When the user enters the customer name and customer ID and presses the information retrieval button, the terminal sends that information to the server. The server confirms the request and proceeds to the next processing step.

[1179] Step 3: Information Gathering

[1180] Based on the received request, the server first accesses the company's internal customer database to retrieve the customer's past sales history. Next, it sends queries to external sources such as news APIs and company websites to gather the latest information. It also accesses the internal information database to retrieve internal memos and past deal information related to that customer.

[1181] Input: Customer name, Customer ID

[1182] Output: Collected customer information data

[1183] Specific operation: The server generates a database search query based on the customer name and customer ID, and searches the customer database. Furthermore, it accesses external APIs to collect the latest information.

[1184] Step 4: Information Summary

[1185] The server passes the collected information to a natural language processing engine (e.g., Google Cloud Natural Language API) to generate a summary. This summary includes the latest customer trends and key points from past deals.

[1186] Input: Collected customer information data

[1187] Output: Summarized customer information

[1188] Specific operation: The server inputs text data collected from the database into a natural language processing engine and generates summarized data.

[1189] Step 5: Providing information to users

[1190] The server sends summarized information to the user's terminal. The user can then use the provided information to prepare for the business negotiation.

[1191] Input: Summarized customer information

[1192] Output: Providing information to users

[1193] Specific operation: The server sends summary data to the user's terminal, and the user receives and displays it.

[1194] Step 6: Start the negotiation

[1195] The user starts the business negotiation and enables the recording function on their device.

[1196] Input: Instruction to start the business negotiation

[1197] Output: Enable recording function

[1198] Specific operation: When the user presses the record start button, the device starts recording and prepares to send the audio data to the server.

[1199] Step 7: Real-time transmission of audio data

[1200] The terminal transmits audio data of the business negotiation to the server in real time.

[1201] Input: Sales negotiation audio data

[1202] Output: Transmission of real-time audio data

[1203] Specific operation: The terminal divides the recorded audio and sends it to the server sequentially.

[1204] Step 8: Speech to Text

[1205] The server passes the received audio data to a real-time speech recognition engine (for example, Google Cloud Speech-to-Text) to convert the audio into text.

[1206] Input: Sales negotiation audio data

[1207] Output: Text-converted deal details

[1208] Specific operation: The server passes the audio data to the speech recognition engine and receives the output as text data.

[1209] Step 9: Keyword Extraction

[1210] The server extracts important keywords and phrases from the converted text.

[1211] Input: Text-converted sales opportunity details

[1212] Output: Extracted keywords

[1213] Specific operation: The server uses an extraction algorithm to identify and extract specific keywords or phrases.

[1214] Step 10: Real-time product proposal

[1215] The server suggests relevant products to the user in real time based on the extracted keywords. The suggestions are displayed as pop-up notifications on the device.

[1216] Input: Extracted keywords

[1217] Output: Product proposal notification

[1218] Specific operation: The server searches the product database corresponding to the extracted keywords and notifies the user's terminal of the relevant products.

[1219] Step 11: Ending the Deal

[1220] The user ends the deal and clicks the "End Deal" button.

[1221] Input: Instruction to end the business negotiation

[1222] Output: Start of deal closing process

[1223] Specific operation: When the user presses the "End Deal" button, the device stops recording and sends a notification to the server that the deal has ended.

[1224] Step 12: Send a follow-up request

[1225] The device sends a follow-up request to the server.

[1226] Input: Follow-up request

[1227] Output: Confirmation of the start of follow-up processing.

[1228] Specific operation: When the user presses the follow-up request button, the device sends a request to the server.

[1229] Step 13: Finalize the meeting minutes

[1230] The server finalizes the meeting minutes at the end of the meeting and sends them to the user's terminal. The minutes include key points and agreements from the business negotiation.

[1231] Input: Completed meeting minutes data

[1232] Output: Providing meeting minutes to users

[1233] Specific operation: The server generates meeting minutes from the text data of the business negotiation and sends them to the user's terminal.

[1234] Step 14: Re-analyzing the details of the business negotiation

[1235] The server re-analyzes the meeting minutes and sales negotiation details to generate the next proposals and schedules. It also extracts relevant information from a database of similar past sales negotiations and sends that information to the user's terminal.

[1236] Input: Meeting minutes data, sales negotiation text data

[1237] Output: Next proposal and schedule

[1238] Specific operation: The server uses meeting minutes and sales negotiation text data to generate the content and schedule for the next proposal, and collects and summarizes information on similar sales negotiations.

[1239] Step 15: Generating the next proposal and schedule

[1240] The server summarizes relevant information from a database of similar past business opportunities and generates the content and schedule for the next proposal. The user can then create a proposal based on this information.

[1241] Input: Past similar deal data

[1242] Output: Next proposal content, proposal schedule

[1243] Specific operation: The server searches the database for information on similar business opportunities and provides the user with the next proposal and its schedule.

[1244] In this way, this system enables users to efficiently utilize information at each stage—before, during, and after a business negotiation—through a series of processing steps, thereby facilitating smooth sales activities.

[1245] (Application Example 1)

[1246] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1247] During maintenance and troubleshooting on production lines within factories, there are problems with efficiently gathering and analyzing necessary information. Furthermore, the lack of means to provide useful information in real time during maintenance work can potentially reduce the accuracy and speed of the work.

[1248] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1249] In this invention, the server includes means for analyzing customer information, means for summarizing and presenting the results of the analysis of the customer information, means for generating proposal content related to the business negotiation, means for real-time speech recognition of the conversation content during the business negotiation and converting it into text, means for analyzing the conversation content and summarizing related information, means for generating and presenting meeting minutes after the business negotiation is completed, means for generating the next proposal schedule and means for extracting and presenting information on similar business negotiations, means for acquiring maintenance information based on identification information of the work target, and means for analyzing voice commands and presenting related maintenance information. This makes it possible to perform maintenance and troubleshooting work in the factory efficiently and accurately.

[1250] "Customer information" is a general term for data such as basic customer information, past transaction history, and inquiry history.

[1251] "Analysis means" refers to software or hardware configurations used to analyze collected information and generate useful insights.

[1252] A "summarization tool" is a function for concisely summarizing and presenting the analysis results.

[1253] "Proposal content generation means" refers to a system function that automatically creates proposal items related to business negotiations.

[1254] "Speech recognition means" refers to technology that converts speech data into text data.

[1255] "Text conversion means" refers to a function that converts speech into text information.

[1256] An "information summarization tool" is a function that displays collected or analyzed data in a shortened form.

[1257] A "meeting minutes generation system" is a system for recording and documenting the content of business negotiations and meetings.

[1258] The "proposal schedule generation method" is a function that plans the date for the next proposal or meeting.

[1259] The "similar business opportunity information extraction method" is a function that searches for and extracts information on similar business opportunities from a database of past business opportunities.

[1260] "Identification information" refers to information used to uniquely identify a specific work object or piece of equipment.

[1261] A "maintenance information acquisition method" is a function that acquires maintenance history and procedures related to equipment and systems based on identification information.

[1262] A "voice command analysis means" is a function that analyzes voice input and presents specific instructions or information based on its content.

[1263] This invention provides a system to support maintenance and troubleshooting within a factory. Specific embodiments are described below.

[1264] This system includes means for analyzing customer information, means for summarizing and presenting the results of the analysis of the customer information, means for generating proposals related to business negotiations, means for real-time speech recognition of conversations during business negotiations and converting them into text, means for analyzing the conversations and summarizing related information, means for generating and presenting meeting minutes after the business negotiation is completed, means for generating a schedule for the next proposal and means for extracting and presenting information on similar business negotiations, means for acquiring maintenance information based on identification information of the work target, and means for analyzing voice commands and presenting related maintenance information.

[1265] Hardware and software configuration

[1266] The hardware configuration of this system includes a microphone mounted on the robot body used in the factory, a server, and an internet connection. The software configuration includes a Python program, a speech recognition library (speech_recognition), and an HTTP request library (requests).

[1267] Process Overview

[1268] 1. Voice input recognition

[1269] The user inputs voice commands to the robot to obtain maintenance information for the object being worked on. This voice is collected via a microphone.

[1270] 2. Text conversion of audio data

[1271] The server converts the collected audio data into text data using a speech recognition library. At this stage, the sensitivity is adjusted to account for ambient noise.

[1272] 3. Information acquisition based on identification information

[1273] The server retrieves relevant maintenance information based on identification information (e.g., equipment ID) in the text data. The retrieved information is parsed in JSON format, and the necessary information is extracted.

[1274] 4. Display of maintenance information

[1275] The server provides users with acquired maintenance information in real time. This information is displayed on the robot's display and on a separate terminal, making it immediately available to the worker.

[1276] Specific example

[1277] For example, if a sudden malfunction occurs in "device 123" on a production line in a factory, the worker would input a voice command to the robot saying, "Tell me the maintenance information for device 123." The robot would recognize this command, retrieve the latest maintenance information and logs related to "device 123" from a server via the internet, and provide them to the worker.

[1278] Example of a prompt

[1279] "Please enter questions to provide appropriate maintenance and troubleshooting information for equipment problems that have occurred on the factory production line. For example, 'Please provide the latest maintenance information for equipment 123,' or 'Please provide the past trouble history for equipment 456.'"

[1280] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1281] Step 1:

[1282] The user inputs voice commands to the robot to obtain maintenance information for the object being worked on. The input voice is collected through the microphone on the robot itself.

[1283] Step 2:

[1284] The terminal (robot) sends the collected audio data to the server. The server converts this audio data into text data using a speech recognition library (speech_recognition). The input is audio data, and the output is the corresponding text data.

[1285] Step 3:

[1286] The server analyzes the converted text data and extracts identification information (e.g., device ID). This identification information is identified as a specific keyword from the input voice command, and the output is this identification information.

[1287] Step 4:

[1288] The server sends a request to the maintenance information retrieval API based on the identification information. The input is the identification information, and the output is the maintenance information response data. This response data is returned in JSON format.

[1289] Step 5:

[1290] The server parses the acquired maintenance information, extracts relevant information, and summarizes it. The input is maintenance information in JSON format, and the output is the summarized maintenance information.

[1291] Step 6:

[1292] The terminal (robot) displays summarized maintenance information sent from the server to the user. The input is summarized maintenance information, and the output is text-based information presented to the user's visual input. This information is displayed on the robot's screen and is immediately available to the worker.

[1293] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1294] This invention relates to a sales support system that combines customer information collection and analysis, real-time processing during sales negotiations, post-sales follow-up, and an emotion engine that recognizes user emotions and optimizes proposals. Detailed embodiments of this system are described below.

[1295] Customer information collection and analysis

[1296] User actions:

[1297] The user logs into the system and sends a request to retrieve information about a specific customer. The user's device sends this request to the server.

[1298] Server processing:

[1299] After receiving a customer information retrieval request, the server accesses the customer database to retrieve past sales negotiation history. Next, it gathers the latest information from external sources (news API, company website, etc.). Furthermore, it accesses the internal information database to retrieve relevant internal memos and past project information. The collected information is summarized by a natural language processing engine, and the server sends the summarized information to the user's terminal.

[1300] Specific example:

[1301] If a user has a business meeting scheduled with "Company A," the server collects Company A's past business meeting history, latest news, and relevant internal records, and provides this summarized information to the user's terminal. The user then inputs their proposal based on this information, and the server generates a proposal document based on that input.

[1302] Information processing and emotional engine during business negotiations

[1303] User actions:

[1304] The user initiates a business negotiation and activates the recording and emotion recognition functions on their device. The audio data of the negotiation and emotion data such as the user's facial expressions are transmitted to the server in real time.

[1305] Server processing:

[1306] The server converts received audio data into text using a real-time speech recognition engine. It extracts important keywords and phrases from the converted text and embeds them into a meeting minutes template. It also features an emotion engine that analyzes the user's emotional state and adjusts the proposed content in real time based on that analysis. Furthermore, the server analyzes the sales negotiation content and proposes relevant products to the user in real time.

[1307] Specific example:

[1308] When a user conducts an online business meeting with a representative from "Company A," the terminal sends audio of the meeting and the user's facial expression data to the server. The server converts the audio to text and detects statements such as "I am interested in cost reduction." In this case, if the emotion engine detects that the user's emotional state is "attractive," the server immediately presents the user with products related to cost reduction.

[1309] Post-sales follow-up and utilization of emotional data

[1310] User actions:

[1311] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[1312] Server processing:

[1313] The server finalizes the meeting minutes at the end of the meeting and sends them to the user's terminal. Next, it re-analyzes the meeting minutes and sales negotiation content to generate the next proposal and schedule. It also extracts relevant deals from a database of similar past deals, summarizes that information, and sends it to the user's terminal. Furthermore, it optimizes the proposal content based on sentiment data obtained by the sentiment engine.

[1314] Specific example:

[1315] After the user concludes a business meeting with "Company A," the server reviews the meeting minutes and presents a proposal and timeline for the next meeting. Furthermore, if there have been similar business meetings with "Company B" in the past, the server summarizes that information and provides it to the user. If the emotion engine predicts that the representative from "Company A" will have a positive sentiment towards the next meeting, the server adjusts the proposal based on that prediction. Once the user enters the proposal again, the server generates a new proposal.

[1316] In this way, this system enables users to efficiently utilize information at each stage—before, during, and after a business negotiation—and also takes into account the user's emotional state, allowing sales activities to proceed smoothly.

[1317] The following describes the processing flow.

[1318] Customer information collection and analysis

[1319] Step 1:

[1320] A user logs into the system and sends a request from their terminal to the server to retrieve information about a specific customer.

[1321] Step 2:

[1322] The server receives a request to retrieve customer information.

[1323] Step 3:

[1324] The server accesses the customer database and retrieves the customer's past sales history.

[1325] Step 4:

[1326] The server collects the latest information from external sources (news APIs, company websites, etc.).

[1327] Step 5:

[1328] The server accesses the company's internal information database to retrieve relevant internal memos and past project information.

[1329] Step 6:

[1330] The server collects information and then uses a natural language processing engine to summarize it.

[1331] Step 7:

[1332] The server sends the summarized information to the user's terminal.

[1333] Step 8:

[1334] The user reviews the summary information and enters the suggested content into their device.

[1335] Step 9:

[1336] The terminal sends the user's inputted suggestions to the server.

[1337] Step 10:

[1338] The server analyzes the proposal and generates a proposal document based on that information.

[1339] Step 11:

[1340] The server sends the generated proposal to the user's terminal.

[1341] Information processing and emotional engine during business negotiations

[1342] Step 1:

[1343] The user initiates a business negotiation and activates the recording and emotion recognition functions on their device.

[1344] Step 2:

[1345] The terminal transmits audio data of the business negotiation and emotional data such as the user's facial expressions to the server in real time.

[1346] Step 3:

[1347] The server converts the received audio data into text using a real-time speech recognition engine.

[1348] Step 4:

[1349] The server extracts important keywords and phrases from the converted text.

[1350] Step 5:

[1351] The server extracts the information and embeds it into the meeting minutes template.

[1352] Step 6:

[1353] The emotion engine analyzes the user's emotional state and adjusts the suggested content in real time based on that analysis.

[1354] Step 7:

[1355] The server analyzes the details of the business negotiation and suggests relevant products to the user in real time.

[1356] Step 8:

[1357] Users can view real-time suggestions from the server and use them immediately.

[1358] Post-sales follow-up and utilization of emotional data

[1359] Step 1:

[1360] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[1361] Step 2:

[1362] The server finalizes the meeting minutes at the time of shutdown and sends them to the user's terminal.

[1363] Step 3:

[1364] The server re-analyzes the meeting minutes and negotiation details and generates a schedule of what should be proposed next.

[1365] Step 4:

[1366] The server extracts relevant sales case examples from a database of similar past sales opportunities.

[1367] Step 5:

[1368] The server extracts similar business opportunity information, summarizes it, and sends it to the user's terminal.

[1369] Step 6:

[1370] The suggested content is optimized based on emotional data acquired by the emotion engine.

[1371] Step 7:

[1372] Based on the information from Step 6, the server generates a new proposal for the next business meeting.

[1373] Step 8:

[1374] The user enters a new proposal into the terminal for the next business meeting.

[1375] Step 9:

[1376] The terminal sends the new suggestions entered by the user to the server.

[1377] Step 10:

[1378] The server analyzes the new proposal and generates a new proposal document based on that information.

[1379] Step 11:

[1380] The server sends the new proposal to the user's terminal.

[1381] (Example 2)

[1382] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1383] Traditional sales support systems lacked the ability to integrate customer information collection, real-time information processing during negotiations, emotional data analysis, and post-sales follow-up. As a result, sales representatives had to manage each process individually, leading to decreased efficiency. Furthermore, there was no effective way to utilize emotional data during negotiations, making it difficult to respond quickly to changes in customer emotions. This resulted in insufficient optimization of proposals and a lower success rate for sales negotiations.

[1384] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1385] In this invention, the server includes means for analyzing customer information, means for summarizing and presenting the results of the analysis of the customer information, means for real-time speech recognition of conversation content during negotiations and conversion to text, and means for analyzing emotional data acquired during negotiations and adjusting the content of proposals. This makes it possible to effectively collect and analyze customer information at each stage before, during, and after negotiations, and to further optimize the content of proposals in real time by utilizing emotional data.

[1386] "Customer information" refers to data, history, and related notes about customers.

[1387] "Analysis" refers to the process of processing data and extracting meaningful information or patterns.

[1388] "Summary" refers to providing a concise overview of detailed information.

[1389] "Presentation" refers to displaying information in a way that is easy for users to understand.

[1390] "Business negotiation" refers to discussions or meetings regarding sales or purchases.

[1391] "Proposed content" refers to the products, services, or their terms and conditions presented during business negotiations.

[1392] "Speech recognition" refers to the technology that converts speech into text.

[1393] "Converting to text" refers to replacing audio data with text data.

[1394] "Conversation content" refers to the words and statements exchanged during a business negotiation.

[1395] "Emotional data" refers to information about a user's emotions obtained during a business negotiation, such as their facial expressions and voice.

[1396] "Meeting minutes" refers to a document that records and summarizes the key points of a business negotiation.

[1397] "Next proposal schedule" refers to the planned visits and proposals for the next business meeting.

[1398] "Information on similar deals" refers to data and history related to similar deals that have taken place in the past.

[1399] "Real-time" refers to processing being performed at the very moment an event occurs.

[1400] "Adjustment" refers to changing the content or method depending on the situation.

[1401] This invention relates to a sales support system that combines customer information collection and analysis, real-time processing during sales negotiations, post-sales follow-up, and an emotion engine that recognizes user emotions and optimizes proposals. Specific embodiments for carrying out the invention are described below.

[1402] Customer information collection and analysis

[1403] User actions:

[1404] The user logs into the system and sends a request to retrieve information about a specific customer. The user's device sends this request to the server.

[1405] Server processing:

[1406] After receiving a customer information retrieval request, the server accesses the customer database to retrieve past sales history. Next, it gathers the latest information from external sources (e.g., news APIs or company websites). Furthermore, it accesses the internal information database to retrieve relevant internal memos and past project information. This information is summarized by a natural language processing engine (NLP engine), and the server sends the summarized information to the user's terminal.

[1407] Specific example:

[1408] If a user has an upcoming business meeting with "Company A," they log into the system and request customer information. The server retrieves "Company A's" past business meeting history, latest news, and related internal records, and summarizes the information using an NLP engine. The summarized information is then provided to the user's device. For example, the Google News API could be used as the news API, and a CRM system could be used for internal information management.

[1409] Information processing and emotional engine during business negotiations

[1410] User actions:

[1411] The user initiates a business negotiation and activates the recording and emotion recognition functions on their device. The audio data of the negotiation and the user's emotion data (facial expression data, etc.) are transmitted to the server in real time.

[1412] Server processing:

[1413] The server converts received audio data into text using a real-time speech recognition engine. It extracts important keywords and phrases from the converted text and embeds them into a meeting minutes template. The server also has the capability to analyze the user's emotional state using an emotion recognition engine and adjust the proposed content in real time. Furthermore, it analyzes the content of business negotiations and proposes relevant products and services to the user in real time.

[1414] Specific example:

[1415] When a user conducts an online business meeting with a representative from "Company A," the user enables recording and emotion recognition features in advance. The device sends the audio of the meeting and the user's facial expression data to the server. The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the audio to text and extract keywords. For example, if the representative says, "I'm interested in cost reduction," the server uses an emotion recognition algorithm to determine the emotional state is "attractive" and quickly presents the user with product information related to cost reduction.

[1416] Post-sales follow-up and utilization of emotional data

[1417] User actions:

[1418] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[1419] Server processing:

[1420] The server finalizes the meeting minutes at the end of the business negotiation and sends them to the user's terminal. Next, it re-analyzes the minutes and negotiation details to generate the next proposed content and schedule. Furthermore, it extracts relevant information from a database of similar past negotiations and provides a summary to the user's terminal. Finally, it optimizes the proposed content based on sentiment data obtained by the sentiment recognition engine.

[1421] Specific example:

[1422] After the user concludes a business meeting with "Company A," the server finalizes the meeting minutes and sends them to the user. The server searches for similar past business meeting data and provides a summary of successful cases, such as those with "Company B." It also generates and proposes a schedule for the next visit and proposals to the user. The emotion engine determines that the emotional state of the "Company A" representative is positive and adjusts the next proposal based on that prediction.

[1423] In this way, this system enables users to efficiently utilize information at each stage—before, during, and after a business negotiation—and also takes into account the user's emotional state, allowing sales activities to proceed smoothly.

[1424] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1425] Customer information collection and analysis

[1426] Step 1:

[1427] The user logs into the system and sends a request to retrieve information about a specific customer. Specifically, the user clicks the "Retrieve Customer Information" button on the terminal's operation screen and enters information such as the company name and customer name. The terminal sends this request data to the server. The input is "Company Name: Company A" and the output is "Customer Information Retrieval Request".

[1428] Step 2:

[1429] The server receives a customer information retrieval request. The server first accesses the customer database to retrieve past sales negotiation history for "Company A". At this stage, the input is the "customer information retrieval request" and the output is the "past sales negotiation history".

[1430] Step 3:

[1431] The server accesses external sources (such as news APIs and company websites) to collect the latest information related to "Company A". At this stage, the input is "Company Name: Company A" and the output is "Latest News". Specifically, the server calls APIs to collect data and stores it internally.

[1432] Step 4:

[1433] The server accesses the company's internal information database to retrieve internal memos and past project information related to "Company A". At this stage, the input is "Company Name: Company A", and the output is "Internal Memos and Project Information". Specifically, it executes database queries to extract the necessary information.

[1434] Step 5:

[1435] The collected information is summarized by a natural language processing engine (NLP engine). The server uses past sales history, latest news, and internal memos as input data to perform summarization and generate the results. At this stage, the input is "all collected information," and the output is "summarized customer information." Specifically, the NLP engine's algorithm is executed to generate the summary.

[1436] Step 6:

[1437] The server sends summarized information to the user's terminal. At this stage, the input is "summarized customer information," and the output is "customer information displayed on the user's terminal." Specifically, the server sends an HTTP response, and the terminal displays that information.

[1438] ---

[1439] Information processing and emotional engine during business negotiations

[1440] Step 1:

[1441] The user initiates a business meeting and activates the recording and emotion recognition functions on their device. The user clicks the "Start Recording" and "Start Emotion Recognition" buttons on the device. The input is "Notification of Business Meeting Start," and the output is "Start of Recording and Emotion Recognition."

[1442] Step 2:

[1443] The terminal transmits audio and facial expression data from the business negotiation to the server in real time. At this stage, the input is "audio and facial expression data," and the output is "data sent to the server." Specifically, the terminal transmits the data in real time using streaming technology.

[1444] Step 3:

[1445] The server converts the received audio data into text using a real-time speech recognition engine. At this stage, the input is "audio data" and the output is "converted text". Specifically, it calls the Google Cloud Speech-to-Text API to convert audio to text.

[1446] Step 4:

[1447] The server extracts important keywords and phrases from the text and embeds them into a meeting minutes template. At this stage, the input is the "converted text," and the output is the "keywords embedded in the meeting minutes." Specifically, a text analysis algorithm is executed to extract keywords.

[1448] Step 5:

[1449] The server uses an emotion recognition engine to analyze the user's emotional state and adjusts the suggestions in real time. At this stage, the input is "facial expression data and converted text," and the output is "adjusted suggestions." Specifically, it uses an emotion recognition algorithm to analyze the emotional state and adjusts the suggestions based on that data.

[1450] Step 6:

[1451] The server analyzes the details of the business negotiation and proposes relevant products and services to the user's terminal in real time. The input at this stage is "text data of the adjusted proposal and business negotiation," and the output is "product information presented to the user." Specifically, it queries a related product database to generate appropriate proposals.

[1452] ---

[1453] Post-sales follow-up and utilization of emotional data

[1454] Step 1:

[1455] The user concludes the business negotiation and sends a follow-up request from their device to the server. The input is "Follow-up notification," and the output is "Follow-up request sent."

[1456] Step 2:

[1457] The server finalizes the meeting minutes and sends them to the user's terminal. At this stage, the input is "meeting minutes created in real time," and the output is "sending the finalized meeting minutes to the user." Specifically, it completes the meeting minutes template and provides it to the user.

[1458] Step 3:

[1459] The server re-analyzes the meeting minutes and negotiation details to generate the next proposal and schedule. At this stage, the input is the "finalized meeting minutes and negotiation details," and the output is the "next proposal and schedule." Specifically, it runs a data analysis algorithm to plan the next proposal.

[1460] Step 4:

[1461] The server checks a database of similar past deals, extracts relevant information, and provides it to the user. At this stage, the input is "finalized meeting minutes and deal details," and the output is "provision of information on similar deals." Specifically, it performs a database search to identify similar past cases, summarizes that information, and provides it.

[1462] Step 5:

[1463] The server optimizes the next proposal based on sentiment data. At this stage, the input is "sentiment data and finalized meeting minutes," and the output is "optimized next proposal." Specifically, the proposal is readjusted based on sentiment analysis data.

[1464] Step 6:

[1465] The server sends the optimized suggestions to the user's terminal. At this stage, the input is the "optimized next suggestion," and the output is the "optimized suggestion presented to the user." Specifically, the server sends an HTTP response, and the terminal displays that information.

[1466] As described above, by clearly defining the specific operation and input / output of the system at each step, and by clearly indicating the roles of the user, terminal, and server, the present invention can be effectively implemented.

[1467] (Application Example 2)

[1468] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1469] In modern brick-and-mortar stores, customer service often relies heavily on the experience and skills of store staff, leading to inconsistencies in customer satisfaction. Furthermore, accurately understanding customer needs and emotions in real time and providing optimal suggestions based on that understanding is challenging. Traditional systems fail to adequately optimize suggestions based on customer emotions, making efficient sales support difficult. Moreover, real-time information processing during sales negotiations and effective follow-up after negotiations are often lacking.

[1470] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1471] In this invention, the server includes means for analyzing customer information, means for summarizing and presenting the results of the customer information analysis, means for generating proposal content related to the business negotiation, means for real-time speech recognition and text conversion of conversation content during the business negotiation, means for analyzing the conversation content and summarizing related information, means for analyzing video data during the business negotiation to recognize the user's emotions, means for optimizing the proposal content based on the user's emotion data, means for generating and presenting meeting minutes after the business negotiation is completed, means for generating a schedule for the next proposal, and means for extracting and presenting information on similar business negotiations. This makes it possible to grasp customer needs and emotions in real time at physical stores and make optimal proposals. Furthermore, it is expected that follow-up during and after business negotiations can be efficiently carried out, leading to improved customer satisfaction and sales efficiency.

[1472] "Customer information" refers to data and historical information about customers, including, for example, past purchase history, inquiry history, and personal profile information.

[1473] "Means of analysis" refers to methods and techniques for analyzing collected data and extracting useful information from it. Specifically, this includes machine learning algorithms and statistical analysis techniques.

[1474] "Speech recognition" is a technology that converts speech data into text data, and includes the process of automatically transcribing the user's speech into text.

[1475] "Means of converting to text" refers to technologies and systems for converting collected audio data into text, such as speech recognition engines.

[1476] "Methods of summarization" refer to methods and techniques for organizing analyzed information into a concise and easily understandable form. Natural language processing and summarization algorithms are used in this process.

[1477] "Means of recognizing emotions" refers to technologies that determine a user's emotional state at a given time from their video and audio data. This includes emotion recognition engines and image processing technologies.

[1478] "Methods for optimizing suggestions" refer to technologies that generate the most suitable suggestions by considering user sentiment data and other factors. This includes recommendation algorithms and personalization engines.

[1479] "Means for generating and presenting meeting minutes" refers to technologies and systems that automatically create meeting minutes based on text data recorded during business negotiations and provide them to users.

[1480] "Methods for generating proposal schedules" refer to technologies that generate the content and schedule of the next proposal based on the details of the business negotiation and past history. This includes scheduling management systems and task management tools.

[1481] "Methods for extracting and presenting information on similar business opportunities" refers to technologies that extract cases similar to the current business opportunity from a database of past business opportunities and provide them as reference information. Data mining and clustering algorithms are used in this process.

[1482] In this invention, to specifically realize a sales support system for physical stores, smart glasses, a server, a speech recognition engine, a natural language processing engine, an emotion recognition engine, and a recommendation algorithm work together as a single unit. The operation of this system will be described below in order.

[1483] System configuration and operation

[1484] 1. Data Collection

[1485] The user (staff member) wears smart glasses.

[1486] When a conversation with a customer begins, the smart glasses collect audio and video data in real time and send it to the server.

[1487] 2. Data Processing

[1488] The server converts the received audio data into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[1489] The converted text data is sent to a natural language processing engine (e.g., a BERT model) where important keywords and phrases are extracted.

[1490] The video data is used to analyze the emotional state of customers using an emotion recognition engine (e.g., Microsoft Azure Face API).

[1491] Based on extracted keywords and sentiment data, a recommendation algorithm (e.g., collaborative filtering algorithm) is used to select the most suitable product suggestions.

[1492] 3. Data Output

[1493] The selected product information is displayed on the smart glasses' screen and presented to the user (staff).

[1494] The user suggests selected products to customers who visit the store.

[1495] Specific example

[1496] 1. Examples of customer service

[1497] When a user asks, "Hello, what can I help you with?", the customer replies, "I've come to look at winter coats."

[1498] The smart glasses send the audio data to a server, where speech recognition and natural language processing extract the keyword "winter coat."

[1499] The emotion recognition engine detects emotions such as "excitement" and "interest" from video data of customers visiting the store.

[1500] A recommendation algorithm selects multiple coats based on inventory information and popular product data, and displays them on smart glasses.

[1501] A user who has checked the display of smart glasses suggests to a customer, "These two coats are very popular."

[1502] Example of a prompt

[1503] "Enter the following user voice data into the speech recognition engine: Hello, I'm looking for a winter coat."

[1504] "The converted text data is passed to a natural language processing engine to extract keywords."

[1505] "Video data is input into an emotion recognition engine to detect the user's emotional state."

[1506] "Next, the extracted keywords and sentiment data are passed to a product recommendation algorithm to select the most suitable product."

[1507] In this way, it becomes possible to provide optimal suggestions tailored to customer needs in physical stores in real time, thereby improving customer satisfaction.

[1508] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1509] Step 1:

[1510] The user (staff member) wears smart glasses and begins a conversation with a customer. The smart glasses collect audio and video data in real time and send it to the server. The input is the audio and video data of the conversation with the customer, and the output is sending this data to the server.

[1511] Step 2:

[1512] The server converts the received audio data into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text). Audio data is the input, and text data is the output. Specifically, the audio data is analyzed, and the spoken content is extracted as a string of characters.

[1513] Step 3:

[1514] The server passes the converted text data to a natural language processing engine (e.g., a BERT model) to extract important keywords and phrases. The input is text data, and the output is important keywords and phrases. Specifically, it analyzes the context of the text and extracts information relevant to the business deal.

[1515] Step 4:

[1516] Video data is sent to a server, which uses an emotion recognition engine (e.g., Microsoft Azure Face API) to analyze the user's emotional state. The input is video data, and the output is recognized emotion data. Specifically, emotions are determined from facial expressions, and levels of excitement and interest are quantified.

[1517] Step 5:

[1518] The server uses a recommendation algorithm (e.g., collaborative filtering algorithm) to select the most suitable product based on extracted keywords and sentiment data. The input is keywords and sentiment data, and the output is a list of recommended products. Specifically, it refers to past sales history and inventory information to select the most relevant products.

[1519] Step 6:

[1520] The server displays selected product information on the smart glasses' screen and suggests it to the user (staff). The input is a list of recommended products, and the output is the product information displayed on the smart glasses' screen. Specifically, it provides the user with optimized suggestions immediately.

[1521] Step 7:

[1522] The user (staff) checks the information displayed on the smart glasses and makes suggestions to customers. The input is product information displayed on the screen, and the output is suggestions to customers. Specifically, the staff makes verbal suggestions based on the information they visually confirm.

[1523] In this way, the system of the present invention can efficiently and effectively handle customer interactions in physical stores.

[1524] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1525] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1526] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1527] [Fourth Embodiment]

[1528] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1529] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1530] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1531] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1532] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1533] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1534] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1535] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1536] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1537] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1538] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1539] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1540] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1541] This invention relates to a sales support system for efficiently collecting and analyzing customer information, processing sales negotiations in real time, and following up after sales negotiations. Detailed embodiments of this system are described below.

[1542] Customer information collection and analysis

[1543] User actions:

[1544] The user logs into the system and sends a request to retrieve information about a specific customer. The user's device sends this request to the server.

[1545] Server processing:

[1546] After receiving a customer information retrieval request, the server accesses the customer database to retrieve past sales negotiation history. Next, it gathers the latest information from external sources (news API, company website, etc.). Furthermore, it accesses the internal information database to retrieve relevant internal memos and past project information. The collected information is summarized by a natural language processing engine, and the server sends the summarized information to the user's terminal.

[1547] Specific example:

[1548] If a user has an upcoming business meeting with "Company A," the server collects and summarizes Company A's past business meeting history, latest news, and relevant internal records, and provides this information to the user's terminal. The user then inputs their proposal based on this information, and the server generates a proposal document based on that input.

[1549] Information processing during business negotiations

[1550] User actions:

[1551] The user initiates a business meeting and activates the recording function on their device. The audio data of the meeting is transmitted to the server in real time.

[1552] Server processing:

[1553] The server converts received audio data into text using a real-time speech recognition engine. It extracts important keywords and phrases from the converted text and embeds them into a meeting minutes template. The server further analyzes the sales meeting content and suggests relevant products to the user in real time.

[1554] Specific example:

[1555] When a user conducts an online business meeting with a representative from "Company A," the terminal sends the audio of the meeting to the server. The server converts the audio to text and detects statements such as "I am interested in cost reduction." In this case, the server immediately proposes products related to cost reduction to the user.

[1556] Follow-up after business negotiations

[1557] User actions:

[1558] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[1559] Server processing:

[1560] The server finalizes the meeting minutes at the end of the meeting and sends them to the user's terminal. Next, it re-analyzes the meeting minutes and sales negotiation details to generate the next proposed content and schedule. It also extracts relevant sales opportunities from a database of similar past sales opportunities, summarizes that information, and sends it to the user's terminal.

[1561] Specific example:

[1562] After the user concludes a business negotiation with "Company A," the server reviews the meeting minutes and presents the content and timeline for the next proposal. Furthermore, if there have been similar negotiations with "Company B" in the past, the server summarizes that information and provides it to the user. When the user enters proposal details again, the server generates a new proposal document.

[1563] In this way, this system enables users to efficiently utilize information at each stage—before, during, and after a business negotiation—and to conduct sales activities smoothly.

[1564] The following describes the processing flow.

[1565] Customer information collection and analysis

[1566] Step 1:

[1567] A user logs into the system and sends a request from their terminal to the server to retrieve information about a specific customer.

[1568] Step 2:

[1569] The server receives a request to retrieve customer information.

[1570] Step 3:

[1571] The server accesses the customer database and retrieves the customer's past sales history.

[1572] Step 4:

[1573] The server collects the latest information from external sources (news APIs, company websites, etc.).

[1574] Step 5:

[1575] The server accesses the company's internal information database to retrieve relevant internal memos and past project information.

[1576] Step 6:

[1577] The server collects information and then uses a natural language processing engine to summarize it.

[1578] Step 7:

[1579] The server sends the summarized information to the user's terminal.

[1580] Step 8:

[1581] The user reviews the summary information and enters the suggested content into their device.

[1582] Step 9:

[1583] The terminal sends the user's inputted suggestions to the server.

[1584] Step 10:

[1585] The server analyzes the proposal and generates a proposal document based on that information.

[1586] Step 11:

[1587] The server sends the generated proposal to the user's terminal.

[1588] Information processing during business negotiations

[1589] Step 1:

[1590] The user initiates a business meeting and activates the recording function on their device.

[1591] Step 2:

[1592] The terminal transmits the audio data of the business negotiation to the server in real time.

[1593] Step 3:

[1594] The server converts the received audio data into text using a real-time speech recognition engine.

[1595] Step 4:

[1596] The server extracts important keywords and phrases from the converted text.

[1597] Step 5:

[1598] The server extracts the information and embeds it into the meeting minutes template.

[1599] Step 6:

[1600] The server analyzes the details of the business negotiation and suggests relevant products to the user in real time.

[1601] Step 7:

[1602] Users can view real-time suggestions from the server and use them immediately.

[1603] Follow-up after business negotiations

[1604] Step 1:

[1605] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[1606] Step 2:

[1607] The server finalizes the meeting minutes at the time of shutdown and sends them to the user's terminal.

[1608] Step 3:

[1609] The server re-analyzes the meeting minutes and negotiation details and generates a schedule of what should be proposed next.

[1610] Step 4:

[1611] The server extracts relevant sales case examples from a database of similar past sales opportunities.

[1612] Step 5:

[1613] The server extracts similar business opportunity information, summarizes it, and sends it to the user's terminal.

[1614] Step 6:

[1615] The user enters a new proposal into the terminal for the next business meeting.

[1616] Step 7:

[1617] The terminal sends the new suggestions entered by the user to the server.

[1618] Step 8:

[1619] The server analyzes the new proposal and generates a new proposal document based on that information.

[1620] Step 9:

[1621] The server sends the new proposal to the user's terminal.

[1622] (Example 1)

[1623] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1624] In conventional sales support systems, customer information collection and analysis, real-time processing during negotiations, and post-negotiation follow-up were all handled independently, making it difficult to utilize information consistently and conduct efficient sales activities. Furthermore, there was a need for improved accuracy in real-time proposals during negotiations and in the content of follow-up proposals after negotiations concluded.

[1625] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1626] In this invention, the server includes means for analyzing customer information, means for summarizing and presenting the results of the customer information analysis, means for receiving information collection requests and collecting information from a customer database and external information sources, means for summarizing the collected information using a natural language processing engine, means for generating proposals related to business negotiations, means for real-time speech recognition and text conversion of conversation content during business negotiations, means for analyzing conversation content, extracting important keywords, and proposing related products, means for generating and presenting meeting minutes after the conclusion of business negotiations, means for generating a schedule for the next proposal, and means for extracting and presenting information on similar business negotiations. This makes it possible to efficiently utilize information at each stage before, during, and after business negotiations and to smoothly carry out sales activities.

[1627] "Customer data analysis" is the process of collecting data about customers and analyzing patterns and trends.

[1628] A "summary of customer information analysis results" is a concise compilation of the most important points extracted from the analyzed information.

[1629] An "information gathering request" is the process of requesting detailed information or up-to-date data about a specific customer.

[1630] A "customer database" is a data storage system that aggregates and stores detailed information about customers.

[1631] "External information sources" refer to information providers other than internal databases, such as company websites and news APIs.

[1632] A "natural language processing engine" is software that analyzes text data and executes algorithms to understand and generate human language.

[1633] "Generating proposals related to business negotiations" is the process of creating the optimal proposal based on the objectives of the negotiation and the customer's needs.

[1634] "Real-time speech recognition" is a technology that instantly converts audio generated during business negotiations into text data.

[1635] "Keyword extraction" is the process of selecting particularly important words and phrases from conversations or texts.

[1636] "Proposing products or services" refers to suggesting suitable products or services based on the needs identified during a business negotiation.

[1637] "Generating meeting minutes after a business negotiation" is the process of organizing the content and conclusions of a business negotiation and documenting them so that they can be referenced later.

[1638] "Generating the next proposal schedule" is the process of planning what should be proposed next and when.

[1639] "Extracting information on similar business opportunities" means detecting cases similar to the current business opportunity from past business opportunity history and reusing that information.

[1640] The present invention relates to a sales support system for efficiently collecting and analyzing customer information, processing sales negotiations in real time, and following up after sales negotiations. This system includes means for analyzing customer information, means for summarizing and presenting the analysis results, means for generating proposals related to sales negotiations, means for real-time speech recognition and text conversion of conversations during sales negotiations, means for analyzing conversations and summarizing related information, means for generating and presenting meeting minutes after the sales negotiation is completed, means for generating a schedule for the next proposal, and means for extracting and presenting information on similar sales negotiations.

[1641] Customer information collection and analysis

[1642] User actions:

[1643] The user logs into the sales support system and sends a request to retrieve information about a specific customer. The user's device sends this request to the server.

[1644] Server processing:

[1645] After receiving a customer information retrieval request, the server accesses the customer database to retrieve past sales history. Next, it gathers the latest information from external sources such as news APIs and company websites. Furthermore, it accesses internal information databases to retrieve relevant internal memos and past project information. The collected information is summarized by a natural language processing engine (e.g., Google Cloud Natural Language API), and the server sends the summarized information to the user's terminal.

[1646] Specific example:

[1647] For example, if a user has a business meeting scheduled with "Company A," the server collects past business meeting history, latest news, and relevant internal records of "Company A," and provides summarized information to the user's terminal. The user then inputs their proposal based on this information, and the server generates a proposal document based on that input.

[1648] Examples of prompts for a generative AI model:

[1649] "Please provide a summary of past business negotiations with a certain company, the latest news, and relevant internal memos."

[1650] Information processing during business negotiations

[1651] User actions:

[1652] The user initiates the business negotiation and activates the recording function on their device.

[1653] Server processing:

[1654] The terminal transmits audio data of the business negotiation to the server in real time. The server passes the received audio data to a real-time speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the audio to text. It extracts important keywords and phrases from the converted text and suggests relevant products to the user in real time based on that.

[1655] Specific example:

[1656] When a user conducts an online business meeting with a representative from "a certain company," the terminal transmits the audio of the meeting to the server. The server converts the audio to text and detects statements such as "I am interested in cost reduction." In this case, the server immediately proposes products related to cost reduction to the user.

[1657] Examples of prompts for a generative AI model:

[1658] "Extract key keywords from the following text and propose corresponding products / services: 'A representative from a certain company has stated they are interested in cost reduction.'"

[1659] Follow-up after business negotiations

[1660] User actions:

[1661] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[1662] Server processing:

[1663] The server finalizes the meeting minutes at the end of the meeting and sends them to the user's terminal. Next, it re-analyzes the minutes and negotiation details to generate the next proposal and schedule. It also extracts relevant deals from a database of similar past deals, summarizes that information, and sends it to the user's terminal. The user can then create a proposal based on the next proposal.

[1664] Specific example:

[1665] After the user concludes a business negotiation with "a certain company," the server reviews the meeting minutes and presents the content and timeline for the next proposal. Furthermore, if there have been similar negotiations with "other companies" in the past, the server summarizes that information and provides it to the user. Based on this, the user creates a new proposal.

[1666] Examples of prompts for a generative AI model:

[1667] "Please generate the next proposal and timeline based on the following meeting minutes: 'We need a proposal regarding cost reduction in negotiations with a certain company. Please provide a schedule for creating the next proposal.'"

[1668] In this way, this system enables users to efficiently utilize information at each stage—before, during, and after a business negotiation—and to conduct sales activities smoothly.

[1669] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1670] Step 1: User Login

[1671] The user accesses the login screen of the sales support system and enters their authentication information (user ID and password). The terminal sends this information to the server, which verifies it against the database to perform authentication. If authentication is successful, the user's session begins.

[1672] Input: User ID, Password

[1673] Output: Authentication result (success or failure)

[1674] Specific operation: When a user enters their user ID and password and presses the login button, the terminal sends the authentication information to the server, which then authenticates the user by comparing it with the information stored in the database.

[1675] Step 2: Customer Information Request

[1676] The user enters the customer's name and customer ID to retrieve information about a specific customer. The terminal sends this request to the server, and the server receives the request.

[1677] Input: Customer name, Customer ID

[1678] Output: Confirmation of receipt of customer information retrieval request

[1679] Specific operation: When the user enters the customer name and customer ID and presses the information retrieval button, the terminal sends that information to the server. The server confirms the request and proceeds to the next processing step.

[1680] Step 3: Information Gathering

[1681] Based on the received request, the server first accesses the company's internal customer database to retrieve the customer's past sales history. Next, it sends queries to external sources such as news APIs and company websites to gather the latest information. It also accesses the internal information database to retrieve internal memos and past deal information related to that customer.

[1682] Input: Customer name, Customer ID

[1683] Output: Collected customer information data

[1684] Specific operation: The server generates a database search query based on the customer name and customer ID, and searches the customer database. Furthermore, it accesses external APIs to collect the latest information.

[1685] Step 4: Information Summary

[1686] The server passes the collected information to a natural language processing engine (e.g., Google Cloud Natural Language API) to generate a summary. This summary includes the latest customer trends and key points from past deals.

[1687] Input: Collected customer information data

[1688] Output: Summarized customer information

[1689] Specific operation: The server inputs text data collected from the database into a natural language processing engine and generates summarized data.

[1690] Step 5: Providing information to users

[1691] The server sends summarized information to the user's terminal. The user can then use the provided information to prepare for the business negotiation.

[1692] Input: Summarized customer information

[1693] Output: Providing information to users

[1694] Specific operation: The server sends summary data to the user's terminal, and the user receives and displays it.

[1695] Step 6: Start the negotiation

[1696] The user starts the business negotiation and enables the recording function on their device.

[1697] Input: Instruction to start the business negotiation

[1698] Output: Enable recording function

[1699] Specific operation: When the user presses the record start button, the device starts recording and prepares to send the audio data to the server.

[1700] Step 7: Real-time transmission of audio data

[1701] The terminal transmits audio data of the business negotiation to the server in real time.

[1702] Input: Sales negotiation audio data

[1703] Output: Transmission of real-time audio data

[1704] Specific operation: The terminal divides the recorded audio and sends it to the server sequentially.

[1705] Step 8: Speech to Text

[1706] The server passes the received audio data to a real-time speech recognition engine (for example, Google Cloud Speech-to-Text) to convert the audio into text.

[1707] Input: Sales negotiation audio data

[1708] Output: Text-converted deal details

[1709] Specific operation: The server passes the audio data to the speech recognition engine and receives the output as text data.

[1710] Step 9: Keyword Extraction

[1711] The server extracts important keywords and phrases from the converted text.

[1712] Input: Text-converted sales opportunity details

[1713] Output: Extracted keywords

[1714] Specific operation: The server uses an extraction algorithm to identify and extract specific keywords or phrases.

[1715] Step 10: Real-time product proposal

[1716] The server suggests relevant products to the user in real time based on the extracted keywords. The suggestions are displayed as pop-up notifications on the device.

[1717] Input: Extracted keywords

[1718] Output: Product proposal notification

[1719] Specific operation: The server searches the product database corresponding to the extracted keywords and notifies the user's terminal of the relevant products.

[1720] Step 11: Ending the Deal

[1721] The user ends the deal and clicks the "End Deal" button.

[1722] Input: Instruction to end the business negotiation

[1723] Output: Start of deal closing process

[1724] Specific operation: When the user presses the "End Deal" button, the device stops recording and sends a notification to the server that the deal has ended.

[1725] Step 12: Send a follow-up request

[1726] The device sends a follow-up request to the server.

[1727] Input: Follow-up request

[1728] Output: Confirmation of the start of follow-up processing.

[1729] Specific operation: When the user presses the follow-up request button, the device sends a request to the server.

[1730] Step 13: Finalize the meeting minutes

[1731] The server finalizes the meeting minutes at the end of the meeting and sends them to the user's terminal. The minutes include key points and agreements from the business negotiation.

[1732] Input: Completed meeting minutes data

[1733] Output: Providing meeting minutes to users

[1734] Specific operation: The server generates meeting minutes from the text data of the business negotiation and sends them to the user's terminal.

[1735] Step 14: Re-analyzing the details of the business negotiation

[1736] The server re-analyzes the meeting minutes and sales negotiation details to generate the next proposals and schedules. It also extracts relevant information from a database of similar past sales negotiations and sends that information to the user's terminal.

[1737] Input: Meeting minutes data, sales negotiation text data

[1738] Output: Next proposal and schedule

[1739] Specific operation: The server uses meeting minutes and sales negotiation text data to generate the content and schedule for the next proposal, and collects and summarizes information on similar sales negotiations.

[1740] Step 15: Generating the next proposal and schedule

[1741] The server summarizes relevant information from a database of similar past business opportunities and generates the content and schedule for the next proposal. The user can then create a proposal based on this information.

[1742] Input: Past similar deal data

[1743] Output: Next proposal content, proposal schedule

[1744] Specific operation: The server searches the database for information on similar business opportunities and provides the user with the next proposal and its schedule.

[1745] In this way, this system enables users to efficiently utilize information at each stage—before, during, and after a business negotiation—through a series of processing steps, thereby facilitating smooth sales activities.

[1746] (Application Example 1)

[1747] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1748] During maintenance and troubleshooting on production lines within factories, there are problems with efficiently gathering and analyzing necessary information. Furthermore, the lack of means to provide useful information in real time during maintenance work can potentially reduce the accuracy and speed of the work.

[1749] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1750] In this invention, the server includes means for analyzing customer information, means for summarizing and presenting the results of the analysis of the customer information, means for generating proposal content related to the business negotiation, means for real-time speech recognition of the conversation content during the business negotiation and converting it into text, means for analyzing the conversation content and summarizing related information, means for generating and presenting meeting minutes after the business negotiation is completed, means for generating the next proposal schedule and means for extracting and presenting information on similar business negotiations, means for acquiring maintenance information based on identification information of the work target, and means for analyzing voice commands and presenting related maintenance information. This makes it possible to perform maintenance and troubleshooting work in the factory efficiently and accurately.

[1751] "Customer information" is a general term for data such as basic customer information, past transaction history, and inquiry history.

[1752] "Analysis means" refers to software or hardware configurations used to analyze collected information and generate useful insights.

[1753] A "summarization tool" is a function for concisely summarizing and presenting the analysis results.

[1754] "Proposal content generation means" refers to a system function that automatically creates proposal items related to business negotiations.

[1755] "Speech recognition means" refers to technology that converts speech data into text data.

[1756] "Text conversion means" refers to a function that converts speech into text information.

[1757] An "information summarization tool" is a function that displays collected or analyzed data in a shortened form.

[1758] A "meeting minutes generation system" is a system for recording and documenting the content of business negotiations and meetings.

[1759] The "proposal schedule generation method" is a function that plans the date for the next proposal or meeting.

[1760] The "similar business opportunity information extraction method" is a function that searches for and extracts information on similar business opportunities from a database of past business opportunities.

[1761] "Identification information" refers to information used to uniquely identify a specific work object or piece of equipment.

[1762] A "maintenance information acquisition method" is a function that acquires maintenance history and procedures related to equipment and systems based on identification information.

[1763] A "voice command analysis means" is a function that analyzes voice input and presents specific instructions or information based on its content.

[1764] This invention provides a system to support maintenance and troubleshooting within a factory. Specific embodiments are described below.

[1765] This system includes means for analyzing customer information, means for summarizing and presenting the results of the analysis of the customer information, means for generating proposals related to business negotiations, means for real-time speech recognition of conversations during business negotiations and converting them into text, means for analyzing the conversations and summarizing related information, means for generating and presenting meeting minutes after the business negotiation is completed, means for generating a schedule for the next proposal and means for extracting and presenting information on similar business negotiations, means for acquiring maintenance information based on identification information of the work target, and means for analyzing voice commands and presenting related maintenance information.

[1766] Hardware and software configuration

[1767] The hardware configuration of this system includes a microphone mounted on the robot body used in the factory, a server, and an internet connection. The software configuration includes a Python program, a speech recognition library (speech_recognition), and an HTTP request library (requests).

[1768] Process Overview

[1769] 1. Voice input recognition

[1770] The user inputs voice commands to the robot to obtain maintenance information for the object being worked on. This voice is collected via a microphone.

[1771] 2. Text conversion of audio data

[1772] The server converts the collected audio data into text data using a speech recognition library. At this stage, the sensitivity is adjusted to account for ambient noise.

[1773] 3. Information acquisition based on identification information

[1774] The server retrieves relevant maintenance information based on identification information (e.g., equipment ID) in the text data. The retrieved information is parsed in JSON format, and the necessary information is extracted.

[1775] 4. Display of maintenance information

[1776] The server provides users with acquired maintenance information in real time. This information is displayed on the robot's display and on a separate terminal, making it immediately available to the worker.

[1777] Specific example

[1778] For example, if a sudden malfunction occurs in "device 123" on a production line in a factory, the worker would input a voice command to the robot saying, "Tell me the maintenance information for device 123." The robot would recognize this command, retrieve the latest maintenance information and logs related to "device 123" from a server via the internet, and provide them to the worker.

[1779] Example of a prompt

[1780] "Please enter questions to provide appropriate maintenance and troubleshooting information for equipment problems that have occurred on the factory production line. For example, 'Please provide the latest maintenance information for equipment 123,' or 'Please provide the past trouble history for equipment 456.'"

[1781] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1782] Step 1:

[1783] The user inputs voice commands to the robot to obtain maintenance information for the object being worked on. The input voice is collected through the microphone on the robot itself.

[1784] Step 2:

[1785] The terminal (robot) sends the collected audio data to the server. The server converts this audio data into text data using a speech recognition library (speech_recognition). The input is audio data, and the output is the corresponding text data.

[1786] Step 3:

[1787] The server analyzes the converted text data and extracts identification information (e.g., device ID). This identification information is identified as a specific keyword from the input voice command, and the output is this identification information.

[1788] Step 4:

[1789] The server sends a request to the maintenance information retrieval API based on the identification information. The input is the identification information, and the output is the maintenance information response data. This response data is returned in JSON format.

[1790] Step 5:

[1791] The server parses the acquired maintenance information, extracts relevant information, and summarizes it. The input is maintenance information in JSON format, and the output is the summarized maintenance information.

[1792] Step 6:

[1793] The terminal (robot) displays summarized maintenance information sent from the server to the user. The input is summarized maintenance information, and the output is text-based information presented to the user's visual input. This information is displayed on the robot's screen and is immediately available to the worker.

[1794] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1795] This invention relates to a sales support system that combines customer information collection and analysis, real-time processing during sales negotiations, post-sales follow-up, and an emotion engine that recognizes user emotions and optimizes proposals. Detailed embodiments of this system are described below.

[1796] Customer information collection and analysis

[1797] User actions:

[1798] The user logs into the system and sends a request to retrieve information about a specific customer. The user's device sends this request to the server.

[1799] Server processing:

[1800] After receiving a customer information retrieval request, the server accesses the customer database to retrieve past sales negotiation history. Next, it gathers the latest information from external sources (news API, company website, etc.). Furthermore, it accesses the internal information database to retrieve relevant internal memos and past project information. The collected information is summarized by a natural language processing engine, and the server sends the summarized information to the user's terminal.

[1801] Specific example:

[1802] If a user has a business meeting scheduled with "Company A," the server collects Company A's past business meeting history, latest news, and relevant internal records, and provides this summarized information to the user's terminal. The user then inputs their proposal based on this information, and the server generates a proposal document based on that input.

[1803] Information processing and emotional engine during business negotiations

[1804] User actions:

[1805] The user initiates a business negotiation and activates the recording and emotion recognition functions on their device. The audio data of the negotiation and emotion data such as the user's facial expressions are transmitted to the server in real time.

[1806] Server processing:

[1807] The server converts received audio data into text using a real-time speech recognition engine. It extracts important keywords and phrases from the converted text and embeds them into a meeting minutes template. It also features an emotion engine that analyzes the user's emotional state and adjusts the proposed content in real time based on that analysis. Furthermore, the server analyzes the sales negotiation content and proposes relevant products to the user in real time.

[1808] Specific example:

[1809] When a user conducts an online business meeting with a representative from "Company A," the terminal sends audio of the meeting and the user's facial expression data to the server. The server converts the audio to text and detects statements such as "I am interested in cost reduction." In this case, if the emotion engine detects that the user's emotional state is "attractive," the server immediately presents the user with products related to cost reduction.

[1810] Post-sales follow-up and utilization of emotional data

[1811] User actions:

[1812] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[1813] Server processing:

[1814] The server finalizes the meeting minutes at the end of the meeting and sends them to the user's terminal. Next, it re-analyzes the meeting minutes and sales negotiation content to generate the next proposal and schedule. It also extracts relevant deals from a database of similar past deals, summarizes that information, and sends it to the user's terminal. Furthermore, it optimizes the proposal content based on sentiment data obtained by the sentiment engine.

[1815] Specific example:

[1816] After the user concludes a business meeting with "Company A," the server reviews the meeting minutes and presents a proposal and timeline for the next meeting. Furthermore, if there have been similar business meetings with "Company B" in the past, the server summarizes that information and provides it to the user. If the emotion engine predicts that the representative from "Company A" will have a positive sentiment towards the next meeting, the server adjusts the proposal based on that prediction. Once the user enters the proposal again, the server generates a new proposal.

[1817] In this way, this system enables users to efficiently utilize information at each stage—before, during, and after a business negotiation—and also takes into account the user's emotional state, allowing sales activities to proceed smoothly.

[1818] The following describes the processing flow.

[1819] Customer information collection and analysis

[1820] Step 1:

[1821] A user logs into the system and sends a request from their terminal to the server to retrieve information about a specific customer.

[1822] Step 2:

[1823] The server receives a request to retrieve customer information.

[1824] Step 3:

[1825] The server accesses the customer database and retrieves the customer's past sales history.

[1826] Step 4:

[1827] The server collects the latest information from external sources (news APIs, company websites, etc.).

[1828] Step 5:

[1829] The server accesses the company's internal information database to retrieve relevant internal memos and past project information.

[1830] Step 6:

[1831] The server collects information and then uses a natural language processing engine to summarize it.

[1832] Step 7:

[1833] The server sends the summarized information to the user's terminal.

[1834] Step 8:

[1835] The user reviews the summary information and enters the suggested content into their device.

[1836] Step 9:

[1837] The terminal sends the user's inputted suggestions to the server.

[1838] Step 10:

[1839] The server analyzes the proposal and generates a proposal document based on that information.

[1840] Step 11:

[1841] The server sends the generated proposal to the user's terminal.

[1842] Information processing and emotional engine during business negotiations

[1843] Step 1:

[1844] The user initiates a business negotiation and activates the recording and emotion recognition functions on their device.

[1845] Step 2:

[1846] The terminal transmits audio data of the business negotiation and emotional data such as the user's facial expressions to the server in real time.

[1847] Step 3:

[1848] The server converts the received audio data into text using a real-time speech recognition engine.

[1849] Step 4:

[1850] The server extracts important keywords and phrases from the converted text.

[1851] Step 5:

[1852] The server extracts the information and embeds it into the meeting minutes template.

[1853] Step 6:

[1854] The emotion engine analyzes the user's emotional state and adjusts the suggested content in real time based on that analysis.

[1855] Step 7:

[1856] The server analyzes the details of the business negotiation and suggests relevant products to the user in real time.

[1857] Step 8:

[1858] Users can view real-time suggestions from the server and use them immediately.

[1859] Post-sales follow-up and utilization of emotional data

[1860] Step 1:

[1861] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[1862] Step 2:

[1863] The server finalizes the meeting minutes at the time of shutdown and sends them to the user's terminal.

[1864] Step 3:

[1865] The server re-analyzes the meeting minutes and negotiation details and generates a schedule of what should be proposed next.

[1866] Step 4:

[1867] The server extracts relevant sales case examples from a database of similar past sales opportunities.

[1868] Step 5:

[1869] The server extracts similar business opportunity information, summarizes it, and sends it to the user's terminal.

[1870] Step 6:

[1871] The suggested content is optimized based on emotional data acquired by the emotion engine.

[1872] Step 7:

[1873] Based on the information from Step 6, the server generates a new proposal for the next business meeting.

[1874] Step 8:

[1875] The user enters a new proposal into the terminal for the next business meeting.

[1876] Step 9:

[1877] The terminal sends the new suggestions entered by the user to the server.

[1878] Step 10:

[1879] The server analyzes the new proposal and generates a new proposal document based on that information.

[1880] Step 11:

[1881] The server sends the new proposal to the user's terminal.

[1882] (Example 2)

[1883] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1884] Traditional sales support systems lacked the ability to integrate customer information collection, real-time information processing during negotiations, emotional data analysis, and post-sales follow-up. As a result, sales representatives had to manage each process individually, leading to decreased efficiency. Furthermore, there was no effective way to utilize emotional data during negotiations, making it difficult to respond quickly to changes in customer emotions. This resulted in insufficient optimization of proposals and a lower success rate for sales negotiations.

[1885] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1886] In this invention, the server includes means for analyzing customer information, means for summarizing and presenting the results of the analysis of the customer information, means for real-time speech recognition of conversation content during negotiations and conversion to text, and means for analyzing emotional data acquired during negotiations and adjusting the content of proposals. This makes it possible to effectively collect and analyze customer information at each stage before, during, and after negotiations, and to further optimize the content of proposals in real time by utilizing emotional data.

[1887] "Customer information" refers to data, history, and related notes about customers.

[1888] "Analysis" refers to the process of processing data and extracting meaningful information or patterns.

[1889] "Summary" refers to providing a concise overview of detailed information.

[1890] "Presentation" refers to displaying information in a way that is easy for users to understand.

[1891] "Business negotiation" refers to discussions or meetings regarding sales or purchases.

[1892] "Proposed content" refers to the products, services, or their terms and conditions presented during business negotiations.

[1893] "Speech recognition" refers to the technology that converts speech into text.

[1894] "Converting to text" refers to replacing audio data with text data.

[1895] "Conversation content" refers to the words and statements exchanged during a business negotiation.

[1896] "Emotional data" refers to information about a user's emotions obtained during a business negotiation, such as their facial expressions and voice.

[1897] "Meeting minutes" refers to a document that records and summarizes the key points of a business negotiation.

[1898] "Next proposal schedule" refers to the planned visits and proposals for the next business meeting.

[1899] "Information on similar deals" refers to data and history related to similar deals that have taken place in the past.

[1900] "Real-time" refers to processing being performed at the very moment an event occurs.

[1901] "Adjustment" refers to changing the content or method depending on the situation.

[1902] This invention relates to a sales support system that combines customer information collection and analysis, real-time processing during sales negotiations, post-sales follow-up, and an emotion engine that recognizes user emotions and optimizes proposals. Specific embodiments for carrying out the invention are described below.

[1903] Customer information collection and analysis

[1904] User actions:

[1905] The user logs into the system and sends a request to retrieve information about a specific customer. The user's device sends this request to the server.

[1906] Server processing:

[1907] After receiving a customer information retrieval request, the server accesses the customer database to retrieve past sales history. Next, it gathers the latest information from external sources (e.g., news APIs or company websites). Furthermore, it accesses the internal information database to retrieve relevant internal memos and past project information. This information is summarized by a natural language processing engine (NLP engine), and the server sends the summarized information to the user's terminal.

[1908] Specific example:

[1909] If a user has an upcoming business meeting with "Company A," they log into the system and request customer information. The server retrieves "Company A's" past business meeting history, latest news, and related internal records, and summarizes the information using an NLP engine. The summarized information is then provided to the user's device. For example, the Google News API could be used as the news API, and a CRM system could be used for internal information management.

[1910] Information processing and emotional engine during business negotiations

[1911] User actions:

[1912] The user initiates a business negotiation and activates the recording and emotion recognition functions on their device. The audio data of the negotiation and the user's emotion data (facial expression data, etc.) are transmitted to the server in real time.

[1913] Server processing:

[1914] The server converts received audio data into text using a real-time speech recognition engine. It extracts important keywords and phrases from the converted text and embeds them into a meeting minutes template. The server also has the capability to analyze the user's emotional state using an emotion recognition engine and adjust the proposed content in real time. Furthermore, it analyzes the content of business negotiations and proposes relevant products and services to the user in real time.

[1915] Specific example:

[1916] When a user conducts an online business meeting with a representative from "Company A," the user enables recording and emotion recognition features in advance. The device sends the audio of the meeting and the user's facial expression data to the server. The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the audio to text and extract keywords. For example, if the representative says, "I'm interested in cost reduction," the server uses an emotion recognition algorithm to determine the emotional state is "attractive" and quickly presents the user with product information related to cost reduction.

[1917] Post-sales follow-up and utilization of emotional data

[1918] User actions:

[1919] The user concludes the business negotiation and sends a follow-up request from their device to the server.

[1920] Server processing:

[1921] The server finalizes the meeting minutes at the end of the business negotiation and sends them to the user's terminal. Next, it re-analyzes the minutes and negotiation details to generate the next proposed content and schedule. Furthermore, it extracts relevant information from a database of similar past negotiations and provides a summary to the user's terminal. Finally, it optimizes the proposed content based on sentiment data obtained by the sentiment recognition engine.

[1922] Specific example:

[1923] After the user concludes a business meeting with "Company A," the server finalizes the meeting minutes and sends them to the user. The server searches for similar past business meeting data and provides a summary of successful cases, such as those with "Company B." It also generates and proposes a schedule for the next visit and proposals to the user. The emotion engine determines that the emotional state of the "Company A" representative is positive and adjusts the next proposal based on that prediction.

[1924] In this way, this system enables users to efficiently utilize information at each stage—before, during, and after a business negotiation—and also takes into account the user's emotional state, allowing sales activities to proceed smoothly.

[1925] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1926] Customer information collection and analysis

[1927] Step 1:

[1928] The user logs into the system and sends a request to retrieve information about a specific customer. Specifically, the user clicks the "Retrieve Customer Information" button on the terminal's operation screen and enters information such as the company name and customer name. The terminal sends this request data to the server. The input is "Company Name: Company A" and the output is "Customer Information Retrieval Request".

[1929] Step 2:

[1930] The server receives a customer information retrieval request. The server first accesses the customer database to retrieve past sales negotiation history for "Company A". At this stage, the input is the "customer information retrieval request" and the output is the "past sales negotiation history".

[1931] Step 3:

[1932] The server accesses external sources (such as news APIs and company websites) to collect the latest information related to "Company A". At this stage, the input is "Company Name: Company A" and the output is "Latest News". Specifically, the server calls APIs to collect data and stores it internally.

[1933] Step 4:

[1934] The server accesses the company's internal information database to retrieve internal memos and past project information related to "Company A". At this stage, the input is "Company Name: Company A", and the output is "Internal Memos and Project Information". Specifically, it executes database queries to extract the necessary information.

[1935] Step 5:

[1936] The collected information is summarized by a natural language processing engine (NLP engine). The server uses past sales history, latest news, and internal memos as input data to perform summarization and generate the results. At this stage, the input is "all collected information," and the output is "summarized customer information." Specifically, the NLP engine's algorithm is executed to generate the summary.

[1937] Step 6:

[1938] The server sends summarized information to the user's terminal. At this stage, the input is "summarized customer information," and the output is "customer information displayed on the user's terminal." Specifically, the server sends an HTTP response, and the terminal displays that information.

[1939] ---

[1940] Information processing and emotional engine during business negotiations

[1941] Step 1:

[1942] The user initiates a business meeting and activates the recording and emotion recognition functions on their device. The user clicks the "Start Recording" and "Start Emotion Recognition" buttons on the device. The input is "Notification of Business Meeting Start," and the output is "Start of Recording and Emotion Recognition."

[1943] Step 2:

[1944] The terminal transmits audio and facial expression data from the business negotiation to the server in real time. At this stage, the input is "audio and facial expression data," and the output is "data sent to the server." Specifically, the terminal transmits the data in real time using streaming technology.

[1945] Step 3:

[1946] The server converts the received audio data into text using a real-time speech recognition engine. At this stage, the input is "audio data" and the output is "converted text". Specifically, it calls the Google Cloud Speech-to-Text API to convert audio to text.

[1947] Step 4:

[1948] The server extracts important keywords and phrases from the text and embeds them into a meeting minutes template. At this stage, the input is the "converted text," and the output is the "keywords embedded in the meeting minutes." Specifically, a text analysis algorithm is executed to extract keywords.

[1949] Step 5:

[1950] The server uses an emotion recognition engine to analyze the user's emotional state and adjusts the suggestions in real time. At this stage, the input is "facial expression data and converted text," and the output is "adjusted suggestions." Specifically, it uses an emotion recognition algorithm to analyze the emotional state and adjusts the suggestions based on that data.

[1951] Step 6:

[1952] The server analyzes the details of the business negotiation and proposes relevant products and services to the user's terminal in real time. The input at this stage is "text data of the adjusted proposal and business negotiation," and the output is "product information presented to the user." Specifically, it queries a related product database to generate appropriate proposals.

[1953] ---

[1954] Post-sales follow-up and utilization of emotional data

[1955] Step 1:

[1956] The user concludes the business negotiation and sends a follow-up request from their device to the server. The input is "Follow-up notification," and the output is "Follow-up request sent."

[1957] Step 2:

[1958] The server finalizes the meeting minutes and sends them to the user's terminal. At this stage, the input is "meeting minutes created in real time," and the output is "sending the finalized meeting minutes to the user." Specifically, it completes the meeting minutes template and provides it to the user.

[1959] Step 3:

[1960] The server re-analyzes the meeting minutes and negotiation details to generate the next proposal and schedule. At this stage, the input is the "finalized meeting minutes and negotiation details," and the output is the "next proposal and schedule." Specifically, it runs a data analysis algorithm to plan the next proposal.

[1961] Step 4:

[1962] The server checks a database of similar past deals, extracts relevant information, and provides it to the user. At this stage, the input is "finalized meeting minutes and deal details," and the output is "provision of information on similar deals." Specifically, it performs a database search to identify similar past cases, summarizes that information, and provides it.

[1963] Step 5:

[1964] The server optimizes the next proposal based on sentiment data. At this stage, the input is "sentiment data and finalized meeting minutes," and the output is "optimized next proposal." Specifically, the proposal is readjusted based on sentiment analysis data.

[1965] Step 6:

[1966] The server sends the optimized suggestions to the user's terminal. At this stage, the input is the "optimized next suggestion," and the output is the "optimized suggestion presented to the user." Specifically, the server sends an HTTP response, and the terminal displays that information.

[1967] As described above, by clearly defining the specific operation and input / output of the system at each step, and by clearly indicating the roles of the user, terminal, and server, the present invention can be effectively implemented.

[1968] (Application Example 2)

[1969] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1970] In modern brick-and-mortar stores, customer service often relies heavily on the experience and skills of store staff, leading to inconsistencies in customer satisfaction. Furthermore, accurately understanding customer needs and emotions in real time and providing optimal suggestions based on that understanding is challenging. Traditional systems fail to adequately optimize suggestions based on customer emotions, making efficient sales support difficult. Moreover, real-time information processing during sales negotiations and effective follow-up after negotiations are often lacking.

[1971] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1972] In this invention, the server includes means for analyzing customer information, means for summarizing and presenting the results of the customer information analysis, means for generating proposal content related to the business negotiation, means for real-time speech recognition and text conversion of conversation content during the business negotiation, means for analyzing the conversation content and summarizing related information, means for analyzing video data during the business negotiation to recognize the user's emotions, means for optimizing the proposal content based on the user's emotion data, means for generating and presenting meeting minutes after the business negotiation is completed, means for generating a schedule for the next proposal, and means for extracting and presenting information on similar business negotiations. This makes it possible to grasp customer needs and emotions in real time at physical stores and make optimal proposals. Furthermore, it is expected that follow-up during and after business negotiations can be efficiently carried out, leading to improved customer satisfaction and sales efficiency.

[1973] "Customer information" refers to data and historical information about customers, including, for example, past purchase history, inquiry history, and personal profile information.

[1974] "Means of analysis" refers to methods and techniques for analyzing collected data and extracting useful information from it. Specifically, this includes machine learning algorithms and statistical analysis techniques.

[1975] "Speech recognition" is a technology that converts speech data into text data, and includes the process of automatically transcribing the user's speech into text.

[1976] "Means of converting to text" refers to technologies and systems for converting collected audio data into text, such as speech recognition engines.

[1977] "Methods of summarization" refer to methods and techniques for organizing analyzed information into a concise and easily understandable form. Natural language processing and summarization algorithms are used in this process.

[1978] "Means of recognizing emotions" refers to technologies that determine a user's emotional state at a given time from their video and audio data. This includes emotion recognition engines and image processing technologies.

[1979] "Methods for optimizing suggestions" refer to technologies that generate the most suitable suggestions by considering user sentiment data and other factors. This includes recommendation algorithms and personalization engines.

[1980] "Means for generating and presenting meeting minutes" refers to technologies and systems that automatically create meeting minutes based on text data recorded during business negotiations and provide them to users.

[1981] "Methods for generating proposal schedules" refer to technologies that generate the content and schedule of the next proposal based on the details of the business negotiation and past history. This includes scheduling management systems and task management tools.

[1982] "Methods for extracting and presenting information on similar business opportunities" refers to technologies that extract cases similar to the current business opportunity from a database of past business opportunities and provide them as reference information. Data mining and clustering algorithms are used in this process.

[1983] In this invention, to specifically realize a sales support system for physical stores, smart glasses, a server, a speech recognition engine, a natural language processing engine, an emotion recognition engine, and a recommendation algorithm work together as a single unit. The operation of this system will be described below in order.

[1984] System configuration and operation

[1985] 1. Data Collection

[1986] The user (staff member) wears smart glasses.

[1987] When a conversation with a customer begins, the smart glasses collect audio and video data in real time and send it to the server.

[1988] 2. Data Processing

[1989] The server converts the received audio data into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[1990] The converted text data is sent to a natural language processing engine (e.g., a BERT model) where important keywords and phrases are extracted.

[1991] The video data is used to analyze the emotional state of customers using an emotion recognition engine (e.g., Microsoft Azure Face API).

[1992] Based on extracted keywords and sentiment data, a recommendation algorithm (e.g., collaborative filtering algorithm) is used to select the most suitable product suggestions.

[1993] 3. Data Output

[1994] The selected product information is displayed...

Claims

1. Means for analyzing customer information, A means for summarizing and presenting the results of the analysis of the aforementioned customer information, A means of generating proposals related to business negotiations, A method for real-time speech recognition and text conversion of conversations during business negotiations, A means for analyzing the content of the aforementioned conversation and summarizing the relevant information, A method for generating and presenting meeting minutes after the business negotiation has concluded, A means for generating the next proposal schedule and a means for extracting and presenting information on similar business deals, A system that includes this.

2. The system according to claim 1, further comprising means for generating proposals in real time during a business negotiation.

3. The system according to claim 1, further comprising means for automatically generating a proposal based on the aforementioned proposed content.

Images