system

Abstract

Provide a system. The present invention provides a means for recognizing the voice of a customer on hold; A means of identifying customer interests and needs from recognized speech; means for generating personalized advertisements in real time based on identified interests and needs; means for delivering the generated personalized advertisement to a customer's device; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] For companies providing customer service, call center hold times are one of the factors that reduce customer satisfaction. Long hold times in particular can lead to customer dissatisfaction, which can ultimately damage a company's brand image and revenue opportunities. Conventional hold music and messages make it difficult to make effective use of customers' waiting time, limiting the value they can provide to customers. Therefore, a system is needed to convert customer waiting time into something meaningful and valuable. [Means for solving the problem]

[0005] The present invention provides a system that uses voice recognition and trend analysis to provide personalized advertisements to customers whose calls are on hold. The system includes the following means: first, a means for recognizing the voice of a customer whose call is on hold; second, a means for identifying the customer's interests and needs from the recognized voice; third, a means for generating personalized advertisements in real time based on the identified interests and needs; and finally, a means for delivering the generated personalized advertisements to the customer's terminal. This can turn customer waiting time into a valuable experience, improving customer engagement and corporate profitability.

[0006] A "customer on hold" is a customer who is connected to a call center or customer service center by telephone but whose direct conversation with an operator is temporarily interrupted.

[0007] "Voice recognition means" refers to technology or devices for collecting customer speech and converting that speech data into text data.

[0008] "Means for identifying interests and needs" refers to technologies and algorithms that analyze text data obtained through voice recognition to identify customer interests and information needs.

[0009] "Personalized advertising" refers to marketing information and advertising content that is customized and optimized according to the interests and needs of a particular customer.

[0010] "Means of generating in real time" refers to technology and software that instantly creates advertisements and information based on analysis results and makes them available for use on the spot.

[0011] "Means for delivering to customer devices" refers to the technology and systems used to deliver generated advertising content to the telephone or other communications device used by the customer. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0016] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0017] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0018] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0020] [First embodiment]

[0021] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0022] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0025] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0028] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0032] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0033] The present invention is a system for providing personalized advertisements in real time to customers on hold. This system integrates speech recognition, trend analysis, advertisement generation, and advertisement distribution functions, with the aim of turning customers' waiting time into a valuable experience.

[0034] Program implementation example

[0035] This system is operated based on the following program processing procedures.

[0036] Speech recognition and text conversion

[0037] User: A customer calls a call center and states a specific question or request to an operator. For example, "I want to know about credit card reward points."

[0038] Device: Captures what the customer says and temporarily stores it as audio data, which is then sent to a server in encrypted form to ensure security.

[0039] Server: The received voice data is analyzed using a voice recognition engine and converted into text data.

[0040] Analyzing text data and identifying customer needs

[0041] Server: The speech recognition engine converts the data into text and sends it to the trend analysis module, which extracts keywords from the text and compares them with past trends and patterns stored in a database to identify customer interests and needs.

[0042] Generating personalized ads

[0043] Server: Based on the results of trend analysis, the server generates advertising content that matches the customer's interests. The advertising generation module creates appropriate advertisements in real time and encodes them as audio files. For example, based on past data, it may determine that new point campaign information would be effective for customers who are interested in "credit card points."

[0044] Delivering personalized ads

[0045] Server: Prepares the encoded audio ad file for sending to the customer's terminal, and sends the data to the terminal via the communication module.

[0046] Terminal: The received audio ad file is played in real time and made available to customers on hold. Example: An audio ad saying "A new points campaign has started" is played.

[0047] Customer response tracking and data updates

[0048] Server: After delivering the ad, the server monitors the conversation after the customer is returned to the operator and tracks their response to the ad. If the customer shows interest in the ad content, the server records this information in a database. This will help improve ad generation in the future.

[0049] Specific examples

[0050] A customer calls a call center and inquires about credit card points. The operator puts the call on hold, and the conversation is captured as voice data by the terminal. The voice data is sent to the server and converted into text by a voice recognition engine. A trend analysis module then extracts the keywords "credit card" and "points" and compares them with a past database. The server generates a personalized advertisement for a "new points campaign" in real time and sends the advertisement to the customer's terminal. The terminal plays the received advertisement to the customer while they are on hold. If the customer then asks about the advertisement when they return to the operator, the server tracks their response and records it in a database. This series of processes transforms customer waiting time into a meaningful experience, improving the company's profitability and customer satisfaction.

[0051] According to these modes for implementation, the present invention is applied to provide personalized announcements to customers whose calls are on hold.

[0052] The processing flow will be explained below.

[0053] Step 1:

[0054] Server: When a customer calls the call center, the server monitors the call status and detects that the operator has put the customer on hold. It sets a hold flag and records that the customer's waiting time has begun.

[0055] Step 2:

[0056] Terminal: Captures the conversation between the operator and the customer as voice data in real time, temporarily stores the voice data before the conversation is put on hold, and encrypts it for security purposes.

[0057] Step 3:

[0058] Server: Receives the encrypted voice data and sends it to the voice recognition engine, which analyzes the voice data and converts it into text data.

[0059] Step 4:

[0060] Server: Sends the converted data to a trend analysis module, which extracts keywords from the text data and matches them with historical data in a database to identify customer interests and needs.

[0061] Step 5:

[0062] Server: Based on the results of trend analysis, the server generates optimal advertising content in real time that matches the customer's interests. The generated advertisements are encoded as audio files and prepared for distribution through the advertisement generation module.

[0063] Step 6:

[0064] Server: Sends the encoded audio advertisement file to the customer's device. The distribution module distributes the advertisement to the customer on hold at the appropriate time.

[0065] Step 7:

[0066] Terminal: The received audio advertisement file is played in real time and made available to customers on hold, turning their waiting time into valuable information time.

[0067] Step 8:

[0068] Server: After delivering the advertisement, monitor the conversation when the customer returns to the operator and track their response to the advertisement. If the customer shows interest in the advertisement content, record that information in the database and use it to generate future advertisements.

[0069] Step 9:

[0070] Server: The accumulated response data is analyzed using machine learning algorithms, forming a feedback loop to improve the accuracy of ad generation and delivery, thereby optimizing the performance of the entire system.

[0071] Through this processing step, the system can turn customer waiting time into a positive experience, providing them with personalized advertising, and improving business profitability and customer satisfaction.

[0072] Example 1

[0073] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0074] Conventional advertisements provided while a call is on hold are generally uniform and do not provide information appropriate to the customer's interests and needs, resulting in lower customer satisfaction and limited advertising effectiveness for companies.

[0075] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0076] In this invention, the server includes means for recognizing the voice of a customer while the call is on hold, means for converting the recognized voice into text data, means for extracting keywords from the text data, means for identifying the interests and needs of the customer by analyzing the extracted keywords, means for generating a personalized advertisement in real time based on the identified interests and needs, means for encoding the generated personalized advertisement as an audio file, means for delivering the encoded personalized advertisement to the customer's terminal, means for playing the advertisement on the customer's terminal, and means for monitoring the customer's response after the advertisement is played and recording the result in a database. This makes it possible to turn the customer's waiting time into a valuable experience, improve customer satisfaction, and maximize the effectiveness of a company's advertising.

[0077] "Call on hold" means that a customer is temporarily placed on hold by a telephone operator or automated answering system.

[0078] "Means for recognizing voice" refers to technology for converting customer utterances into text data, including a voice recognition engine.

[0079] "Means for converting into text data" refers to the technology for analyzing voice data and converting it into text information.

[0080] "Keyword extraction methods" refers to techniques for identifying and extracting specific important words or phrases from text data.

[0081] "Means for identifying customer interests and needs" refers to technology that analyzes extracted keywords and determines customer concerns and requests.

[0082] "Means for generating personalized advertisements" refers to technology that creates advertisements that are individually tailored to a customer's interests and needs.

[0083] "Means for encoding as an audio file" refers to a technique for converting the generated advertising content into an audio data format.

[0084] "Means for delivery to Customer Device" means the technology that transmits the encoded audio file to the Customer Device.

[0085] "Means for playing advertisements on a customer's device" refers to technology for playing received audio files on a customer's device.

[0086] "Means for monitoring customer responses after an advertisement is played" refers to technology that tracks customer behavior and comments after an advertisement is played and collects data based on that.

[0087] "Means of recording in a database" refers to the technology used to store collected customer response data and manage it for analysis and future use.

[0088] MODE FOR CARRYING OUT THE INVENTION

[0089] The present invention relates to a system that provides personalized advertisements to customers on hold in real time. The system integrates speech recognition, trend analysis, advertisement generation, and advertisement distribution functions. The purpose of the system is to turn customers' waiting time into a valuable experience.

[0090] Speech recognition and text conversion

[0091] User: A customer calls the call center with a question or request. For example, they might ask, "I want to know about my credit card points."

[0092] Device: Captures what the customer says and temporarily stores it as audio data, which is then sent to a server in encrypted form to ensure security.

[0093] Server: The received voice data is analyzed using a voice recognition engine (e.g., Google® Cloud Speech-to-Text or IBM Watson®) and converted into text data.

[0094] Analyzing text data and identifying customer needs

[0095] Server: The data converted to text by the speech recognition engine is sent to a trend analysis module (e.g., Python's NLTK library). This module extracts keywords from the text data and compares them with past trends and patterns stored in a database, thereby identifying customer interests and needs.

[0096] Generating personalized ads

[0097] Server: Based on the results of the trend analysis, advertising content tailored to customer interests is generated using automated speech generation software (e.g., Amazon Polly or Google Text-to-Speech). The generated advertisements are encoded as audio files in real time.

[0098] Delivering personalized ads

[0099] Server: Prepares the encoded audio ad file for sending to the customer's device. Sends the data to the device via a communication module (e.g. HTTP / HTTPS communication).

[0100] Terminal: The received audio ad file is played in real time and made available to customers on hold. For example, an audio ad saying "A new points campaign has started" is played.

[0101] Customer response tracking and data updates

[0102] Server: After delivering the ad, the server monitors the conversation after the customer is returned to the operator and tracks their response to the ad. If the customer shows interest in the ad content, the server records this information in a database. This will help improve ad generation in the future.

[0103] Specific examples

[0104] A customer calls a call center and inquires about credit card points. The operator puts the call on hold, and the conversation is captured as voice data by the terminal. The voice data is sent to the server and converted into text by a voice recognition engine. A trend analysis module then extracts the keywords "credit card" and "points" and compares them with a past database. The server generates a personalized advertisement for a "new points campaign" in real time and sends the advertisement to the customer's terminal. The terminal plays the received advertisement to the customer while they are on hold. If the customer then asks about the advertisement when they return to the operator, the server tracks their response and records it in a database. This series of processes transforms customer waiting time into a meaningful experience, improving the company's profitability and customer satisfaction.

[0105] Prompt Sentence Examples

[0106] A customer calls a call center and asks, "Tell me about credit card points." The operator puts the customer on hold, and the system uses speech recognition to convert the conversation into text. The server then analyzes the text and extracts the keywords "credit card" and "points." Based on this, the system generates a personalized advertisement about a "new points campaign" in real time and plays it to the customer while they are on hold.

[0107] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0108] Step 1:

[0109] User: A customer calls the call center and says, "I want to know about credit card points." The input is the customer's voice. The output is the voice data of what the customer said.

[0110] Step 2:

[0111] Terminal: Captures and temporarily stores what the customer says. The captured voice data is encrypted using AES. The input is the customer's voice data, and the output is encrypted voice data.

[0112] Step 3:

[0113] Terminal: Sends encrypted audio data to the server. The input is the encrypted audio data, and the output is the data sent to the server.

[0114] Step 4:

[0115] Server: The received voice data is analyzed using a speech recognition engine (e.g., Google Cloud Speech-to-Text or IBM Watson) and converted into text data. At this time, the voice data is analyzed and text data such as "I would like to know about credit card points" is generated. The input is encrypted voice data, and the output is analyzed text data.

[0116] Step 5:

[0117] Server: The text data generated by the speech recognition engine is sent to a trend analysis module (e.g., Python's NLTK library). This module extracts keywords such as "credit card" and "points" from the text data and compares them with a historical database. This identifies the customer's interests and needs. The input is text data, and the output is the keywords and their analysis results.

[0118] Step 6:

[0119] Server: Based on the results of the trend analysis, an audio ad containing information about the new points campaign is generated using automatic speech generation software (e.g., Amazon Polly or Google Text-to-Speech). The generated ad content is encoded as an audio file. The input is the analysis results and appropriate ad content information, and the output is an audio file.

[0120] Step 7:

[0121] Server: Sends the encoded audio ad file to the customer's terminal via a communication module (e.g. HTTP / HTTPS communication). The input is the encoded audio file, and the output is the data sent to the terminal.

[0122] Step 8:

[0123] Terminal: Plays the received audio ad file in real time and lets customers on hold hear the audio ad, "A new points campaign has started." The input is the audio ad file received by the terminal, and the output is the audio ad heard by customers on hold.

[0124] Step 9:

[0125] Server: After delivering the ad, monitor the customer's response when they return to the operator. If the customer asks, "Tell me more about the points campaign," track that response and record it in a database. This will help improve future ad generation. The input is the customer's response data after the ad is delivered, and the output is the tracking data recorded in the database.

[0126] (Application example 1)

[0127] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0128] Conventional systems for providing personalized advertisements to customers on hold have had difficulty in fully identifying customer interests and needs and generating and delivering advertisements in real time. Furthermore, it has not been possible to track customer responses and provide feedback to future advertisement generation. The present invention aims to solve these problems and improve the customer shopping experience in virtual stores.

[0129] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0130] In this invention, the server includes means for recognizing the voice of a customer while the call is on hold, means for identifying the interests and needs of the customer from the recognized voice, means for generating personalized advertisements in real time, means for delivering the generated personalized advertisements to the customer's terminal, means for supporting the customer's purchasing experience through voice interaction in a virtual store, and means for tracking the customer's responses and recording them in a database. This makes it possible to provide advertisements based on the customer's interests and needs in real time and improve the purchasing experience.

[0131] A "customer on hold" is a customer who has called a company or support center and is waiting for a response.

[0132] "Speech recognition means" refers to means that uses technology to analyze voice data and convert it into text data.

[0133] "Means for identifying customer interests and needs" refers to a process of identifying what a customer is interested in or needs by analyzing text data obtained by speech recognition means.

[0134] "Personalized advertising" refers to advertising that is individually optimized based on a customer's specific interests and needs.

[0135] A "generative AI model" is a model that uses artificial intelligence technology to generate content such as advertisements from given information.

[0136] The "advertising generation means" refers to a means for automatically generating advertising content suitable for a customer based on the analysis results.

[0137] The "advertising distribution means" refers to a means for transmitting the generated advertisement to the customer's terminal in real time and playing it back.

[0138] "Customer Device" means a communication device used by a Customer, such as a smartphone, smart glasses, or head-mounted display.

[0139] A "virtual store" is an online shop or shopping platform operated via the Internet or virtual reality technology.

[0140] "Means to support customer purchasing experience" refers to means that support customers through voice interaction when browsing, selecting, and purchasing products in a virtual store.

[0141] "Means for tracking customer responses" refers to means for monitoring and collecting data on how customers respond to the advertisements that are played.

[0142] "Means for recording in a database" refers to means for storing tracked customer responses in a database and using them for later analysis and advertisement generation.

[0143] The present invention provides a personalized advertisement providing system for improving a customer's shopping experience in a virtual store. The system is implemented by the following components:

[0144] 1. Speech Recognition and Text Conversion

[0145] server:

[0146] The server is equipped with a speech recognition engine that recognizes the speech customers make in the virtual store and converts it into text data. This speech recognition uses technologies such as Google Cloud Speech-to-Text. The text data is encrypted and stored securely.

[0147] As a specific example, the server captures the audio of a customer saying, "Please tell me about a camera you recommend," and converts the audio into text data.

[0148] 2. Identifying customer interests and needs

[0149] server:

[0150] The server analyzes the text data obtained by the speech recognition tool to extract the customer's interests and needs. This analysis uses trend analysis modules such as "IBM Watson Natural Language Understanding." Here, the server compares the data with past data to identify the customer's interests and needs.

[0151] 3. Generating personalized ads

[0152] server:

[0153] The server generates personalized ads for customers based on the results of trend analysis. Generative AI models such as OpenAI (registered trademark) GPT-4 (registered trademark) are used to generate these ads. The generated ads are encoded as audio files.

[0154] Here is an example prompt:

[0155] "Generate an ad for a new digital camera model. Based on past purchase history, you know your customer is interested in high-quality photography. Write compelling copy that includes the features and price of the new model."

[0156] 4. Delivering personalized advertising

[0157] server:

[0158] The server then sends the generated audio ad file to the customer's device, using technologies such as Firebase Cloud Messaging.

[0159] Device:

[0160] The customer's device will play the received audio advertisement file in real time, for example, "New camera model available at special price."

[0161] 5. Tracking customer responses and updating data

[0162] server:

[0163] The server tracks customer responses after delivering the ads and records the data in a database, using database technologies such as MySQL® and ElasticSearch®, which provides feedback to generate more effective ads in the future.

[0164] For example, if a customer listens to an advertisement and then asks the operator again, "Can you tell me more about that camera?", that response will be recorded in the database.

[0165] The present invention enables the generation and delivery of real-time personalized advertisements based on customer interests and needs, improving the customer's shopping experience.

[0166] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0167] Step 1:

[0168] Input: User voice input

[0169] Action: A user enters a virtual store and says, "What camera do you recommend?"

[0170] Server: The server captures the user's voice input and temporarily stores it as audio data. The audio data is stored in encrypted form.

[0171] Output: Encrypted audio data

[0172] Step 2:

[0173] Input: Encrypted audio data

[0174] Processing: The server sends the audio data to a speech recognition engine (e.g., Google Cloud Speech-to-Text) and converts the audio into text data.

[0175] Server: The server sends voice data and receives text data converted by a voice recognition engine.

[0176] Output: Text data (e.g., "What camera would you recommend?")

[0177] Step 3:

[0178] Input: Text data

[0179] Processing: The server sends the text data to a trend analysis module (e.g., IBM Watson Natural Language Understanding) to extract keywords and identify customer interests and needs.

[0180] Server: Extracts keywords (e.g., "camera") from the text data and matches them with past trends and patterns in a database to identify customer interests and needs.

[0181] Output: Customer interests and needs (e.g., "I'm interested in cameras")

[0182] Step 4:

[0183] Input: Customer interests and needs

[0184] Processing: The server creates prompts to generate personalized advertisements based on the identified interests and needs.

[0185] Server: For example, generate a prompt like this: "Generate an advertisement for a new digital camera model. Based on past purchase history, we know that the customer is interested in high-quality photography. Write an appealing ad copy that includes the features and price of the new model."

[0186] Output: prompt statement

[0187] Step 5:

[0188] Input: prompt statement

[0189] Processing: The server sends prompts to a generative AI model (e.g., OpenAI GPT-4) to generate personalized ads.

[0190] Server: The generative AI model generates advertising content based on the prompt and encodes it as an audio file.

[0191] Output: Advertising content in the form of an audio file (e.g., "Special Offer on New Camera Models")

[0192] Step 6:

[0193] Input: Ad content in the form of an audio file

[0194] Processing: The server delivers the audio file to the client's device.

[0195] Server: Sends advertisements to the customer's device via a communication module (e.g., Firebase Cloud Messaging).

[0196] Output: Audio file delivered to the customer's device

[0197] Step 7:

[0198] Input: Streamed audio file

[0199] Processing: The device plays the received audio file in real time.

[0200] Device: The customer's device plays an audio ad (e.g., "Special Offer on New Camera Models").

[0201] Output: Audio ad played to customer

[0202] Step 8:

[0203] Input: Customer response to audio ad

[0204] Processing: The server tracks customer responses to the audio advertisements and records them in a database.

[0205] Server: For example, if the customer asks the operator again, "Please tell me the details of that camera," the response is tracked and stored in a database.

[0206] Output: Customer responses recorded in a database

[0207] All these processing steps result in the generation and delivery of real-time personalized ads based on customer interests and needs, improving the customer shopping experience.

[0208] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0209] The present invention combines a system that provides personalized advertisements to customers on hold in real time with an emotion engine that recognizes user emotions. This system integrates speech recognition, trend analysis, advertisement generation, advertisement distribution, and emotion recognition functions, with the aim of turning customers' waiting time into a valuable experience.

[0210] Program implementation example

[0211] This system is operated based on the following program processing procedures.

[0212] Speech recognition and text conversion

[0213] User: A customer calls a call center and states a specific question or request to an operator. For example, "I want to know about credit card reward points."

[0214] Device: Captures and temporarily stores the customer's speech as audio data in real time. This data is encrypted and sent to a server for security purposes.

[0215] Server: Sends the received voice data to the voice recognition engine and converts the voice data into text data.

[0216] Analyzing text data and identifying customer needs

[0217] Server: Sends the text data to a trend analysis module, which extracts keywords from the text data and matches them with historical data in a database to identify customer interests and needs.

[0218] emotion recognition

[0219] Server: The text data is also sent to the emotion engine, which analyzes the tone of the customer's voice and the content of what they say to recognize their emotional state (e.g., joy, anger, sadness, etc.).

[0220] Example: Recognizing emotions such as "customer dissatisfaction" from the customer's tone of voice and speech patterns.

[0221] Generating personalized ads

[0222] Server: Based on the results of trend analysis and the analysis results of the emotion engine, the server generates appropriate advertising content in real time that matches the customer's interests and emotions. For example, if a customer is dissatisfied, the server will display an apology message or information about shortening waiting times instead of a general advertisement.

[0223] Example: A customer who is interested in "credit card points" but dissatisfied with them can be informed of the start of a "new points campaign" and also be given information about the expected waiting time.

[0224] Delivering personalized ads

[0225] Server: Sends the encoded audio ad file to the customer's device. Through the distribution module, the ad is delivered to the customer on hold at the appropriate time.

[0226] Terminal: The received audio ad file is played in real time and made available to customers on hold, turning waiting time into valuable information time.

[0227] Customer response tracking and data updates

[0228] Server: After the ad is delivered, the server monitors the conversation after the customer is returned to the operator and tracks their response to the ad. If the customer shows interest in the ad content or responds indicative of their emotional state, this information is recorded in a database. This will be used to improve the accuracy of future ad generation and emotion recognition.

[0229] Specific examples

[0230] A customer calls a call center and inquires about credit card points. The operator puts the call on hold, and the conversation is captured as voice data by the terminal and sent to the server. The voice data is converted into text by a speech recognition engine. A trend analysis module then extracts the keywords "credit card" and "points," and an emotion engine recognizes the customer's emotional state. The server generates a personalized advertisement about a "new points campaign" in real time, along with information about the expected waiting time, and sends it to the customer's terminal as an audio file. The terminal plays the advertisement to the customer while they are on hold. The customer is then returned to the operator, and if they ask a question about the advertisement or respond emotionally, the server tracks that data and records it in a database. This process turns customer waiting time into a meaningful experience, improving the company's profitability and customer satisfaction.

[0231] In this way, the present invention is implemented as a system that provides optimal advertisements while a call is on hold, taking into consideration the customer's emotions, and achieves both customer satisfaction and corporate profits.

[0232] The processing flow will be explained below.

[0233] Step 1:

[0234] Server: When a customer calls the call center, the server monitors the call status and detects that the operator has put the customer on hold. It sets a hold flag and records that the customer's wait time has begun.

[0235] Step 2:

[0236] Terminal: Captures the conversation between the operator and the customer as voice data in real time. The voice data before the conversation is put on hold is temporarily stored, encrypted, and sent to the server.

[0237] Step 3:

[0238] Server: Receives the encrypted voice data and sends it to the voice recognition engine, which analyzes the voice data and converts it into text data.

[0239] Step 4:

[0240] Server: Sends the text data to the trend analysis module, which extracts keywords from the text data and matches them with historical data in the database to identify customer interests and needs.

[0241] Step 5:

[0242] Server: Sends text data to the emotion engine. The emotion engine analyzes the tone of the customer's voice and the content of what they say to recognize their emotional state. For example, it recognizes that "the customer's tone of voice indicates dissatisfaction."

[0243] Step 6:

[0244] Server: Based on the results of trend analysis and the emotion engine, the server generates personalized advertisements in real time that match the customer's interests and emotions. For example, if a customer is interested in and dissatisfied with "credit card points," the server generates information about new campaigns related to points and an apology message for keeping them waiting.

[0245] Step 7:

[0246] Server: Encodes the generated personalized ad into an audio file and prepares it for delivery.

[0247] Step 8:

[0248] Server: Sends the encoded audio ad file to the customer's device. The distribution module distributes the ad to the pending customer at the appropriate time.

[0249] Step 9:

[0250] Terminal: The received audio advertisement file is played in real time, allowing customers on hold to listen to the advertisement, thereby turning the customer's waiting time into valuable information time.

[0251] Step 10:

[0252] Server: After the advertisement is delivered, the server monitors the conversation when the customer is returned to the operator. It tracks the response to the advertisement, and if the customer shows interest in the advertisement content, it records that information in a database.

[0253] Step 11:

[0254] Server: Machine learning algorithms analyze the accumulated response data, creating a feedback loop to improve the accuracy of ad generation and delivery, thereby optimizing the performance of the entire system.

[0255] When a customer calls a call center to inquire about credit card points, the operator puts the call on hold. The terminal captures what the customer says as voice data and sends it to the server. The voice data is converted into text by a speech recognition engine. A trend analysis module then extracts keywords, and an emotion engine recognizes the customer's emotional state. The server generates a personalized advertisement in real time and sends it to the customer's terminal as an audio file. The terminal plays the advertisement to the customer while they are on hold. If the customer is then returned to the operator and asks a question about the advertisement, the server tracks their response and records it in a database. Through this process, the customer's waiting time is transformed into a meaningful experience, improving the company's profitability and customer satisfaction.

[0256] Example 2

[0257] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0258] In conventional call center systems, advertisements and messages provided to customers while they were on hold were typically not tailored to the individual customer's interests and needs. As a result, customers were unable to spend their waiting time meaningfully, which could lead to lower customer satisfaction. Furthermore, delivery that did not take into account the customer's emotional state could exacerbate customer dissatisfaction. To solve these problems, the present invention aims to effectively utilize waiting time and improve customer satisfaction by analyzing the customer's interests, needs, and even their emotional state in real time and providing personalized advertisements.

[0259] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0260] In this invention, the server includes means for recognizing the voice of a customer while the call is on hold, means for identifying the interests and needs of the customer from the recognized voice, means for analyzing the emotional state of the customer, means for generating personalized advertisements in real time based on the identified interests, needs and emotional state, and means for delivering the generated personalized advertisements to the customer's device, thereby enabling the customer's waiting time while on hold to be transformed into time for providing information suited to their individual needs and emotions.

[0261] "Customers on hold" means customers who have called the call center and are on hold waiting for an operator to respond.

[0262] "Voice recognition means" refers to devices or software that have the ability to collect customer voice data, analyze it, and convert it into text.

[0263] "Means for identifying customer interests and needs" refers to algorithms or devices that analyze and extract customer interests and requests from text data.

[0264] "Means for analyzing the emotional state of the customer" refers to algorithms or devices that recognize emotions based on the customer's voice and text data.

[0265] "Personalized advertising" refers to advertising that is individually tailored to take into account the interests, needs, and emotional state of a particular customer.

[0266] "Generating means" refers to algorithms or devices that generate specific content or advertisements based on the results of data analysis.

[0267] "Delivery means" refers to the algorithms and / or equipment used to transmit and play generated content and advertisements on Customer's devices.

[0268] "Real-time" refers to instantaneous processing or response with very low latency.

[0269] "Device" means an electronic device capable of sending and receiving voice and data that is used by a customer while a call is on hold.

[0270] The present invention relates to a system that provides personalized advertisements to customers on hold in real time, and further combines it with an emotion engine that recognizes user emotions. This system integrates speech recognition, trend analysis, advertisement generation, advertisement distribution, and emotion recognition functions, with the aim of turning customers' waiting time into a valuable experience.

[0271] The whole system consists of three main components: server, terminal, and user. Here, the server is responsible for data processing and analysis, the terminal is the interface with the customer, and the user is the customer who uses the system.

[0272] First, a user calls the call center and states a specific question or request to an operator. For example, "I want to know about credit card points." At this time, the terminal captures the user's speech as voice data in real time and temporarily stores it. The captured voice data is AES encrypted to ensure security and sent to the server.

[0273] The server sends the received voice data to a speech recognition engine (e.g., Google Cloud Speech-to-Text), which converts the voice data into text data. The text data is then sent to a natural language processing (NLP) module, which extracts keywords related to the customer's interests and needs from the text data. Furthermore, an emotion engine (e.g., IBM Watson Tone Analyzer) analyzes the customer's tone of voice and the content of what is being said to recognize the customer's emotional state (e.g., joy, anger, sadness).

[0274] Next, the server generates appropriate advertising content in real time based on the results of trend analysis and the emotion engine's analysis, matching the customer's interests and emotions. For example, if a customer is interested in "credit card points" but is dissatisfied, an advertisement containing information about a new points campaign and a message of goodwill will be generated.

[0275] The generated advertising content is encoded as an audio file and sent from the server to the customer's device. The device then plays the received advertising audio file in real time and lets the customer listen to it while on hold, thereby turning the customer's waiting time into valuable information time.

[0276] After the ad is delivered, the server monitors the conversation and tracks the customer's response to the ad. If the customer shows interest in the ad content or responds in a way that indicates a particular emotional state, that information is recorded in a database. This information can be used to generate future ads and improve the accuracy of emotion recognition.

[0277] For example, a user calls a call center and inquires about credit card points. The conversation is captured as voice data by the terminal and sent to the server. The voice data is then converted into text by the Google Cloud Speech-to-Text engine. An NLP module then extracts the keywords "credit card" and "points," and an emotion engine (IBM Watson Tone Analyzer) recognizes the customer's emotional state. The server generates a personalized advertisement about a "new points campaign" in real time, along with information about the expected waiting time, and sends it to the customer's terminal as an audio file. The terminal plays the advertisement to the customer while they are on hold. The customer is then returned to the operator, and if they ask a question about the advertisement or respond emotionally, the server tracks that data and records it in a database. This process transforms customer waiting time into a meaningful experience, improving the company's profitability and customer satisfaction.

[0278] The above is an embodiment of the present invention.

[0279] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0280] Step 1:

[0281] A user calls a call center and states a specific question or request to an operator.

[0282] Input: User's spoken utterance

[0283] Output: Audio data

[0284] Specific operation: A user calls and asks, "I want to know about credit card points."

[0285] Step 2:

[0286] The device captures the user's speech as audio data in real time.

[0287] Input: User's voice data

[0288] Output: Encrypted audio data

[0289] Specific operation: The device encrypts the captured audio data using AES and temporarily stores it.

[0290] Step 3:

[0291] The device sends the encrypted audio data to the server.

[0292] Input: Encrypted audio data

[0293] Output: Encrypted audio data sent to the server

[0294] Specific operation: The device sends encrypted voice data to the server via the network.

[0295] Step 4:

[0296] The server sends the received voice data to a voice recognition engine, which converts the voice data into text data.

[0297] Input: Encrypted audio data

[0298] Output: Text data

[0299] Specific operation: The server uses the Google Cloud Speech-to-Text engine to convert the audio data into text data.

[0300] Step 5:

[0301] The server sends the converted data to a natural language processing (NLP) module to extract keywords.

[0302] Input: Text data

[0303] Output: Extracted keywords

[0304] Specific operation: The server uses an NLP module to analyze and extract keywords such as "credit card" and "points."

[0305] Step 6:

[0306] The server simultaneously sends the text data to the emotion engine to analyze the customer's emotional state.

[0307] Input: Text data

[0308] Output: Customer's emotional state

[0309] What it does: The server uses IBM Watson Tone Analyzer to identify the customer's emotional state from the text data.

[0310] Step 7:

[0311] The server generates personalized advertisements based on keywords and emotional states.

[0312] Input: extracted keywords and emotional states

[0313] Output: Personalized advertising content

[0314] Specific operation: The server generates appropriate advertising content (e.g., "New points campaign" and "Information on shortened waiting times") in accordance with the customer's interests and emotions.

[0315] Step 8:

[0316] The server encodes the generated advertising content as an audio file and transmits it to the terminal.

[0317] Input: Personalized advertising content

[0318] Output: Audio file

[0319] Specific operation: The server encodes the advertising content into an audio file and sends it to the device.

[0320] Step 9:

[0321] The terminal plays the received audio advertisement file and lets the customer on hold hear it.

[0322] Input: Audio ad file

[0323] Output: Customer hears audio ad

[0324] What it does: The terminal plays audio ads in real time on the customer's device.

[0325] Step 10:

[0326] The server monitors the conversation between the customer and the operator after the advertisement is delivered and tracks the response to the advertisement.

[0327] Input: Customer and operator conversation data

[0328] Output: Customer response data

[0329] What it does: The server analyzes the conversation data and records whether the customer has a specific reaction or question to the ad.

[0330] These are the processing steps of the system, which will make customer waiting time more meaningful and increase the company's profitability and customer satisfaction.

[0331] (Application example 2)

[0332] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0333] Conventional on-hold ad delivery systems have struggled to deliver personalized ads that reflect customer emotions and specific interests. This has resulted in customers' waiting time not being fully utilized as an opportunity to provide information, preventing customer satisfaction and corporate profitability from being fully improved. Furthermore, in brick-and-mortar stores, there has been a lack of timely product recommendations that reflect customer emotions, resulting in missed sales opportunities.

[0334] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing the voice of a customer while the call is on hold, means for identifying the customer's interests and needs from the recognized voice, means for analyzing the customer's emotions, means for generating a personalized advertisement in real time based on the identified interests and needs and the customer's emotions, means for delivering the generated personalized advertisement to the customer's terminal, and means for displaying the generated advertisement on the smart glasses. This makes it possible to provide personalized advertisements and product suggestions in real time according to the customer's emotions and interests.

[0335] "Call on hold" refers to a state in which a user pauses a telephone call while waiting for another operation or a response.

[0336] A "voice recognition means" is a device or program that captures customer speech as voice data and converts it into text.

[0337] The "means for identifying customer interests and needs" refers to a device or program that extracts keywords from the customer's text data and identifies the customer's interests and requests based on this information.

[0338] The "means for analyzing customer emotions" refers to a device or program that analyzes the tone of a customer's voice and the content of their statements to recognize the customer's emotional state.

[0339] "Personalized advertising" is advertising content that is specific to a customer's individual interests, needs, and emotional state.

[0340] "Generating means" refers to a device or program that automatically creates advertisements or information based on the analyzed data.

[0341] "Distribution means" refers to a device or program that transmits the generated advertisements and information to the customer's device (such as a smartphone or smart glasses).

[0342] "Smart glasses" are wearable devices equipped with functions such as a display, camera, and microphone, and capable of displaying information and capturing audio.

[0343] "Real-time" refers to processing and information being provided immediately without delay.

[0344] This invention is a system that provides personalized advertisements in real time while taking into account customer emotions during call on-hold or in-store interactions. The system operates by integrating speech recognition, trend analysis, advertisement generation, advertisement distribution, and emotion recognition functions.

[0345] The server converts the customer's voice into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text). The converted data is sent to a trend analysis module (e.g., Python library - spaCy) to extract keywords. After that, an emotion recognition engine (e.g., IBM Watson Tone Analyzer) is used to analyze the customer's emotional state.

[0346] Based on the identified keywords and emotional state, the server automatically generates personalized advertisements using a generative AI model (e.g., OpenAI GPT-3® API). These advertisements are encoded as audio files or text information and delivered to the customer's device. Specifically, the advertisements are displayed on the display of smart glasses (e.g., Google Glass®).

[0347] For example, if a customer is talking to a store staff member about a new smartphone model, the smart glasses will capture the conversation and send the audio data to the server. The server will then convert the speech into text using a speech recognition engine and extract keywords such as "smartphone" or "new model" using a trend analysis module. At the same time, the emotion recognition engine will analyze the customer's emotions and recognize whether they are expressing positive emotions.

[0348] Based on these results, a generative AI model is used to generate appropriate ads, with example prompts such as:

[0349] "If a customer is interested in a new smartphone model and has positive sentiment:

[0350] Explain the features and benefits of the new smartphone model that your customer is interested in and suggest any special promotions you can offer them on the spot.

[0351] The generated advertisements and information are displayed in real time on the smart glasses display, providing customers with valuable information. This transforms customer waiting time into a meaningful experience, improving corporate profitability and customer satisfaction. This system can be applied not only while on hold, but also during interactions in physical stores, making it applicable in a wide range of situations.

[0352] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0353] Step 1:

[0354] A user interacts with a store staff member and asks a question about a specific product or service, such as a new smartphone. The microphone in the smart glasses captures the voice in real time and captures the voice data, which is then encrypted and sent to a server.

[0355] Step 2:

[0356] The server runs the received voice data through a speech recognition engine (Google Cloud Speech-to-Text) and converts the voice data into text data. The input is voice data and the output is text data. Once the voice data has been converted into text, it is ready to proceed to the next analysis step.

[0357] Step 3:

[0358] The server sends the converted data to a trend analysis module (a Python library - spaCy) to extract keywords. The input is the text data and the output is the extracted keywords. The trend analysis module extracts keywords of interest, such as "smartphone" or "new model."

[0359] Step 4:

[0360] The server sends the converted text data to an emotion recognition engine (IBM Watson Tone Analyzer) to analyze the customer's emotions. The input is text data, and the output is the emotional state (positive, negative, neutral, etc.). As a result of the analysis, it is recognized that the customer is expressing positive emotions, for example.

[0361] Step 5:

[0362] The server generates personalized ads using a generative AI model (OpenAI GPT-3 API) based on the results of trend analysis and emotion recognition. The input is the extracted keywords and emotional state, and the output is the ad content. The generative AI model generates ads that suggest, for example, "features of new smartphone models" or "special promotions."

[0363] Step 6:

[0364] The server distributes the generated advertising content to the smart glasses terminal. The input is the advertising content, and the output is the display data for the smart glasses. The smart glasses display the received advertisement on the display in real time and notify the customer.

[0365] Step 7:

[0366] The user (customer) can view the advertisements displayed on the smart glasses display, for example, to view details of special promotions, which increases the customer's motivation to purchase and provides valuable information while they wait.

[0367] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0368] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0369] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0370] [Second embodiment]

[0371] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0372] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0373] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0374] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0375] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0376] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0377] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0378] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0379] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0380] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0381] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0382] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0383] The present invention is a system for providing personalized advertisements in real time to customers on hold. This system integrates speech recognition, trend analysis, advertisement generation, and advertisement distribution functions, with the aim of turning customers' waiting time into a valuable experience.

[0384] Program implementation example

[0385] This system is operated based on the following program processing procedures.

[0386] Speech recognition and text conversion

[0387] User: A customer calls a call center and states a specific question or request to an operator. For example, "I want to know about credit card reward points."

[0388] Device: Captures what the customer says and temporarily stores it as audio data, which is then sent to a server in encrypted form to ensure security.

[0389] Server: The received voice data is analyzed using a voice recognition engine and converted into text data.

[0390] Analyzing text data and identifying customer needs

[0391] Server: The speech recognition engine converts the data into text and sends it to the trend analysis module, which extracts keywords from the text and compares them with past trends and patterns stored in a database to identify customer interests and needs.

[0392] Generating personalized ads

[0393] Server: Based on the results of trend analysis, the server generates advertising content that matches the customer's interests. The advertising generation module creates appropriate advertisements in real time and encodes them as audio files. For example, based on past data, it may determine that new point campaign information would be effective for customers who are interested in "credit card points."

[0394] Delivering personalized ads

[0395] Server: Prepares the encoded audio ad file for sending to the customer's terminal, and sends the data to the terminal via the communication module.

[0396] Terminal: The received audio ad file is played in real time and made available to customers on hold. Example: An audio ad saying "A new points campaign has started" is played.

[0397] Customer response tracking and data updates

[0398] Server: After delivering the ad, the server monitors the conversation after the customer is returned to the operator and tracks their response to the ad. If the customer shows interest in the ad content, the server records this information in a database. This will help improve ad generation in the future.

[0399] Specific examples

[0400] A customer calls a call center and inquires about credit card points. The operator puts the call on hold, and the conversation is captured as voice data by the terminal. The voice data is sent to the server and converted into text by a voice recognition engine. A trend analysis module then extracts the keywords "credit card" and "points" and compares them with a past database. The server generates a personalized advertisement for a "new points campaign" in real time and sends the advertisement to the customer's terminal. The terminal plays the received advertisement to the customer while they are on hold. If the customer then asks about the advertisement when they return to the operator, the server tracks their response and records it in a database. This series of processes transforms customer waiting time into a meaningful experience, improving the company's profitability and customer satisfaction.

[0401] According to these modes for implementation, the present invention is applied to provide personalized announcements to customers whose calls are on hold.

[0402] The processing flow will be explained below.

[0403] Step 1:

[0404] Server: When a customer calls the call center, the server monitors the call status and detects that the operator has put the customer on hold. It sets a hold flag and records that the customer's waiting time has begun.

[0405] Step 2:

[0406] Terminal: Captures the conversation between the operator and the customer as voice data in real time, temporarily stores the voice data before the conversation is put on hold, and encrypts it for security purposes.

[0407] Step 3:

[0408] Server: Receives the encrypted voice data and sends it to the voice recognition engine, which analyzes the voice data and converts it into text data.

[0409] Step 4:

[0410] Server: Sends the converted data to a trend analysis module, which extracts keywords from the text data and matches them with historical data in a database to identify customer interests and needs.

[0411] Step 5:

[0412] Server: Based on the results of trend analysis, the server generates optimal advertising content in real time that matches the customer's interests. The generated advertisements are encoded as audio files and prepared for distribution through the advertisement generation module.

[0413] Step 6:

[0414] Server: Sends the encoded audio advertisement file to the customer's device. The distribution module distributes the advertisement to the customer on hold at the appropriate time.

[0415] Step 7:

[0416] Terminal: The received audio advertisement file is played in real time and made available to customers on hold, turning their waiting time into valuable information time.

[0417] Step 8:

[0418] Server: After delivering the advertisement, monitor the conversation when the customer returns to the operator and track their response to the advertisement. If the customer shows interest in the advertisement content, record that information in the database and use it to generate future advertisements.

[0419] Step 9:

[0420] Server: The accumulated response data is analyzed using machine learning algorithms, forming a feedback loop to improve the accuracy of ad generation and delivery, thereby optimizing the performance of the entire system.

[0421] Through this processing step, the system can turn customer waiting time into a positive experience, providing them with personalized advertising, and improving business profitability and customer satisfaction.

[0422] Example 1

[0423] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0424] Conventional advertisements provided while a call is on hold are generally uniform and do not provide information appropriate to the customer's interests and needs, resulting in lower customer satisfaction and limited advertising effectiveness for companies.

[0425] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0426] In this invention, the server includes means for recognizing the voice of a customer while the call is on hold, means for converting the recognized voice into text data, means for extracting keywords from the text data, means for identifying the interests and needs of the customer by analyzing the extracted keywords, means for generating a personalized advertisement in real time based on the identified interests and needs, means for encoding the generated personalized advertisement as an audio file, means for delivering the encoded personalized advertisement to the customer's terminal, means for playing the advertisement on the customer's terminal, and means for monitoring the customer's response after the advertisement is played and recording the result in a database. This makes it possible to turn the customer's waiting time into a valuable experience, improve customer satisfaction, and maximize the effectiveness of a company's advertising.

[0427] "Call on hold" means that a customer is temporarily placed on hold by a telephone operator or automated answering system.

[0428] "Means for recognizing voice" refers to technology for converting customer utterances into text data, including a voice recognition engine.

[0429] "Means for converting into text data" refers to the technology for analyzing voice data and converting it into text information.

[0430] "Keyword extraction methods" refers to techniques for identifying and extracting specific important words or phrases from text data.

[0431] "Means for identifying customer interests and needs" refers to technology that analyzes extracted keywords and determines customer concerns and requests.

[0432] "Means for generating personalized advertisements" refers to technology that creates advertisements that are individually tailored to a customer's interests and needs.

[0433] "Means for encoding as an audio file" refers to a technique for converting the generated advertising content into an audio data format.

[0434] "Means for delivery to Customer Device" means the technology that transmits the encoded audio file to the Customer Device.

[0435] "Means for playing advertisements on a customer's device" refers to technology for playing received audio files on a customer's device.

[0436] "Means for monitoring customer responses after an advertisement is played" refers to technology that tracks customer behavior and comments after an advertisement is played and collects data based on that.

[0437] "Means of recording in a database" refers to the technology used to store collected customer response data and manage it for analysis and future use.

[0438] MODE FOR CARRYING OUT THE INVENTION

[0439] The present invention relates to a system that provides personalized advertisements to customers on hold in real time. The system integrates speech recognition, trend analysis, advertisement generation, and advertisement distribution functions. The purpose of the system is to turn customers' waiting time into a valuable experience.

[0440] Speech recognition and text conversion

[0441] User: A customer calls the call center with a question or request. For example, they might ask, "I want to know about my credit card points."

[0442] Device: Captures what the customer says and temporarily stores it as audio data, which is then sent to a server in encrypted form to ensure security.

[0443] Server: The received voice data is analyzed using a voice recognition engine (e.g., Google Cloud Speech-to-Text or IBM Watson) and converted into text data.

[0444] Analyzing text data and identifying customer needs

[0445] Server: The data converted to text by the speech recognition engine is sent to a trend analysis module (e.g., Python's NLTK library). This module extracts keywords from the text data and compares them with past trends and patterns stored in a database, thereby identifying customer interests and needs.

[0446] Generating personalized ads

[0447] Server: Based on the results of the trend analysis, advertising content tailored to customer interests is generated using automated speech generation software (e.g., Amazon Polly or Google Text-to-Speech). The generated advertisements are encoded as audio files in real time.

[0448] Delivering personalized ads

[0449] Server: Prepares the encoded audio ad file for sending to the customer's device. Sends the data to the device via a communication module (e.g. HTTP / HTTPS communication).

[0450] Terminal: The received audio ad file is played in real time and made available to customers on hold. For example, an audio ad saying "A new points campaign has started" is played.

[0451] Customer response tracking and data updates

[0452] Server: After delivering the ad, the server monitors the conversation after the customer is returned to the operator and tracks their response to the ad. If the customer shows interest in the ad content, the server records this information in a database. This will help improve ad generation in the future.

[0453] Specific examples

[0454] A customer calls a call center and inquires about credit card points. The operator puts the call on hold, and the conversation is captured as voice data by the terminal. The voice data is sent to the server and converted into text by a voice recognition engine. A trend analysis module then extracts the keywords "credit card" and "points" and compares them with a past database. The server generates a personalized advertisement for a "new points campaign" in real time and sends the advertisement to the customer's terminal. The terminal plays the received advertisement to the customer while they are on hold. If the customer then asks about the advertisement when they return to the operator, the server tracks their response and records it in a database. This series of processes transforms customer waiting time into a meaningful experience, improving the company's profitability and customer satisfaction.

[0455] Prompt Sentence Examples

[0456] A customer calls a call center and asks, "Tell me about credit card points." The operator puts the customer on hold, and the system uses speech recognition to convert the conversation into text. The server then analyzes the text and extracts the keywords "credit card" and "points." Based on this, the system generates a personalized advertisement about a "new points campaign" in real time and plays it to the customer while they are on hold.

[0457] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0458] Step 1:

[0459] User: A customer calls the call center and says, "I want to know about credit card points." The input is the customer's voice. The output is the voice data of what the customer said.

[0460] Step 2:

[0461] Terminal: Captures and temporarily stores what the customer says. The captured voice data is encrypted using AES. The input is the customer's voice data, and the output is encrypted voice data.

[0462] Step 3:

[0463] Terminal: Sends encrypted audio data to the server. The input is the encrypted audio data, and the output is the data sent to the server.

[0464] Step 4:

[0465] Server: The received voice data is analyzed using a speech recognition engine (e.g., Google Cloud Speech-to-Text or IBM Watson) and converted into text data. At this time, the voice data is analyzed and text data such as "I would like to know about credit card points" is generated. The input is encrypted voice data, and the output is analyzed text data.

[0466] Step 5:

[0467] Server: The text data generated by the speech recognition engine is sent to a trend analysis module (e.g., Python's NLTK library). This module extracts keywords such as "credit card" and "points" from the text data and compares them with a historical database. This identifies the customer's interests and needs. The input is text data, and the output is the keywords and their analysis results.

[0468] Step 6:

[0469] Server: Based on the results of the trend analysis, an audio ad containing information about the new points campaign is generated using automatic speech generation software (e.g., Amazon Polly or Google Text-to-Speech). The generated ad content is encoded as an audio file. The input is the analysis results and appropriate ad content information, and the output is an audio file.

[0470] Step 7:

[0471] Server: Sends the encoded audio ad file to the customer's terminal via a communication module (e.g. HTTP / HTTPS communication). The input is the encoded audio file, and the output is the data sent to the terminal.

[0472] Step 8:

[0473] Terminal: Plays the received audio ad file in real time and lets customers on hold hear the audio ad, "A new points campaign has started." The input is the audio ad file received by the terminal, and the output is the audio ad heard by customers on hold.

[0474] Step 9:

[0475] Server: After delivering the ad, monitor the customer's response when they return to the operator. If the customer asks, "Tell me more about the points campaign," track that response and record it in a database. This will help improve future ad generation. The input is the customer's response data after the ad is delivered, and the output is the tracking data recorded in the database.

[0476] (Application example 1)

[0477] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0478] Conventional systems for providing personalized advertisements to customers on hold have had difficulty in fully identifying customer interests and needs and generating and delivering advertisements in real time. Furthermore, it has not been possible to track customer responses and provide feedback to future advertisement generation. The present invention aims to solve these problems and improve the customer shopping experience in virtual stores.

[0479] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0480] In this invention, the server includes means for recognizing the voice of a customer while the call is on hold, means for identifying the interests and needs of the customer from the recognized voice, means for generating personalized advertisements in real time, means for delivering the generated personalized advertisements to the customer's terminal, means for supporting the customer's purchasing experience through voice interaction in a virtual store, and means for tracking the customer's responses and recording them in a database. This makes it possible to provide advertisements based on the customer's interests and needs in real time and improve the purchasing experience.

[0481] A "customer on hold" is a customer who has called a company or support center and is waiting for a response.

[0482] "Speech recognition means" refers to means that uses technology to analyze voice data and convert it into text data.

[0483] "Means for identifying customer interests and needs" refers to a process of identifying what a customer is interested in or needs by analyzing text data obtained by speech recognition means.

[0484] "Personalized advertising" refers to advertising that is individually optimized based on a customer's specific interests and needs.

[0485] A "generative AI model" is a model that uses artificial intelligence technology to generate content such as advertisements from given information.

[0486] The "advertising generation means" refers to a means for automatically generating advertising content suitable for a customer based on the analysis results.

[0487] The "advertising distribution means" refers to a means for transmitting the generated advertisement to the customer's terminal in real time and playing it back.

[0488] "Customer Device" means a communication device used by a Customer, such as a smartphone, smart glasses, or head-mounted display.

[0489] A "virtual store" is an online shop or shopping platform operated via the Internet or virtual reality technology.

[0490] "Means to support customer purchasing experience" refers to means that support customers through voice interaction when browsing, selecting, and purchasing products in a virtual store.

[0491] "Means for tracking customer responses" refers to means for monitoring and collecting data on how customers respond to the advertisements that are played.

[0492] "Means for recording in a database" refers to means for storing tracked customer responses in a database and using them for later analysis and advertisement generation.

[0493] The present invention provides a personalized advertisement providing system for improving a customer's shopping experience in a virtual store. The system is implemented by the following components:

[0494] 1. Speech Recognition and Text Conversion

[0495] server:

[0496] The server is equipped with a speech recognition engine that recognizes the speech customers make in the virtual store and converts it into text data. This speech recognition uses technologies such as Google Cloud Speech-to-Text. The text data is encrypted and stored securely.

[0497] As a specific example, the server captures the audio of a customer saying, "Please tell me about a camera you recommend," and converts the audio into text data.

[0498] 2. Identifying customer interests and needs

[0499] server:

[0500] The server analyzes the text data obtained by the speech recognition tool to extract the customer's interests and needs. This analysis uses trend analysis modules such as "IBM Watson Natural Language Understanding." Here, the server compares the data with past data to identify the customer's interests and needs.

[0501] 3. Generating personalized ads

[0502] server:

[0503] The server generates personalized ads for customers based on the results of trend analysis, using generative AI models such as OpenAI GPT-4, and encodes the ads as audio files.

[0504] Here is an example prompt:

[0505] "Generate an ad for a new digital camera model. Based on past purchase history, you know your customer is interested in high-quality photography. Write compelling copy that includes the features and price of the new model."

[0506] 4. Delivering personalized advertising

[0507] server:

[0508] The server then sends the generated audio ad file to the customer's device, using technologies such as Firebase Cloud Messaging.

[0509] Device:

[0510] The customer's device will play the received audio advertisement file in real time, for example, "New camera model available at special price."

[0511] 5. Tracking customer responses and updating data

[0512] server:

[0513] The server tracks customer responses after serving ads and records the data in a database, which can use database technologies such as MySQL or Elasticsearch, providing feedback to generate more effective ads in the future.

[0514] For example, if a customer listens to an advertisement and then asks the operator again, "Can you tell me more about that camera?", that response will be recorded in the database.

[0515] The present invention enables the generation and delivery of real-time personalized advertisements based on customer interests and needs, improving the customer's shopping experience.

[0516] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0517] Step 1:

[0518] Input: User voice input

[0519] Action: A user enters a virtual store and says, "What camera do you recommend?"

[0520] Server: The server captures the user's voice input and temporarily stores it as audio data. The audio data is stored in encrypted form.

[0521] Output: Encrypted audio data

[0522] Step 2:

[0523] Input: Encrypted audio data

[0524] Processing: The server sends the audio data to a speech recognition engine (e.g., Google Cloud Speech-to-Text) and converts the audio into text data.

[0525] Server: The server sends voice data and receives text data converted by a voice recognition engine.

[0526] Output: Text data (e.g., "What camera would you recommend?")

[0527] Step 3:

[0528] Input: Text data

[0529] Processing: The server sends the text data to a trend analysis module (e.g., IBM Watson Natural Language Understanding) to extract keywords and identify customer interests and needs.

[0530] Server: Extracts keywords (e.g., "camera") from the text data and matches them with past trends and patterns in a database to identify customer interests and needs.

[0531] Output: Customer interests and needs (e.g., "I'm interested in cameras")

[0532] Step 4:

[0533] Input: Customer interests and needs

[0534] Processing: The server creates prompts to generate personalized advertisements based on the identified interests and needs.

[0535] Server: For example, generate a prompt like this: "Generate an advertisement for a new digital camera model. Based on past purchase history, we know that the customer is interested in high-quality photography. Write an appealing ad copy that includes the features and price of the new model."

[0536] Output: prompt statement

[0537] Step 5:

[0538] Input: prompt statement

[0539] Processing: The server sends prompts to a generative AI model (e.g., OpenAI GPT-4) to generate personalized ads.

[0540] Server: The generative AI model generates advertising content based on the prompt and encodes it as an audio file.

[0541] Output: Advertising content in the form of an audio file (e.g., "Special Offer on New Camera Models")

[0542] Step 6:

[0543] Input: Ad content in the form of an audio file

[0544] Processing: The server delivers the audio file to the client's device.

[0545] Server: Sends advertisements to the customer's device via a communication module (e.g., Firebase Cloud Messaging).

[0546] Output: Audio file delivered to the customer's device

[0547] Step 7:

[0548] Input: Streamed audio file

[0549] Processing: The device plays the received audio file in real time.

[0550] Device: The customer's device plays an audio ad (e.g., "Special Offer on New Camera Models").

[0551] Output: Audio ad played to customer

[0552] Step 8:

[0553] Input: Customer response to audio ad

[0554] Processing: The server tracks customer responses to the audio advertisements and records them in a database.

[0555] Server: For example, if the customer asks the operator again, "Please tell me the details of that camera," the response is tracked and stored in a database.

[0556] Output: Customer responses recorded in a database

[0557] All these processing steps result in the generation and delivery of real-time personalized ads based on customer interests and needs, improving the customer shopping experience.

[0558] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0559] The present invention combines a system that provides personalized advertisements to customers on hold in real time with an emotion engine that recognizes user emotions. This system integrates speech recognition, trend analysis, advertisement generation, advertisement distribution, and emotion recognition functions, with the aim of turning customers' waiting time into a valuable experience.

[0560] Program implementation example

[0561] This system is operated based on the following program processing procedures.

[0562] Speech recognition and text conversion

[0563] User: A customer calls a call center and states a specific question or request to an operator. For example, "I want to know about credit card reward points."

[0564] Device: Captures and temporarily stores the customer's speech as audio data in real time. This data is encrypted and sent to a server for security purposes.

[0565] Server: Sends the received voice data to the voice recognition engine and converts the voice data into text data.

[0566] Analyzing text data and identifying customer needs

[0567] Server: Sends the text data to a trend analysis module, which extracts keywords from the text data and matches them with historical data in a database to identify customer interests and needs.

[0568] emotion recognition

[0569] Server: The text data is also sent to the emotion engine, which analyzes the tone of the customer's voice and the content of what they say to recognize their emotional state (e.g., joy, anger, sadness, etc.).

[0570] Example: Recognizing emotions such as "customer dissatisfaction" from the customer's tone of voice and speech patterns.

[0571] Generating personalized ads

[0572] Server: Based on the results of trend analysis and the analysis results of the emotion engine, the server generates appropriate advertising content in real time that matches the customer's interests and emotions. For example, if a customer is dissatisfied, the server will display an apology message or information about shortening waiting times instead of a general advertisement.

[0573] Example: A customer who is interested in "credit card points" but dissatisfied with them can be informed of the start of a "new points campaign" and also be given information about the expected waiting time.

[0574] Delivering personalized ads

[0575] Server: Sends the encoded audio ad file to the customer's device. Through the distribution module, the ad is delivered to the customer on hold at the appropriate time.

[0576] Terminal: The received audio ad file is played in real time and made available to customers on hold, turning waiting time into valuable information time.

[0577] Customer response tracking and data updates

[0578] Server: After the ad is delivered, the server monitors the conversation after the customer is returned to the operator and tracks their response to the ad. If the customer shows interest in the ad content or responds indicative of their emotional state, this information is recorded in a database. This will be used to improve the accuracy of future ad generation and emotion recognition.

[0579] Specific examples

[0580] A customer calls a call center and inquires about credit card points. The operator puts the call on hold, and the conversation is captured as voice data by the terminal and sent to the server. The voice data is converted into text by a speech recognition engine. A trend analysis module then extracts the keywords "credit card" and "points," and an emotion engine recognizes the customer's emotional state. The server generates a personalized advertisement about a "new points campaign" in real time, along with information about the expected waiting time, and sends it to the customer's terminal as an audio file. The terminal plays the advertisement to the customer while they are on hold. The customer is then returned to the operator, and if they ask a question about the advertisement or respond emotionally, the server tracks that data and records it in a database. This process turns customer waiting time into a meaningful experience, improving the company's profitability and customer satisfaction.

[0581] In this way, the present invention is implemented as a system that provides optimal advertisements while a call is on hold, taking into consideration the customer's emotions, and achieves both customer satisfaction and corporate profits.

[0582] The processing flow will be explained below.

[0583] Step 1:

[0584] Server: When a customer calls the call center, the server monitors the call status and detects that the operator has put the customer on hold. It sets a hold flag and records that the customer's wait time has begun.

[0585] Step 2:

[0586] Terminal: Captures the conversation between the operator and the customer as voice data in real time. The voice data before the conversation is put on hold is temporarily stored, encrypted, and sent to the server.

[0587] Step 3:

[0588] Server: Receives the encrypted voice data and sends it to the voice recognition engine, which analyzes the voice data and converts it into text data.

[0589] Step 4:

[0590] Server: Sends the text data to the trend analysis module, which extracts keywords from the text data and matches them with historical data in the database to identify customer interests and needs.

[0591] Step 5:

[0592] Server: Sends text data to the emotion engine. The emotion engine analyzes the tone of the customer's voice and the content of what they say to recognize their emotional state. For example, it recognizes that "the customer's tone of voice indicates dissatisfaction."

[0593] Step 6:

[0594] Server: Based on the results of trend analysis and the emotion engine, the server generates personalized advertisements in real time that match the customer's interests and emotions. For example, if a customer is interested in and dissatisfied with "credit card points," the server generates information about new campaigns related to points and an apology message for keeping them waiting.

[0595] Step 7:

[0596] Server: Encodes the generated personalized ad into an audio file and prepares it for delivery.

[0597] Step 8:

[0598] Server: Sends the encoded audio ad file to the customer's device. The distribution module distributes the ad to the pending customer at the appropriate time.

[0599] Step 9:

[0600] Terminal: The received audio advertisement file is played in real time, allowing customers on hold to listen to the advertisement, thereby turning the customer's waiting time into valuable information time.

[0601] Step 10:

[0602] Server: After the advertisement is delivered, the server monitors the conversation when the customer is returned to the operator. It tracks the response to the advertisement, and if the customer shows interest in the advertisement content, it records that information in a database.

[0603] Step 11:

[0604] Server: Machine learning algorithms analyze the accumulated response data, creating a feedback loop to improve the accuracy of ad generation and delivery, thereby optimizing the performance of the entire system.

[0605] When a customer calls a call center to inquire about credit card points, the operator puts the call on hold. The terminal captures what the customer says as voice data and sends it to the server. The voice data is converted into text by a speech recognition engine. A trend analysis module then extracts keywords, and an emotion engine recognizes the customer's emotional state. The server generates a personalized advertisement in real time and sends it to the customer's terminal as an audio file. The terminal plays the advertisement to the customer while they are on hold. If the customer is then returned to the operator and asks a question about the advertisement, the server tracks their response and records it in a database. Through this process, the customer's waiting time is transformed into a meaningful experience, improving the company's profitability and customer satisfaction.

[0606] Example 2

[0607] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0608] In conventional call center systems, advertisements and messages provided to customers while they were on hold were typically not tailored to the individual customer's interests and needs. As a result, customers were unable to spend their waiting time meaningfully, which could lead to lower customer satisfaction. Furthermore, delivery that did not take into account the customer's emotional state could exacerbate customer dissatisfaction. To solve these problems, the present invention aims to effectively utilize waiting time and improve customer satisfaction by analyzing the customer's interests, needs, and even their emotional state in real time and providing personalized advertisements.

[0609] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0610] In this invention, the server includes means for recognizing the voice of a customer while the call is on hold, means for identifying the interests and needs of the customer from the recognized voice, means for analyzing the emotional state of the customer, means for generating personalized advertisements in real time based on the identified interests, needs and emotional state, and means for delivering the generated personalized advertisements to the customer's device, thereby enabling the customer's waiting time while on hold to be transformed into time for providing information suited to their individual needs and emotions.

[0611] "Customers on hold" means customers who have called the call center and are on hold waiting for an operator to respond.

[0612] "Voice recognition means" refers to devices or software that have the ability to collect customer voice data, analyze it, and convert it into text.

[0613] "Means for identifying customer interests and needs" refers to algorithms or devices that analyze and extract customer interests and requests from text data.

[0614] "Means for analyzing the emotional state of the customer" refers to algorithms or devices that recognize emotions based on the customer's voice and text data.

[0615] "Personalized advertising" refers to advertising that is individually tailored to take into account the interests, needs, and emotional state of a particular customer.

[0616] "Generating means" refers to algorithms or devices that generate specific content or advertisements based on the results of data analysis.

[0617] "Delivery means" refers to the algorithms and / or equipment used to transmit and play generated content and advertisements on Customer's devices.

[0618] "Real-time" refers to instantaneous processing or response with very low latency.

[0619] "Device" means an electronic device capable of sending and receiving voice and data that is used by a customer while a call is on hold.

[0620] The present invention relates to a system that provides personalized advertisements to customers on hold in real time, and further combines it with an emotion engine that recognizes user emotions. This system integrates speech recognition, trend analysis, advertisement generation, advertisement distribution, and emotion recognition functions, with the aim of turning customers' waiting time into a valuable experience.

[0621] The whole system consists of three main components: server, terminal, and user. Here, the server is responsible for data processing and analysis, the terminal is the interface with the customer, and the user is the customer who uses the system.

[0622] First, a user calls the call center and states a specific question or request to an operator. For example, "I want to know about credit card points." At this time, the terminal captures the user's speech as voice data in real time and temporarily stores it. The captured voice data is AES encrypted to ensure security and sent to the server.

[0623] The server sends the received voice data to a speech recognition engine (e.g., Google Cloud Speech-to-Text), which converts the voice data into text data. The text data is then sent to a natural language processing (NLP) module, which extracts keywords related to the customer's interests and needs from the text data. Furthermore, an emotion engine (e.g., IBM Watson Tone Analyzer) analyzes the customer's tone of voice and the content of what is being said to recognize the customer's emotional state (e.g., joy, anger, sadness).

[0624] Next, the server generates appropriate advertising content in real time based on the results of trend analysis and the emotion engine's analysis, matching the customer's interests and emotions. For example, if a customer is interested in "credit card points" but is dissatisfied, an advertisement containing information about a new points campaign and a message of goodwill will be generated.

[0625] The generated advertising content is encoded as an audio file and sent from the server to the customer's device. The device then plays the received advertising audio file in real time and lets the customer listen to it while on hold, thereby turning the customer's waiting time into valuable information time.

[0626] After the ad is delivered, the server monitors the conversation and tracks the customer's response to the ad. If the customer shows interest in the ad content or responds in a way that indicates a particular emotional state, that information is recorded in a database. This information can be used to generate future ads and improve the accuracy of emotion recognition.

[0627] For example, a user calls a call center and inquires about credit card points. The conversation is captured as voice data by the terminal and sent to the server. The voice data is then converted into text by the Google Cloud Speech-to-Text engine. An NLP module then extracts the keywords "credit card" and "points," and an emotion engine (IBM Watson Tone Analyzer) recognizes the customer's emotional state. The server generates a personalized advertisement about a "new points campaign" in real time, along with information about the expected waiting time, and sends it to the customer's terminal as an audio file. The terminal plays the advertisement to the customer while they are on hold. The customer is then returned to the operator, and if they ask a question about the advertisement or respond emotionally, the server tracks that data and records it in a database. This process transforms customer waiting time into a meaningful experience, improving the company's profitability and customer satisfaction.

[0628] The above is an embodiment of the present invention.

[0629] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0630] Step 1:

[0631] A user calls a call center and states a specific question or request to an operator.

[0632] Input: User's spoken utterance

[0633] Output: Audio data

[0634] Specific operation: A user calls and asks, "I want to know about credit card points."

[0635] Step 2:

[0636] The device captures the user's speech as audio data in real time.

[0637] Input: User's voice data

[0638] Output: Encrypted audio data

[0639] Specific operation: The device encrypts the captured audio data using AES and temporarily stores it.

[0640] Step 3:

[0641] The device sends the encrypted audio data to the server.

[0642] Input: Encrypted audio data

[0643] Output: Encrypted audio data sent to the server

[0644] Specific operation: The device sends encrypted voice data to the server via the network.

[0645] Step 4:

[0646] The server sends the received voice data to a voice recognition engine, which converts the voice data into text data.

[0647] Input: Encrypted audio data

[0648] Output: Text data

[0649] Specific operation: The server uses the Google Cloud Speech-to-Text engine to convert the audio data into text data.

[0650] Step 5:

[0651] The server sends the converted data to a natural language processing (NLP) module to extract keywords.

[0652] Input: Text data

[0653] Output: Extracted keywords

[0654] Specific operation: The server uses an NLP module to analyze and extract keywords such as "credit card" and "points."

[0655] Step 6:

[0656] The server simultaneously sends the text data to the emotion engine to analyze the customer's emotional state.

[0657] Input: Text data

[0658] Output: Customer's emotional state

[0659] What it does: The server uses IBM Watson Tone Analyzer to identify the customer's emotional state from the text data.

[0660] Step 7:

[0661] The server generates personalized advertisements based on keywords and emotional states.

[0662] Input: extracted keywords and emotional states

[0663] Output: Personalized advertising content

[0664] Specific operation: The server generates appropriate advertising content (e.g., "New points campaign" and "Information on shortened waiting times") in accordance with the customer's interests and emotions.

[0665] Step 8:

[0666] The server encodes the generated advertising content as an audio file and transmits it to the terminal.

[0667] Input: Personalized advertising content

[0668] Output: Audio file

[0669] Specific operation: The server encodes the advertising content into an audio file and sends it to the device.

[0670] Step 9:

[0671] The terminal plays the received audio advertisement file and lets the customer on hold hear it.

[0672] Input: Audio ad file

[0673] Output: Customer hears audio ad

[0674] What it does: The terminal plays audio ads in real time on the customer's device.

[0675] Step 10:

[0676] The server monitors the conversation between the customer and the operator after the advertisement is delivered and tracks the response to the advertisement.

[0677] Input: Customer and operator conversation data

[0678] Output: Customer response data

[0679] What it does: The server analyzes the conversation data and records whether the customer has a specific reaction or question to the ad.

[0680] These are the processing steps of the system, which will make customer waiting time more meaningful and increase the company's profitability and customer satisfaction.

[0681] (Application example 2)

[0682] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0683] Conventional on-hold ad delivery systems have struggled to deliver personalized ads that reflect customer emotions and specific interests. This has resulted in customers' waiting time not being fully utilized as an opportunity to provide information, preventing customer satisfaction and corporate profitability from being fully improved. Furthermore, in brick-and-mortar stores, there has been a lack of timely product recommendations that reflect customer emotions, resulting in missed sales opportunities.

[0684] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing the voice of a customer while the call is on hold, means for identifying the customer's interests and needs from the recognized voice, means for analyzing the customer's emotions, means for generating a personalized advertisement in real time based on the identified interests and needs and the customer's emotions, means for delivering the generated personalized advertisement to the customer's terminal, and means for displaying the generated advertisement on the smart glasses. This makes it possible to provide personalized advertisements and product suggestions in real time according to the customer's emotions and interests.

[0685] "Call on hold" refers to a state in which a user pauses a telephone call while waiting for another operation or a response.

[0686] A "voice recognition means" is a device or program that captures customer speech as voice data and converts it into text.

[0687] The "means for identifying customer interests and needs" refers to a device or program that extracts keywords from the customer's text data and identifies the customer's interests and requests based on this information.

[0688] The "means for analyzing customer emotions" refers to a device or program that analyzes the tone of a customer's voice and the content of their statements to recognize the customer's emotional state.

[0689] "Personalized advertising" is advertising content that is specific to a customer's individual interests, needs, and emotional state.

[0690] "Generating means" refers to a device or program that automatically creates advertisements or information based on the analyzed data.

[0691] "Distribution means" refers to a device or program that transmits the generated advertisements and information to the customer's device (such as a smartphone or smart glasses).

[0692] "Smart glasses" are wearable devices equipped with functions such as a display, camera, and microphone, and capable of displaying information and capturing audio.

[0693] "Real-time" refers to processing and information being provided immediately without delay.

[0694] This invention is a system that provides personalized advertisements in real time while taking into account customer emotions during call on-hold or in-store interactions. The system operates by integrating speech recognition, trend analysis, advertisement generation, advertisement distribution, and emotion recognition functions.

[0695] The server converts the customer's voice into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text). The converted data is sent to a trend analysis module (e.g., Python library - spaCy) to extract keywords. After that, an emotion recognition engine (e.g., IBM Watson Tone Analyzer) is used to analyze the customer's emotional state.

[0696] Based on the identified keywords and emotional state, the server automatically generates personalized ads using a generative AI model (e.g., OpenAI GPT-3 API), which are encoded as audio files or text information and delivered to the customer's device. Specifically, the ads are displayed on the display of smart glasses (e.g., Google Glass).

[0697] For example, if a customer is talking to a store staff member about a new smartphone model, the smart glasses will capture the conversation and send the audio data to the server. The server will then convert the speech into text using a speech recognition engine and extract keywords such as "smartphone" or "new model" using a trend analysis module. At the same time, the emotion recognition engine will analyze the customer's emotions and recognize whether they are expressing positive emotions.

[0698] Based on these results, a generative AI model is used to generate appropriate ads, with example prompts such as:

[0699] "If a customer is interested in a new smartphone model and has positive sentiment:

[0700] Explain the features and benefits of the new smartphone model that your customer is interested in and suggest any special promotions you can offer them on the spot.

[0701] The generated advertisements and information are displayed in real time on the smart glasses display, providing customers with valuable information. This transforms customer waiting time into a meaningful experience, improving corporate profitability and customer satisfaction. This system can be applied not only while on hold, but also during interactions in physical stores, making it applicable in a wide range of situations.

[0702] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0703] Step 1:

[0704] A user interacts with a store staff member and asks a question about a specific product or service, such as a new smartphone. The microphone in the smart glasses captures the voice in real time and captures the voice data, which is then encrypted and sent to a server.

[0705] Step 2:

[0706] The server runs the received voice data through a speech recognition engine (Google Cloud Speech-to-Text) and converts the voice data into text data. The input is voice data and the output is text data. Once the voice data has been converted into text, it is ready to proceed to the next analysis step.

[0707] Step 3:

[0708] The server sends the converted data to a trend analysis module (a Python library - spaCy) to extract keywords. The input is the text data and the output is the extracted keywords. The trend analysis module extracts keywords of interest, such as "smartphone" or "new model."

[0709] Step 4:

[0710] The server sends the converted text data to an emotion recognition engine (IBM Watson Tone Analyzer) to analyze the customer's emotions. The input is text data, and the output is the emotional state (positive, negative, neutral, etc.). As a result of the analysis, it is recognized that the customer is expressing positive emotions, for example.

[0711] Step 5:

[0712] The server generates personalized ads using a generative AI model (OpenAI GPT-3 API) based on the results of trend analysis and emotion recognition. The input is the extracted keywords and emotional state, and the output is the ad content. The generative AI model generates ads that suggest, for example, "features of new smartphone models" or "special promotions."

[0713] Step 6:

[0714] The server distributes the generated advertising content to the smart glasses terminal. The input is the advertising content, and the output is the display data for the smart glasses. The smart glasses display the received advertisement on the display in real time and notify the customer.

[0715] Step 7:

[0716] The user (customer) can view the advertisements displayed on the smart glasses display, for example, to view details of special promotions, which increases the customer's motivation to purchase and provides valuable information while they wait.

[0717] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0718] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0719] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0720] [Third embodiment]

[0721] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0722] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0723] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0724] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0725] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0726] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0727] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0728] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0729] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0730] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0731] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0732] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0733] The present invention is a system for providing personalized advertisements in real time to customers on hold. This system integrates speech recognition, trend analysis, advertisement generation, and advertisement distribution functions, with the aim of turning customers' waiting time into a valuable experience.

[0734] Program implementation example

[0735] This system is operated based on the following program processing procedures.

[0736] Speech recognition and text conversion

[0737] User: A customer calls a call center and states a specific question or request to an operator. For example, "I want to know about credit card reward points."

[0738] Device: Captures what the customer says and temporarily stores it as audio data, which is then sent to a server in encrypted form to ensure security.

[0739] Server: The received voice data is analyzed using a voice recognition engine and converted into text data.

[0740] Analyzing text data and identifying customer needs

[0741] Server: The speech recognition engine converts the data into text and sends it to the trend analysis module, which extracts keywords from the text and compares them with past trends and patterns stored in a database to identify customer interests and needs.

[0742] Generating personalized ads

[0743] Server: Based on the results of trend analysis, the server generates advertising content that matches the customer's interests. The advertising generation module creates appropriate advertisements in real time and encodes them as audio files. For example, based on past data, it may determine that new point campaign information would be effective for customers who are interested in "credit card points."

[0744] Delivering personalized ads

[0745] Server: Prepares the encoded audio ad file for sending to the customer's terminal, and sends the data to the terminal via the communication module.

[0746] Terminal: The received audio ad file is played in real time and made available to customers on hold. Example: An audio ad saying "A new points campaign has started" is played.

[0747] Customer response tracking and data updates

[0748] Server: After delivering the ad, the server monitors the conversation after the customer is returned to the operator and tracks their response to the ad. If the customer shows interest in the ad content, the server records this information in a database. This will help improve ad generation in the future.

[0749] Specific examples

[0750] A customer calls a call center and inquires about credit card points. The operator puts the call on hold, and the conversation is captured as voice data by the terminal. The voice data is sent to the server and converted into text by a voice recognition engine. A trend analysis module then extracts the keywords "credit card" and "points" and compares them with a past database. The server generates a personalized advertisement for a "new points campaign" in real time and sends the advertisement to the customer's terminal. The terminal plays the received advertisement to the customer while they are on hold. If the customer then asks about the advertisement when they return to the operator, the server tracks their response and records it in a database. This series of processes transforms customer waiting time into a meaningful experience, improving the company's profitability and customer satisfaction.

[0751] According to these modes for implementation, the present invention is applied to provide personalized announcements to customers whose calls are on hold.

[0752] The processing flow will be explained below.

[0753] Step 1:

[0754] Server: When a customer calls the call center, the server monitors the call status and detects that the operator has put the customer on hold. It sets a hold flag and records that the customer's waiting time has begun.

[0755] Step 2:

[0756] Terminal: Captures the conversation between the operator and the customer as voice data in real time, temporarily stores the voice data before the conversation is put on hold, and encrypts it for security purposes.

[0757] Step 3:

[0758] Server: Receives the encrypted voice data and sends it to the voice recognition engine, which analyzes the voice data and converts it into text data.

[0759] Step 4:

[0760] Server: Sends the converted data to a trend analysis module, which extracts keywords from the text data and matches them with historical data in a database to identify customer interests and needs.

[0761] Step 5:

[0762] Server: Based on the results of trend analysis, the server generates optimal advertising content in real time that matches the customer's interests. The generated advertisements are encoded as audio files and prepared for distribution through the advertisement generation module.

[0763] Step 6:

[0764] Server: Sends the encoded audio advertisement file to the customer's device. The distribution module distributes the advertisement to the customer on hold at the appropriate time.

[0765] Step 7:

[0766] Terminal: The received audio advertisement file is played in real time and made available to customers on hold, turning their waiting time into valuable information time.

[0767] Step 8:

[0768] Server: After delivering the advertisement, monitor the conversation when the customer returns to the operator and track their response to the advertisement. If the customer shows interest in the advertisement content, record that information in the database and use it to generate future advertisements.

[0769] Step 9:

[0770] Server: The accumulated response data is analyzed using machine learning algorithms, forming a feedback loop to improve the accuracy of ad generation and delivery, thereby optimizing the performance of the entire system.

[0771] Through this processing step, the system can turn customer waiting time into a positive experience, providing them with personalized advertising, and improving business profitability and customer satisfaction.

[0772] Example 1

[0773] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0774] Conventional advertisements provided while a call is on hold are generally uniform and do not provide information appropriate to the customer's interests and needs, resulting in lower customer satisfaction and limited advertising effectiveness for companies.

[0775] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0776] In this invention, the server includes means for recognizing the voice of a customer while the call is on hold, means for converting the recognized voice into text data, means for extracting keywords from the text data, means for identifying the interests and needs of the customer by analyzing the extracted keywords, means for generating a personalized advertisement in real time based on the identified interests and needs, means for encoding the generated personalized advertisement as an audio file, means for delivering the encoded personalized advertisement to the customer's terminal, means for playing the advertisement on the customer's terminal, and means for monitoring the customer's response after the advertisement is played and recording the result in a database. This makes it possible to turn the customer's waiting time into a valuable experience, improve customer satisfaction, and maximize the effectiveness of a company's advertising.

[0777] "Call on hold" means that a customer is temporarily placed on hold by a telephone operator or automated answering system.

[0778] "Means for recognizing voice" refers to technology for converting customer utterances into text data, including a voice recognition engine.

[0779] "Means for converting into text data" refers to the technology for analyzing voice data and converting it into text information.

[0780] "Keyword extraction methods" refers to techniques for identifying and extracting specific important words or phrases from text data.

[0781] "Means for identifying customer interests and needs" refers to technology that analyzes extracted keywords and determines customer concerns and requests.

[0782] "Means for generating personalized advertisements" refers to technology that creates advertisements that are individually tailored to a customer's interests and needs.

[0783] "Means for encoding as an audio file" refers to a technique for converting the generated advertising content into an audio data format.

[0784] "Means for delivery to Customer Device" means the technology that transmits the encoded audio file to the Customer Device.

[0785] "Means for playing advertisements on a customer's device" refers to technology for playing received audio files on a customer's device.

[0786] "Means for monitoring customer responses after an advertisement is played" refers to technology that tracks customer behavior and comments after an advertisement is played and collects data based on that.

[0787] "Means of recording in a database" refers to the technology used to store collected customer response data and manage it for analysis and future use.

[0788] MODE FOR CARRYING OUT THE INVENTION

[0789] The present invention relates to a system that provides personalized advertisements to customers on hold in real time. The system integrates speech recognition, trend analysis, advertisement generation, and advertisement distribution functions. The purpose of the system is to turn customers' waiting time into a valuable experience.

[0790] Speech recognition and text conversion

[0791] User: A customer calls the call center with a question or request. For example, they might ask, "I want to know about my credit card points."

[0792] Device: Captures what the customer says and temporarily stores it as audio data, which is then sent to a server in encrypted form to ensure security.

[0793] Server: The received voice data is analyzed using a voice recognition engine (e.g., Google Cloud Speech-to-Text or IBM Watson) and converted into text data.

[0794] Analyzing text data and identifying customer needs

[0795] Server: The data converted to text by the speech recognition engine is sent to a trend analysis module (e.g., Python's NLTK library). This module extracts keywords from the text data and compares them with past trends and patterns stored in a database, thereby identifying customer interests and needs.

[0796] Generating personalized ads

[0797] Server: Based on the results of the trend analysis, advertising content tailored to customer interests is generated using automated speech generation software (e.g., Amazon Polly or Google Text-to-Speech). The generated advertisements are encoded as audio files in real time.

[0798] Delivering personalized ads

[0799] Server: Prepares the encoded audio ad file for sending to the customer's device. Sends the data to the device via a communication module (e.g. HTTP / HTTPS communication).

[0800] Terminal: The received audio ad file is played in real time and made available to customers on hold. For example, an audio ad saying "A new points campaign has started" is played.

[0801] Customer response tracking and data updates

[0802] Server: After delivering the ad, the server monitors the conversation after the customer is returned to the operator and tracks their response to the ad. If the customer shows interest in the ad content, the server records this information in a database. This will help improve ad generation in the future.

[0803] Specific examples

[0804] A customer calls a call center and inquires about credit card points. The operator puts the call on hold, and the conversation is captured as voice data by the terminal. The voice data is sent to the server and converted into text by a voice recognition engine. A trend analysis module then extracts the keywords "credit card" and "points" and compares them with a past database. The server generates a personalized advertisement for a "new points campaign" in real time and sends the advertisement to the customer's terminal. The terminal plays the received advertisement to the customer while they are on hold. If the customer then asks about the advertisement when they return to the operator, the server tracks their response and records it in a database. This series of processes transforms customer waiting time into a meaningful experience, improving the company's profitability and customer satisfaction.

[0805] Prompt Sentence Examples

[0806] A customer calls a call center and asks, "Tell me about credit card points." The operator puts the customer on hold, and the system uses speech recognition to convert the conversation into text. The server then analyzes the text and extracts the keywords "credit card" and "points." Based on this, the system generates a personalized advertisement about a "new points campaign" in real time and plays it to the customer while they are on hold.

[0807] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0808] Step 1:

[0809] User: A customer calls the call center and says, "I want to know about credit card points." The input is the customer's voice. The output is the voice data of what the customer said.

[0810] Step 2:

[0811] Terminal: Captures and temporarily stores what the customer says. The captured voice data is encrypted using AES. The input is the customer's voice data, and the output is encrypted voice data.

[0812] Step 3:

[0813] Terminal: Sends encrypted audio data to the server. The input is the encrypted audio data, and the output is the data sent to the server.

[0814] Step 4:

[0815] Server: The received voice data is analyzed using a speech recognition engine (e.g., Google Cloud Speech-to-Text or IBM Watson) and converted into text data. At this time, the voice data is analyzed and text data such as "I would like to know about credit card points" is generated. The input is encrypted voice data, and the output is analyzed text data.

[0816] Step 5:

[0817] Server: The text data generated by the speech recognition engine is sent to a trend analysis module (e.g., Python's NLTK library). This module extracts keywords such as "credit card" and "points" from the text data and compares them with a historical database. This identifies the customer's interests and needs. The input is text data, and the output is the keywords and their analysis results.

[0818] Step 6:

[0819] Server: Based on the results of the trend analysis, an audio ad containing information about the new points campaign is generated using automatic speech generation software (e.g., Amazon Polly or Google Text-to-Speech). The generated ad content is encoded as an audio file. The input is the analysis results and appropriate ad content information, and the output is an audio file.

[0820] Step 7:

[0821] Server: Sends the encoded audio ad file to the customer's terminal via a communication module (e.g. HTTP / HTTPS communication). The input is the encoded audio file, and the output is the data sent to the terminal.

[0822] Step 8:

[0823] Terminal: Plays the received audio ad file in real time and lets customers on hold hear the audio ad, "A new points campaign has started." The input is the audio ad file received by the terminal, and the output is the audio ad heard by customers on hold.

[0824] Step 9:

[0825] Server: After delivering the ad, monitor the customer's response when they return to the operator. If the customer asks, "Tell me more about the points campaign," track that response and record it in a database. This will help improve future ad generation. The input is the customer's response data after the ad is delivered, and the output is the tracking data recorded in the database.

[0826] (Application example 1)

[0827] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0828] Conventional systems for providing personalized advertisements to customers on hold have had difficulty in fully identifying customer interests and needs and generating and delivering advertisements in real time. Furthermore, it has not been possible to track customer responses and provide feedback to future advertisement generation. The present invention aims to solve these problems and improve the customer shopping experience in virtual stores.

[0829] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0830] In this invention, the server includes means for recognizing the voice of a customer while the call is on hold, means for identifying the interests and needs of the customer from the recognized voice, means for generating personalized advertisements in real time, means for delivering the generated personalized advertisements to the customer's terminal, means for supporting the customer's purchasing experience through voice interaction in a virtual store, and means for tracking the customer's responses and recording them in a database. This makes it possible to provide advertisements based on the customer's interests and needs in real time and improve the purchasing experience.

[0831] A "customer on hold" is a customer who has called a company or support center and is waiting for a response.

[0832] "Speech recognition means" refers to means that uses technology to analyze voice data and convert it into text data.

[0833] "Means for identifying customer interests and needs" refers to a process of identifying what a customer is interested in or needs by analyzing text data obtained by speech recognition means.

[0834] "Personalized advertising" refers to advertising that is individually optimized based on a customer's specific interests and needs.

[0835] A "generative AI model" is a model that uses artificial intelligence technology to generate content such as advertisements from given information.

[0836] The "advertising generation means" refers to a means for automatically generating advertising content suitable for a customer based on the analysis results.

[0837] The "advertising distribution means" refers to a means for transmitting the generated advertisement to the customer's terminal in real time and playing it back.

[0838] "Customer Device" means a communication device used by a Customer, such as a smartphone, smart glasses, or head-mounted display.

[0839] A "virtual store" is an online shop or shopping platform operated via the Internet or virtual reality technology.

[0840] "Means to support customer purchasing experience" refers to means that support customers through voice interaction when browsing, selecting, and purchasing products in a virtual store.

[0841] "Means for tracking customer responses" refers to means for monitoring and collecting data on how customers respond to the advertisements that are played.

[0842] "Means for recording in a database" refers to means for storing tracked customer responses in a database and using them for later analysis and advertisement generation.

[0843] The present invention provides a personalized advertisement providing system for improving a customer's shopping experience in a virtual store. The system is implemented by the following components:

[0844] 1. Speech Recognition and Text Conversion

[0845] server:

[0846] The server is equipped with a speech recognition engine that recognizes the speech customers make in the virtual store and converts it into text data. This speech recognition uses technologies such as Google Cloud Speech-to-Text. The text data is encrypted and stored securely.

[0847] As a specific example, the server captures the audio of a customer saying, "Please tell me about a camera you recommend," and converts the audio into text data.

[0848] 2. Identifying customer interests and needs

[0849] server:

[0850] The server analyzes the text data obtained by the speech recognition tool to extract the customer's interests and needs. This analysis uses trend analysis modules such as "IBM Watson Natural Language Understanding." Here, the server compares the data with past data to identify the customer's interests and needs.

[0851] 3. Generating personalized ads

[0852] server:

[0853] The server generates personalized ads for customers based on the results of trend analysis, using generative AI models such as OpenAI GPT-4, and encodes the ads as audio files.

[0854] Here is an example prompt:

[0855] "Generate an ad for a new digital camera model. Based on past purchase history, you know your customer is interested in high-quality photography. Write compelling copy that includes the features and price of the new model."

[0856] 4. Delivering personalized advertising

[0857] server:

[0858] The server then sends the generated audio ad file to the customer's device, using technologies such as Firebase Cloud Messaging.

[0859] Device:

[0860] The customer's device will play the received audio advertisement file in real time, for example, "New camera model available at special price."

[0861] 5. Tracking customer responses and updating data

[0862] server:

[0863] The server tracks customer responses after serving ads and records the data in a database, which can use database technologies such as MySQL or Elasticsearch, providing feedback to generate more effective ads in the future.

[0864] For example, if a customer listens to an advertisement and then asks the operator again, "Can you tell me more about that camera?", that response will be recorded in the database.

[0865] The present invention enables the generation and delivery of real-time personalized advertisements based on customer interests and needs, improving the customer's shopping experience.

[0866] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0867] Step 1:

[0868] Input: User voice input

[0869] Action: A user enters a virtual store and says, "What camera do you recommend?"

[0870] Server: The server captures the user's voice input and temporarily stores it as audio data. The audio data is stored in encrypted form.

[0871] Output: Encrypted audio data

[0872] Step 2:

[0873] Input: Encrypted audio data

[0874] Processing: The server sends the audio data to a speech recognition engine (e.g., Google Cloud Speech-to-Text) and converts the audio into text data.

[0875] Server: The server sends voice data and receives text data converted by a voice recognition engine.

[0876] Output: Text data (e.g., "What camera would you recommend?")

[0877] Step 3:

[0878] Input: Text data

[0879] Processing: The server sends the text data to a trend analysis module (e.g., IBM Watson Natural Language Understanding) to extract keywords and identify customer interests and needs.

[0880] Server: Extracts keywords (e.g., "camera") from the text data and matches them with past trends and patterns in a database to identify customer interests and needs.

[0881] Output: Customer interests and needs (e.g., "I'm interested in cameras")

[0882] Step 4:

[0883] Input: Customer interests and needs

[0884] Processing: The server creates prompts to generate personalized advertisements based on the identified interests and needs.

[0885] Server: For example, generate a prompt like this: "Generate an advertisement for a new digital camera model. Based on past purchase history, we know that the customer is interested in high-quality photography. Write an appealing ad copy that includes the features and price of the new model."

[0886] Output: prompt statement

[0887] Step 5:

[0888] Input: prompt statement

[0889] Processing: The server sends prompts to a generative AI model (e.g., OpenAI GPT-4) to generate personalized ads.

[0890] Server: The generative AI model generates advertising content based on the prompt and encodes it as an audio file.

[0891] Output: Advertising content in the form of an audio file (e.g., "Special Offer on New Camera Models")

[0892] Step 6:

[0893] Input: Ad content in the form of an audio file

[0894] Processing: The server delivers the audio file to the client's device.

[0895] Server: Sends advertisements to the customer's device via a communication module (e.g., Firebase Cloud Messaging).

[0896] Output: Audio file delivered to the customer's device

[0897] Step 7:

[0898] Input: Streamed audio file

[0899] Processing: The device plays the received audio file in real time.

[0900] Device: The customer's device plays an audio ad (e.g., "Special Offer on New Camera Models").

[0901] Output: Audio ad played to customer

[0902] Step 8:

[0903] Input: Customer response to audio ad

[0904] Processing: The server tracks customer responses to the audio advertisements and records them in a database.

[0905] Server: For example, if the customer asks the operator again, "Please tell me the details of that camera," the response is tracked and stored in a database.

[0906] Output: Customer responses recorded in a database

[0907] All these processing steps result in the generation and delivery of real-time personalized ads based on customer interests and needs, improving the customer shopping experience.

[0908] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0909] The present invention combines a system that provides personalized advertisements to customers on hold in real time with an emotion engine that recognizes user emotions. This system integrates speech recognition, trend analysis, advertisement generation, advertisement distribution, and emotion recognition functions, with the aim of turning customers' waiting time into a valuable experience.

[0910] Program implementation example

[0911] This system is operated based on the following program processing procedures.

[0912] Speech recognition and text conversion

[0913] User: A customer calls a call center and states a specific question or request to an operator. For example, "I want to know about credit card reward points."

[0914] Device: Captures and temporarily stores the customer's speech as audio data in real time. This data is encrypted and sent to a server for security purposes.

[0915] Server: Sends the received voice data to the voice recognition engine and converts the voice data into text data.

[0916] Analyzing text data and identifying customer needs

[0917] Server: Sends the text data to a trend analysis module, which extracts keywords from the text data and matches them with historical data in a database to identify customer interests and needs.

[0918] emotion recognition

[0919] Server: The text data is also sent to the emotion engine, which analyzes the tone of the customer's voice and the content of what they say to recognize their emotional state (e.g., joy, anger, sadness, etc.).

[0920] Example: Recognizing emotions such as "customer dissatisfaction" from the customer's tone of voice and speech patterns.

[0921] Generating personalized ads

[0922] Server: Based on the results of trend analysis and the analysis results of the emotion engine, the server generates appropriate advertising content in real time that matches the customer's interests and emotions. For example, if a customer is dissatisfied, the server will display an apology message or information about shortening waiting times instead of a general advertisement.

[0923] Example: A customer who is interested in "credit card points" but dissatisfied with them can be informed of the start of a "new points campaign" and also be given information about the expected waiting time.

[0924] Delivering personalized ads

[0925] Server: Sends the encoded audio ad file to the customer's device. Through the distribution module, the ad is delivered to the customer on hold at the appropriate time.

[0926] Terminal: The received audio ad file is played in real time and made available to customers on hold, turning waiting time into valuable information time.

[0927] Customer response tracking and data updates

[0928] Server: After the ad is delivered, the server monitors the conversation after the customer is returned to the operator and tracks their response to the ad. If the customer shows interest in the ad content or responds indicative of their emotional state, this information is recorded in a database. This will be used to improve the accuracy of future ad generation and emotion recognition.

[0929] Specific examples

[0930] A customer calls a call center and inquires about credit card points. The operator puts the call on hold, and the conversation is captured as voice data by the terminal and sent to the server. The voice data is converted into text by a speech recognition engine. A trend analysis module then extracts the keywords "credit card" and "points," and an emotion engine recognizes the customer's emotional state. The server generates a personalized advertisement about a "new points campaign" in real time, along with information about the expected waiting time, and sends it to the customer's terminal as an audio file. The terminal plays the advertisement to the customer while they are on hold. The customer is then returned to the operator, and if they ask a question about the advertisement or respond emotionally, the server tracks that data and records it in a database. This process turns customer waiting time into a meaningful experience, improving the company's profitability and customer satisfaction.

[0931] In this way, the present invention is implemented as a system that provides optimal advertisements while a call is on hold, taking into consideration the customer's emotions, and achieves both customer satisfaction and corporate profits.

[0932] The processing flow will be explained below.

[0933] Step 1:

[0934] Server: When a customer calls the call center, the server monitors the call status and detects that the operator has put the customer on hold. It sets a hold flag and records that the customer's wait time has begun.

[0935] Step 2:

[0936] Terminal: Captures the conversation between the operator and the customer as voice data in real time. The voice data before the conversation is put on hold is temporarily stored, encrypted, and sent to the server.

[0937] Step 3:

[0938] Server: Receives the encrypted voice data and sends it to the voice recognition engine, which analyzes the voice data and converts it into text data.

[0939] Step 4:

[0940] Server: Sends the text data to the trend analysis module, which extracts keywords from the text data and matches them with historical data in the database to identify customer interests and needs.

[0941] Step 5:

[0942] Server: Sends text data to the emotion engine. The emotion engine analyzes the tone of the customer's voice and the content of what they say to recognize their emotional state. For example, it recognizes that "the customer's tone of voice indicates dissatisfaction."

[0943] Step 6:

[0944] Server: Based on the results of trend analysis and the emotion engine, the server generates personalized advertisements in real time that match the customer's interests and emotions. For example, if a customer is interested in and dissatisfied with "credit card points," the server generates information about new campaigns related to points and an apology message for keeping them waiting.

[0945] Step 7:

[0946] Server: Encodes the generated personalized ad into an audio file and prepares it for delivery.

[0947] Step 8:

[0948] Server: Sends the encoded audio ad file to the customer's device. The distribution module distributes the ad to the pending customer at the appropriate time.

[0949] Step 9:

[0950] Terminal: The received audio advertisement file is played in real time, allowing customers on hold to listen to the advertisement, thereby turning the customer's waiting time into valuable information time.

[0951] Step 10:

[0952] Server: After the advertisement is delivered, the server monitors the conversation when the customer is returned to the operator. It tracks the response to the advertisement, and if the customer shows interest in the advertisement content, it records that information in a database.

[0953] Step 11:

[0954] Server: Machine learning algorithms analyze the accumulated response data, creating a feedback loop to improve the accuracy of ad generation and delivery, thereby optimizing the performance of the entire system.

[0955] When a customer calls a call center to inquire about credit card points, the operator puts the call on hold. The terminal captures what the customer says as voice data and sends it to the server. The voice data is converted into text by a speech recognition engine. A trend analysis module then extracts keywords, and an emotion engine recognizes the customer's emotional state. The server generates a personalized advertisement in real time and sends it to the customer's terminal as an audio file. The terminal plays the advertisement to the customer while they are on hold. If the customer is then returned to the operator and asks a question about the advertisement, the server tracks their response and records it in a database. Through this process, the customer's waiting time is transformed into a meaningful experience, improving the company's profitability and customer satisfaction.

[0956] Example 2

[0957] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0958] In conventional call center systems, advertisements and messages provided to customers while they were on hold were typically not tailored to the individual customer's interests and needs. As a result, customers were unable to spend their waiting time meaningfully, which could lead to lower customer satisfaction. Furthermore, delivery that did not take into account the customer's emotional state could exacerbate customer dissatisfaction. To solve these problems, the present invention aims to effectively utilize waiting time and improve customer satisfaction by analyzing the customer's interests, needs, and even their emotional state in real time and providing personalized advertisements.

[0959] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0960] In this invention, the server includes means for recognizing the voice of a customer while the call is on hold, means for identifying the interests and needs of the customer from the recognized voice, means for analyzing the emotional state of the customer, means for generating personalized advertisements in real time based on the identified interests, needs and emotional state, and means for delivering the generated personalized advertisements to the customer's device, thereby enabling the customer's waiting time while on hold to be transformed into time for providing information suited to their individual needs and emotions.

[0961] "Customers on hold" means customers who have called the call center and are on hold waiting for an operator to respond.

[0962] "Voice recognition means" refers to devices or software that have the ability to collect customer voice data, analyze it, and convert it into text.

[0963] "Means for identifying customer interests and needs" refers to algorithms or devices that analyze and extract customer interests and requests from text data.

[0964] "Means for analyzing the emotional state of the customer" refers to algorithms or devices that recognize emotions based on the customer's voice and text data.

[0965] "Personalized advertising" refers to advertising that is individually tailored to take into account the interests, needs, and emotional state of a particular customer.

[0966] "Generating means" refers to algorithms or devices that generate specific content or advertisements based on the results of data analysis.

[0967] "Delivery means" refers to the algorithms and / or equipment used to transmit and play generated content and advertisements on Customer's devices.

[0968] "Real-time" refers to instantaneous processing or response with very low latency.

[0969] "Device" means an electronic device capable of sending and receiving voice and data that is used by a customer while a call is on hold.

[0970] The present invention relates to a system that provides personalized advertisements to customers on hold in real time, and further combines it with an emotion engine that recognizes user emotions. This system integrates speech recognition, trend analysis, advertisement generation, advertisement distribution, and emotion recognition functions, with the aim of turning customers' waiting time into a valuable experience.

[0971] The whole system consists of three main components: server, terminal, and user. Here, the server is responsible for data processing and analysis, the terminal is the interface with the customer, and the user is the customer who uses the system.

[0972] First, a user calls the call center and states a specific question or request to an operator. For example, "I want to know about credit card points." At this time, the terminal captures the user's speech as voice data in real time and temporarily stores it. The captured voice data is AES encrypted to ensure security and sent to the server.

[0973] The server sends the received voice data to a speech recognition engine (e.g., Google Cloud Speech-to-Text), which converts the voice data into text data. The text data is then sent to a natural language processing (NLP) module, which extracts keywords related to the customer's interests and needs from the text data. Furthermore, an emotion engine (e.g., IBM Watson Tone Analyzer) analyzes the customer's tone of voice and the content of what is being said to recognize the customer's emotional state (e.g., joy, anger, sadness).

[0974] Next, the server generates appropriate advertising content in real time based on the results of trend analysis and the emotion engine's analysis, matching the customer's interests and emotions. For example, if a customer is interested in "credit card points" but is dissatisfied, an advertisement containing information about a new points campaign and a message of goodwill will be generated.

[0975] The generated advertising content is encoded as an audio file and sent from the server to the customer's device. The device then plays the received advertising audio file in real time and lets the customer listen to it while on hold, thereby turning the customer's waiting time into valuable information time.

[0976] After the ad is delivered, the server monitors the conversation and tracks the customer's response to the ad. If the customer shows interest in the ad content or responds in a way that indicates a particular emotional state, that information is recorded in a database. This information can be used to generate future ads and improve the accuracy of emotion recognition.

[0977] For example, a user calls a call center and inquires about credit card points. The conversation is captured as voice data by the terminal and sent to the server. The voice data is then converted into text by the Google Cloud Speech-to-Text engine. An NLP module then extracts the keywords "credit card" and "points," and an emotion engine (IBM Watson Tone Analyzer) recognizes the customer's emotional state. The server generates a personalized advertisement about a "new points campaign" in real time, along with information about the expected waiting time, and sends it to the customer's terminal as an audio file. The terminal plays the advertisement to the customer while they are on hold. The customer is then returned to the operator, and if they ask a question about the advertisement or respond emotionally, the server tracks that data and records it in a database. This process transforms customer waiting time into a meaningful experience, improving the company's profitability and customer satisfaction.

[0978] The above is an embodiment of the present invention.

[0979] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0980] Step 1:

[0981] A user calls a call center and states a specific question or request to an operator.

[0982] Input: User's spoken utterance

[0983] Output: Audio data

[0984] Specific operation: A user calls and asks, "I want to know about credit card points."

[0985] Step 2:

[0986] The device captures the user's speech as audio data in real time.

[0987] Input: User's voice data

[0988] Output: Encrypted audio data

[0989] Specific operation: The device encrypts the captured audio data using AES and temporarily stores it.

[0990] Step 3:

[0991] The device sends the encrypted audio data to the server.

[0992] Input: Encrypted audio data

[0993] Output: Encrypted audio data sent to the server

[0994] Specific operation: The device sends encrypted voice data to the server via the network.

[0995] Step 4:

[0996] The server sends the received voice data to a voice recognition engine, which converts the voice data into text data.

[0997] Input: Encrypted audio data

[0998] Output: Text data

[0999] Specific operation: The server uses the Google Cloud Speech-to-Text engine to convert the audio data into text data.

[1000] Step 5:

[1001] The server sends the converted data to a natural language processing (NLP) module to extract keywords.

[1002] Input: Text data

[1003] Output: Extracted keywords

[1004] Specific operation: The server uses an NLP module to analyze and extract keywords such as "credit card" and "points."

[1005] Step 6:

[1006] The server simultaneously sends the text data to the emotion engine to analyze the customer's emotional state.

[1007] Input: Text data

[1008] Output: Customer's emotional state

[1009] What it does: The server uses IBM Watson Tone Analyzer to identify the customer's emotional state from the text data.

[1010] Step 7:

[1011] The server generates personalized advertisements based on keywords and emotional states.

[1012] Input: extracted keywords and emotional states

[1013] Output: Personalized advertising content

[1014] Specific operation: The server generates appropriate advertising content (e.g., "New points campaign" and "Information on shortened waiting times") in accordance with the customer's interests and emotions.

[1015] Step 8:

[1016] The server encodes the generated advertising content as an audio file and transmits it to the terminal.

[1017] Input: Personalized advertising content

[1018] Output: Audio file

[1019] Specific operation: The server encodes the advertising content into an audio file and sends it to the device.

[1020] Step 9:

[1021] The terminal plays the received audio advertisement file and lets the customer on hold hear it.

[1022] Input: Audio ad file

[1023] Output: Customer hears audio ad

[1024] What it does: The terminal plays audio ads in real time on the customer's device.

[1025] Step 10:

[1026] The server monitors the conversation between the customer and the operator after the advertisement is delivered and tracks the response to the advertisement.

[1027] Input: Customer and operator conversation data

[1028] Output: Customer response data

[1029] What it does: The server analyzes the conversation data and records whether the customer has a specific reaction or question to the ad.

[1030] These are the processing steps of the system, which will make customer waiting time more meaningful and increase the company's profitability and customer satisfaction.

[1031] (Application example 2)

[1032] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1033] Conventional on-hold ad delivery systems have struggled to deliver personalized ads that reflect customer emotions and specific interests. This has resulted in customers' waiting time not being fully utilized as an opportunity to provide information, preventing customer satisfaction and corporate profitability from being fully improved. Furthermore, in brick-and-mortar stores, there has been a lack of timely product recommendations that reflect customer emotions, resulting in missed sales opportunities.

[1034] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing the voice of a customer while the call is on hold, means for identifying the customer's interests and needs from the recognized voice, means for analyzing the customer's emotions, means for generating a personalized advertisement in real time based on the identified interests and needs and the customer's emotions, means for delivering the generated personalized advertisement to the customer's terminal, and means for displaying the generated advertisement on the smart glasses. This makes it possible to provide personalized advertisements and product suggestions in real time according to the customer's emotions and interests.

[1035] "Call on hold" refers to a state in which a user pauses a telephone call while waiting for another operation or a response.

[1036] A "voice recognition means" is a device or program that captures customer speech as voice data and converts it into text.

[1037] The "means for identifying customer interests and needs" refers to a device or program that extracts keywords from the customer's text data and identifies the customer's interests and requests based on this information.

[1038] The "means for analyzing customer emotions" refers to a device or program that analyzes the tone of a customer's voice and the content of their statements to recognize the customer's emotional state.

[1039] "Personalized advertising" is advertising content that is specific to a customer's individual interests, needs, and emotional state.

[1040] "Generating means" refers to a device or program that automatically creates advertisements or information based on the analyzed data.

[1041] "Distribution means" refers to a device or program that transmits the generated advertisements and information to the customer's device (such as a smartphone or smart glasses).

[1042] "Smart glasses" are wearable devices equipped with functions such as a display, camera, and microphone, and capable of displaying information and capturing audio.

[1043] "Real-time" refers to processing and information being provided immediately without delay.

[1044] This invention is a system that provides personalized advertisements in real time while taking into account customer emotions during call on-hold or in-store interactions. The system operates by integrating speech recognition, trend analysis, advertisement generation, advertisement distribution, and emotion recognition functions.

[1045] The server converts the customer's voice into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text). The converted data is sent to a trend analysis module (e.g., Python library - spaCy) to extract keywords. After that, an emotion recognition engine (e.g., IBM Watson Tone Analyzer) is used to analyze the customer's emotional state.

[1046] Based on the identified keywords and emotional state, the server automatically generates personalized ads using a generative AI model (e.g., OpenAI GPT-3 API), which are encoded as audio files or text information and delivered to the customer's device. Specifically, the ads are displayed on the display of smart glasses (e.g., Google Glass).

[1047] For example, if a customer is talking to a store staff member about a new smartphone model, the smart glasses will capture the conversation and send the audio data to the server. The server will then convert the speech into text using a speech recognition engine and extract keywords such as "smartphone" or "new model" using a trend analysis module. At the same time, the emotion recognition engine will analyze the customer's emotions and recognize whether they are expressing positive emotions.

[1048] Based on these results, a generative AI model is used to generate appropriate ads, with example prompts such as:

[1049] "If a customer is interested in a new smartphone model and has positive sentiment:

[1050] Explain the features and benefits of the new smartphone model that your customer is interested in and suggest any special promotions you can offer them on the spot.

[1051] The generated advertisements and information are displayed in real time on the smart glasses display, providing customers with valuable information. This transforms customer waiting time into a meaningful experience, improving corporate profitability and customer satisfaction. This system can be applied not only while on hold, but also during interactions in physical stores, making it applicable in a wide range of situations.

[1052] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1053] Step 1:

[1054] A user interacts with a store staff member and asks a question about a specific product or service, such as a new smartphone. The microphone in the smart glasses captures the voice in real time and captures the voice data, which is then encrypted and sent to a server.

[1055] Step 2:

[1056] The server runs the received voice data through a speech recognition engine (Google Cloud Speech-to-Text) and converts the voice data into text data. The input is voice data and the output is text data. Once the voice data has been converted into text, it is ready to proceed to the next analysis step.

[1057] Step 3:

[1058] The server sends the converted data to a trend analysis module (a Python library - spaCy) to extract keywords. The input is the text data and the output is the extracted keywords. The trend analysis module extracts keywords of interest, such as "smartphone" or "new model."

[1059] Step 4:

[1060] The server sends the converted text data to an emotion recognition engine (IBM Watson Tone Analyzer) to analyze the customer's emotions. The input is text data, and the output is the emotional state (positive, negative, neutral, etc.). As a result of the analysis, it is recognized that the customer is expressing positive emotions, for example.

[1061] Step 5:

[1062] The server generates personalized ads using a generative AI model (OpenAI GPT-3 API) based on the results of trend analysis and emotion recognition. The input is the extracted keywords and emotional state, and the output is the ad content. The generative AI model generates ads that suggest, for example, "features of new smartphone models" or "special promotions."

[1063] Step 6:

[1064] The server distributes the generated advertising content to the smart glasses terminal. The input is the advertising content, and the output is the display data for the smart glasses. The smart glasses display the received advertisement on the display in real time and notify the customer.

[1065] Step 7:

[1066] The user (customer) can view the advertisements displayed on the smart glasses display, for example, to view details of special promotions, which increases the customer's motivation to purchase and provides valuable information while they wait.

[1067] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1068] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1069] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1070] [Fourth embodiment]

[1071] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1072] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1073] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1074] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1075] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1076] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1077] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1078] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1079] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1080] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1081] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1082] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1083] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1084] The present invention is a system for providing personalized advertisements in real time to customers on hold. This system integrates speech recognition, trend analysis, advertisement generation, and advertisement distribution functions, with the aim of turning customers' waiting time into a valuable experience.

[1085] Program implementation example

[1086] This system is operated based on the following program processing procedures.

[1087] Speech recognition and text conversion

[1088] User: A customer calls a call center and states a specific question or request to an operator. For example, "I want to know about credit card reward points."

[1089] Device: Captures what the customer says and temporarily stores it as audio data, which is then sent to a server in encrypted form to ensure security.

[1090] Server: The received voice data is analyzed using a voice recognition engine and converted into text data.

[1091] Analyzing text data and identifying customer needs

[1092] Server: The speech recognition engine converts the data into text and sends it to the trend analysis module, which extracts keywords from the text and compares them with past trends and patterns stored in a database to identify customer interests and needs.

[1093] Generating personalized ads

[1094] Server: Based on the results of trend analysis, the server generates advertising content that matches the customer's interests. The advertising generation module creates appropriate advertisements in real time and encodes them as audio files. For example, based on past data, it may determine that new point campaign information would be effective for customers who are interested in "credit card points."

[1095] Delivering personalized ads

[1096] Server: Prepares the encoded audio ad file for sending to the customer's terminal, and sends the data to the terminal via the communication module.

[1097] Terminal: The received audio ad file is played in real time and made available to customers on hold. Example: An audio ad saying "A new points campaign has started" is played.

[1098] Customer response tracking and data updates

[1099] Server: After delivering the ad, the server monitors the conversation after the customer is returned to the operator and tracks their response to the ad. If the customer shows interest in the ad content, the server records this information in a database. This will help improve ad generation in the future.

[1100] Specific examples

[1101] A customer calls a call center and inquires about credit card points. The operator puts the call on hold, and the conversation is captured as voice data by the terminal. The voice data is sent to the server and converted into text by a voice recognition engine. A trend analysis module then extracts the keywords "credit card" and "points" and compares them with a past database. The server generates a personalized advertisement for a "new points campaign" in real time and sends the advertisement to the customer's terminal. The terminal plays the received advertisement to the customer while they are on hold. If the customer then asks about the advertisement when they return to the operator, the server tracks their response and records it in a database. This series of processes transforms customer waiting time into a meaningful experience, improving the company's profitability and customer satisfaction.

[1102] According to these modes for implementation, the present invention is applied to provide personalized announcements to customers whose calls are on hold.

[1103] The processing flow will be explained below.

[1104] Step 1:

[1105] Server: When a customer calls the call center, the server monitors the call status and detects that the operator has put the customer on hold. It sets a hold flag and records that the customer's waiting time has begun.

[1106] Step 2:

[1107] Terminal: Captures the conversation between the operator and the customer as voice data in real time, temporarily stores the voice data before the conversation is put on hold, and encrypts it for security purposes.

[1108] Step 3:

[1109] Server: Receives the encrypted voice data and sends it to the voice recognition engine, which analyzes the voice data and converts it into text data.

[1110] Step 4:

[1111] Server: Sends the converted data to a trend analysis module, which extracts keywords from the text data and matches them with historical data in a database to identify customer interests and needs.

[1112] Step 5:

[1113] Server: Based on the results of trend analysis, the server generates optimal advertising content in real time that matches the customer's interests. The generated advertisements are encoded as audio files and prepared for distribution through the advertisement generation module.

[1114] Step 6:

[1115] Server: Sends the encoded audio advertisement file to the customer's device. The distribution module distributes the advertisement to the customer on hold at the appropriate time.

[1116] Step 7:

[1117] Terminal: The received audio advertisement file is played in real time and made available to customers on hold, turning their waiting time into valuable information time.

[1118] Step 8:

[1119] Server: After delivering the advertisement, monitor the conversation when the customer returns to the operator and track their response to the advertisement. If the customer shows interest in the advertisement content, record that information in the database and use it to generate future advertisements.

[1120] Step 9:

[1121] Server: The accumulated response data is analyzed using machine learning algorithms, forming a feedback loop to improve the accuracy of ad generation and delivery, thereby optimizing the performance of the entire system.

[1122] Through this processing step, the system can turn customer waiting time into a positive experience, providing them with personalized advertising, and improving business profitability and customer satisfaction.

[1123] Example 1

[1124] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1125] Conventional advertisements provided while a call is on hold are generally uniform and do not provide information appropriate to the customer's interests and needs, resulting in lower customer satisfaction and limited advertising effectiveness for companies.

[1126] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1127] In this invention, the server includes means for recognizing the voice of a customer while the call is on hold, means for converting the recognized voice into text data, means for extracting keywords from the text data, means for identifying the interests and needs of the customer by analyzing the extracted keywords, means for generating a personalized advertisement in real time based on the identified interests and needs, means for encoding the generated personalized advertisement as an audio file, means for delivering the encoded personalized advertisement to the customer's terminal, means for playing the advertisement on the customer's terminal, and means for monitoring the customer's response after the advertisement is played and recording the result in a database. This makes it possible to turn the customer's waiting time into a valuable experience, improve customer satisfaction, and maximize the effectiveness of a company's advertising.

[1128] "Call on hold" means that a customer is temporarily placed on hold by a telephone operator or automated answering system.

[1129] "Means for recognizing voice" refers to technology for converting customer utterances into text data, including a voice recognition engine.

[1130] "Means for converting into text data" refers to the technology for analyzing voice data and converting it into text information.

[1131] "Keyword extraction methods" refers to techniques for identifying and extracting specific important words or phrases from text data.

[1132] "Means for identifying customer interests and needs" refers to technology that analyzes extracted keywords and determines customer concerns and requests.

[1133] "Means for generating personalized advertisements" refers to technology that creates advertisements that are individually tailored to a customer's interests and needs.

[1134] "Means for encoding as an audio file" refers to a technique for converting the generated advertising content into an audio data format.

[1135] "Means for delivery to Customer Device" means the technology that transmits the encoded audio file to the Customer Device.

[1136] "Means for playing advertisements on a customer's device" refers to technology for playing received audio files on a customer's device.

[1137] "Means for monitoring customer responses after an advertisement is played" refers to technology that tracks customer behavior and comments after an advertisement is played and collects data based on that.

[1138] "Means of recording in a database" refers to the technology used to store collected customer response data and manage it for analysis and future use.

[1139] MODE FOR CARRYING OUT THE INVENTION

[1140] The present invention relates to a system that provides personalized advertisements to customers on hold in real time. The system integrates speech recognition, trend analysis, advertisement generation, and advertisement distribution functions. The purpose of the system is to turn customers' waiting time into a valuable experience.

[1141] Speech recognition and text conversion

[1142] User: A customer calls the call center with a question or request. For example, they might ask, "I want to know about my credit card points."

[1143] Device: Captures what the customer says and temporarily stores it as audio data, which is then sent to a server in encrypted form to ensure security.

[1144] Server: The received voice data is analyzed using a voice recognition engine (e.g., Google Cloud Speech-to-Text or IBM Watson) and converted into text data.

[1145] Analyzing text data and identifying customer needs

[1146] Server: The data converted to text by the speech recognition engine is sent to a trend analysis module (e.g., Python's NLTK library). This module extracts keywords from the text data and compares them with past trends and patterns stored in a database, thereby identifying customer interests and needs.

[1147] Generating personalized ads

[1148] Server: Based on the results of the trend analysis, advertising content tailored to customer interests is generated using automated speech generation software (e.g., Amazon Polly or Google Text-to-Speech). The generated advertisements are encoded as audio files in real time.

[1149] Delivering personalized ads

[1150] Server: Prepares the encoded audio ad file for sending to the customer's device. Sends the data to the device via a communication module (e.g. HTTP / HTTPS communication).

[1151] Terminal: The received audio ad file is played in real time and made available to customers on hold. For example, an audio ad saying "A new points campaign has started" is played.

[1152] Customer response tracking and data updates

[1153] Server: After delivering the ad, the server monitors the conversation after the customer is returned to the operator and tracks their response to the ad. If the customer shows interest in the ad content, the server records this information in a database. This will help improve ad generation in the future.

[1154] Specific examples

[1155] A customer calls a call center and inquires about credit card points. The operator puts the call on hold, and the conversation is captured as voice data by the terminal. The voice data is sent to the server and converted into text by a voice recognition engine. A trend analysis module then extracts the keywords "credit card" and "points" and compares them with a past database. The server generates a personalized advertisement for a "new points campaign" in real time and sends the advertisement to the customer's terminal. The terminal plays the received advertisement to the customer while they are on hold. If the customer then asks about the advertisement when they return to the operator, the server tracks their response and records it in a database. This series of processes transforms customer waiting time into a meaningful experience, improving the company's profitability and customer satisfaction.

[1156] Prompt Sentence Examples

[1157] A customer calls a call center and asks, "Tell me about credit card points." The operator puts the customer on hold, and the system uses speech recognition to convert the conversation into text. The server then analyzes the text and extracts the keywords "credit card" and "points." Based on this, the system generates a personalized advertisement about a "new points campaign" in real time and plays it to the customer while they are on hold.

[1158] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1159] Step 1:

[1160] User: A customer calls the call center and says, "I want to know about credit card points." The input is the customer's voice. The output is the voice data of what the customer said.

[1161] Step 2:

[1162] Terminal: Captures and temporarily stores what the customer says. The captured voice data is encrypted using AES. The input is the customer's voice data, and the output is encrypted voice data.

[1163] Step 3:

[1164] Terminal: Sends encrypted audio data to the server. The input is the encrypted audio data, and the output is the data sent to the server.

[1165] Step 4:

[1166] Server: The received voice data is analyzed using a speech recognition engine (e.g., Google Cloud Speech-to-Text or IBM Watson) and converted into text data. At this time, the voice data is analyzed and text data such as "I would like to know about credit card points" is generated. The input is encrypted voice data, and the output is analyzed text data.

[1167] Step 5:

[1168] Server: The text data generated by the speech recognition engine is sent to a trend analysis module (e.g., Python's NLTK library). This module extracts keywords such as "credit card" and "points" from the text data and compares them with a historical database. This identifies the customer's interests and needs. The input is text data, and the output is the keywords and their analysis results.

[1169] Step 6:

[1170] Server: Based on the results of the trend analysis, an audio ad containing information about the new points campaign is generated using automatic speech generation software (e.g., Amazon Polly or Google Text-to-Speech). The generated ad content is encoded as an audio file. The input is the analysis results and appropriate ad content information, and the output is an audio file.

[1171] Step 7:

[1172] Server: Sends the encoded audio ad file to the customer's terminal via a communication module (e.g. HTTP / HTTPS communication). The input is the encoded audio file, and the output is the data sent to the terminal.

[1173] Step 8:

[1174] Terminal: Plays the received audio ad file in real time and lets customers on hold hear the audio ad, "A new points campaign has started." The input is the audio ad file received by the terminal, and the output is the audio ad heard by customers on hold.

[1175] Step 9:

[1176] Server: After delivering the ad, monitor the customer's response when they return to the operator. If the customer asks, "Tell me more about the points campaign," track that response and record it in a database. This will help improve future ad generation. The input is the customer's response data after the ad is delivered, and the output is the tracking data recorded in the database.

[1177] (Application example 1)

[1178] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1179] Conventional systems for providing personalized advertisements to customers on hold have had difficulty in fully identifying customer interests and needs and generating and delivering advertisements in real time. Furthermore, it has not been possible to track customer responses and provide feedback to future advertisement generation. The present invention aims to solve these problems and improve the customer shopping experience in virtual stores.

[1180] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1181] In this invention, the server includes means for recognizing the voice of a customer while the call is on hold, means for identifying the interests and needs of the customer from the recognized voice, means for generating personalized advertisements in real time, means for delivering the generated personalized advertisements to the customer's terminal, means for supporting the customer's purchasing experience through voice interaction in a virtual store, and means for tracking the customer's responses and recording them in a database. This makes it possible to provide advertisements based on the customer's interests and needs in real time and improve the purchasing experience.

[1182] A "customer on hold" is a customer who has called a company or support center and is waiting for a response.

[1183] "Speech recognition means" refers to means that uses technology to analyze voice data and convert it into text data.

[1184] "Means for identifying customer interests and needs" refers to a process of identifying what a customer is interested in or needs by analyzing text data obtained by speech recognition means.

[1185] "Personalized advertising" refers to advertising that is individually optimized based on a customer's specific interests and needs.

[1186] A "generative AI model" is a model that uses artificial intelligence technology to generate content such as advertisements from given information.

[1187] The "advertising generation means" refers to a means for automatically generating advertising content suitable for a customer based on the analysis results.

[1188] The "advertising distribution means" refers to a means for transmitting the generated advertisement to the customer's terminal in real time and playing it back.

[1189] "Customer Device" means a communication device used by a Customer, such as a smartphone, smart glasses, or head-mounted display.

[1190] A "virtual store" is an online shop or shopping platform operated via the Internet or virtual reality technology.

[1191] "Means to support customer purchasing experience" refers to means that support customers through voice interaction when browsing, selecting, and purchasing products in a virtual store.

[1192] "Means for tracking customer responses" refers to means for monitoring and collecting data on how customers respond to the advertisements that are played.

[1193] "Means for recording in a database" refers to means for storing tracked customer responses in a database and using them for later analysis and advertisement generation.

[1194] The present invention provides a personalized advertisement providing system for improving a customer's shopping experience in a virtual store. The system is implemented by the following components:

[1195] 1. Speech Recognition and Text Conversion

[1196] server:

[1197] The server is equipped with a speech recognition engine that recognizes the speech customers make in the virtual store and converts it into text data. This speech recognition uses technologies such as Google Cloud Speech-to-Text. The text data is encrypted and stored securely.

[1198] As a specific example, the server captures the audio of a customer saying, "Please tell me about a camera you recommend," and converts the audio into text data.

[1199] 2. Identifying customer interests and needs

[1200] server:

[1201] The server analyzes the text data obtained by the speech recognition tool to extract the customer's interests and needs. This analysis uses trend analysis modules such as "IBM Watson Natural Language Understanding." Here, the server compares the data with past data to identify the customer's interests and needs.

[1202] 3. Generating personalized ads

[1203] server:

[1204] The server generates personalized ads for customers based on the results of trend analysis, using generative AI models such as OpenAI GPT-4, and encodes the ads as audio files.

[1205] Here is an example prompt:

[1206] "Generate an ad for a new digital camera model. Based on past purchase history, you know your customer is interested in high-quality photography. Write compelling copy that includes the features and price of the new model."

[1207] 4. Delivering personalized advertising

[1208] server:

[1209] The server then sends the generated audio ad file to the customer's device, using technologies such as Firebase Cloud Messaging.

[1210] Device:

[1211] The customer's device will play the received audio advertisement file in real time, for example, "New camera model available at special price."

[1212] 5. Tracking customer responses and updating data

[1213] server:

[1214] The server tracks customer responses after serving ads and records the data in a database, which can use database technologies such as MySQL or Elasticsearch, providing feedback to generate more effective ads in the future.

[1215] For example, if a customer listens to an advertisement and then asks the operator again, "Can you tell me more about that camera?", that response will be recorded in the database.

[1216] The present invention enables the generation and delivery of real-time personalized advertisements based on customer interests and needs, improving the customer's shopping experience.

[1217] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1218] Step 1:

[1219] Input: User voice input

[1220] Action: A user enters a virtual store and says, "What camera do you recommend?"

[1221] Server: The server captures the user's voice input and temporarily stores it as audio data. The audio data is stored in encrypted form.

[1222] Output: Encrypted audio data

[1223] Step 2:

[1224] Input: Encrypted audio data

[1225] Processing: The server sends the audio data to a speech recognition engine (e.g., Google Cloud Speech-to-Text) and converts the audio into text data.

[1226] Server: The server sends voice data and receives text data converted by a voice recognition engine.

[1227] Output: Text data (e.g., "What camera would you recommend?")

[1228] Step 3:

[1229] Input: Text data

[1230] Processing: The server sends the text data to a trend analysis module (e.g., IBM Watson Natural Language Understanding) to extract keywords and identify customer interests and needs.

[1231] Server: Extracts keywords (e.g., "camera") from the text data and matches them with past trends and patterns in a database to identify customer interests and needs.

[1232] Output: Customer interests and needs (e.g., "I'm interested in cameras")

[1233] Step 4:

[1234] Input: Customer interests and needs

[1235] Processing: The server creates prompts to generate personalized advertisements based on the identified interests and needs.

[1236] Server: For example, generate a prompt like this: "Generate an advertisement for a new digital camera model. Based on past purchase history, we know that the customer is interested in high-quality photography. Write an appealing ad copy that includes the features and price of the new model."

[1237] Output: prompt statement

[1238] Step 5:

[1239] Input: prompt statement

[1240] Processing: The server sends prompts to a generative AI model (e.g., OpenAI GPT-4) to generate personalized ads.

[1241] Server: The generative AI model generates advertising content based on the prompt and encodes it as an audio file.

[1242] Output: Advertising content in the form of an audio file (e.g., "Special Offer on New Camera Models")

[1243] Step 6:

[1244] Input: Ad content in the form of an audio file

[1245] Processing: The server delivers the audio file to the client's device.

[1246] Server: Sends advertisements to the customer's device via a communication module (e.g., Firebase Cloud Messaging).

[1247] Output: Audio file delivered to the customer's device

[1248] Step 7:

[1249] Input: Streamed audio file

[1250] Processing: The device plays the received audio file in real time.

[1251] Device: The customer's device plays an audio ad (e.g., "Special Offer on New Camera Models").

[1252] Output: Audio ad played to customer

[1253] Step 8:

[1254] Input: Customer response to audio ad

[1255] Processing: The server tracks customer responses to the audio advertisements and records them in a database.

[1256] Server: For example, if the customer asks the operator again, "Please tell me the details of that camera," the response is tracked and stored in a database.

[1257] Output: Customer responses recorded in a database

[1258] All these processing steps result in the generation and delivery of real-time personalized ads based on customer interests and needs, improving the customer shopping experience.

[1259] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1260] The present invention combines a system that provides personalized advertisements to customers on hold in real time with an emotion engine that recognizes user emotions. This system integrates speech recognition, trend analysis, advertisement generation, advertisement distribution, and emotion recognition functions, with the aim of turning customers' waiting time into a valuable experience.

[1261] Program implementation example

[1262] This system is operated based on the following program processing procedures.

[1263] Speech recognition and text conversion

[1264] User: A customer calls a call center and states a specific question or request to an operator. For example, "I want to know about credit card reward points."

[1265] Device: Captures and temporarily stores the customer's speech as audio data in real time. This data is encrypted and sent to a server for security purposes.

[1266] Server: Sends the received voice data to the voice recognition engine and converts the voice data into text data.

[1267] Analyzing text data and identifying customer needs

[1268] Server: Sends the text data to a trend analysis module, which extracts keywords from the text data and matches them with historical data in a database to identify customer interests and needs.

[1269] emotion recognition

[1270] Server: The text data is also sent to the emotion engine, which analyzes the tone of the customer's voice and the content of what they say to recognize their emotional state (e.g., joy, anger, sadness, etc.).

[1271] Example: Recognizing emotions such as "customer dissatisfaction" from the customer's tone of voice and speech patterns.

[1272] Generating personalized ads

[1273] Server: Based on the results of trend analysis and the analysis results of the emotion engine, the server generates appropriate advertising content in real time that matches the customer's interests and emotions. For example, if a customer is dissatisfied, the server will display an apology message or information about shortening waiting times instead of a general advertisement.

[1274] Example: A customer who is interested in "credit card points" but dissatisfied with them can be informed of the start of a "new points campaign" and also be given information about the expected waiting time.

[1275] Delivering personalized ads

[1276] Server: Sends the encoded audio ad file to the customer's device. Through the distribution module, the ad is delivered to the customer on hold at the appropriate time.

[1277] Terminal: The received audio ad file is played in real time and made available to customers on hold, turning waiting time into valuable information time.

[1278] Customer response tracking and data updates

[1279] Server: After the ad is delivered, the server monitors the conversation after the customer is returned to the operator and tracks their response to the ad. If the customer shows interest in the ad content or responds indicative of their emotional state, this information is recorded in a database. This will be used to improve the accuracy of future ad generation and emotion recognition.

[1280] Specific examples

[1281] A customer calls a call center and inquires about credit card points. The operator puts the call on hold, and the conversation is captured as voice data by the terminal and sent to the server. The voice data is converted into text by a speech recognition engine. A trend analysis module then extracts the keywords "credit card" and "points," and an emotion engine recognizes the customer's emotional state. The server generates a personalized advertisement about a "new points campaign" in real time, along with information about the expected waiting time, and sends it to the customer's terminal as an audio file. The terminal plays the advertisement to the customer while they are on hold. The customer is then returned to the operator, and if they ask a question about the advertisement or respond emotionally, the server tracks that data and records it in a database. This process turns customer waiting time into a meaningful experience, improving the company's profitability and customer satisfaction.

[1282] In this way, the present invention is implemented as a system that provides optimal advertisements while a call is on hold, taking into consideration the customer's emotions, and achieves both customer satisfaction and corporate profits.

[1283] The processing flow will be explained below.

[1284] Step 1:

[1285] Server: When a customer calls the call center, the server monitors the call status and detects that the operator has put the customer on hold. It sets a hold flag and records that the customer's wait time has begun.

[1286] Step 2:

[1287] Terminal: Captures the conversation between the operator and the customer as voice data in real time. The voice data before the conversation is put on hold is temporarily stored, encrypted, and sent to the server.

[1288] Step 3:

[1289] Server: Receives the encrypted voice data and sends it to the voice recognition engine, which analyzes the voice data and converts it into text data.

[1290] Step 4:

[1291] Server: Sends the text data to the trend analysis module, which extracts keywords from the text data and matches them with historical data in the database to identify customer interests and needs.

[1292] Step 5:

[1293] Server: Sends text data to the emotion engine. The emotion engine analyzes the tone of the customer's voice and the content of what they say to recognize their emotional state. For example, it recognizes that "the customer's tone of voice indicates dissatisfaction."

[1294] Step 6:

[1295] Server: Based on the results of trend analysis and the emotion engine, the server generates personalized advertisements in real time that match the customer's interests and emotions. For example, if a customer is interested in and dissatisfied with "credit card points," the server generates information about new campaigns related to points and an apology message for keeping them waiting.

[1296] Step 7:

[1297] Server: Encodes the generated personalized ad into an audio file and prepares it for delivery.

[1298] Step 8:

[1299] Server: Sends the encoded audio ad file to the customer's device. The distribution module distributes the ad to the pending customer at the appropriate time.

[1300] Step 9:

[1301] Terminal: The received audio advertisement file is played in real time, allowing customers on hold to listen to the advertisement, thereby turning the customer's waiting time into valuable information time.

[1302] Step 10:

[1303] Server: After the advertisement is delivered, the server monitors the conversation when the customer is returned to the operator. It tracks the response to the advertisement, and if the customer shows interest in the advertisement content, it records that information in a database.

[1304] Step 11:

[1305] Server: Machine learning algorithms analyze the accumulated response data, creating a feedback loop to improve the accuracy of ad generation and delivery, thereby optimizing the performance of the entire system.

[1306] When a customer calls a call center to inquire about credit card points, the operator puts the call on hold. The terminal captures what the customer says as voice data and sends it to the server. The voice data is converted into text by a speech recognition engine. A trend analysis module then extracts keywords, and an emotion engine recognizes the customer's emotional state. The server generates a personalized advertisement in real time and sends it to the customer's terminal as an audio file. The terminal plays the advertisement to the customer while they are on hold. If the customer is then returned to the operator and asks a question about the advertisement, the server tracks their response and records it in a database. Through this process, the customer's waiting time is transformed into a meaningful experience, improving the company's profitability and customer satisfaction.

[1307] Example 2

[1308] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1309] In conventional call center systems, advertisements and messages provided to customers while they were on hold were typically not tailored to the individual customer's interests and needs. As a result, customers were unable to spend their waiting time meaningfully, which could lead to lower customer satisfaction. Furthermore, delivery that did not take into account the customer's emotional state could exacerbate customer dissatisfaction. To solve these problems, the present invention aims to effectively utilize waiting time and improve customer satisfaction by analyzing the customer's interests, needs, and even their emotional state in real time and providing personalized advertisements.

[1310] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1311] In this invention, the server includes means for recognizing the voice of a customer while the call is on hold, means for identifying the interests and needs of the customer from the recognized voice, means for analyzing the emotional state of the customer, means for generating personalized advertisements in real time based on the identified interests, needs and emotional state, and means for delivering the generated personalized advertisements to the customer's device, thereby enabling the customer's waiting time while on hold to be transformed into time for providing information suited to their individual needs and emotions.

[1312] "Customers on hold" means customers who have called the call center and are on hold waiting for an operator to respond.

[1313] "Voice recognition means" refers to devices or software that have the ability to collect customer voice data, analyze it, and convert it into text.

[1314] "Means for identifying customer interests and needs" refers to algorithms or devices that analyze and extract customer interests and requests from text data.

[1315] "Means for analyzing the emotional state of the customer" refers to algorithms or devices that recognize emotions based on the customer's voice and text data.

[1316] "Personalized advertising" refers to advertising that is individually tailored to take into account the interests, needs, and emotional state of a particular customer.

[1317] "Generating means" refers to algorithms or devices that generate specific content or advertisements based on the results of data analysis.

[1318] "Delivery means" refers to the algorithms and / or equipment used to transmit and play generated content and advertisements on Customer's devices.

[1319] "Real-time" refers to instantaneous processing or response with very low latency.

[1320] "Device" means an electronic device capable of sending and receiving voice and data that is used by a customer while a call is on hold.

[1321] The present invention relates to a system that provides personalized advertisements to customers on hold in real time, and further combines it with an emotion engine that recognizes user emotions. This system integrates speech recognition, trend analysis, advertisement generation, advertisement distribution, and emotion recognition functions, with the aim of turning customers' waiting time into a valuable experience.

[1322] The whole system consists of three main components: server, terminal, and user. Here, the server is responsible for data processing and analysis, the terminal is the interface with the customer, and the user is the customer who uses the system.

[1323] First, a user calls the call center and states a specific question or request to an operator. For example, "I want to know about credit card points." At this time, the terminal captures the user's speech as voice data in real time and temporarily stores it. The captured voice data is AES encrypted to ensure security and sent to the server.

[1324] The server sends the received voice data to a speech recognition engine (e.g., Google Cloud Speech-to-Text), which converts the voice data into text data. The text data is then sent to a natural language processing (NLP) module, which extracts keywords related to the customer's interests and needs from the text data. Furthermore, an emotion engine (e.g., IBM Watson Tone Analyzer) analyzes the customer's tone of voice and the content of what is being said to recognize the customer's emotional state (e.g., joy, anger, sadness).

[1325] Next, the server generates appropriate advertising content in real time based on the results of trend analysis and the emotion engine's analysis, matching the customer's interests and emotions. For example, if a customer is interested in "credit card points" but is dissatisfied, an advertisement containing information about a new points campaign and a message of goodwill will be generated.

[1326] The generated advertising content is encoded as an audio file and sent from the server to the customer's device. The device then plays the received advertising audio file in real time and lets the customer listen to it while on hold, thereby turning the customer's waiting time into valuable information time.

[1327] After the ad is delivered, the server monitors the conversation and tracks the customer's response to the ad. If the customer shows interest in the ad content or responds in a way that indicates a particular emotional state, that information is recorded in a database. This information can be used to generate future ads and improve the accuracy of emotion recognition.

[1328] For example, a user calls a call center and inquires about credit card points. The conversation is captured as voice data by the terminal and sent to the server. The voice data is then converted into text by the Google Cloud Speech-to-Text engine. An NLP module then extracts the keywords "credit card" and "points," and an emotion engine (IBM Watson Tone Analyzer) recognizes the customer's emotional state. The server generates a personalized advertisement about a "new points campaign" in real time, along with information about the expected waiting time, and sends it to the customer's terminal as an audio file. The terminal plays the advertisement to the customer while they are on hold. The customer is then returned to the operator, and if they ask a question about the advertisement or respond emotionally, the server tracks that data and records it in a database. This process transforms customer waiting time into a meaningful experience, improving the company's profitability and customer satisfaction.

[1329] The above is an embodiment of the present invention.

[1330] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1331] Step 1:

[1332] A user calls a call center and states a specific question or request to an operator.

[1333] Input: User's spoken utterance

[1334] Output: Audio data

[1335] Specific operation: A user calls and asks, "I want to know about credit card points."

[1336] Step 2:

[1337] The device captures the user's speech as audio data in real time.

[1338] Input: User's voice data

[1339] Output: Encrypted audio data

[1340] Specific operation: The device encrypts the captured audio data using AES and temporarily stores it.

[1341] Step 3:

[1342] The device sends the encrypted audio data to the server.

[1343] Input: Encrypted audio data

[1344] Output: Encrypted audio data sent to the server

[1345] Specific operation: The device sends encrypted voice data to the server via the network.

[1346] Step 4:

[1347] The server sends the received voice data to a voice recognition engine, which converts the voice data into text data.

[1348] Input: Encrypted audio data

[1349] Output: Text data

[1350] Specific operation: The server uses the Google Cloud Speech-to-Text engine to convert the audio data into text data.

[1351] Step 5:

[1352] The server sends the converted data to a natural language processing (NLP) module to extract keywords.

[1353] Input: Text data

[1354] Output: Extracted keywords

[1355] Specific operation: The server uses an NLP module to analyze and extract keywords such as "credit card" and "points."

[1356] Step 6:

[1357] The server simultaneously sends the text data to the emotion engine to analyze the customer's emotional state.

[1358] Input: Text data

[1359] Output: Customer's emotional state

[1360] What it does: The server uses IBM Watson Tone Analyzer to identify the customer's emotional state from the text data.

[1361] Step 7:

[1362] The server generates personalized advertisements based on keywords and emotional states.

[1363] Input: extracted keywords and emotional states

[1364] Output: Personalized advertising content

[1365] Specific operation: The server generates appropriate advertising content (e.g., "New points campaign" and "Information on shortened waiting times") in accordance with the customer's interests and emotions.

[1366] Step 8:

[1367] The server encodes the generated advertising content as an audio file and transmits it to the terminal.

[1368] Input: Personalized advertising content

[1369] Output: Audio file

[1370] Specific operation: The server encodes the advertising content into an audio file and sends it to the device.

[1371] Step 9:

[1372] The terminal plays the received audio advertisement file and lets the customer on hold hear it.

[1373] Input: Audio ad file

[1374] Output: Customer hears audio ad

[1375] What it does: The terminal plays audio ads in real time on the customer's device.

[1376] Step 10:

[1377] The server monitors the conversation between the customer and the operator after the advertisement is delivered and tracks the response to the advertisement.

[1378] Input: Customer and operator conversation data

[1379] Output: Customer response data

[1380] What it does: The server analyzes the conversation data and records whether the customer has a specific reaction or question to the ad.

[1381] These are the processing steps of the system, which will make customer waiting time more meaningful and increase the company's profitability and customer satisfaction.

[1382] (Application example 2)

[1383] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1384] Conventional on-hold ad delivery systems have struggled to deliver personalized ads that reflect customer emotions and specific interests. This has resulted in customers' waiting time not being fully utilized as an opportunity to provide information, preventing customer satisfaction and corporate profitability from being fully improved. Furthermore, in brick-and-mortar stores, there has been a lack of timely product recommendations that reflect customer emotions, resulting in missed sales opportunities.

[1385] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing the voice of a customer while the call is on hold, means for identifying the customer's interests and needs from the recognized voice, means for analyzing the customer's emotions, means for generating a personalized advertisement in real time based on the identified interests and needs and the customer's emotions, means for delivering the generated personalized advertisement to the customer's terminal, and means for displaying the generated advertisement on the smart glasses. This makes it possible to provide personalized advertisements and product suggestions in real time according to the customer's emotions and interests.

[1386] "Call on hold" refers to a state in which a user pauses a telephone call while waiting for another operation or a response.

[1387] A "voice recognition means" is a device or program that captures customer speech as voice data and converts it into text.

[1388] The "means for identifying customer interests and needs" refers to a device or program that extracts keywords from the customer's text data and identifies the customer's interests and requests based on this information.

[1389] The "means for analyzing customer emotions" refers to a device or program that analyzes the tone of a customer's voice and the content of their statements to recognize the customer's emotional state.

[1390] "Personalized advertising" is advertising content that is specific to a customer's individual interests, needs, and emotional state.

[1391] "Generating means" refers to a device or program that automatically creates advertisements or information based on the analyzed data.

[1392] "Distribution means" refers to a device or program that transmits the generated advertisements and information to the customer's device (such as a smartphone or smart glasses).

[1393] "Smart glasses" are wearable devices equipped with functions such as a display, camera, and microphone, and capable of displaying information and capturing audio.

[1394] "Real-time" refers to processing and information being provided immediately without delay.

[1395] This invention is a system that provides personalized advertisements in real time while taking into account customer emotions during call on-hold or in-store interactions. The system operates by integrating speech recognition, trend analysis, advertisement generation, advertisement distribution, and emotion recognition functions.

[1396] The server converts the customer's voice into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text). The converted data is sent to a trend analysis module (e.g., Python library - spaCy) to extract keywords. After that, an emotion recognition engine (e.g., IBM Watson Tone Analyzer) is used to analyze the customer's emotional state.

[1397] Based on the identified keywords and emotional state, the server automatically generates personalized ads using a generative AI model (e.g., OpenAI GPT-3 API), which are encoded as audio files or text information and delivered to the customer's device. Specifically, the ads are displayed on the display of smart glasses (e.g., Google Glass).

[1398] For example, if a customer is talking to a store staff member about a new smartphone model, the smart glasses will capture the conversation and send the audio data to the server. The server will then convert the speech into text using a speech recognition engine and extract keywords such as "smartphone" or "new model" using a trend analysis module. At the same time, the emotion recognition engine will analyze the customer's emotions and recognize whether they are expressing positive emotions.

[1399] Based on these results, a generative AI model is used to generate appropriate ads, with example prompts such as:

[1400] "If a customer is interested in a new smartphone model and has positive sentiment:

[1401] Explain the features and benefits of the new smartphone model that your customer is interested in and suggest any special promotions you can offer them on the spot.

[1402] The generated advertisements and information are displayed in real time on the smart glasses display, providing customers with valuable information. This transforms customer waiting time into a meaningful experience, improving corporate profitability and customer satisfaction. This system can be applied not only while on hold, but also during interactions in physical stores, making it applicable in a wide range of situations.

[1403] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1404] Step 1:

[1405] A user interacts with a store staff member and asks a question about a specific product or service, such as a new smartphone. The microphone in the smart glasses captures the voice in real time and captures the voice data, which is then encrypted and sent to a server.

[1406] Step 2:

[1407] The server runs the received voice data through a speech recognition engine (Google Cloud Speech-to-Text) and converts the voice data into text data. The input is voice data and the output is text data. Once the voice data has been converted into text, it is ready to proceed to the next analysis step.

[1408] Step 3:

[1409] The server sends the converted data to a trend analysis module (a Python library - spaCy) to extract keywords. The input is the text data and the output is the extracted keywords. The trend analysis module extracts keywords of interest, such as "smartphone" or "new model."

[1410] Step 4:

[1411] The server sends the converted text data to an emotion recognition engine (IBM Watson Tone Analyzer) to analyze the customer's emotions. The input is text data, and the output is the emotional state (positive, negative, neutral, etc.). As a result of the analysis, it is recognized that the customer is expressing positive emotions, for example.

[1412] Step 5:

[1413] The server generates personalized ads using a generative AI model (OpenAI GPT-3 API) based on the results of trend analysis and emotion recognition. The input is the extracted keywords and emotional state, and the output is the ad content. The generative AI model generates ads that suggest, for example, "features of new smartphone models" or "special promotions."

[1414] Step 6:

[1415] The server distributes the generated advertising content to the smart glasses terminal. The input is the advertising content, and the output is the display data for the smart glasses. The smart glasses display the received advertisement on the display in real time and notify the customer.

[1416] Step 7:

[1417] The user (customer) can view the advertisements displayed on the smart glasses display, for example, to view details of special promotions, which increases the customer's motivation to purchase and provides valuable information while they wait.

[1418] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1419] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1420] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1421] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1422] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1423] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1424] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1425] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1426] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1427] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1428] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1429] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1430] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1431] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1432] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1433] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1434] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1435] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1436] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1437] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1438] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1439] The following is further disclosed regarding the above embodiment.

[1440] (Claim 1)

[1441] a means for recognizing the voice of a customer while the call is on hold;

[1442] A means of identifying customer interests and needs from recognized speech;

[1443] means for generating personalized advertisements in real time based on identified interests and needs;

[1444] means for delivering the generated personalized advertisement to a customer's device;

[1445] A system including:

[1446] (Claim 2)

[1447] 2. The system according to claim 1, wherein the system analyzes data converted into text by a voice recognition means and extracts customer interests and needs.

[1448] (Claim 3)

[1449] The system according to claim 1, wherein the analysis results are compared with past data to identify customer interests and needs.

[1450] "Example 1"

[1451] (Claim 1)

[1452] a means for recognizing the voice of a customer while the call is on hold;

[1453] means for converting the recognized speech into text data;

[1454] A means for extracting keywords from text data;

[1455] A means of analyzing the extracted keywords to identify customer interests and needs;

[1456] means for generating personalized advertisements in real time based on identified interests and needs;

[1457] means for encoding the generated personalized advertisement as an audio file;

[1458] means for delivering the encoded personalized advertisements to the customer's device;

[1459] a means for playing the advertisement on the customer's device;

[1460] a means for monitoring and recording customer responses after the advertisement is played in a database;

[1461] A system including:

[1462] (Claim 2)

[1463] 2. The system according to claim 1, wherein the system analyzes data converted into text by a voice recognition means and extracts customer interests and needs.

[1464] (Claim 3)

[1465] The system according to claim 1, wherein the analysis results are compared with past data to identify customer interests and needs.

[1466] "Application Example 1"

[1467] (Claim 1)

[1468] a means for recognizing the voice of a customer while the call is on hold;

[1469] A means of identifying customer interests and needs from recognized speech;

[1470] means for generating personalized advertisements in real time based on identified interests and needs;

[1471] means for delivering the generated personalized advertisement to a customer's device;

[1472] A means for supporting customer purchasing experiences through voice interaction in a virtual store;

[1473] A means of tracking customer responses and recording them in a database;

[1474] A system including:

[1475] (Claim 2)

[1476] A means for analyzing data converted into text by a voice recognition means and extracting customer interests and needs;

[1477] A trend analysis module is used to compare historical data to identify customer interests and needs;

[1478] 10. The system of claim 1, comprising:

[1479] (Claim 3)

[1480] a means for generating prompt sentences and creating personalized advertising content using a generative AI model;

[1481] After the advertisement is delivered, a means of recording customer responses in a database to improve future advertisement generation;

[1482] 10. The system of claim 1, comprising:

[1483] "Example 2: Combining Emotion Engines"

[1484] (Claim 1)

[1485] a means for recognizing the voice of a customer while the call is on hold;

[1486] A means of identifying customer interests and needs from recognized speech;

[1487] a means for analyzing the emotional state of a customer;

[1488] means for generating personalized advertisements in real time based on identified interests, needs and emotional states;

[1489] a means for delivering the generated personalized advertisements to the customer's device;

[1490] A system including:

[1491] (Claim 2)

[1492] 2. The system according to claim 1, wherein the system analyzes data converted into text by a voice recognition means and extracts customer interests and needs.

[1493] (Claim 3)

[1494] The system according to claim 1, wherein the analysis results are compared with past data to identify customer interests and needs.

[1495] "Application example 2 when combining emotion engines"

[1496] (Claim 1)

[1497] a means for recognizing the voice of a customer while the call is on hold;

[1498] A means of identifying customer interests and needs from recognized speech;

[1499] A means of analyzing customer sentiment,

[1500] means for generating personalized advertisements in real time based on identified interests, needs and customer sentiment;

[1501] means for delivering the generated personalized advertisement to a customer's device;

[1502] means for displaying the generated advertisement on the smart glasses;

[1503] A system including:

[1504] (Claim 2)

[1505] 2. The system according to claim 1, wherein the system analyzes data converted into text by a voice recognition means and extracts customer interests and needs.

[1506] (Claim 3)

[1507] 2. The system of claim 1, wherein the analysis results and the customer's emotional state are compared with past data to identify the customer's interests and needs. [Explanation of symbols]

[1508] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for recognizing the voice of a customer while the call is on hold; A means of identifying customer interests and needs from recognized speech; means for generating personalized advertisements in real time based on identified interests and needs; means for delivering the generated personalized advertisement to a customer's device; A system including:

2. 2. The system according to claim 1, wherein the data converted into text by the voice recognition means is analyzed to extract customer interests and needs.

3. The system according to claim 1, wherein the analysis results are compared with past data to identify customer interests and needs.

Images