System

Abstract

To provide a system that can analyze a request of a user and purchase a commercial product.SOLUTION: A system comprises: voice recognition means; means for analyzing a request of a user on the basis of voice data recognized by the voice recognition means to add a commercial product that is selected on the basis of the analyzed result of the request of the user to a cart of a mail-order site, and performing a purchase procedure of the commercial product depending on the request of the user using a prompt sentence instructing performance of the purchase procedure and a generative AI model; feeling recognition means that recognizes a feeling from a voice of the user; and recommendation means for selecting the commercial product of the mail-order site using a prompt sentence instructing selection of the commercial product of the mail-order site and the generative AI model on the basis of a feeling state recognized by the feeling recognition means, and presenting the user as an additional recommended commercial product.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] One issue is that there is no system in place that allows elderly people who are not good at using the Internet to order online simply by speaking what they want and when they want it. [Means for solving the problem]

[0005] The system uses a voice recognition means to recognize voice input from a user in real time, and an analysis means to analyze the user's request from the voice data.Then, based on the analysis result, a mail order means automatically adds the product to a cart and performs a purchase procedure. [Brief description of the drawings]

[0006] [Figure 1]1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15] FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16] FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, the communication I / F (Interface) with the code is a communication processor and

[0013] The communication I / F is an interface including a power supply, a power source, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0014] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0015] [First embodiment]

[0016] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0021] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0024] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0026] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0027] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0028] "Example 1"

[0029] The present invention is embodied as a system including a voice recognition means, an analysis means, and a mail order means. The voice recognition means recognizes voice input from a user in real time. For example, when a user says, "I would like to buy three apples," the voice recognition means converts the voice into text data. Next, the analysis means analyzes the user's request from the text data. In this example, the request "three apples" is analyzed. Finally, the mail order means automatically adds the product to the cart based on the analysis result and completes the purchase. In this example, three apples are added to the cart, and the user is prompted to complete the purchase.

[0030] "Example 2"

[0031] The present invention is embodied as a system including a voice recognition means, an analysis means, and a mail order means. The voice recognition means recognizes voice input from a user in real time. For example, when a user says, "I would like to buy three apples," the voice recognition means converts the voice into text data. Next, the analysis means analyzes the user's request from the text data. In this example, the request "three apples" is analyzed. Finally, the mail order means automatically adds the product to the cart based on the analysis result and completes the purchase. In this example, three apples are added to the cart, and the user is prompted to complete the purchase.

[0032] The process flow of each embodiment will be described below.

[0033] "Example 1"

[0034] Step 1: The user opens the application and says, "I want to buy 3 apples."

[0035] Step 2: The speech recognition means recognizes the speech input from the user in real time and converts the speech into text data. In this example, the speech "I want to buy three apples" is converted into text data.

[0036] Step 3: The analysis means analyzes the user's request from the text data. In this example, the request "3 apples" is analyzed.

[0037] Step 4: The online retailer automatically adds the products to the cart based on the analysis and checks out. In this example, it adds 3 apples to the cart and prompts the user to check out.

[0038] Example 1

[0039] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0040] Conventional voice recognition systems have had difficulty in accurately recognizing and analyzing users' voice inputs and conducting e-commerce based on the results. In addition, the accuracy and speed of the process of converting voice data into text data and analyzing the text data was insufficient, making it impossible to quickly and accurately reflect users' requests. This resulted in a decrease in user convenience and a decrease in the efficiency of e-commerce.

[0041] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0042] In this invention, the server includes a voice recognition means, an analysis means, an electronic commerce means, a conversion means for converting voice data into text data, a natural language processing means for analyzing the text data, a cart addition means for adding products to a cart based on the analysis result, and a procedure promotion means for promoting the purchase procedure. This makes it possible to recognize and analyze a user's voice input in real time, and to quickly and accurately add products to the cart based on the result, and promote the purchase procedure.

[0043] A "voice recognition means" is a device or software that recognizes voice input from a user in real time and acquires it as voice data.

[0044] The "analysis means" is a device or software that analyzes the user's request based on the voice data acquired by the voice recognition means.

[0045] The "electronic commerce means" refers to a device or software that provides electronic commerce services based on the user's requests analyzed by the analysis means.

[0046] A "conversion means" is a device or software that converts voice data into text data.

[0047] A "natural language processing means" is a device or software that uses natural language processing technology to analyze text data and extract user needs.

[0048] The "cart addition means" refers to a device or software that automatically adds products to a cart based on the analysis results.

[0049] The "procedure promotion means" refers to a device or software that displays a message to the user to encourage the user to complete the purchase procedure, and advances the purchase procedure.

[0050] The present invention is a system including a voice recognition unit, an analysis unit, an electronic commerce unit, a conversion unit, a natural language processing unit, an add-to-cart unit, and a procedure promotion unit. A specific embodiment of this system will be described below.

[0051] Hardware and Software Used

[0052] Speech recognition means: Uses a speech recognition API (e.g., Google® Cloud Speech-to-Text API) to recognize voice input in real time.

[0053] Analysis methods: Use Python scripts and natural language processing libraries (e.g. spaCy) to analyze the audio data.

[0054] E-commerce tools: Use the API of an e-commerce site (e.g. Amazon API) to provide e-commerce services.

[0055] Conversion method: A speech recognition API is used to convert voice data into text data.

[0056] Natural language processing tools: Use natural language processing libraries to analyze text data.

[0057] Method for adding to cart: Use the API of the e-commerce site to add products to the cart.

[0058] Checkout Facilitators: Use the user interface to facilitate checkout.

[0059] System Operation

[0060] 1. Acquiring voice input

[0061] The user says to the terminal, "I want to buy three apples."

[0062] The device uses a built-in microphone to capture and temporarily store audio data.

[0063] 2. Converting voice data to text

[0064] The device sends the saved voice data to the voice recognition API.

[0065] The server receives the voice data and converts it into text data in real time.

[0066] The server returns the converted text data "I want to buy three apples" to the terminal.

[0067] 3. Analysis of text data

[0068] The terminal passes the received text data to the Python script.

[0069] The server runs a Python script and uses natural language processing libraries to analyze the text data.

[0070] The server extracts the request, such as "three apples," and returns the analysis results to the terminal.

[0071] 4. Add to Cart

[0072] The terminal sends the analysis results to the EC site's API.

[0073] The server adds "3 apples" to the cart via the API.

[0074] The server confirms that the item has been added to the cart and notifies the terminal.

[0075] 5. Facilitating checkout

[0076] The terminal displays a message prompting the user to complete the purchase procedure.

[0077] The user confirms the displayed message and proceeds with the purchase procedure.

[0078] The terminal receives the user's input and completes the final purchase transaction.

[0079] Examples

[0080] User dictation: "I want to buy 3 apples."

[0081] Example prompt for a generative AI model: “If the user says they want to buy three apples, explain how you would proceed.”

[0082] In this way, the system specifically executes a series of processes from the user's voice input to product purchase.

[0083] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0084] Step 1: Getting voice input

[0085] The user says to the terminal, "I want to buy three apples." The terminal uses the built-in microphone to capture the voice data and temporarily stores it. The input is the user's voice, and the output is the stored voice data. Specifically, the terminal displays "Recording..." and captures the voice data.

[0086] Step 2: Convert audio to text

[0087] The device sends the saved voice data to the voice recognition API. The server receives the voice data and converts it into text data in real time. The input is voice data, and the output is the text data "I would like to buy three apples." In concrete terms, the device displays "Recognizing voice..." and the server converts the voice data into text data.

[0088] Step 3: Analyze the text data

[0089] The terminal passes the received text data to the Python script. The server executes the Python script and analyzes the text data using a natural language processing library. The input is the text data "I want to buy three apples," and the output is the analysis result "three apples." In concrete terms, the terminal displays "Analyzing text..." and the server extracts the request "three apples."

[0090] Step 4: Adding products to cart

[0091] The terminal sends the analysis result to the API of the EC site. The server adds "3 apples" to the cart via the API. The input is the analysis result "3 apples" and the output is information about the product added to the cart. In concrete terms, the terminal displays "Adding product to cart..." and the server adds the product to the cart.

[0092] Step 5: Facilitate checkout

[0093] The terminal displays a message encouraging the user to complete the purchase. The user confirms the displayed message and proceeds with the purchase. The input is information about the products added to the cart, and the output is a notification that the purchase has been completed. Specifically, the terminal displays the message "You have added three apples to your cart. Would you like to proceed with the purchase?" and the user selects "Yes" to complete the purchase.

[0094] (Application example 1)

[0095] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0096] Conventional online shopping systems have the problem that it takes a lot of time and effort for users to search for products, add them to their cart, and complete the purchase procedure. In addition, these operations are often difficult, especially for elderly people and people with visual impairments. Furthermore, even in systems that use voice recognition technology, it can be difficult to accurately recognize voice input and properly analyze the user's requests. This can result in a poor user experience and discourage users from using online shopping sites.

[0097] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0098] In this invention, the server includes a voice recognition means, an analysis means, and a mail order means. This allows a user to search for products through voice input, add them to a cart, and complete a purchase procedure. Specifically, the voice recognition means recognizes voice input from the user in real time, and the analysis means analyzes the user's request based on the recognized voice data. Furthermore, the mail order means automatically adds the product to the cart based on the analysis result, and completes the purchase procedure. This allows the user to easily purchase products using only voice, and provides a system that is particularly easy to use for the elderly and visually impaired. In addition, by using a generative AI model to analyze the user's request and generate a prompt sentence, more accurate analysis is possible, improving the user experience.

[0099] A "voice recognition means" is a device or software that recognizes voice input from a user in real time and converts the voice data into text data.

[0100] The "analysis means" is a device or software that analyzes the text data generated by the voice recognition means and extracts the user's request.

[0101] The "mail order means" refers to a device or software that automatically adds products to a cart and completes the purchase procedure based on the user's desires analyzed by the analysis means.

[0102] A "smartphone" is a portable information and communication terminal that implements voice recognition means, analysis means, and mail order means.

[0103] A "generative AI model" is an artificial intelligence model that analyzes user requests and generates prompt sentences.

[0104] A "prompt sentence" is text that specifically expresses the user's request, generated by a generative AI model.

[0105] "API" stands for Application Program Interface, an interface for exchanging data and functions between different software programs.

[0106] A "cart" is a virtual shopping basket on an online shopping site that temporarily stores products that a user wishes to purchase.

[0107] A "checkout" is a series of operations and processes that a user performs to actually purchase the items that they have added to their cart.

[0108] As an embodiment of the present invention, a system including a voice recognition unit, an analysis unit, and a mail order unit is implemented as an application installed on a smartphone. A specific embodiment of this system will be described below.

[0109] Voice Recognition Means

[0110] The speech recognition means recognizes voice input from the user in real time and converts the voice data into text data. Specifically, it collects voice using a microphone built into the smartphone and converts the voice data into text data using the SpeechRecognition library. When a user says, "I want to buy three apples," this speech recognition means converts the voice into text data.

[0111] Analysis means

[0112] The analysis means analyzes the text data generated by the voice recognition means and extracts the user's request. Specifically, the analysis means analyzes the text data using a generative AI model and identifies the user's request. The analysis means extracts the request "3 apples."

[0113] Mail order methods

[0114] The mail order means automatically adds products to the cart and completes the purchase procedure based on the user's request analyzed by the analysis means. Specifically, it sends a request to the API of the mail order site to add the specified products to the cart. The mail order means adds three apples to the cart and prompts the user to complete the purchase procedure.

[0115] Hardware and Software Used

[0116] Hardware: Smartphone, microphone

[0117] Software: Python, SpeechRecognition library, requests library, generative AI model

[0118] Examples

[0119] When a user says to their smartphone, "I want to buy three apples," the speech recognition means converts the speech into text data. Next, the analysis means uses a generative AI model to analyze the text data and extracts the request, "three apples." Finally, the mail order means sends a request to the API of the mail order site to add three apples to the cart and prompt the user to complete the purchase.

[0120] Examples of prompt statements

[0121] "When a user says to their smartphone, 'I want to buy three apples,' the voice recognition system converts the speech into text, and the analysis system extracts the request, 'three apples.' Then it sends a request to the API of the online shopping site to add three apples to the cart."

[0122] In this way, users can easily purchase products using only their voice, providing a system that is particularly easy to use for the elderly and visually impaired.

[0123] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0124] Step 1:

[0125] The user inputs voice into the smartphone.

[0126] Input: User speech (e.g. "I want to buy 3 apples")

[0127] Output: Audio data

[0128] Specific operation: Voice data is collected when the user speaks into the smartphone's microphone.

[0129] Step 2:

[0130] The terminal converts the voice data into text data using a voice recognition means.

[0131] Input: Audio data

[0132] Output: Text data (e.g. "I want to buy 3 apples")

[0133] Specific operation: The SpeechRecognition library installed on the smartphone analyzes the voice data and converts it into text data.

[0134] Step 3:

[0135] The terminal uses an analysis means to analyze the text data and extract the user's request.

[0136] Input: Text data (e.g. "I want to buy 3 apples")

[0137] Output: Demand data (e.g. "3 apples")

[0138] How it works: The generative AI model analyzes text data and identifies the user's request. For example, the request "three apples" is extracted.

[0139] Step 4:

[0140] The terminal uses the mail order means to send a request to the API of the mail order site based on the analyzed request data.

[0141] Input: Request data (e.g. "3 apples")

[0142] Output: The result of the API request (e.g., a response indicating that a product was added to the cart)

[0143] Specific operation: The smartphone uses the requests library to send a request to the online shopping site's API to "add three apples to the cart."

[0144] Step 5:

[0145] The server receives the API request, adds the specified product to the cart, and completes the purchase.

[0146] Input: API request (e.g. "Add 3 apples to cart")

[0147] Output: A confirmation that the purchase has been completed (e.g. "3 apples added to cart")

[0148] Specific operation: The mail-order site's server processes the API request, adds the specified product to the cart, and completes the purchase. Once the process is complete, a notification of the completion of the purchase is sent to the smartphone.

[0149] Step 6:

[0150] The terminal displays a notification to the user that the purchase process is complete.

[0151] Input: Checkout completion notification (e.g. "3 apples added to cart")

[0152] Output: Display a notification to the user

[0153] Specific behavior: The smartphone receives a notification that the purchase procedure has been completed and displays the notification to the user, allowing the user to confirm that the purchase procedure has been completed.

[0154] Example 2

[0155] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0156] In conventional e-commerce systems, when users wanted to purchase a product, they had to manually search for the product, add it to their cart, and then complete the purchase procedure, which was time-consuming and inconvenient. Even in systems that used voice input, the accuracy and real-time nature of voice recognition was insufficient, and users' requests were sometimes not accurately reflected. This resulted in a poor user experience and made users avoid using the system.

[0157] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0158] In this invention, the server includes a voice recognition means, an analysis means, and an electronic commerce means, so that when a user purchases a product through voice input, the voice can be converted into text in real time, the text can be analyzed to understand the user's request, and the product can be automatically added to a cart based on the analysis results, and the user can be prompted to complete the purchase procedure.

[0159] A "voice recognition means" is a device or software that has the function of recognizing voice input from a user in real time and converting the voice data into text data.

[0160] The "analysis means" refers to a device or software having a function for analyzing the text data converted by the voice recognition means and understanding the user's request.

[0161] An "electronic commerce means" is a device or software having a function of automatically adding products to a cart and carrying out a purchasing procedure based on the user's desires analyzed by an analysis means.

[0162] "User" means an individual or organization that uses the system through voice input to purchase goods.

[0163] A "server" is a computer system that manages the entire system including the speech recognition means, analysis means, and electronic commerce means, and executes various processes.

[0164] "Voice input" refers to voice data generated by a user through an input device such as a microphone.

[0165] "Text data" refers to character string data converted from voice data by a voice recognition means.

[0166] A "cart" is a virtual shopping basket in an electronic commerce means for temporarily storing products that a user wishes to purchase.

[0167] A "checkout" is a series of operations and processes that a user performs to actually purchase an item that has been added to their cart.

[0168] The present invention relates to a system including a speech recognition unit, an analysis unit, and an electronic commerce unit. A specific embodiment of the system will be described below.

[0169] Hardware and Software Used

[0170] Hardware: Microphone, server, user device (smartphone, PC, etc.)

[0171] Software: voice recognition software (e.g. voice recognition APIs), analytics software (e.g. natural language processing libraries), e-commerce platforms (e.g. APIs for e-commerce sites)

[0172] System Operation

[0173] 1. Receiving voice input

[0174] The user provides voice input through a microphone.

[0175] For example, a user might say, "I want to buy three apples."

[0176] 2. Converting voice data to text

[0177] The server uses voice recognition software (voice recognition API) to convert the user's voice into text data in real time.

[0178] Specifically, the server sends voice data to the API, which converts the voice into text and returns it.

[0179] 3. Analysis of text data

[0180] The server passes the converted text data to analysis software (natural language processing library).

[0181] Analysis software analyzes user requests from text data.

[0182] Specifically, the product name "apple" and the quantity "3" are extracted from the text "three apples."

[0183] 4. Add to Cart

[0184] Based on the analysis results, the server calls the API of the e-commerce platform.

[0185] Specifically, the server sends the product name and quantity to the API, and the API adds the specified product to the cart.

[0186] 5. Prompt for purchase

[0187] The server sends a notification to the user's terminal prompting them to complete the purchase procedure.

[0188] Specifically, a message is displayed on the user's device stating, "Three apples have been added to your cart. Please complete the purchase."

[0189] Examples

[0190] User dictation: "I want to buy 3 apples."

[0191] Speech recognition result: "I want to buy three apples."

[0192] Analysis result: Product name: "Apple", quantity: "3"

[0193] E-commerce transaction action: Uses the API of the e-commerce site to add three apples to the cart and prompts the user to complete the purchase.

[0194] Example prompts for generative AI models

[0195] If a user says, "I want to buy three apples," the server uses speech recognition software to convert the speech to text, analyzes the text to understand what the user wants, and then calls the e-commerce platform's API to add three apples to the cart and prompt the user to check out.

[0196] In this way, the system automates the entire process from user voice input to product purchase.

[0197] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0198] Step 1:

[0199] Receiving voice input

[0200] The user provides voice input through a microphone.

[0201] For example, a user might say, "I want to buy three apples."

[0202] Input: User's voice data

[0203] Output: Audio data captured from the microphone.

[0204] Step 2:

[0205] Converting audio data to text

[0206] The server uses voice recognition software (voice recognition API) to convert the user's voice into text data in real time.

[0207] Specifically, the server sends voice data to the API, which converts the voice into text and returns it.

[0208] Input: Audio data

[0209] Data processing: Convert voice data into text data using speech recognition API

[0210] Output: Text data (e.g. "I want to buy 3 apples")

[0211] Step 3:

[0212] Analyzing Text Data

[0213] The server passes the converted text data to analysis software (natural language processing library).

[0214] Analysis software analyzes user requests from text data.

[0215] Specifically, the product name "apple" and the quantity "3" are extracted from the text "three apples."

[0216] Input: Text data

[0217] Data Calculation: Analyze text data using natural language processing libraries to extract product names and quantities

[0218] Output: Analysis result (product name "Apple", quantity "3")

[0219] Step 4:

[0220] Add to Cart

[0221] Based on the analysis results, the server calls the API of the e-commerce platform.

[0222] Specifically, the server sends the product name and quantity to the API, and the API adds the specified product to the cart.

[0223] Input: Analysis result (product name "Apple", quantity "3")

[0224] Data processing: Add products to cart using API of e-commerce platform

[0225] Output: Product information added to cart

[0226] Step 5:

[0227] Prompt for purchase

[0228] The server sends a notification to the user's terminal prompting them to complete the purchase procedure.

[0229] Specifically, a message is displayed on the user's device stating, "Three apples have been added to your cart. Please complete the purchase."

[0230] Input: Product information added to cart

[0231] Data processing: generating notification messages

[0232] Output: A notification message displayed on the user's terminal.

[0233] In this way, the system automates the entire process from user voice input to product purchase.

[0234] (Application example 2)

[0235] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0236] Conventional mail-order systems require users to search for products, add them to their cart, and complete the purchase process in multiple steps, which is time-consuming. Even in systems that use voice input, there are issues with the accuracy of voice recognition and real-time performance, which means the user experience does not improve.

[0237] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0238] In this invention, the server includes a voice recognition means, an analysis means, a mail order means, and a means including an application to be installed on a smartphone, which allows a user to search for products through voice input, add them to a cart, and complete the purchase procedure in real time.

[0239] A "voice recognition means" is a device or software that recognizes voice input from a user in real time and converts the voice data into text data.

[0240] The "analysis means" refers to a device or software that analyzes the text data obtained by the voice recognition means and extracts the user's request.

[0241] The "mail order means" refers to a device or software that selects products based on the user's needs analyzed by the analysis means, adds them to a cart, and completes the purchase procedure.

[0242] "Means including an application installed on a smartphone" refers to an application that runs on a smartphone and integrates speech recognition means, analysis means, and mail order means to enable a user to purchase products through voice input.

[0243] The system for carrying out the present invention includes a voice recognition unit, an analysis unit, a mail order unit, and a unit including an application installed on a smartphone. Specific embodiments of each unit will be described below.

[0244] First, the speech recognition means recognizes the voice input from the user in real time and converts the voice data into text data. This is done using the smartphone's microphone and a speech recognition library (for example, the speech_recognition library). When the user says, "I want to buy three apples," the speech recognition means converts the voice into text data.

[0245] Next, the analysis means analyzes the text data obtained by the voice recognition means and extracts the user's request. Specifically, an analysis logic is used to extract the product name and quantity from the text data. For example, from the text data "I want to buy three apples," "apples" and "three" are extracted.

[0246] The mail order means then selects products based on the user's needs analyzed by the analysis means, adds them to the cart, and completes the purchase procedure. This means uses the API of the mail order site to add the specified products and quantities to the cart and proceeds with the purchase procedure. For example, it adds three apples to the cart based on the analysis results, and prompts the user to complete the purchase procedure.

[0247] The means including the application installed on the smartphone integrates the voice recognition means, analysis means, and mail order means, and enables the user to purchase products through voice input. The user can search for products, add them to a cart, and complete the purchase process simply by speaking to the smartphone.

[0248] As a specific example, when a user says, "I would like to buy two bottles of milk," the voice recognition means converts the voice into text data, the analysis means extracts "milk" and "two bottles," and the mail order means adds the two bottles of milk to the cart and prompts the user to proceed with the purchase.

[0249] Example of a prompt to be input to a generative AI model:

[0250] If a user says, "I would like to buy two bottles of milk," generate a Python program that performs speech recognition, converts the speech to text, analyzes it, and adds two bottles of milk to the shopping cart on an online shopping site.

[0251] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0252] Step 1:

[0253] A user speaks to a smartphone, for example, "I want to buy two bottles of milk." This voice input is captured by the smartphone's microphone.

[0254] Step 2:

[0255] The smartphone's voice recognition means converts the captured voice data into text data. Specifically, it uses the speech_recognition library to analyze the voice data and generate text data such as "I would like to buy two bottles of milk." The input is voice data, and the output is text data.

[0256] Step 3:

[0257] The smartphone's analysis means analyzes the text data generated by the voice recognition means. Specifically, it uses analysis logic to extract product names and quantities from the text data. For example, from the text data "I would like to buy two bottles of milk," it extracts "milk" and "two bottles." The input is text data, and the output is a pair of product name and quantity.

[0258] Step 4:

[0259] The mail-order means of the smartphone calls the API of the mail-order site based on the product name and quantity extracted by the analysis means. Specifically, the requests library is used to send the product name and quantity to the API endpoint of the mail-order site and add it to the cart. For example, an API request is sent to add "Milk" and "2 bottles" to the cart. The input is the product name and quantity, and the output is the response to the API request.

[0260] Step 5:

[0261] The smartphone shopping tool receives the response to the API request and displays a message to the user urging them to complete the purchase. For example, it displays a message saying, "You have added two bottles of milk to your cart. Would you like to complete the purchase?" The input is the response to the API request, and the output is the message to the user.

[0262] Step 6:

[0263] The user performs an operation to confirm the purchase procedure, for example by tapping a "Purchase" button on the smartphone screen. This operation is captured by the smartphone's mail order processing device.

[0264] Step 7:

[0265] The smartphone mail order tool sends an API request to complete the purchase procedure based on the user's confirmation operation. Specifically, it uses the requests library to send a request to an API endpoint for completing the purchase procedure. The input is the user's confirmation operation, and the output is the API request response indicating the completion of the purchase procedure.

[0266] Step 8:

[0267] The smartphone mail order tool receives the response to the API request indicating the completion of the purchase procedure and displays a purchase completion message to the user. For example, it displays a message saying "The purchase procedure has been completed. Thank you." The input is the response to the API request indicating the completion of the purchase procedure, and the output is the purchase completion message to the user.

[0268] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0269] "Example 1"

[0270] As one embodiment of the present invention, a system incorporating an emotion recognition means is provided. This system includes an emotion recognition means for recognizing emotions from the user's voice, in addition to a voice recognition means, an analysis means, and a mail order means. Specifically, when a user opens an application and speaks "I want to buy three apples," the voice recognition means recognizes the voice input from the user in real time and converts the voice into text data. In this example, the voice "I want to buy three apples" is converted into text data. The analysis means analyzes the user's request from this text data. In this example, the request "three apples" is analyzed. The emotion recognition means recognizes the user's emotional state from the tone, pitch, volume, etc. of the user's voice. In this example, if the user speaks in a calm tone about wanting to buy three apples, the user feels relaxed. The mail order means automatically adds the product to the cart based on the analysis result and the emotion recognition result, and performs the purchase procedure. In this example, three apples are added to the cart, and the user is prompted to perform the purchase procedure. In addition, if the user feels relaxed, a product such as a relaxing music CD is recommended.

[0271] "Example 2"

[0272] As one embodiment of the present invention, a system incorporating an emotion recognition means is provided. This system includes an emotion recognition means for recognizing emotions from the user's voice, in addition to a voice recognition means, an analysis means, and a mail order means. Specifically, when a user opens an application and speaks "I want to buy three apples," the voice recognition means recognizes the voice input from the user in real time and converts the voice into text data. In this example, the voice "I want to buy three apples" is converted into text data. The analysis means analyzes the user's request from this text data. In this example, the request "three apples" is analyzed. The emotion recognition means recognizes the user's emotional state from the tone, pitch, volume, etc. of the user's voice. In this example, if the user speaks in a calm tone about wanting to buy three apples, the user feels relaxed. The mail order means automatically adds the product to the cart based on the analysis result and the emotion recognition result, and performs the purchase procedure. In this example, three apples are added to the cart, and the user is prompted to perform the purchase procedure. In addition, if the user feels relaxed, a product such as a relaxing music CD is recommended.

[0273] The process flow of each embodiment will be described below.

[0274] "Example 1"

[0275] Step 1: The user opens the application and says, "I want to buy three apples."

[0276] Step 2: The speech recognition means recognizes the speech input from the user in real time and converts the speech into text data. In this example, the speech "I want to buy three apples" is converted into text data.

[0277] Step 3: The analysis means analyzes the user's request from the text data. In this example, the request "3 apples" is analyzed.

[0278] Step 4: The emotion recognition means recognizes the user's emotional state from the tone, pitch, volume, etc. of the user's voice. In this example, if the user speaks in a calm tone about his / her desire to buy three apples, the user is perceived as relaxed.

[0279] Step 5: The online retailer automatically adds the products to the cart based on the analysis and emotion recognition results, and checks out. In this example, it adds three apples to the cart and prompts the user to check out.

[0280] Step 6: If the user feels relaxed, recommend a product, such as a relaxing music CD.

[0281] Example 1

[0282] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0283] Conventional voice recognition systems were able to analyze users' requests and provide online shopping services, but they did not recommend products that took into account the user's emotional state. As a result, they were unable to recommend appropriate products according to the user's emotions, which resulted in a lack of improvement in the user experience.

[0284] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0285] In this invention, the server includes a voice recognition means, a prompt sentence for analyzing a user's request based on the voice data recognized by the voice recognition means, and adding a product selected based on the analysis result of the user's request to a shopping cart on a shopping site, and a means for carrying out a purchase procedure for a product according to the user's request using a generative AI model, an emotion recognition means for recognizing emotion from the user's voice, a prompt sentence for instructing the user to select a product on the shopping site based on the emotional state recognized by the emotion recognition means, and a recommendation means for selecting a product on the shopping site using the generative AI model and presenting it to the user as an additional recommended product. In the server, the means for carrying out the purchase procedure carries out the purchase procedure by sending a request for the purchase procedure to the API of the online shopping site. In the server, the recommendation means includes, when the emotional state recognized by the emotion recognition means is negative, the prompt sentence for instructing the user to select a product that suppresses the emotional state, and the generative AI model, and selecting a product on the shopping site and presenting it to the user as an additional recommended product. This makes it possible to not only recognize the user's voice input in real time and analyze the request, but also to recommend appropriate products based on the user's emotional state.

[0286] A "voice recognition means" is a means for recognizing voice input from a user in real time and converting the voice data into text data.

[0287] The "analysis means" is a means for analyzing a user's request based on the text data recognized by the voice recognition means.

[0288] The "mail order means" is a means for automatically adding products to a shopping cart and carrying out a purchasing procedure based on the user's desires analyzed by the analysis means.

[0289] An "emotion recognition means" is a means for analyzing the tone, pitch, volume, etc. of a user's voice and recognizing the user's emotional state.

[0290] The "recommendation means" is a means for presenting appropriate additional recommended products to the user based on the emotional state recognized by the emotion recognition means.

[0291] The present invention is a system including a voice recognition unit, an analysis unit, a mail order sales unit, an emotion recognition unit, and a recommendation unit. A specific embodiment of the system will be described below.

[0292] Hardware and Software Used

[0293] Speech recognition method: General speech recognition API (e.g., speech recognition cloud service)

[0294] Generative AI model: GPT-4 Analysis method: Natural language processing library (e.g. Python natural language processing library)

[0295] Mail order method: EC site API (e.g. online shopping API)

[0296] Emotion recognition means: Emotion analysis service (e.g. emotion analysis cloud service)

[0297] Recommendation method: Recommendation engine (e.g., recommendation system API)

[0298] System operation explanation

[0299] 1. Voice Recognition Method

[0300] The user opens the application and says, "I want to buy three apples."

[0301] The device captures the user's voice with a microphone and transmits the voice data to the server.

[0302] The server uses a voice recognition cloud service to convert the voice data into text data.

[0303] For example, speech saying "I want to buy three apples" is converted into text data saying "I want to buy three apples."

[0304] 2. Analysis method

[0305] The server uses a natural language processing library to analyze the user's request from the text data.

[0306] In this example, the request "3 apples" is extracted.

[0307] The server stores the analysis results in an internal database.

[0308] 3. Emotion recognition means

[0309] The server calls an emotion analysis cloud service to recognize the user's emotional state from the tone, pitch, volume, etc. of their voice.

[0310] For example, if the user speaks in a calm tone, the server will recognize that the user is relaxed.

[0311] The server stores the emotion recognition results in an internal database.

[0312] 4. Mail Order Methods

[0313] The server analyzes the user's request based on the recognized voice data, and adds the product selected based on the analysis result of the user's request to a shopping cart on the online shopping site, and performs the purchase procedure of the product according to the user's request using a prompt sentence and the generative AI model that instructs the user to complete the purchase procedure. Specifically, the server calls an online shopping API based on the analysis result and adds three apples to the cart. The server then sends a message to the terminal to prompt the user to complete the purchase procedure.

[0314] The terminal displays to the user, "You have added three apples to your cart. Would you like to complete the purchase?" If the user answers "Yes," the server executes the purchase procedure on the online shopping site. At this time, the purchase procedure is carried out by sending a request for the purchase procedure to the API of the online shopping site.

[0315] 5. Recommendation method

[0316] The server uses a prompt sentence instructing the user to select a product from the mail-order site based on the emotional state recognized by the emotion recognition means and a generative AI model to select a product from the mail-order site and presents the product to the user as an additional recommended product. At this time, if the recognized emotional state is negative, the server uses a prompt sentence instructing the user to select a product that suppresses the emotional state and a generative AI model to select a product from the mail-order site and presents the product to the user as an additional recommended product. For example, if the emotion recognition result is negative, the server selects a relaxing product (e.g., a music CD).

[0317] The server uses the recommendation system API to obtain information about the recommended products.

[0318] The server transmits information about the recommended products to the terminal.

[0319] The terminal displays to the user, "Would you like to listen to some relaxing music on CD?"

[0320] Examples

[0321] Example 1 of a prompt to check out a product "The user says, 'I want to buy three apples.' Analyze the user's request, select the product that best suits the user's request from the products on the online shopping site AA, and add it to the cart. Ask the user if they want to purchase the product, and then proceed with the purchase."

[0322] Example 2 of a prompt to check out a product "The user says, 'I'm looking for a relaxing music CD.' Analyze the user's request, select the product that best suits the user's request from the products on the online shopping site AA, and add it to the cart. Ask the user if they want to purchase the product, and then proceed with the purchase."

[0323] Example 2 of a prompt to recommend additional products "The user's emotion has been recognized as 'anger.' Please select a product from the AA online shopping site that can suppress the user's emotion and add it to the cart. Please present it to the user as an additional recommended product, ask if they would like to purchase it, and then proceed with the purchase." In this way, it is possible to not only recognize the user's voice input in real time and analyze their needs, but also to recommend appropriate products based on the user's emotional state.

[0324] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0325] Step 1:

[0326] The user performs voice input.

[0327] Input: The user says, "I would like to buy three apples."

[0328] What happens: The device captures the user's voice with the microphone.

[0329] Output: The captured audio data.

[0330] Step 2:

[0331] The server converts the audio into text data.

[0332] Input: Audio data sent from the device.

[0333] Specific operation: The server calls a voice recognition cloud service and converts the voice data into text data.

[0334] Output: The text "I want to buy three apples".

[0335] Step 3:

[0336] The server analyzes the text data.

[0337] Input: Text data generated by a speech recognition tool.

[0338] How it works: The server uses the generative AI model to analyze the user's request from the text data.

[0339] Output: The request "3 apples".

[0340] Step 4:

[0341] The server recognizes the emotion.

[0342] Input: Audio and text data.

[0343] Specific operation: The server calls the emotion analysis cloud service and recognizes the user's emotional state from the tone, pitch, volume, etc. of their voice.

[0344] Output: Emotion recognition result that the user is relaxed.

[0345] Step 5:

[0346] The server uses the generative AI model to add products selected based on the analysis results of the user's requests to a cart on the mail-order site, and prompts the user to complete the purchase process, and using the generative AI model, adds products according to the user's requests to the cart and prompts the user to complete the purchase process.

[0347] Input: User request as analysis result.

[0348] What happens: The server calls an online shopping API to add three apples to the cart. The server then sends a message to the device prompting the user to complete the purchase.

[0349] Output: The message "3 apples added to cart. Would you like to checkout?"

[0350] Step 6:

[0351] The server uses a prompt sentence instructing the user to select a product on the online shopping site based on the recognized emotional state and a generative AI model to select a product on the online shopping site and presents the product to the user as an additional recommended product.

[0352] Input: Emotion recognition results.

[0353] Specific operation: The server uses the generated AI model and the API of the shopping site to select relaxing products (e.g. music CDs) and obtains information about the recommended products. The server then sends the information about the recommended products to the device.

[0354] Output: The message "Would you like a relaxing music CD?"

[0355] (Application example 1)

[0356] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0357] In conventional mail-order systems, when a user orders a product by voice, product recommendations do not take into account the user's emotional state, limiting the user experience. In addition, it is difficult to improve user satisfaction because the system is unable to recommend appropriate products according to the user's emotional state, such as when the user is relaxed or stressed.

[0358] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0359] In this invention, the server includes a voice recognition means, an analysis means, a mail order means, an emotion recognition means, and a recommendation means, which makes it possible to recognize a user's voice input in real time, analyze the user's request based on the voice data, and further recognize the user's emotional state to recommend products according to the emotional state.

[0360] A "voice recognition means" is a device or software that recognizes voice input from a user in real time and converts the voice into text data.

[0361] The "analysis means" refers to a device or software that analyzes the user's request from the text data recognized by the voice recognition means.

[0362] The "mail order means" refers to a device or software that automatically adds products to a cart based on the user's desires analyzed by the analysis means and carries out the purchasing procedure.

[0363] An "emotion recognition means" is a device or software that analyzes the tone, pitch, volume, etc. of a user's voice and recognizes the user's emotional state.

[0364] The "recommendation means" is a device or software that recommends products suitable for the user based on the emotional state recognized by the emotion recognition means.

[0365] As an embodiment of the present invention, a system including a voice recognition unit, an analysis unit, a mail order sales unit, an emotion recognition unit, and a recommendation unit is constructed. A specific embodiment of this system will be described below.

[0366] First, the server uses a voice recognition means to recognize voice input from the user in real time and convert the voice into text data. For this voice recognition means, for example, the "speech_recognition" library can be used.

[0367] Next, the server uses an analysis means to analyze the user's request from the text data recognized by the voice recognition means, which may use, for example, the "textblob" library.

[0368] The server then uses the mail order means to automatically add products to the cart based on the user's needs analyzed by the analysis means and to carry out the purchase procedure. The mail order means can send API requests to the mail order means using, for example, the "requests" library.

[0369] In addition, the server uses emotion recognition means to analyze the tone, pitch, volume, etc., of the user's voice to recognize the user's emotional state. For example, the emotion recognition means may use the "textblob" library.

[0370] Finally, the server uses a recommendation means to recommend products suitable for the user based on the emotional state recognized by the emotion recognition means. The recommendation means recommends products according to the emotional state of the user, for example, recommending relaxing products when the user is relaxed.

[0371] As a specific example, when a user says "I want to buy three apples," the speech recognition means converts the speech into text data, and the analysis means analyzes the request "three apples" from the text data. Next, the mail order means adds three apples to the cart, and the emotion recognition means recognizes the user's emotional state. If it is determined that the user is relaxed, the recommendation means recommends a product that will help the user relax (e.g., a music CD).

[0372] Examples of prompts to be input to a generative AI model include:

[0373] "If a user says, 'I want to buy three apples,' convert that speech to text, analyze the request and add it to the cart. Also, recognize the user's emotional state and recommend products that will help them relax if they're relaxed."

[0374] In this way, it is possible to recognize the user's voice input in real time, analyze their needs, and make product recommendations according to their emotional state.

[0375] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0376] Step 1:

[0377] The server uses a voice recognition means to recognize voice input from the user in real time. Specifically, when the user says, "I want to buy three apples," the voice recognition means converts the voice into text data. The input is the user's voice data, and the output is the text data "I want to buy three apples."

[0378] Step 2:

[0379] The server uses the analysis means to analyze the user's request from the text data recognized by the voice recognition means. Specifically, the server analyzes the text data "I want to buy three apples" and extracts the request "three apples." The input is the text data, and the output is the analyzed request data.

[0380] Step 3:

[0381] The server uses the mail order means to automatically add products to the cart based on the user's request analyzed by the analysis means, and performs the purchase procedure. Specifically, based on the request "three apples," three apples are added to the cart. The input is the analyzed request data, and the output is the product data added to the cart.

[0382] Step 4:

[0383] The server uses emotion recognition means to analyze the tone, pitch, volume, etc. of the user's voice and recognize the user's emotional state. Specifically, the server analyzes the tone, pitch, and volume of the user's voice data and determines whether the user is relaxed. The input is the user's voice data, and the output is emotional state data.

[0384] Step 5:

[0385] The server uses the recommendation means to recommend products suitable for the user based on the emotional state recognized by the emotion recognition means. Specifically, if it is determined that the user is relaxed, a relaxing product (e.g., a music CD) is recommended. The input is the emotional state data, and the output is the recommended product data.

[0386] Example 2

[0387] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0388] Conventional voice recognition systems can analyze user requests, but they have difficulty providing services that take into account the user's emotional state. This makes it difficult to provide personalized services to increase user satisfaction.

[0389] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0390] In this invention, the server includes a voice recognition means, an analysis means, an emotion recognition means, and a mail order means, which allows the server to recognize the user's voice input in real time, analyze the user's request, recognize the user's emotional state, and provide personalized product recommendations and purchase procedures based on the user's emotional state.

[0391] A "voice recognition means" is a means for recognizing voice input from a user in real time and converting the voice into text data.

[0392] The "analysis means" is a means for analyzing a user's request based on the text data recognized by the voice recognition means.

[0393] The "emotion recognition means" is a means for recognizing an emotional state of a user from the tone, pitch, volume, etc. of the user's voice based on the user's desire analyzed by the analysis means.

[0394] The "mail order means" is a means for automatically adding products that meet the user's needs to a cart based on the emotional state recognized by the emotion recognition means, and carrying out the purchasing procedure.

[0395] The present invention is a system including a voice recognition unit, an analysis unit, an emotion recognition unit, and a mail order sales unit. A specific embodiment of the system will be described below.

[0396] Voice Recognition Means

[0397] A user opens an application using a device such as a smartphone or tablet and speaks, "I would like to buy three apples." The device uses voice recognition software such as Google Cloud Speech-to-Text API to recognize the user's voice input in real time and converts the voice into text data. For example, the voice saying "I would like to buy three apples" is converted into text data such as "I would like to buy three apples."

[0398] Analysis means

[0399] The server receives the text data sent from the terminal and analyzes the user's request using natural language processing (NLP) techniques such as Python's NLTK library and spaCy. For example, the request "3 apples" is analyzed and the product name "apples" and the quantity "3" are extracted.

[0400] emotion recognition means

[0401] The server receives the data from the analysis means and recognizes the emotional state of the user from the tone, pitch, volume, etc. of the user's voice using emotion recognition software such as IBM Watson (registered trademark) Tone Analyzer or Microsoft (registered trademark) Azure (registered trademark) Emotion API. For example, if the user speaks in a calm tone, it is determined that the user is relaxed.

[0402] Mail order methods

[0403] Based on the analysis and emotion recognition results, the server automatically adds products to the cart and completes the purchase using APIs of mail-order platforms such as Amazon API and Rakuten API. For example, it adds three apples to the cart and recommends a relaxing music CD.

[0404] Examples

[0405] When a user opens the application and says "I want to buy three apples," the device converts the speech into text data using the Google Cloud Speech-to-Text API. Next, the server uses Python's NLTK library to analyze the request "three apples" and extracts the product name "apples" and the quantity "3." The server then uses IBM Watson Tone Analyzer to recognize the user's emotions and determine that the user is relaxed. Finally, the server uses the Amazon API to add three apples to the cart and recommend a relaxing music CD.

[0406] Examples of prompt statements

[0407] "If a user says, 'I want to buy three apples,' the device converts the speech to text data using the Google Cloud Speech-to-Text API, and the server analyzes the request 'three apples' using Python's NLTK library. Explain the process of the system that recognizes the user's emotions using IBM Watson Tone Analyzer, adds three apples to the cart using Amazon APIs, and recommends a relaxing music CD."

[0408] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0409] Step 1:

[0410] The user provides voice input.

[0411] A user opens an application using a device such as a smartphone or tablet and says, "I want to buy three apples." The input is the user's voice data, and the output is the voice signal sent to the device.

[0412] Step 2:

[0413] The device converts the voice into text data.

[0414] The device calls the Google Cloud Speech-to-Text API and sends voice data. The API converts the voice to text and returns the text data "I want to buy three apples." The input is a voice signal and the output is text data.

[0415] Step 3:

[0416] The server analyzes the text data.

[0417] The server receives the text data sent from the terminal and uses Python's NLTK library and spaCy to analyze the request "3 apples." As a result of the analysis, the product name "apples" and the quantity "3" are extracted. The input is text data, and the output is the analysis result.

[0418] Step 4:

[0419] The server recognizes emotions

[0420] The server receives the data from the analysis means and uses IBM Watson Tone Analyzer to analyze the tone, pitch, and volume of the user's voice. As a result of the analysis, it is determined that the user is relaxed. The inputs are the analysis results and the voice data, and the output is the emotion recognition result.

[0421] Step 5:

[0422] The server adds the item to the cart and checks out

[0423] Based on the analysis results and emotion recognition results, the server uses Amazon API to add three apples to the cart. It also recommends a relaxing music CD and sends a notification to the user to prompt the user to proceed with the purchase. The input is the analysis results and emotion recognition results, and the output is the products added to the cart and the recommended products.

[0424] (Application example 2)

[0425] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0426] Conventional mail-order systems have a function to add products to a cart based on the user's voice input, but lack a product recommendation function that takes into account the user's emotional state. This limits the user's purchasing experience, and there are issues with not being able to recommend products that are appropriate for the user's emotions. Furthermore, if product recommendations could be made based on the user's emotional state, such as when they are relaxed or stressed, a more personalized purchasing experience could be provided.

[0427] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0428] In this invention, the server includes a voice recognition means, an analysis means, a mail order means, an emotion recognition means, and a recommendation means, which makes it possible to not only recognize the user's voice input in real time and analyze the user's request, but also to recognize the user's emotional state and recommend appropriate products based on the recognition.

[0429] A "voice recognition means" is a device or software that recognizes voice input from a user in real time and converts the voice data into text data.

[0430] The "analysis means" refers to a device or software that analyzes the user's request from the text data converted by the voice recognition means.

[0431] The "mail order means" refers to a device or software that automatically adds products to a cart and completes the purchase procedure based on the user's desires analyzed by the analysis means.

[0432] An "emotion recognition means" is a device or software that recognizes a user's emotional state from the tone, pitch, volume, etc. of the user's voice.

[0433] The "recommendation means" is a device or software that recommends appropriate products to a user based on the emotional state recognized by the emotion recognition means.

[0434] As an embodiment of the present invention, an application installed on a smartphone will be described as an example. This application is a system including a voice recognition unit, an analysis unit, a mail order sales unit, an emotion recognition unit, and a recommendation unit.

[0435] System Configuration

[0436] 1. Voice Recognition Method

[0437] The speech recognition means recognizes speech input from the user in real time and converts the speech data into text data. Specifically, it uses the speech_recognition library to convert speech to text.

[0438] 2. Analysis method

[0439] The analysis means analyzes the user's request from the text data converted by the voice recognition means, and extracts the user's request by analyzing the text data using the TextBlob library.

[0440] 3. Mail Order Methods

[0441] The mail order means automatically adds the product to the cart and performs the purchase procedure based on the user's request analyzed by the analysis means. The mail order means uses the requests library to send an API request to add the product to the cart.

[0442] 4. Emotion recognition means

[0443] The emotion recognition means recognizes the user's emotional state from the user's voice tone, pitch, volume, etc. A simple emotion recognition algorithm is used to determine whether the user is relaxed.

[0444] 5. Recommendation method

[0445] The recommendation means recommends appropriate products to the user based on the emotional state recognized by the emotion recognition means. For example, if the user is relaxed, a relaxing music CD is recommended.

[0446] Processing flow

[0447] 1. A user opens an application on their smartphone and says, "I want to buy three apples."

[0448] 2. The speech recognition means recognizes the user's voice in real time and converts the voice into text data.

[0449] 3. The analytical means analyzes the user's request from the text data and extracts the request "three apples."

[0450] 4. The emotion recognition means recognizes the user's emotional state from the tone, pitch, volume, etc. of the user's voice and determines that the user is relaxed.

[0451] 5. Based on the analysis, the online shopping channel adds three apples to the cart and completes the purchase.

[0452] 6. The recommendation means recommends products such as relaxing music CDs based on the emotion recognition results.

[0453] Examples

[0454] When a user says "I want to buy three apples," the application recognizes the speech and converts it into text. It then parses the request "three apples" from the text, and if emotion recognition determines that the user is relaxed, it adds apples to the cart and recommends a relaxing music CD.

[0455] Examples of prompt statements

[0456] User: "I want to buy 3 apples."

[0457] Application: "I've added 3 apples to your cart. Would you like a relaxing music CD as well?"

[0458] In this way, we can create a smart shopping assistant that automatically adds products to the cart based on the user's voice input and makes product recommendations according to their emotional state.

[0459] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0460] Step 1:

[0461] A user opens an application on their smartphone and says, "I want to buy three apples."

[0462] Input: User's voice input

[0463] Output: Audio data

[0464] Specific behavior: The user launches the application and speaks their request into the microphone.

[0465] Step 2:

[0466] The voice recognition means of the terminal recognizes the user's voice in real time and converts the voice into text data.

[0467] Input: Audio data

[0468] Output: Text data

[0469] Specific behavior: Uses the speech_recognition library to convert audio data into text data.

[0470] Step 3:

[0471] The terminal's analysis means analyzes the user's request from the text data and extracts the request "three apples."

[0472] Input: Text data

[0473] Output: User's request (3 apples)

[0474] Specific operation: Uses the TextBlob library to parse text data and extract user requests.

[0475] Step 4:

[0476] An emotion recognition means of the terminal recognizes the emotional state of the user from the tone, pitch, volume, etc. of the user's voice and determines that the user is relaxed.

[0477] Input: Audio data

[0478] Output: Emotional state (Relaxed)

[0479] What it does: Analyzes the tone, pitch, and volume of audio data and uses a simple emotion recognition algorithm to determine emotional state.

[0480] Step 5:

[0481] Based on the analysis, the server's mail order solution adds three apples to the cart and completes the purchase.

[0482] Input: User request (3 apples)

[0483] Output: Items added to cart (3 apples)

[0484] What it does: Uses the requests library to send an API request to add a product to the cart.

[0485] Step 6:

[0486] The server's recommendation means recommends products such as relaxing music CDs based on the emotion recognition results.

[0487] Input: Emotional state (relaxed)

[0488] Output: Recommended Products (Relaxing Music CD)

[0489] Specific operation: Based on the emotion recognition results, select relaxing products and recommend them to the user.

[0490] In this way, we can create a smart shopping assistant that automatically adds products to the cart based on the user's voice input and makes product recommendations according to their emotional state.

[0491] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0492] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0493] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0494] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0495] [Second embodiment]

[0496] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0497] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0498] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0499] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0500] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0501] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0502] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0503] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0504] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0505] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0506] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0507] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0508] "Example 1"

[0509] The present invention is embodied as a system including a voice recognition means, an analysis means, and a mail order means. The voice recognition means recognizes voice input from a user in real time. For example, when a user says, "I would like to buy three apples," the voice recognition means converts the voice into text data. Next, the analysis means analyzes the user's request from the text data. In this example, the request "three apples" is analyzed. Finally, the mail order means automatically adds the product to the cart based on the analysis result and completes the purchase. In this example, three apples are added to the cart, and the user is prompted to complete the purchase.

[0510] "Example 2"

[0511] The present invention is embodied as a system including a voice recognition means, an analysis means, and a mail order means. The voice recognition means recognizes voice input from a user in real time. For example, when a user says, "I would like to buy three apples," the voice recognition means converts the voice into text data. Next, the analysis means analyzes the user's request from the text data. In this example, the request "three apples" is analyzed. Finally, the mail order means automatically adds the product to the cart based on the analysis result and completes the purchase. In this example, three apples are added to the cart, and the user is prompted to complete the purchase.

[0512] The process flow of each embodiment will be described below.

[0513] "Example 1"

[0514] Step 1: The user opens the application and says, "I want to buy 3 apples."

[0515] Step 2: The speech recognition means recognizes the speech input from the user in real time and converts the speech into text data. In this example, the speech "I want to buy three apples" is converted into text data.

[0516] Step 3: The analysis means analyzes the user's request from the text data. In this example, the request "3 apples" is analyzed.

[0517] Step 4: The online retailer automatically adds the products to the cart based on the analysis and checks out. In this example, it adds 3 apples to the cart and prompts the user to check out.

[0518] Example 1

[0519] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0520] Conventional voice recognition systems have had difficulty in accurately recognizing and analyzing users' voice inputs and conducting e-commerce based on the results. In addition, the accuracy and speed of the process of converting voice data into text data and analyzing the text data was insufficient, making it impossible to quickly and accurately reflect users' requests. This resulted in a decrease in user convenience and a decrease in the efficiency of e-commerce.

[0521] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0522] In this invention, the server includes a voice recognition means, an analysis means, an electronic commerce means, a conversion means for converting voice data into text data, a natural language processing means for analyzing the text data, a cart addition means for adding products to a cart based on the analysis result, and a procedure promotion means for promoting the purchase procedure. This makes it possible to recognize and analyze a user's voice input in real time, and to quickly and accurately add products to the cart based on the result, and promote the purchase procedure.

[0523] A "voice recognition means" is a device or software that recognizes voice input from a user in real time and acquires it as voice data.

[0524] The "analysis means" is a device or software that analyzes the user's request based on the voice data acquired by the voice recognition means.

[0525] The "electronic commerce means" refers to a device or software that provides electronic commerce services based on the user's requests analyzed by the analysis means.

[0526] A "conversion means" is a device or software that converts voice data into text data.

[0527] A "natural language processing means" is a device or software that uses natural language processing technology to analyze text data and extract user needs.

[0528] The "cart addition means" refers to a device or software that automatically adds products to a cart based on the analysis results.

[0529] The "procedure promotion means" refers to a device or software that displays a message to the user to encourage the user to complete the purchase procedure, and advances the purchase procedure.

[0530] The present invention is a system including a voice recognition unit, an analysis unit, an electronic commerce unit, a conversion unit, a natural language processing unit, an add-to-cart unit, and a procedure promotion unit. A specific embodiment of this system will be described below.

[0531] Hardware and Software Used

[0532] Speech recognition method: Use a speech recognition API (e.g. Google Cloud Speech-to-Text API) to recognize voice input in real time.

[0533] Analysis methods: Use Python scripts and natural language processing libraries (e.g. spaCy) to analyze the audio data.

[0534] E-commerce tools: Use the API of an e-commerce site (e.g. Amazon API) to provide e-commerce services.

[0535] Conversion method: A speech recognition API is used to convert voice data into text data.

[0536] Natural language processing tools: Use natural language processing libraries to analyze text data.

[0537] Method for adding to cart: Use the API of the e-commerce site to add products to the cart.

[0538] Checkout Facilitators: Use the user interface to facilitate checkout.

[0539] System Operation

[0540] 1. Acquiring voice input

[0541] The user says to the terminal, "I want to buy three apples."

[0542] The device uses a built-in microphone to capture and temporarily store audio data.

[0543] 2. Converting voice data to text

[0544] The device sends the saved voice data to the voice recognition API.

[0545] The server receives the voice data and converts it into text data in real time.

[0546] The server returns the converted text data "I want to buy three apples" to the terminal.

[0547] 3. Analysis of text data

[0548] The terminal passes the received text data to the Python script.

[0549] The server runs a Python script and uses natural language processing libraries to analyze the text data.

[0550] The server extracts the request, such as "three apples," and returns the analysis results to the terminal.

[0551] 4. Add to Cart

[0552] The terminal sends the analysis results to the EC site's API.

[0553] The server adds "3 apples" to the cart via the API.

[0554] The server confirms that the item has been added to the cart and notifies the terminal.

[0555] 5. Facilitating checkout

[0556] The terminal displays a message prompting the user to complete the purchase procedure.

[0557] The user confirms the displayed message and proceeds with the purchase procedure.

[0558] The terminal receives the user's input and completes the final purchase transaction.

[0559] Examples

[0560] User dictation: "I want to buy 3 apples."

[0561] Example prompt for a generative AI model: “If the user says they want to buy three apples, explain how you would proceed.”

[0562] In this way, the system specifically executes a series of processes from the user's voice input to product purchase.

[0563] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0564] Step 1: Getting voice input

[0565] The user says to the terminal, "I want to buy three apples." The terminal uses the built-in microphone to capture the voice data and temporarily stores it. The input is the user's voice, and the output is the stored voice data. Specifically, the terminal displays "Recording..." and captures the voice data.

[0566] Step 2: Convert audio to text

[0567] The device sends the saved voice data to the voice recognition API. The server receives the voice data and converts it into text data in real time. The input is voice data, and the output is the text data "I would like to buy three apples." In concrete terms, the device displays "Recognizing voice..." and the server converts the voice data into text data.

[0568] Step 3: Analyze the text data

[0569] The terminal passes the received text data to the Python script. The server executes the Python script and analyzes the text data using a natural language processing library. The input is the text data "I want to buy three apples," and the output is the analysis result "three apples." In concrete terms, the terminal displays "Analyzing text..." and the server extracts the request "three apples."

[0570] Step 4: Adding products to cart

[0571] The terminal sends the analysis result to the API of the EC site. The server adds "3 apples" to the cart via the API. The input is the analysis result "3 apples" and the output is information about the product added to the cart. In concrete terms, the terminal displays "Adding product to cart..." and the server adds the product to the cart.

[0572] Step 5: Facilitate checkout

[0573] The terminal displays a message encouraging the user to complete the purchase. The user confirms the displayed message and proceeds with the purchase. The input is information about the products added to the cart, and the output is a notification that the purchase has been completed. Specifically, the terminal displays the message "You have added three apples to your cart. Would you like to proceed with the purchase?" and the user selects "Yes" to complete the purchase.

[0574] (Application example 1)

[0575] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0576] Conventional online shopping systems have the problem that it takes a lot of time and effort for users to search for products, add them to their cart, and complete the purchase procedure. In addition, these operations are often difficult, especially for elderly people and people with visual impairments. Furthermore, even in systems that use voice recognition technology, it can be difficult to accurately recognize voice input and properly analyze the user's requests. This can result in a poor user experience and discourage users from using online shopping sites.

[0577] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0578] In this invention, the server includes a voice recognition means, an analysis means, and a mail order means. This allows a user to search for products through voice input, add them to a cart, and complete a purchase procedure. Specifically, the voice recognition means recognizes voice input from the user in real time, and the analysis means analyzes the user's request based on the recognized voice data. Furthermore, the mail order means automatically adds the product to the cart based on the analysis result, and completes the purchase procedure. This allows the user to easily purchase products using only voice, and provides a system that is particularly easy to use for the elderly and visually impaired. In addition, by using a generative AI model to analyze the user's request and generate a prompt sentence, more accurate analysis is possible, improving the user experience.

[0579] A "voice recognition means" is a device or software that recognizes voice input from a user in real time and converts the voice data into text data.

[0580] The "analysis means" is a device or software that analyzes the text data generated by the voice recognition means and extracts the user's request.

[0581] The "mail order means" refers to a device or software that automatically adds products to a cart and completes the purchase procedure based on the user's desires analyzed by the analysis means.

[0582] A "smartphone" is a portable information and communication terminal that implements voice recognition means, analysis means, and mail order means.

[0583] A "generative AI model" is an artificial intelligence model that analyzes user requests and generates prompt sentences.

[0584] A "prompt sentence" is text that specifically expresses the user's request, generated by a generative AI model.

[0585] "API" stands for Application Program Interface, an interface for exchanging data and functions between different software programs.

[0586] A "cart" is a virtual shopping basket on an online shopping site that temporarily stores products that a user wishes to purchase.

[0587] A "checkout" is a series of operations and processes that a user performs to actually purchase the items that they have added to their cart.

[0588] As an embodiment of the present invention, a system including a voice recognition unit, an analysis unit, and a mail order unit is implemented as an application installed on a smartphone. A specific embodiment of this system will be described below.

[0589] Voice Recognition Means

[0590] The speech recognition means recognizes voice input from the user in real time and converts the voice data into text data. Specifically, it collects voice using a microphone built into the smartphone and converts the voice data into text data using the SpeechRecognition library. When a user says, "I want to buy three apples," this speech recognition means converts the voice into text data.

[0591] Analysis means

[0592] The analysis means analyzes the text data generated by the voice recognition means and extracts the user's request. Specifically, the analysis means analyzes the text data using a generative AI model and identifies the user's request. The analysis means extracts the request "3 apples."

[0593] Mail order methods

[0594] The mail order means automatically adds products to the cart and completes the purchase procedure based on the user's request analyzed by the analysis means. Specifically, it sends a request to the API of the mail order site to add the specified products to the cart. The mail order means adds three apples to the cart and prompts the user to complete the purchase procedure.

[0595] Hardware and Software Used

[0596] Hardware: Smartphone, microphone

[0597] Software: Python, SpeechRecognition library, requests library, generative AI model

[0598] Examples

[0599] When a user says to their smartphone, "I want to buy three apples," the speech recognition means converts the speech into text data. Next, the analysis means uses a generative AI model to analyze the text data and extracts the request, "three apples." Finally, the mail order means sends a request to the API of the mail order site to add three apples to the cart and prompt the user to complete the purchase.

[0600] Examples of prompt statements

[0601] "When a user says to their smartphone, 'I want to buy three apples,' the voice recognition system converts the speech into text, and the analysis system extracts the request, 'three apples.' Then it sends a request to the API of the online shopping site to add three apples to the cart."

[0602] In this way, users can easily purchase products using only their voice, providing a system that is particularly easy to use for the elderly and visually impaired.

[0603] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0604] Step 1:

[0605] The user inputs voice into the smartphone.

[0606] Input: User speech (e.g. "I want to buy 3 apples")

[0607] Output: Audio data

[0608] Specific operation: Voice data is collected when the user speaks into the smartphone's microphone.

[0609] Step 2:

[0610] The terminal converts the voice data into text data using a voice recognition means.

[0611] Input: Audio data

[0612] Output: Text data (e.g. "I want to buy 3 apples")

[0613] Specific operation: The SpeechRecognition library installed on the smartphone analyzes the voice data and converts it into text data.

[0614] Step 3:

[0615] The terminal uses an analysis means to analyze the text data and extract the user's request.

[0616] Input: Text data (e.g. "I want to buy 3 apples")

[0617] Output: Demand data (e.g. "3 apples")

[0618] How it works: The generative AI model analyzes the text data and identifies the user's request. For example, the request "three apples" is extracted.

[0619] Step 4:

[0620] The terminal uses the mail order means to send a request to the API of the mail order site based on the analyzed request data.

[0621] Input: Request data (e.g. "3 apples")

[0622] Output: The result of the API request (e.g., a response indicating that a product was added to the cart)

[0623] Specific operation: The smartphone uses the requests library to send a request to the online shopping site's API to "add three apples to the cart."

[0624] Step 5:

[0625] The server receives the API request, adds the specified product to the cart, and completes the purchase.

[0626] Input: API request (e.g. "Add 3 apples to cart")

[0627] Output: A confirmation that the purchase has been completed (e.g. "3 apples added to cart")

[0628] Specific operation: The mail-order site's server processes the API request, adds the specified product to the cart, and completes the purchase. Once the process is complete, a notification of the completion of the purchase is sent to the smartphone.

[0629] Step 6:

[0630] The terminal displays a notification to the user that the purchase process is complete.

[0631] Input: Checkout completion notification (e.g. "3 apples added to cart")

[0632] Output: Display a notification to the user

[0633] Specific behavior: The smartphone receives a notification that the purchase procedure has been completed and displays the notification to the user, allowing the user to confirm that the purchase procedure has been completed.

[0634] Example 2

[0635] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0636] In conventional e-commerce systems, when users wanted to purchase a product, they had to manually search for the product, add it to their cart, and then complete the purchase procedure, which was time-consuming and inconvenient. Even in systems that used voice input, the accuracy and real-time nature of voice recognition was insufficient, and users' requests were sometimes not accurately reflected. This resulted in a poor user experience and made users avoid using the system.

[0637] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0638] In this invention, the server includes a voice recognition means, an analysis means, and an electronic commerce means, so that when a user purchases a product through voice input, the voice can be converted into text in real time, the text can be analyzed to understand the user's request, and the product can be automatically added to a cart based on the analysis results, and the user can be prompted to complete the purchase procedure.

[0639] A "voice recognition means" is a device or software that has the function of recognizing voice input from a user in real time and converting the voice data into text data.

[0640] The "analysis means" refers to a device or software having a function for analyzing the text data converted by the voice recognition means and understanding the user's request.

[0641] An "electronic commerce means" is a device or software having a function of automatically adding products to a cart and carrying out a purchasing procedure based on the user's desires analyzed by an analysis means.

[0642] "User" means an individual or organization that uses the system through voice input to purchase goods.

[0643] A "server" is a computer system that manages the entire system including the speech recognition means, analysis means, and electronic commerce means, and executes various processes.

[0644] "Voice input" refers to voice data generated by a user through an input device such as a microphone.

[0645] "Text data" refers to character string data converted from voice data by a voice recognition means.

[0646] A "cart" is a virtual shopping basket in an electronic commerce means for temporarily storing products that a user wishes to purchase.

[0647] A "checkout" is a series of operations and processes that a user performs to actually purchase an item that has been added to their cart.

[0648] The present invention relates to a system including a speech recognition unit, an analysis unit, and an electronic commerce unit. A specific embodiment of the system will be described below.

[0649] Hardware and Software Used

[0650] Hardware: Microphone, server, user device (smartphone, PC, etc.)

[0651] Software: voice recognition software (e.g. voice recognition APIs), analytics software (e.g. natural language processing libraries), e-commerce platforms (e.g. APIs for e-commerce sites)

[0652] System Operation

[0653] 1. Receiving voice input

[0654] The user provides voice input through a microphone.

[0655] For example, a user might say, "I want to buy three apples."

[0656] 2. Converting voice data to text

[0657] The server uses voice recognition software (voice recognition API) to convert the user's voice into text data in real time.

[0658] Specifically, the server sends voice data to the API, which converts the voice into text and returns it.

[0659] 3. Analysis of text data

[0660] The server passes the converted text data to analysis software (natural language processing library).

[0661] Analysis software analyzes user requests from text data.

[0662] Specifically, the product name "apple" and the quantity "3" are extracted from the text "three apples."

[0663] 4. Add to Cart

[0664] Based on the analysis results, the server calls the API of the e-commerce platform.

[0665] Specifically, the server sends the product name and quantity to the API, and the API adds the specified product to the cart.

[0666] 5. Prompt for purchase

[0667] The server sends a notification to the user's terminal prompting them to complete the purchase procedure.

[0668] Specifically, a message is displayed on the user's device stating, "Three apples have been added to your cart. Please complete the purchase."

[0669] Examples

[0670] User dictation: "I want to buy 3 apples."

[0671] Speech recognition result: "I want to buy three apples."

[0672] Analysis result: Product name: "Apple", quantity: "3"

[0673] E-commerce transaction action: Uses the API of the e-commerce site to add three apples to the cart and prompts the user to complete the purchase.

[0674] Example prompts for generative AI models

[0675] If a user says, "I want to buy three apples," the server uses speech recognition software to convert the speech to text, analyzes the text to understand what the user wants, and then calls the e-commerce platform's API to add three apples to the cart and prompt the user to check out.

[0676] In this way, the system automates the entire process from user voice input to product purchase.

[0677] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0678] Step 1:

[0679] Receiving voice input

[0680] The user provides voice input through a microphone.

[0681] For example, a user might say, "I want to buy three apples."

[0682] Input: User's voice data

[0683] Output: Audio data captured from the microphone.

[0684] Step 2:

[0685] Converting audio data to text

[0686] The server uses voice recognition software (voice recognition API) to convert the user's voice into text data in real time.

[0687] Specifically, the server sends voice data to the API, which converts the voice into text and returns it.

[0688] Input: Audio data

[0689] Data processing: Convert voice data into text data using speech recognition API

[0690] Output: Text data (e.g. "I want to buy 3 apples")

[0691] Step 3:

[0692] Analyzing Text Data

[0693] The server passes the converted text data to analysis software (natural language processing library).

[0694] Analysis software analyzes user requests from text data.

[0695] Specifically, the product name "apple" and the quantity "3" are extracted from the text "three apples."

[0696] Input: Text data

[0697] Data Calculation: Analyze text data using natural language processing libraries to extract product names and quantities

[0698] Output: Analysis result (product name "Apple", quantity "3")

[0699] Step 4:

[0700] Add to Cart

[0701] Based on the analysis results, the server calls the API of the e-commerce platform.

[0702] Specifically, the server sends the product name and quantity to the API, and the API adds the specified product to the cart.

[0703] Input: Analysis result (product name "Apple", quantity "3")

[0704] Data processing: Adding products to the cart using the API of an e-commerce platform

[0705] Output: Product information added to cart

[0706] Step 5:

[0707] Prompt for purchase

[0708] The server sends a notification to the user's terminal prompting them to complete the purchase procedure.

[0709] Specifically, a message is displayed on the user's device stating, "Three apples have been added to your cart. Please complete the purchase."

[0710] Input: Product information added to cart

[0711] Data processing: generating notification messages

[0712] Output: A notification message displayed on the user's terminal.

[0713] In this way, the system automates the entire process from user voice input to product purchase.

[0714] (Application example 2)

[0715] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0716] Conventional online shopping systems require users to search for products, add them to their cart, and complete the purchase process in multiple steps, which is time-consuming. Even in systems that use voice input, there are issues with the accuracy of voice recognition and real-time performance, which means the user experience does not improve.

[0717] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0718] In this invention, the server includes a voice recognition means, an analysis means, a mail order means, and a means including an application to be installed on a smartphone, which allows a user to search for products through voice input, add them to a cart, and complete the purchase procedure in real time.

[0719] A "voice recognition means" is a device or software that recognizes voice input from a user in real time and converts the voice data into text data.

[0720] The "analysis means" refers to a device or software that analyzes the text data obtained by the voice recognition means and extracts the user's request.

[0721] The "mail order means" refers to a device or software that selects products based on the user's needs analyzed by the analysis means, adds them to a cart, and completes the purchase procedure.

[0722] "Means including an application installed on a smartphone" refers to an application that runs on a smartphone and integrates speech recognition means, analysis means, and mail order means to enable a user to purchase products through voice input.

[0723] The system for carrying out the present invention includes a voice recognition unit, an analysis unit, a mail order unit, and a unit including an application installed on a smartphone. Specific embodiments of each unit will be described below.

[0724] First, the speech recognition means recognizes the voice input from the user in real time and converts the voice data into text data. This is done using the smartphone's microphone and a speech recognition library (for example, the speech_recognition library). When the user says, "I want to buy three apples," the speech recognition means converts the voice into text data.

[0725] Next, the analysis means analyzes the text data obtained by the voice recognition means and extracts the user's request. Specifically, an analysis logic is used to extract the product name and quantity from the text data. For example, from the text data "I want to buy three apples," "apples" and "three" are extracted.

[0726] The mail order means then selects products based on the user's needs analyzed by the analysis means, adds them to the cart, and completes the purchase procedure. This means uses the API of the mail order site to add the specified products and quantities to the cart and proceeds with the purchase procedure. For example, it adds three apples to the cart based on the analysis results, and prompts the user to complete the purchase procedure.

[0727] The means including the application installed on the smartphone integrates the voice recognition means, analysis means, and mail order means, and enables the user to purchase products through voice input. The user can search for products, add them to a cart, and complete the purchase process simply by speaking to the smartphone.

[0728] As a specific example, when a user says, "I would like to buy two bottles of milk," the voice recognition means converts the voice into text data, the analysis means extracts "milk" and "two bottles," and the mail order means adds the two bottles of milk to the cart and prompts the user to proceed with the purchase.

[0729] Example of a prompt to be input to a generative AI model:

[0730] If a user says, "I would like to buy two bottles of milk," generate a Python program that performs speech recognition, converts the speech to text, analyzes it, and adds two bottles of milk to the shopping cart on an online shopping site.

[0731] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0732] Step 1:

[0733] A user speaks to a smartphone, for example, "I want to buy two bottles of milk." This voice input is captured by the smartphone's microphone.

[0734] Step 2:

[0735] The smartphone's voice recognition means converts the captured voice data into text data. Specifically, it uses the speech_recognition library to analyze the voice data and generate text data such as "I would like to buy two bottles of milk." The input is voice data, and the output is text data.

[0736] Step 3:

[0737] The smartphone's analysis means analyzes the text data generated by the voice recognition means. Specifically, it uses analysis logic to extract product names and quantities from the text data. For example, from the text data "I would like to buy two bottles of milk," it extracts "milk" and "two bottles." The input is text data, and the output is a pair of product name and quantity.

[0738] Step 4:

[0739] The mail-order means of the smartphone calls the API of the mail-order site based on the product name and quantity extracted by the analysis means. Specifically, the requests library is used to send the product name and quantity to the API endpoint of the mail-order site and add it to the cart. For example, an API request is sent to add "Milk" and "2 bottles" to the cart. The input is the product name and quantity, and the output is the response to the API request.

[0740] Step 5:

[0741] The smartphone shopping tool receives the response to the API request and displays a message to the user urging them to complete the purchase. For example, it displays a message saying, "You have added two bottles of milk to your cart. Would you like to complete the purchase?" The input is the response to the API request, and the output is the message to the user.

[0742] Step 6:

[0743] The user performs an operation to confirm the purchase procedure, for example by tapping a "Purchase" button on the smartphone screen. This operation is captured by the smartphone's mail order processing device.

[0744] Step 7:

[0745] The smartphone mail order tool sends an API request to complete the purchase procedure based on the user's confirmation operation. Specifically, it uses the requests library to send a request to an API endpoint for completing the purchase procedure. The input is the user's confirmation operation, and the output is the API request response indicating the completion of the purchase procedure.

[0746] Step 8:

[0747] The smartphone mail order tool receives the response to the API request indicating the completion of the purchase procedure and displays a purchase completion message to the user. For example, it displays a message saying "The purchase procedure has been completed. Thank you." The input is the response to the API request indicating the completion of the purchase procedure, and the output is the purchase completion message to the user.

[0748] In addition, an emotion engine that estimates the emotion of the user may be further combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0749] "Example 1"

[0750] As one embodiment of the present invention, a system incorporating an emotion recognition means is provided. This system includes an emotion recognition means for recognizing emotions from the user's voice, in addition to a voice recognition means, an analysis means, and a mail order means. Specifically, when a user opens an application and speaks "I want to buy three apples," the voice recognition means recognizes the voice input from the user in real time and converts the voice into text data. In this example, the voice "I want to buy three apples" is converted into text data. The analysis means analyzes the user's request from this text data. In this example, the request "three apples" is analyzed. The emotion recognition means recognizes the user's emotional state from the tone, pitch, volume, etc. of the user's voice. In this example, if the user speaks in a calm tone about wanting to buy three apples, the user feels relaxed. The mail order means automatically adds the product to the cart based on the analysis result and the emotion recognition result, and performs the purchase procedure. In this example, three apples are added to the cart, and the user is prompted to perform the purchase procedure. In addition, if the user feels relaxed, a product such as a relaxing music CD is recommended.

[0751] "Example 2"

[0752] As one embodiment of the present invention, a system incorporating an emotion recognition means is provided. This system includes an emotion recognition means for recognizing emotions from the user's voice, in addition to a voice recognition means, an analysis means, and a mail order means. Specifically, when a user opens an application and speaks "I want to buy three apples," the voice recognition means recognizes the voice input from the user in real time and converts the voice into text data. In this example, the voice "I want to buy three apples" is converted into text data. The analysis means analyzes the user's request from this text data. In this example, the request "three apples" is analyzed. The emotion recognition means recognizes the user's emotional state from the tone, pitch, volume, etc. of the user's voice. In this example, if the user speaks in a calm tone about wanting to buy three apples, the user feels relaxed. The mail order means automatically adds the product to the cart based on the analysis result and the emotion recognition result, and performs the purchase procedure. In this example, three apples are added to the cart, and the user is prompted to perform the purchase procedure. In addition, if the user feels relaxed, a product such as a relaxing music CD is recommended.

[0753] The process flow of each embodiment will be described below.

[0754] "Example 1"

[0755] Step 1: The user opens the application and says, "I want to buy 3 apples."

[0756] Step 2: The speech recognition means recognizes the speech input from the user in real time and converts the speech into text data. In this example, the speech "I want to buy three apples" is converted into text data.

[0757] Step 3: The analysis means analyzes the user's request from the text data. In this example, the request "3 apples" is analyzed.

[0758] Step 4: The emotion recognition means recognizes the user's emotional state from the tone, pitch, volume, etc. of the user's voice. In this example, if the user speaks in a calm tone about his / her desire to buy three apples, the user is perceived as relaxed.

[0759] Step 5: The online retailer automatically adds the products to the cart based on the analysis and emotion recognition results, and checks out. In this example, it adds three apples to the cart and prompts the user to check out.

[0760] Step 6: If the user feels relaxed, recommend a product, such as a relaxing music CD.

[0761] Example 1

[0762] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0763] Conventional voice recognition systems were able to analyze users' requests and provide online shopping services, but they did not recommend products that took into account the user's emotional state. As a result, they were unable to recommend appropriate products according to the user's emotions, which resulted in a lack of improvement in the user experience.

[0764] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0765] In this invention, the server includes a voice recognition means, a prompt sentence for analyzing a user's request based on the voice data recognized by the voice recognition means, and adding a product selected based on the analysis result of the user's request to a shopping cart on a shopping site, and a means for carrying out a purchase procedure for a product according to the user's request using a generative AI model, an emotion recognition means for recognizing emotion from the user's voice, a prompt sentence for instructing the user to select a product on the shopping site based on the emotional state recognized by the emotion recognition means, and a recommendation means for selecting a product on the shopping site using the generative AI model and presenting it to the user as an additional recommended product. In the server, the means for carrying out the purchase procedure carries out the purchase procedure by sending a request for the purchase procedure to the API of the online shopping site. In the server, the recommendation means includes, when the emotional state recognized by the emotion recognition means is negative, the prompt sentence for instructing the user to select a product that suppresses the emotional state, and the generative AI model, and selecting a product on the shopping site and presenting it to the user as an additional recommended product. This makes it possible to not only recognize the user's voice input in real time and analyze the request, but also to recommend appropriate products based on the user's emotional state.

[0766] A "voice recognition means" is a means for recognizing voice input from a user in real time and converting the voice data into text data.

[0767] The "analysis means" is a means for analyzing a user's request based on the text data recognized by the voice recognition means.

[0768] The "mail order means" is a means for automatically adding products to a shopping cart and carrying out a purchasing procedure based on the user's desires analyzed by the analysis means.

[0769] An "emotion recognition means" is a means for analyzing the tone, pitch, volume, etc. of a user's voice and recognizing the user's emotional state.

[0770] The "recommendation means" is a means for presenting appropriate additional recommended products to the user based on the emotional state recognized by the emotion recognition means.

[0771] The present invention is a system including a voice recognition unit, an analysis unit, a mail order sales unit, an emotion recognition unit, and a recommendation unit. A specific embodiment of the system will be described below.

[0772] Hardware and Software Used

[0773] Speech recognition method: General speech recognition API (e.g., speech recognition cloud service)

[0774] Generative AI model: GPT-4 Analysis method: Natural language processing library (e.g. Python natural language processing library)

[0775] Mail order method: EC site API (e.g. online shopping API)

[0776] Emotion recognition means: Emotion analysis service (e.g. emotion analysis cloud service)

[0777] Recommendation method: Recommendation engine (e.g., recommendation system API)

[0778] System operation explanation

[0779] 1. Voice Recognition Method

[0780] The user opens the application and says, "I want to buy three apples."

[0781] The device captures the user's voice with a microphone and transmits the voice data to the server.

[0782] The server uses a voice recognition cloud service to convert the voice data into text data.

[0783] For example, speech saying "I want to buy three apples" is converted into text data saying "I want to buy three apples."

[0784] 2. Analysis method

[0785] The server uses a natural language processing library to analyze the user's request from the text data.

[0786] In this example, the request "3 apples" is extracted.

[0787] The server stores the analysis results in an internal database.

[0788] 3. Emotion recognition means

[0789] The server calls an emotion analysis cloud service to recognize the user's emotional state from the tone, pitch, volume, etc. of their voice.

[0790] For example, if the user speaks in a calm tone, the server will recognize that the user is relaxed.

[0791] The server stores the emotion recognition results in an internal database.

[0792] 4. Mail Order Methods

[0793] The server analyzes the user's request based on the recognized voice data, and adds the product selected based on the analysis result of the user's request to a shopping cart on the online shopping site, and performs the purchase procedure of the product according to the user's request using a prompt sentence and the generative AI model that instructs the user to complete the purchase procedure. Specifically, the server calls an online shopping API based on the analysis result and adds three apples to the cart. The server then sends a message to the terminal to prompt the user to complete the purchase procedure.

[0794] The terminal displays to the user, "You have added three apples to your cart. Would you like to complete the purchase?" If the user answers "Yes," the server executes the purchase procedure on the online shopping site. At this time, the purchase procedure is carried out by sending a request for the purchase procedure to the API of the online shopping site.

[0795] 5. Recommendation method

[0796] The server uses a prompt sentence instructing the user to select a product from the mail-order site based on the emotional state recognized by the emotion recognition means and a generative AI model to select a product from the mail-order site and presents the product to the user as an additional recommended product. At this time, if the recognized emotional state is negative, the server uses a prompt sentence instructing the user to select a product that suppresses the emotional state and a generative AI model to select a product from the mail-order site and presents the product to the user as an additional recommended product. For example, if the emotion recognition result is negative, the server selects a relaxing product (e.g., a music CD).

[0797] The server uses the recommendation system API to obtain information about the recommended products.

[0798] The server transmits information about the recommended products to the terminal.

[0799] The terminal displays to the user, "Would you like to listen to some relaxing music on CD?"

[0800] Examples

[0801] Example 1 of a prompt to check out a product "The user says, 'I want to buy three apples.' Analyze the user's request, select the product that best suits the user's request from the products on the online shopping site AA, and add it to the cart. Ask the user if they want to purchase the product, and then proceed with the purchase."

[0802] Example 2 of a prompt to check out a product "The user says, 'I'm looking for a relaxing music CD.' Analyze the user's request, select the product that best suits the user's request from the products on the online shopping site AA, and add it to the cart. Ask the user if they want to purchase the product, and then proceed with the purchase."

[0803] Example 2 of a prompt to recommend additional products "The user's emotion has been recognized as 'anger.' Please select a product from the AA online shopping site that can suppress the user's emotion and add it to the cart. Please present it to the user as an additional recommended product, ask if they would like to purchase it, and then proceed with the purchase." In this way, it is possible to not only recognize the user's voice input in real time and analyze their needs, but also to recommend appropriate products based on the user's emotional state.

[0804] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0805] Step 1:

[0806] The user performs voice input.

[0807] Input: The user says, "I would like to buy three apples."

[0808] What happens: The device captures the user's voice with the microphone.

[0809] Output: The captured audio data.

[0810] Step 2:

[0811] The server converts the audio into text data.

[0812] Input: Audio data sent from the device.

[0813] Specific operation: The server calls a voice recognition cloud service and converts the voice data into text data.

[0814] Output: The text "I want to buy three apples".

[0815] Step 3:

[0816] The server analyzes the text data.

[0817] Input: Text data generated by a speech recognition tool.

[0818] How it works: The server uses the generative AI model to analyze the user's request from the text data.

[0819] Output: The request "3 apples".

[0820] Step 4:

[0821] The server recognizes the emotion.

[0822] Input: Audio and text data.

[0823] Specific operation: The server calls the emotion analysis cloud service and recognizes the user's emotional state from the tone, pitch, volume, etc. of their voice.

[0824] Output: Emotion recognition result that the user is relaxed.

[0825] Step 5:

[0826] The server uses the generative AI model to add products selected based on the analysis results of the user's requests to a cart on the mail-order site, and prompts the user to complete the purchase process, and using the generative AI model, adds products according to the user's requests to the cart and prompts the user to complete the purchase process.

[0827] Input: User request as analysis result.

[0828] What happens: The server calls an online shopping API to add three apples to the cart. The server then sends a message to the device prompting the user to complete the purchase.

[0829] Output: The message "3 apples added to cart. Would you like to checkout?"

[0830] Step 6:

[0831] The server uses a prompt sentence instructing the user to select a product on the online shopping site based on the recognized emotional state and a generative AI model to select a product on the online shopping site and presents the product to the user as an additional recommended product.

[0832] Input: Emotion recognition results.

[0833] Specific operation: The server uses the generated AI model and the API of the shopping site to select relaxing products (e.g. music CDs) and obtains information about the recommended products. The server then sends the information about the recommended products to the device.

[0834] Output: The message "Would you like a relaxing music CD?"

[0835] (Application example 1)

[0836] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0837] In conventional mail-order systems, when a user orders a product by voice, product recommendations do not take into account the user's emotional state, limiting the user experience. In addition, it is difficult to improve user satisfaction because the system is unable to recommend appropriate products according to the user's emotional state, such as when the user is relaxed or stressed.

[0838] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0839] In this invention, the server includes a voice recognition means, an analysis means, a mail order means, an emotion recognition means, and a recommendation means, which makes it possible to recognize a user's voice input in real time, analyze the user's request based on the voice data, and further recognize the user's emotional state to recommend products according to the emotional state.

[0840] A "voice recognition means" is a device or software that recognizes voice input from a user in real time and converts the voice into text data.

[0841] The "analysis means" refers to a device or software that analyzes the user's request from the text data recognized by the voice recognition means.

[0842] The "mail order means" refers to a device or software that automatically adds products to a cart based on the user's desires analyzed by the analysis means and carries out the purchasing procedure.

[0843] An "emotion recognition means" is a device or software that analyzes the tone, pitch, volume, etc. of a user's voice and recognizes the user's emotional state.

[0844] The "recommendation means" is a device or software that recommends products suitable for the user based on the emotional state recognized by the emotion recognition means.

[0845] As an embodiment of the present invention, a system including a voice recognition unit, an analysis unit, a mail order sales unit, an emotion recognition unit, and a recommendation unit is constructed. A specific embodiment of this system will be described below.

[0846] First, the server uses a voice recognition means to recognize voice input from the user in real time and convert the voice into text data. For this voice recognition means, for example, the "speech_recognition" library can be used.

[0847] Next, the server uses an analysis means to analyze the user's request from the text data recognized by the voice recognition means, which may use, for example, the "textblob" library.

[0848] The server then uses the mail order means to automatically add products to the cart based on the user's needs analyzed by the analysis means and to carry out the purchase procedure. The mail order means can send API requests to the mail order means using, for example, the "requests" library.

[0849] In addition, the server uses emotion recognition means to analyze the tone, pitch, volume, etc., of the user's voice to recognize the user's emotional state. For example, the emotion recognition means may use the "textblob" library.

[0850] Finally, the server uses a recommendation means to recommend products suitable for the user based on the emotional state recognized by the emotion recognition means. The recommendation means recommends products according to the emotional state of the user, for example, recommending relaxing products when the user is relaxed.

[0851] As a specific example, when a user says "I want to buy three apples," the speech recognition means converts the speech into text data, and the analysis means analyzes the request "three apples" from the text data. Next, the mail order means adds three apples to the cart, and the emotion recognition means recognizes the user's emotional state. If it is determined that the user is relaxed, the recommendation means recommends a product that will help the user relax (e.g., a music CD).

[0852] Examples of prompts to be input to a generative AI model include:

[0853] "If a user says, 'I want to buy three apples,' convert that speech to text, analyze the request and add it to the cart. Also, recognize the user's emotional state and recommend products that will help them relax if they're relaxed."

[0854] In this way, it is possible to recognize the user's voice input in real time, analyze their needs, and make product recommendations according to their emotional state.

[0855] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0856] Step 1:

[0857] The server uses a voice recognition means to recognize voice input from the user in real time. Specifically, when the user says, "I want to buy three apples," the voice recognition means converts the voice into text data. The input is the user's voice data, and the output is the text data "I want to buy three apples."

[0858] Step 2:

[0859] The server uses the analysis means to analyze the user's request from the text data recognized by the voice recognition means. Specifically, the server analyzes the text data "I want to buy three apples" and extracts the request "three apples." The input is the text data, and the output is the analyzed request data.

[0860] Step 3:

[0861] The server uses the mail order means to automatically add products to the cart based on the user's request analyzed by the analysis means, and performs the purchase procedure. Specifically, based on the request "three apples," three apples are added to the cart. The input is the analyzed request data, and the output is the product data added to the cart.

[0862] Step 4:

[0863] The server uses emotion recognition means to analyze the tone, pitch, volume, etc. of the user's voice and recognize the user's emotional state. Specifically, the server analyzes the tone, pitch, and volume of the user's voice data and determines whether the user is relaxed. The input is the user's voice data, and the output is emotional state data.

[0864] Step 5:

[0865] The server uses the recommendation means to recommend products suitable for the user based on the emotional state recognized by the emotion recognition means. Specifically, if it is determined that the user is relaxed, a relaxing product (e.g., a music CD) is recommended. The input is the emotional state data, and the output is the recommended product data.

[0866] Example 2

[0867] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0868] Conventional voice recognition systems can analyze user requests, but they have difficulty providing services that take into account the user's emotional state. This makes it difficult to provide personalized services to increase user satisfaction.

[0869] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0870] In this invention, the server includes a voice recognition means, an analysis means, an emotion recognition means, and a mail order means, which allows the server to recognize the user's voice input in real time, analyze the user's request, recognize the user's emotional state, and provide personalized product recommendations and purchase procedures based on the user's emotional state.

[0871] A "voice recognition means" is a means for recognizing voice input from a user in real time and converting the voice into text data.

[0872] The "analysis means" is a means for analyzing a user's request based on the text data recognized by the voice recognition means.

[0873] The "emotion recognition means" is a means for recognizing an emotional state of a user from the tone, pitch, volume, etc. of the user's voice based on the user's desire analyzed by the analysis means.

[0874] The "mail order means" is a means for automatically adding products that meet the user's needs to a cart based on the emotional state recognized by the emotion recognition means, and carrying out the purchasing procedure.

[0875] The present invention is a system including a voice recognition unit, an analysis unit, an emotion recognition unit, and a mail order sales unit. A specific embodiment of the system will be described below.

[0876] Voice Recognition Means

[0877] A user opens an application using a device such as a smartphone or tablet and speaks, "I would like to buy three apples." The device uses voice recognition software such as Google Cloud Speech-to-Text API to recognize the user's voice input in real time and converts the voice into text data. For example, the voice saying "I would like to buy three apples" is converted into text data such as "I would like to buy three apples."

[0878] Analysis means

[0879] The server receives the text data sent from the terminal and analyzes the user's request using natural language processing (NLP) techniques such as Python's NLTK library and spaCy. For example, the request "3 apples" is analyzed and the product name "apples" and the quantity "3" are extracted.

[0880] emotion recognition means

[0881] The server receives the data from the analysis means and uses emotion recognition software such as IBM Watson Tone Analyzer or Microsoft Azure Emotion API to recognize the user's emotional state from the tone, pitch, volume, etc. of the user's voice. For example, if the user speaks in a calm tone, it is determined that the user is relaxed.

[0882] Mail order methods

[0883] Based on the analysis and emotion recognition results, the server automatically adds products to the cart and completes the purchase using APIs of mail-order platforms such as Amazon API and Rakuten API. For example, it adds three apples to the cart and recommends a relaxing music CD.

[0884] Examples

[0885] When a user opens the application and says "I want to buy three apples," the device converts the speech into text data using the Google Cloud Speech-to-Text API. Next, the server uses Python's NLTK library to analyze the request "three apples" and extracts the product name "apples" and the quantity "3." The server then uses IBM Watson Tone Analyzer to recognize the user's emotions and determine that the user is relaxed. Finally, the server uses the Amazon API to add three apples to the cart and recommend a relaxing music CD.

[0886] Examples of prompt statements

[0887] "If a user says, 'I want to buy three apples,' the device converts the speech to text data using the Google Cloud Speech-to-Text API, and the server analyzes the request 'three apples' using Python's NLTK library. Explain the process of the system that recognizes the user's emotions using IBM Watson Tone Analyzer, adds three apples to the cart using Amazon APIs, and recommends a relaxing music CD."

[0888] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0889] Step 1:

[0890] The user provides voice input.

[0891] A user opens an application using a device such as a smartphone or tablet and says, "I want to buy three apples." The input is the user's voice data, and the output is the voice signal sent to the device.

[0892] Step 2:

[0893] The device converts the voice into text data.

[0894] The device calls the Google Cloud Speech-to-Text API and sends voice data. The API converts the voice to text and returns the text data "I want to buy three apples." The input is a voice signal and the output is text data.

[0895] Step 3:

[0896] The server analyzes the text data.

[0897] The server receives the text data sent from the terminal and uses Python's NLTK library and spaCy to analyze the request "3 apples." As a result of the analysis, the product name "apples" and the quantity "3" are extracted. The input is text data, and the output is the analysis result.

[0898] Step 4:

[0899] The server recognizes emotions

[0900] The server receives the data from the analysis means and uses IBM Watson Tone Analyzer to analyze the tone, pitch, and volume of the user's voice. As a result of the analysis, it is determined that the user is relaxed. The inputs are the analysis results and the voice data, and the output is the emotion recognition result.

[0901] Step 5:

[0902] The server adds the item to the cart and checks out

[0903] Based on the analysis results and emotion recognition results, the server uses Amazon API to add three apples to the cart. It also recommends a relaxing music CD and sends a notification to the user to prompt the user to proceed with the purchase. The input is the analysis results and emotion recognition results, and the output is the products added to the cart and the recommended products.

[0904] (Application example 2)

[0905] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0906] Conventional mail-order systems have a function to add products to a cart based on the user's voice input, but lack a product recommendation function that takes into account the user's emotional state. This limits the user's purchasing experience, and there are issues with not being able to recommend products that are appropriate for the user's emotions. Furthermore, if product recommendations could be made based on the user's emotional state, such as when they are relaxed or stressed, a more personalized purchasing experience could be provided.

[0907] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0908] In this invention, the server includes a voice recognition means, an analysis means, a mail order means, an emotion recognition means, and a recommendation means, which makes it possible to not only recognize the user's voice input in real time and analyze the user's request, but also to recognize the user's emotional state and recommend appropriate products based on the recognition.

[0909] A "voice recognition means" is a device or software that recognizes voice input from a user in real time and converts the voice data into text data.

[0910] The "analysis means" refers to a device or software that analyzes the user's request from the text data converted by the voice recognition means.

[0911] The "mail order means" refers to a device or software that automatically adds products to a cart and completes the purchase procedure based on the user's desires analyzed by the analysis means.

[0912] An "emotion recognition means" is a device or software that recognizes a user's emotional state from the tone, pitch, volume, etc. of the user's voice.

[0913] The "recommendation means" is a device or software that recommends appropriate products to a user based on the emotional state recognized by the emotion recognition means.

[0914] As an embodiment of the present invention, an application installed on a smartphone will be described as an example. This application is a system including a voice recognition unit, an analysis unit, a mail order sales unit, an emotion recognition unit, and a recommendation unit.

[0915] System Configuration

[0916] 1. Voice Recognition Method

[0917] The speech recognition means recognizes speech input from the user in real time and converts the speech data into text data. Specifically, it uses the speech_recognition library to convert speech to text.

[0918] 2. Analysis method

[0919] The analysis means analyzes the user's request from the text data converted by the voice recognition means, and extracts the user's request by analyzing the text data using the TextBlob library.

[0920] 3. Mail Order Methods

[0921] The mail order means automatically adds the product to the cart and performs the purchase procedure based on the user's request analyzed by the analysis means. The mail order means uses the requests library to send an API request to add the product to the cart.

[0922] 4. Emotion recognition means

[0923] The emotion recognition means recognizes the user's emotional state from the user's voice tone, pitch, volume, etc. A simple emotion recognition algorithm is used to determine whether the user is relaxed.

[0924] 5. Recommendation method

[0925] The recommendation means recommends appropriate products to the user based on the emotional state recognized by the emotion recognition means. For example, if the user is relaxed, a relaxing music CD is recommended.

[0926] Processing flow

[0927] 1. A user opens an application on their smartphone and says, "I want to buy three apples."

[0928] 2. The speech recognition means recognizes the user's voice in real time and converts the voice into text data.

[0929] 3. The analytical means analyzes the user's request from the text data and extracts the request "three apples."

[0930] 4. The emotion recognition means recognizes the user's emotional state from the tone, pitch, volume, etc. of the user's voice and determines that the user is relaxed.

[0931] 5. Based on the analysis, the online shopping channel adds three apples to the cart and completes the purchase.

[0932] 6. The recommendation means recommends products such as relaxing music CDs based on the emotion recognition results.

[0933] Examples

[0934] When a user says "I want to buy three apples," the application recognizes the speech and converts it into text. It then parses the request "three apples" from the text, and if emotion recognition determines that the user is relaxed, it adds apples to the cart and recommends a relaxing music CD.

[0935] Examples of prompt statements

[0936] User: "I want to buy 3 apples."

[0937] Application: "I've added 3 apples to your cart. Would you like a relaxing music CD as well?"

[0938] In this way, we can create a smart shopping assistant that automatically adds products to the cart based on the user's voice input and makes product recommendations according to their emotional state.

[0939] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0940] Step 1:

[0941] A user opens an application on their smartphone and says, "I want to buy three apples."

[0942] Input: User's voice input

[0943] Output: Audio data

[0944] Specific behavior: The user launches the application and speaks their request into the microphone.

[0945] Step 2:

[0946] The voice recognition means of the terminal recognizes the user's voice in real time and converts the voice into text data.

[0947] Input: Audio data

[0948] Output: Text data

[0949] Specific behavior: Uses the speech_recognition library to convert audio data into text data.

[0950] Step 3:

[0951] The terminal's analysis means analyzes the user's request from the text data and extracts the request "three apples."

[0952] Input: Text data

[0953] Output: User's request (3 apples)

[0954] Specific operation: Uses the TextBlob library to parse text data and extract user requests.

[0955] Step 4:

[0956] An emotion recognition means of the terminal recognizes the emotional state of the user from the tone, pitch, volume, etc. of the user's voice and determines that the user is relaxed.

[0957] Input: Audio data

[0958] Output: Emotional state (Relaxed)

[0959] What it does: Analyzes the tone, pitch, and volume of audio data and uses a simple emotion recognition algorithm to determine emotional state.

[0960] Step 5:

[0961] Based on the analysis, the server's mail order solution adds three apples to the cart and completes the purchase.

[0962] Input: User request (3 apples)

[0963] Output: Items added to cart (3 apples)

[0964] What it does: Uses the requests library to send an API request to add a product to the cart.

[0965] Step 6:

[0966] The server's recommendation means recommends products such as relaxing music CDs based on the emotion recognition results.

[0967] Input: Emotional state (relaxed)

[0968] Output: Recommended Products (Relaxing Music CD)

[0969] Specific operation: Based on the emotion recognition results, select relaxing products and recommend them to the user.

[0970] In this way, we can create a smart shopping assistant that automatically adds products to the cart based on the user's voice input and makes product recommendations according to their emotional state.

[0971] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0972] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0973] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0974] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0975] [Third embodiment]

[0976] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0977] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0978] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0979] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0980] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0981] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0982] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0983] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0984] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0985] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0986] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0987] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0988] "Example 1"

[0989] The present invention is embodied as a system including a voice recognition means, an analysis means, and a mail order means. The voice recognition means recognizes voice input from a user in real time. For example, when a user says, "I would like to buy three apples," the voice recognition means converts the voice into text data. Next, the analysis means analyzes the user's request from the text data. In this example, the request "three apples" is analyzed. Finally, the mail order means automatically adds the product to the cart based on the analysis result and completes the purchase. In this example, three apples are added to the cart, and the user is prompted to complete the purchase.

[0990] "Example 2"

[0991] The present invention is embodied as a system including a voice recognition means, an analysis means, and a mail order means. The voice recognition means recognizes voice input from a user in real time. For example, when a user says, "I would like to buy three apples," the voice recognition means converts the voice into text data. Next, the analysis means analyzes the user's request from the text data. In this example, the request "three apples" is analyzed. Finally, the mail order means automatically adds the product to the cart based on the analysis result and completes the purchase. In this example, three apples are added to the cart, and the user is prompted to complete the purchase.

[0992] The process flow of each embodiment will be described below.

[0993] "Example 1"

[0994] Step 1: The user opens the application and says, "I want to buy three apples."

[0995] Step 2: The speech recognition means recognizes the speech input from the user in real time and converts the speech into text data. In this example, the speech "I want to buy three apples" is converted into text data.

[0996] Step 3: The analysis means analyzes the user's request from the text data. In this example, the request "3 apples" is analyzed.

[0997] Step 4: The online retailer automatically adds the products to the cart based on the analysis and checks out. In this example, it adds 3 apples to the cart and prompts the user to check out.

[0998] Example 1

[0999] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1000] Conventional voice recognition systems have had difficulty in accurately recognizing and analyzing users' voice inputs and conducting e-commerce based on the results. In addition, the accuracy and speed of the process of converting voice data into text data and analyzing the text data was insufficient, making it impossible to quickly and accurately reflect users' requests. This resulted in a decrease in user convenience and a decrease in the efficiency of e-commerce.

[1001] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1002] In this invention, the server includes a voice recognition means, an analysis means, an electronic commerce means, a conversion means for converting voice data into text data, a natural language processing means for analyzing the text data, a cart addition means for adding products to a cart based on the analysis result, and a procedure promotion means for promoting the purchase procedure. This makes it possible to recognize and analyze a user's voice input in real time, and to quickly and accurately add products to the cart based on the result, and promote the purchase procedure.

[1003] A "voice recognition means" is a device or software that recognizes voice input from a user in real time and acquires it as voice data.

[1004] The "analysis means" is a device or software that analyzes the user's request based on the voice data acquired by the voice recognition means.

[1005] The "electronic commerce means" refers to a device or software that provides electronic commerce services based on the user's requests analyzed by the analysis means.

[1006] A "conversion means" is a device or software that converts voice data into text data.

[1007] A "natural language processing means" is a device or software that uses natural language processing technology to analyze text data and extract user needs.

[1008] The "cart addition means" refers to a device or software that automatically adds products to a cart based on the analysis results.

[1009] The "procedure promotion means" refers to a device or software that displays a message to the user to encourage the user to proceed with the purchase procedure, and advances the purchase procedure.

[1010] The present invention is a system including a voice recognition unit, an analysis unit, an electronic commerce unit, a conversion unit, a natural language processing unit, an add-to-cart unit, and a procedure promotion unit. A specific embodiment of this system will be described below.

[1011] Hardware and Software Used

[1012] Speech recognition method: Use a speech recognition API (e.g. Google Cloud Speech-to-Text API) to recognize voice input in real time.

[1013] Analysis methods: Use Python scripts and natural language processing libraries (e.g. spaCy) to analyze the audio data.

[1014] E-commerce tools: Use the API of an e-commerce site (e.g. Amazon API) to provide e-commerce services.

[1015] Conversion method: A speech recognition API is used to convert voice data into text data.

[1016] Natural language processing tools: Use natural language processing libraries to analyze text data.

[1017] Method for adding to cart: Use the API of the e-commerce site to add products to the cart.

[1018] Checkout Facilitators: Use the user interface to facilitate checkout.

[1019] System Operation

[1020] 1. Acquiring voice input

[1021] The user says to the terminal, "I want to buy three apples."

[1022] The device uses a built-in microphone to capture and temporarily store audio data.

[1023] 2. Converting voice data to text

[1024] The device sends the saved voice data to the voice recognition API.

[1025] The server receives the voice data and converts it into text data in real time.

[1026] The server returns the converted text data "I want to buy three apples" to the terminal.

[1027] 3. Analysis of text data

[1028] The terminal passes the received text data to the Python script.

[1029] The server runs a Python script and uses natural language processing libraries to analyze the text data.

[1030] The server extracts the request, such as "three apples," and returns the analysis results to the terminal.

[1031] 4. Add to Cart

[1032] The terminal sends the analysis results to the EC site's API.

[1033] The server adds "3 apples" to the cart via the API.

[1034] The server confirms that the item has been added to the cart and notifies the terminal.

[1035] 5. Facilitating checkout

[1036] The terminal displays a message prompting the user to complete the purchase procedure.

[1037] The user confirms the displayed message and proceeds with the purchase procedure.

[1038] The terminal receives the user's input and completes the final purchase transaction.

[1039] Examples

[1040] User dictation: "I want to buy 3 apples."

[1041] Example prompt for a generative AI model: “If the user says they want to buy three apples, explain how you would proceed.”

[1042] In this way, the system specifically executes a series of processes from the user's voice input to product purchase.

[1043] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1044] Step 1: Getting voice input

[1045] The user says to the terminal, "I want to buy three apples." The terminal uses the built-in microphone to capture the voice data and temporarily stores it. The input is the user's voice, and the output is the stored voice data. Specifically, the terminal displays "Recording..." and captures the voice data.

[1046] Step 2: Convert audio to text

[1047] The device sends the saved voice data to the voice recognition API. The server receives the voice data and converts it into text data in real time. The input is voice data, and the output is the text data "I would like to buy three apples." In concrete terms, the device displays "Recognizing voice..." and the server converts the voice data into text data.

[1048] Step 3: Analyze the text data

[1049] The terminal passes the received text data to the Python script. The server executes the Python script and analyzes the text data using a natural language processing library. The input is the text data "I want to buy three apples," and the output is the analysis result "three apples." In concrete terms, the terminal displays "Analyzing text..." and the server extracts the request "three apples."

[1050] Step 4: Adding products to cart

[1051] The terminal sends the analysis result to the API of the EC site. The server adds "3 apples" to the cart via the API. The input is the analysis result "3 apples" and the output is information about the product added to the cart. In concrete terms, the terminal displays "Adding product to cart..." and the server adds the product to the cart.

[1052] Step 5: Facilitate checkout

[1053] The terminal displays a message encouraging the user to complete the purchase. The user confirms the displayed message and proceeds with the purchase. The input is information about the products added to the cart, and the output is a notification that the purchase has been completed. Specifically, the terminal displays the message "You have added three apples to your cart. Would you like to proceed with the purchase?" and the user selects "Yes" to complete the purchase.

[1054] (Application example 1)

[1055] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1056] Conventional online shopping systems have the problem that it takes a lot of time and effort for users to search for products, add them to their cart, and complete the purchase procedure. In addition, these operations are often difficult, especially for elderly people and people with visual impairments. Furthermore, even in systems that use voice recognition technology, it can be difficult to accurately recognize voice input and properly analyze the user's requests. This can result in a poor user experience and discourage users from using online shopping sites.

[1057] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1058] In this invention, the server includes a voice recognition means, an analysis means, and a mail order means. This allows a user to search for products through voice input, add them to a cart, and complete a purchase procedure. Specifically, the voice recognition means recognizes voice input from the user in real time, and the analysis means analyzes the user's request based on the recognized voice data. Furthermore, the mail order means automatically adds the product to the cart based on the analysis result, and completes the purchase procedure. This allows the user to easily purchase products using only voice, and provides a system that is particularly easy to use for the elderly and visually impaired. In addition, by using a generative AI model to analyze the user's request and generate a prompt sentence, more accurate analysis is possible, improving the user experience.

[1059] A "voice recognition means" is a device or software that recognizes voice input from a user in real time and converts the voice data into text data.

[1060] The "analysis means" is a device or software that analyzes the text data generated by the voice recognition means and extracts the user's request.

[1061] The "mail order means" refers to a device or software that automatically adds products to a cart and completes the purchase procedure based on the user's desires analyzed by the analysis means.

[1062] A "smartphone" is a portable information and communication terminal that implements voice recognition means, analysis means, and mail order means.

[1063] A "generative AI model" is an artificial intelligence model that analyzes user requests and generates prompt sentences.

[1064] A "prompt sentence" is text that specifically expresses the user's request, generated by a generative AI model.

[1065] "API" stands for Application Program Interface, an interface for exchanging data and functions between different software programs.

[1066] A "cart" is a virtual shopping basket on an online shopping site that temporarily stores products that a user wishes to purchase.

[1067] A "checkout" is a series of operations and processes that a user performs to actually purchase the items that they have added to their cart.

[1068] As an embodiment of the present invention, a system including a voice recognition unit, an analysis unit, and a mail order unit is implemented as an application installed on a smartphone. A specific embodiment of this system will be described below.

[1069] Voice Recognition Means

[1070] The speech recognition means recognizes voice input from the user in real time and converts the voice data into text data. Specifically, it collects voice using a microphone built into the smartphone and converts the voice data into text data using the SpeechRecognition library. When a user says, "I want to buy three apples," this speech recognition means converts the voice into text data.

[1071] Analysis means

[1072] The analysis means analyzes the text data generated by the voice recognition means and extracts the user's request. Specifically, the analysis means analyzes the text data using a generative AI model and identifies the user's request. The analysis means extracts the request "3 apples."

[1073] Mail order methods

[1074] The mail order means automatically adds products to the cart and completes the purchase procedure based on the user's request analyzed by the analysis means. Specifically, it sends a request to the API of the mail order site to add the specified products to the cart. The mail order means adds three apples to the cart and prompts the user to complete the purchase procedure.

[1075] Hardware and Software Used

[1076] Hardware: Smartphone, microphone

[1077] Software: Python, SpeechRecognition library, requests library, generative AI model

[1078] Examples

[1079] When a user says to their smartphone, "I want to buy three apples," the speech recognition means converts the speech into text data. Next, the analysis means uses a generative AI model to analyze the text data and extracts the request, "three apples." Finally, the mail order means sends a request to the API of the mail order site to add three apples to the cart and prompt the user to complete the purchase.

[1080] Examples of prompt statements

[1081] "When a user says to their smartphone, 'I want to buy three apples,' the voice recognition system converts the speech into text, and the analysis system extracts the request, 'three apples.' Then it sends a request to the API of the online shopping site to add three apples to the cart."

[1082] In this way, users can easily purchase products using only their voice, providing a system that is particularly easy to use for the elderly and visually impaired.

[1083] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1084] Step 1:

[1085] The user inputs voice into the smartphone.

[1086] Input: User speech (e.g. "I want to buy 3 apples")

[1087] Output: Audio data

[1088] Specific operation: Voice data is collected when the user speaks into the smartphone's microphone.

[1089] Step 2:

[1090] The terminal converts the voice data into text data using a voice recognition means.

[1091] Input: Audio data

[1092] Output: Text data (e.g. "I want to buy 3 apples")

[1093] Specific operation: The SpeechRecognition library installed on the smartphone analyzes the voice data and converts it into text data.

[1094] Step 3:

[1095] The terminal uses an analysis means to analyze the text data and extract the user's request.

[1096] Input: Text data (e.g. "I want to buy 3 apples")

[1097] Output: Demand data (e.g. "3 apples")

[1098] How it works: The generative AI model analyzes the text data and identifies the user's request. For example, the request "three apples" is extracted.

[1099] Step 4:

[1100] The terminal uses the mail order means to send a request to the API of the mail order site based on the analyzed request data.

[1101] Input: Request data (e.g. "3 apples")

[1102] Output: The result of the API request (e.g., a response indicating that a product was added to the cart)

[1103] Specific operation: The smartphone uses the requests library to send a request to the online shopping site's API to "add three apples to the cart."

[1104] Step 5:

[1105] The server receives the API request, adds the specified product to the cart, and completes the purchase.

[1106] Input: API request (e.g. "Add 3 apples to cart")

[1107] Output: A confirmation that the purchase has been completed (e.g. "3 apples added to cart")

[1108] Specific operation: The mail-order site's server processes the API request, adds the specified product to the cart, and completes the purchase. Once the process is complete, a notification of the completion of the purchase is sent to the smartphone.

[1109] Step 6:

[1110] The terminal displays a notification to the user that the purchase process is complete.

[1111] Input: Checkout completion notification (e.g. "3 apples added to cart")

[1112] Output: Display a notification to the user

[1113] Specific behavior: The smartphone receives a notification that the purchase procedure has been completed and displays the notification to the user, allowing the user to confirm that the purchase procedure has been completed.

[1114] Example 2

[1115] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1116] In conventional e-commerce systems, when users wanted to purchase a product, they had to manually search for the product, add it to their cart, and then complete the purchase procedure, which was time-consuming and inconvenient. Even in systems that used voice input, the accuracy and real-time nature of voice recognition was insufficient, and users' requests were sometimes not accurately reflected. This resulted in a poor user experience and made users avoid using the system.

[1117] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1118] In this invention, the server includes a voice recognition means, an analysis means, and an electronic commerce means, so that when a user purchases a product through voice input, the voice can be converted into text in real time, the text can be analyzed to understand the user's request, and the product can be automatically added to a cart based on the analysis results, and the user can be prompted to complete the purchase procedure.

[1119] A "voice recognition means" is a device or software that has the function of recognizing voice input from a user in real time and converting the voice data into text data.

[1120] The "analysis means" refers to a device or software having a function for analyzing the text data converted by the voice recognition means and understanding the user's request.

[1121] An "electronic commerce means" is a device or software having a function of automatically adding products to a cart and carrying out a purchasing procedure based on the user's desires analyzed by an analysis means.

[1122] "User" means an individual or organization that uses the system through voice input to purchase goods.

[1123] A "server" is a computer system that manages the entire system including the speech recognition means, analysis means, and electronic commerce means, and executes various processes.

[1124] "Voice input" refers to voice data generated by a user through an input device such as a microphone.

[1125] "Text data" refers to character string data converted from voice data by a voice recognition means.

[1126] A "cart" is a virtual shopping basket in an electronic commerce means for temporarily storing products that a user wishes to purchase.

[1127] A "checkout" is a series of operations and processes that a user performs to actually purchase an item that has been added to their cart.

[1128] The present invention relates to a system including a speech recognition unit, an analysis unit, and an electronic commerce unit. A specific embodiment of the system will be described below.

[1129] Hardware and Software Used

[1130] Hardware: Microphone, server, user device (smartphone, PC, etc.)

[1131] Software: voice recognition software (e.g. voice recognition APIs), analytics software (e.g. natural language processing libraries), e-commerce platforms (e.g. APIs for e-commerce sites)

[1132] System Operation

[1133] 1. Receiving voice input

[1134] The user provides voice input through a microphone.

[1135] For example, a user might say, "I want to buy three apples."

[1136] 2. Converting voice data to text

[1137] The server uses voice recognition software (voice recognition API) to convert the user's voice into text data in real time.

[1138] Specifically, the server sends voice data to the API, which converts the voice into text and returns it.

[1139] 3. Analysis of text data

[1140] The server passes the converted text data to analysis software (natural language processing library).

[1141] Analysis software analyzes user requests from text data.

[1142] Specifically, the product name "apple" and the quantity "3" are extracted from the text "three apples."

[1143] 4. Add to Cart

[1144] Based on the analysis results, the server calls the API of the e-commerce platform.

[1145] Specifically, the server sends the product name and quantity to the API, and the API adds the specified product to the cart.

[1146] 5. Prompt for purchase

[1147] The server sends a notification to the user's terminal prompting them to complete the purchase procedure.

[1148] Specifically, a message is displayed on the user's device stating, "Three apples have been added to your cart. Please complete the purchase."

[1149] Examples

[1150] User dictation: "I want to buy 3 apples."

[1151] Speech recognition result: "I want to buy three apples."

[1152] Analysis result: Product name: "Apple", quantity: "3"

[1153] E-commerce transaction action: Uses the API of the e-commerce site to add three apples to the cart and prompts the user to complete the purchase.

[1154] Example prompts for generative AI models

[1155] If a user says, "I want to buy three apples," the server uses speech recognition software to convert the speech to text, analyzes the text to understand what the user wants, and then calls the e-commerce platform's API to add three apples to the cart and prompt the user to check out.

[1156] In this way, the system automates the entire process from user voice input to product purchase.

[1157] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1158] Step 1:

[1159] Receiving voice input

[1160] The user provides voice input through a microphone.

[1161] For example, a user might say, "I want to buy three apples."

[1162] Input: User's voice data

[1163] Output: Audio data captured from the microphone.

[1164] Step 2:

[1165] Converting audio data to text

[1166] The server uses voice recognition software (voice recognition API) to convert the user's voice into text data in real time.

[1167] Specifically, the server sends voice data to the API, which converts the voice into text and returns it.

[1168] Input: Audio data

[1169] Data processing: Convert voice data into text data using speech recognition API

[1170] Output: Text data (e.g. "I want to buy 3 apples")

[1171] Step 3:

[1172] Analyzing Text Data

[1173] The server passes the converted text data to analysis software (natural language processing library).

[1174] Analysis software analyzes user requests from text data.

[1175] Specifically, the product name "apple" and the quantity "3" are extracted from the text "three apples."

[1176] Input: Text data

[1177] Data Calculation: Analyze text data using natural language processing libraries to extract product names and quantities

[1178] Output: Analysis result (product name "Apple", quantity "3")

[1179] Step 4:

[1180] Add to Cart

[1181] Based on the analysis results, the server calls the API of the e-commerce platform.

[1182] Specifically, the server sends the product name and quantity to the API, and the API adds the specified product to the cart.

[1183] Input: Analysis result (product name "Apple", quantity "3")

[1184] Data processing: Adding products to the cart using the API of an e-commerce platform

[1185] Output: Product information added to cart

[1186] Step 5:

[1187] Prompt for purchase

[1188] The server sends a notification to the user's terminal prompting them to complete the purchase procedure.

[1189] Specifically, a message is displayed on the user's device stating, "Three apples have been added to your cart. Please complete the purchase."

[1190] Input: Product information added to cart

[1191] Data processing: generating notification messages

[1192] Output: A notification message displayed on the user's terminal.

[1193] In this way, the system automates the entire process from user voice input to product purchase.

[1194] (Application example 2)

[1195] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1196] Conventional mail-order systems require users to search for products, add them to their cart, and complete the purchase process in multiple steps, which is time-consuming. Even in systems that use voice input, there are issues with the accuracy of voice recognition and real-time performance, which means the user experience does not improve.

[1197] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1198] In this invention, the server includes a voice recognition means, an analysis means, a mail order means, and a means including an application to be installed on a smartphone, which allows a user to search for products through voice input, add them to a cart, and complete the purchase procedure in real time.

[1199] A "voice recognition means" is a device or software that recognizes voice input from a user in real time and converts the voice data into text data.

[1200] The "analysis means" refers to a device or software that analyzes the text data obtained by the voice recognition means and extracts the user's request.

[1201] The "mail order means" refers to a device or software that selects products based on the user's needs analyzed by the analysis means, adds them to a cart, and completes the purchase procedure.

[1202] "Means including an application installed on a smartphone" refers to an application that runs on a smartphone and integrates speech recognition means, analysis means, and mail order means to enable a user to purchase products through voice input.

[1203] The system for carrying out the present invention includes a voice recognition unit, an analysis unit, a mail order unit, and a unit including an application installed on a smartphone. Specific embodiments of each unit will be described below.

[1204] First, the speech recognition means recognizes the voice input from the user in real time and converts the voice data into text data. This is done using the smartphone's microphone and a speech recognition library (for example, the speech_recognition library). When the user says, "I want to buy three apples," the speech recognition means converts the voice into text data.

[1205] Next, the analysis means analyzes the text data obtained by the voice recognition means and extracts the user's request. Specifically, an analysis logic is used to extract the product name and quantity from the text data. For example, from the text data "I want to buy three apples," "apples" and "three" are extracted.

[1206] The mail order means then selects products based on the user's needs analyzed by the analysis means, adds them to the cart, and completes the purchase procedure. This means uses the API of the mail order site to add the specified products and quantities to the cart and proceeds with the purchase procedure. For example, it adds three apples to the cart based on the analysis results, and prompts the user to complete the purchase procedure.

[1207] The means including the application installed on the smartphone integrates the voice recognition means, analysis means, and mail order means, and enables the user to purchase products through voice input. The user can search for products, add them to a cart, and complete the purchase process simply by speaking to the smartphone.

[1208] As a specific example, when a user says, "I would like to buy two bottles of milk," the voice recognition means converts the voice into text data, the analysis means extracts "milk" and "two bottles," and the mail order means adds the two bottles of milk to the cart and prompts the user to proceed with the purchase.

[1209] Example of a prompt to be input to a generative AI model:

[1210] If a user says, "I would like to buy two bottles of milk," generate a Python program that performs speech recognition, converts the speech to text, analyzes it, and adds two bottles of milk to the shopping cart on an online shopping site.

[1211] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1212] Step 1:

[1213] A user speaks to a smartphone, for example, "I want to buy two bottles of milk." This voice input is captured by the smartphone's microphone.

[1214] Step 2:

[1215] The smartphone's voice recognition means converts the captured voice data into text data. Specifically, it uses the speech_recognition library to analyze the voice data and generate text data such as "I would like to buy two bottles of milk." The input is voice data, and the output is text data.

[1216] Step 3:

[1217] The smartphone's analysis means analyzes the text data generated by the voice recognition means. Specifically, it uses analysis logic to extract product names and quantities from the text data. For example, from the text data "I would like to buy two bottles of milk," it extracts "milk" and "two bottles." The input is text data, and the output is a pair of product name and quantity.

[1218] Step 4:

[1219] The mail-order means of the smartphone calls the API of the mail-order site based on the product name and quantity extracted by the analysis means. Specifically, the requests library is used to send the product name and quantity to the API endpoint of the mail-order site and add it to the cart. For example, an API request is sent to add "Milk" and "2 bottles" to the cart. The input is the product name and quantity, and the output is the response to the API request.

[1220] Step 5:

[1221] The smartphone shopping tool receives the response to the API request and displays a message to the user urging them to complete the purchase. For example, it displays a message saying, "You have added two bottles of milk to your cart. Would you like to complete the purchase?" The input is the response to the API request, and the output is the message to the user.

[1222] Step 6:

[1223] The user performs an operation to confirm the purchase procedure, for example by tapping a "Purchase" button on the smartphone screen. This operation is captured by the smartphone's mail order processing device.

[1224] Step 7:

[1225] The smartphone mail order tool sends an API request to complete the purchase procedure based on the user's confirmation operation. Specifically, it uses the requests library to send a request to an API endpoint for completing the purchase procedure. The input is the user's confirmation operation, and the output is the API request response indicating the completion of the purchase procedure.

[1226] Step 8:

[1227] The smartphone mail order tool receives the response to the API request indicating the completion of the purchase procedure and displays a purchase completion message to the user. For example, it displays a message saying "The purchase procedure has been completed. Thank you." The input is the response to the API request indicating the completion of the purchase procedure, and the output is the purchase completion message to the user.

[1228] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1229] "Example 1"

[1230] As one embodiment of the present invention, a system incorporating an emotion recognition means is provided. This system includes an emotion recognition means for recognizing emotions from the user's voice, in addition to a voice recognition means, an analysis means, and a mail order means. Specifically, when a user opens an application and speaks "I want to buy three apples," the voice recognition means recognizes the voice input from the user in real time and converts the voice into text data. In this example, the voice "I want to buy three apples" is converted into text data. The analysis means analyzes the user's request from this text data. In this example, the request "three apples" is analyzed. The emotion recognition means recognizes the user's emotional state from the tone, pitch, volume, etc. of the user's voice. In this example, if the user speaks in a calm tone about wanting to buy three apples, the user feels relaxed. The mail order means automatically adds the product to the cart based on the analysis result and the emotion recognition result, and performs the purchase procedure. In this example, three apples are added to the cart, and the user is prompted to perform the purchase procedure. In addition, if the user feels relaxed, a product such as a relaxing music CD is recommended.

[1231] "Example 2"

[1232] As one embodiment of the present invention, a system incorporating an emotion recognition means is provided. This system includes an emotion recognition means for recognizing emotions from the user's voice, in addition to a voice recognition means, an analysis means, and a mail order means. Specifically, when a user opens an application and speaks "I want to buy three apples," the voice recognition means recognizes the voice input from the user in real time and converts the voice into text data. In this example, the voice "I want to buy three apples" is converted into text data. The analysis means analyzes the user's request from this text data. In this example, the request "three apples" is analyzed. The emotion recognition means recognizes the user's emotional state from the tone, pitch, volume, etc. of the user's voice. In this example, if the user speaks in a calm tone about wanting to buy three apples, the user feels relaxed. The mail order means automatically adds the product to the cart based on the analysis result and the emotion recognition result, and performs the purchase procedure. In this example, three apples are added to the cart, and the user is prompted to perform the purchase procedure. In addition, if the user feels relaxed, a product such as a relaxing music CD is recommended.

[1233] The process flow of each embodiment will be described below.

[1234] "Example 1"

[1235] Step 1: The user opens the application and says, "I want to buy 3 apples."

[1236] Step 2: The speech recognition means recognizes the speech input from the user in real time and converts the speech into text data. In this example, the speech "I want to buy three apples" is converted into text data.

[1237] Step 3: The analysis means analyzes the user's request from the text data. In this example, the request "3 apples" is analyzed.

[1238] Step 4: The emotion recognition means recognizes the user's emotional state from the tone, pitch, volume, etc. of the user's voice. In this example, if the user speaks in a calm tone about his / her desire to buy three apples, the user is perceived as relaxed.

[1239] Step 5: The online retailer automatically adds the products to the cart based on the analysis and emotion recognition results, and checks out. In this example, it adds three apples to the cart and prompts the user to check out.

[1240] Step 6: If the user feels relaxed, recommend a product, such as a relaxing music CD.

[1241] Example 1

[1242] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1243] Conventional voice recognition systems were able to analyze users' requests and provide online shopping services, but they did not recommend products that took into account the user's emotional state. As a result, they were unable to recommend appropriate products according to the user's emotions, which resulted in a lack of improvement in the user experience.

[1244] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1245] In this invention, the server includes a voice recognition means, a prompt sentence for analyzing a user's request based on the voice data recognized by the voice recognition means, and adding a product selected based on the analysis result of the user's request to a shopping cart on a shopping site, and a means for carrying out a purchase procedure for a product according to the user's request using a generative AI model, an emotion recognition means for recognizing emotion from the user's voice, a prompt sentence for instructing the user to select a product on the shopping site based on the emotional state recognized by the emotion recognition means, and a recommendation means for selecting a product on the shopping site using the generative AI model and presenting it to the user as an additional recommended product. In the server, the means for carrying out the purchase procedure carries out the purchase procedure by sending a request for the purchase procedure to the API of the online shopping site. In the server, the recommendation means includes, when the emotional state recognized by the emotion recognition means is negative, the prompt sentence for instructing the user to select a product that suppresses the emotional state, and the generative AI model, and selecting a product on the shopping site and presenting it to the user as an additional recommended product. This makes it possible to not only recognize the user's voice input in real time and analyze the request, but also to recommend appropriate products based on the user's emotional state.

[1246] A "voice recognition means" is a means for recognizing voice input from a user in real time and converting the voice data into text data.

[1247] The "analysis means" is a means for analyzing a user's request based on the text data recognized by the voice recognition means.

[1248] The "mail order means" is a means for automatically adding products to a shopping cart and carrying out a purchasing procedure based on the user's desires analyzed by the analysis means.

[1249] An "emotion recognition means" is a means for analyzing the tone, pitch, volume, etc. of a user's voice and recognizing the user's emotional state.

[1250] The "recommendation means" is a means for presenting appropriate additional recommended products to the user based on the emotional state recognized by the emotion recognition means.

[1251] The present invention is a system including a voice recognition unit, an analysis unit, a mail order sales unit, an emotion recognition unit, and a recommendation unit. A specific embodiment of the system will be described below.

[1252] Hardware and Software Used

[1253] Speech recognition method: General speech recognition API (e.g., speech recognition cloud service)

[1254] Generative AI model: GPT-4 Analysis method: Natural language processing library (e.g. Python natural language processing library)

[1255] Mail order method: EC site API (e.g. online shopping API)

[1256] Emotion recognition means: Emotion analysis service (e.g. emotion analysis cloud service)

[1257] Recommendation method: Recommendation engine (e.g., recommendation system API)

[1258] System operation explanation

[1259] 1. Voice Recognition Method

[1260] The user opens the application and says, "I want to buy three apples."

[1261] The device captures the user's voice with a microphone and transmits the voice data to the server.

[1262] The server uses a voice recognition cloud service to convert the voice data into text data.

[1263] For example, speech saying "I want to buy three apples" is converted into text data saying "I want to buy three apples."

[1264] 2. Analysis method

[1265] The server uses a natural language processing library to analyze the user's request from the text data.

[1266] In this example, the request "3 apples" is extracted.

[1267] The server stores the analysis results in an internal database.

[1268] 3. Emotion recognition means

[1269] The server calls an emotion analysis cloud service to recognize the user's emotional state from the tone, pitch, volume, etc. of their voice.

[1270] For example, if the user speaks in a calm tone, the server will recognize that the user is relaxed.

[1271] The server stores the emotion recognition results in an internal database.

[1272] 4. Mail Order Methods

[1273] The server analyzes the user's request based on the recognized voice data, and adds the product selected based on the analysis result of the user's request to a shopping cart on the online shopping site, and performs the purchase procedure of the product according to the user's request using a prompt sentence and the generative AI model that instructs the user to complete the purchase procedure. Specifically, the server calls an online shopping API based on the analysis result and adds three apples to the cart. The server then sends a message to the terminal to prompt the user to complete the purchase procedure.

[1274] The terminal displays to the user, "You have added three apples to your cart. Would you like to complete the purchase?" If the user answers "Yes," the server executes the purchase procedure on the online shopping site. At this time, the purchase procedure is carried out by sending a request for the purchase procedure to the API of the online shopping site.

[1275] 5. Recommendation method

[1276] The server uses a prompt sentence instructing the user to select a product from the mail-order site based on the emotional state recognized by the emotion recognition means and a generative AI model to select a product from the mail-order site and presents the product to the user as an additional recommended product. At this time, if the recognized emotional state is negative, the server uses a prompt sentence instructing the user to select a product that suppresses the emotional state and a generative AI model to select a product from the mail-order site and presents the product to the user as an additional recommended product. For example, if the emotion recognition result is negative, the server selects a relaxing product (e.g., a music CD).

[1277] The server uses the recommendation system API to obtain information about the recommended products.

[1278] The server transmits information about the recommended products to the terminal.

[1279] The terminal displays to the user, "Would you like to listen to some relaxing music on CD?"

[1280] Examples

[1281] Example 1 of a prompt to check out a product "The user says, 'I want to buy three apples.' Analyze the user's request, select the product that best suits the user's request from the products on the online shopping site AA, and add it to the cart. Ask the user if they want to purchase the product, and then proceed with the purchase."

[1282] Example 2 of a prompt to check out a product "The user says, 'I'm looking for a relaxing music CD.' Analyze the user's request, select the product that best suits the user's request from the products on the online shopping site AA, and add it to the cart. Ask the user if they want to purchase the product, and then proceed with the purchase."

[1283] Example 2 of a prompt to recommend additional products "The user's emotion has been recognized as 'anger.' Please select a product from the AA online shopping site that can suppress the user's emotion and add it to the cart. Please present it to the user as an additional recommended product, ask if they would like to purchase it, and then proceed with the purchase." In this way, it is possible to not only recognize the user's voice input in real time and analyze their needs, but also to recommend appropriate products based on the user's emotional state.

[1284] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1285] Step 1:

[1286] The user performs voice input.

[1287] Input: The user says, "I would like to buy three apples."

[1288] What happens: The device captures the user's voice with the microphone.

[1289] Output: The captured audio data.

[1290] Step 2:

[1291] The server converts the audio into text data.

[1292] Input: Audio data sent from the device.

[1293] Specific operation: The server calls a voice recognition cloud service and converts the voice data into text data.

[1294] Output: The text "I want to buy three apples".

[1295] Step 3:

[1296] The server analyzes the text data.

[1297] Input: Text data generated by a speech recognition tool.

[1298] How it works: The server uses the generative AI model to analyze the user's request from the text data.

[1299] Output: The request "3 apples".

[1300] Step 4:

[1301] The server recognizes the emotion.

[1302] Input: Audio and text data.

[1303] Specific operation: The server calls the emotion analysis cloud service and recognizes the user's emotional state from the tone, pitch, volume, etc. of their voice.

[1304] Output: Emotion recognition result that the user is relaxed.

[1305] Step 5:

[1306] The server uses the generative AI model to add products selected based on the analysis results of the user's requests to a cart on the mail-order site, and prompts the user to complete the purchase process, and using the generative AI model, adds products according to the user's requests to the cart and prompts the user to complete the purchase process.

[1307] Input: User request as analysis result.

[1308] Specific actions: The server calls an online shopping API to add three apples to the cart. The server then sends a message to the device prompting the user to complete the purchase.

[1309] Output: The message "3 apples added to cart. Would you like to checkout?"

[1310] Step 6:

[1311] The server uses a prompt sentence instructing the user to select a product from the online shopping site based on the recognized emotional state and a generative AI model to select a product from the online shopping site and presents it to the user as an additional recommended product.

[1312] Input: Emotion recognition results.

[1313] Specific operation: The server uses the generated AI model and the API of the shopping site to select relaxing products (e.g. music CDs) and obtains information about the recommended products. The server then sends the information about the recommended products to the device.

[1314] Output: The message "Would you like a relaxing music CD?"

[1315] (Application example 1)

[1316] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1317] In conventional mail-order systems, when a user orders a product by voice, product recommendations do not take into account the user's emotional state, limiting the user experience. In addition, it is difficult to improve user satisfaction because the system is unable to recommend appropriate products according to the user's emotional state, such as when the user is relaxed or stressed.

[1318] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1319] In this invention, the server includes a voice recognition means, an analysis means, a mail order means, an emotion recognition means, and a recommendation means, which makes it possible to recognize a user's voice input in real time, analyze the user's request based on the voice data, and further recognize the user's emotional state to recommend products according to the emotional state.

[1320] A "voice recognition means" is a device or software that recognizes voice input from a user in real time and converts the voice into text data.

[1321] The "analysis means" refers to a device or software that analyzes the user's request from the text data recognized by the voice recognition means.

[1322] The "mail order means" refers to a device or software that automatically adds products to a cart based on the user's desires analyzed by the analysis means and carries out the purchasing procedure.

[1323] An "emotion recognition means" is a device or software that analyzes the tone, pitch, volume, etc. of a user's voice and recognizes the user's emotional state.

[1324] The "recommendation means" is a device or software that recommends products suitable for the user based on the emotional state recognized by the emotion recognition means.

[1325] As an embodiment of the present invention, a system including a voice recognition unit, an analysis unit, a mail order sales unit, an emotion recognition unit, and a recommendation unit is constructed. A specific embodiment of this system will be described below.

[1326] First, the server uses a voice recognition means to recognize voice input from the user in real time and convert the voice into text data. For this voice recognition means, for example, the "speech_recognition" library can be used.

[1327] Next, the server uses an analysis means to analyze the user's request from the text data recognized by the voice recognition means, which may use, for example, the "textblob" library.

[1328] The server then uses the mail order means to automatically add products to the cart based on the user's needs analyzed by the analysis means and to carry out the purchase procedure. The mail order means can send API requests to the mail order means using, for example, the "requests" library.

[1329] In addition, the server uses emotion recognition means to analyze the tone, pitch, volume, etc., of the user's voice to recognize the user's emotional state. For example, the emotion recognition means may use the "textblob" library.

[1330] Finally, the server uses a recommendation means to recommend products suitable for the user based on the emotional state recognized by the emotion recognition means. The recommendation means recommends products according to the emotional state of the user, for example, recommending relaxing products when the user is relaxed.

[1331] As a specific example, when a user says "I want to buy three apples," the speech recognition means converts the speech into text data, and the analysis means analyzes the request "three apples" from the text data. Next, the mail order means adds three apples to the cart, and the emotion recognition means recognizes the user's emotional state. If it is determined that the user is relaxed, the recommendation means recommends a product that will help the user relax (e.g., a music CD).

[1332] Examples of prompts to be input to a generative AI model include:

[1333] "If a user says, 'I want to buy three apples,' convert that speech to text, analyze the request and add it to the cart. Also, recognize the user's emotional state and recommend products that will help them relax if they're relaxed."

[1334] In this way, it is possible to recognize the user's voice input in real time, analyze their needs, and make product recommendations according to their emotional state.

[1335] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1336] Step 1:

[1337] The server uses a voice recognition means to recognize voice input from the user in real time. Specifically, when the user says, "I want to buy three apples," the voice recognition means converts the voice into text data. The input is the user's voice data, and the output is the text data "I want to buy three apples."

[1338] Step 2:

[1339] The server uses the analysis means to analyze the user's request from the text data recognized by the voice recognition means. Specifically, the server analyzes the text data "I want to buy three apples" and extracts the request "three apples." The input is the text data, and the output is the analyzed request data.

[1340] Step 3:

[1341] The server uses the mail order means to automatically add products to the cart based on the user's request analyzed by the analysis means, and performs the purchase procedure. Specifically, based on the request "three apples," three apples are added to the cart. The input is the analyzed request data, and the output is the product data added to the cart.

[1342] Step 4:

[1343] The server uses emotion recognition means to analyze the tone, pitch, volume, etc. of the user's voice and recognize the user's emotional state. Specifically, the server analyzes the tone, pitch, and volume of the user's voice data and determines whether the user is relaxed. The input is the user's voice data, and the output is emotional state data.

[1344] Step 5:

[1345] The server uses the recommendation means to recommend products suitable for the user based on the emotional state recognized by the emotion recognition means. Specifically, if it is determined that the user is relaxed, a relaxing product (e.g., a music CD) is recommended. The input is the emotional state data, and the output is the recommended product data.

[1346] Example 2

[1347] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1348] Conventional voice recognition systems can analyze user requests, but they have difficulty providing services that take into account the user's emotional state. This makes it difficult to provide personalized services to increase user satisfaction.

[1349] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1350] In this invention, the server includes a voice recognition means, an analysis means, an emotion recognition means, and a mail order means, which allows the server to recognize the user's voice input in real time, analyze the user's request, recognize the user's emotional state, and provide personalized product recommendations and purchase procedures based on the user's emotional state.

[1351] A "voice recognition means" is a means for recognizing voice input from a user in real time and converting the voice into text data.

[1352] The "analysis means" is a means for analyzing a user's request based on the text data recognized by the voice recognition means.

[1353] The "emotion recognition means" is a means for recognizing an emotional state of a user from the tone, pitch, volume, etc. of the user's voice based on the user's desire analyzed by the analysis means.

[1354] The "mail order means" is a means for automatically adding products that meet the user's needs to a cart based on the emotional state recognized by the emotion recognition means, and carrying out the purchasing procedure.

[1355] The present invention is a system including a voice recognition unit, an analysis unit, an emotion recognition unit, and a mail order sales unit. A specific embodiment of the system will be described below.

[1356] Voice Recognition Means

[1357] A user opens an application using a device such as a smartphone or tablet and speaks, "I would like to buy three apples." The device uses voice recognition software such as Google Cloud Speech-to-Text API to recognize the user's voice input in real time and converts the voice into text data. For example, the voice saying "I would like to buy three apples" is converted into text data such as "I would like to buy three apples."

[1358] Analysis means

[1359] The server receives the text data sent from the terminal and analyzes the user's request using natural language processing (NLP) techniques such as Python's NLTK library and spaCy. For example, the request "3 apples" is analyzed and the product name "apples" and the quantity "3" are extracted.

[1360] emotion recognition means

[1361] The server receives the data from the analysis means and uses emotion recognition software such as IBM Watson Tone Analyzer or Microsoft Azure Emotion API to recognize the user's emotional state from the tone, pitch, volume, etc. of the user's voice. For example, if the user speaks in a calm tone, it is determined that the user is relaxed.

[1362] Mail order methods

[1363] Based on the analysis and emotion recognition results, the server automatically adds products to the cart and completes the purchase using APIs of mail-order platforms such as Amazon API and Rakuten API. For example, it adds three apples to the cart and recommends a relaxing music CD.

[1364] Examples

[1365] When a user opens the application and says "I want to buy three apples," the device converts the speech into text data using the Google Cloud Speech-to-Text API. Next, the server uses Python's NLTK library to analyze the request "three apples" and extracts the product name "apples" and the quantity "3." The server then uses IBM Watson Tone Analyzer to recognize the user's emotions and determine that the user is relaxed. Finally, the server uses the Amazon API to add three apples to the cart and recommend a relaxing music CD.

[1366] Examples of prompt statements

[1367] "If a user says, 'I want to buy three apples,' the device converts the speech to text data using the Google Cloud Speech-to-Text API, and the server analyzes the request 'three apples' using Python's NLTK library. Explain the process of the system that recognizes the user's emotions using IBM Watson Tone Analyzer, adds three apples to the cart using Amazon APIs, and recommends a relaxing music CD."

[1368] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1369] Step 1:

[1370] The user provides voice input.

[1371] A user opens an application using a device such as a smartphone or tablet and says, "I want to buy three apples." The input is the user's voice data, and the output is the voice signal sent to the device.

[1372] Step 2:

[1373] The device converts the voice into text data.

[1374] The device calls the Google Cloud Speech-to-Text API and sends voice data. The API converts the voice to text and returns the text data "I want to buy three apples." The input is a voice signal and the output is text data.

[1375] Step 3:

[1376] The server analyzes the text data.

[1377] The server receives the text data sent from the terminal and uses Python's NLTK library and spaCy to analyze the request "3 apples." As a result of the analysis, the product name "apples" and the quantity "3" are extracted. The input is text data, and the output is the analysis result.

[1378] Step 4:

[1379] The server recognizes emotions

[1380] The server receives the data from the analysis means and uses IBM Watson Tone Analyzer to analyze the tone, pitch, and volume of the user's voice. As a result of the analysis, it is determined that the user is relaxed. The inputs are the analysis results and the voice data, and the output is the emotion recognition result.

[1381] Step 5:

[1382] The server adds the item to the cart and checks out

[1383] Based on the analysis results and emotion recognition results, the server uses Amazon API to add three apples to the cart. It also recommends a relaxing music CD and sends a notification to the user to prompt the user to proceed with the purchase. The input is the analysis results and emotion recognition results, and the output is the products added to the cart and the recommended products.

[1384] (Application example 2)

[1385] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1386] Conventional mail-order systems have a function to add products to a cart based on the user's voice input, but lack a product recommendation function that takes into account the user's emotional state. This limits the user's purchasing experience, and there are issues with not being able to recommend products that are appropriate for the user's emotions. Furthermore, if product recommendations could be made based on the user's emotional state, such as when they are relaxed or stressed, a more personalized purchasing experience could be provided.

[1387] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1388] In this invention, the server includes a voice recognition means, an analysis means, a mail order means, an emotion recognition means, and a recommendation means, which makes it possible to not only recognize the user's voice input in real time and analyze the user's request, but also to recognize the user's emotional state and recommend appropriate products based on the recognition.

[1389] A "voice recognition means" is a device or software that recognizes voice input from a user in real time and converts the voice data into text data.

[1390] The "analysis means" refers to a device or software that analyzes the user's request from the text data converted by the voice recognition means.

[1391] The "mail order means" refers to a device or software that automatically adds products to a cart and completes the purchase procedure based on the user's desires analyzed by the analysis means.

[1392] An "emotion recognition means" is a device or software that recognizes a user's emotional state from the tone, pitch, volume, etc. of the user's voice.

[1393] The "recommendation means" is a device or software that recommends appropriate products to a user based on the emotional state recognized by the emotion recognition means.

[1394] As an embodiment of the present invention, an application installed on a smartphone will be described as an example. This application is a system including a voice recognition unit, an analysis unit, a mail order sales unit, an emotion recognition unit, and a recommendation unit.

[1395] System Configuration

[1396] 1. Voice Recognition Method

[1397] The speech recognition means recognizes speech input from the user in real time and converts the speech data into text data. Specifically, it uses the speech_recognition library to convert speech to text.

[1398] 2. Analysis method

[1399] The analysis means analyzes the user's request from the text data converted by the voice recognition means, and extracts the user's request by analyzing the text data using the TextBlob library.

[1400] 3. Mail Order Methods

[1401] The mail order means automatically adds the product to the cart and performs the purchase procedure based on the user's request analyzed by the analysis means. The mail order means uses the requests library to send an API request to add the product to the cart.

[1402] 4. Emotion recognition means

[1403] The emotion recognition means recognizes the user's emotional state from the user's voice tone, pitch, volume, etc. A simple emotion recognition algorithm is used to determine whether the user is relaxed.

[1404] 5. Recommendation method

[1405] The recommendation means recommends appropriate products to the user based on the emotional state recognized by the emotion recognition means. For example, if the user is relaxed, a relaxing music CD is recommended.

[1406] Processing flow

[1407] 1. A user opens an application on their smartphone and says, "I want to buy three apples."

[1408] 2. The speech recognition means recognizes the user's voice in real time and converts the voice into text data.

[1409] 3. The analytical means analyzes the user's request from the text data and extracts the request "three apples."

[1410] 4. The emotion recognition means recognizes the user's emotional state from the tone, pitch, volume, etc. of the user's voice and determines that the user is relaxed.

[1411] 5. Based on the analysis, the online shopping channel adds three apples to the cart and completes the purchase.

[1412] 6. The recommendation means recommends products such as relaxing music CDs based on the emotion recognition results.

[1413] Examples

[1414] When a user says "I want to buy three apples," the application recognizes the speech and converts it into text. It then parses the request "three apples" from the text, and if emotion recognition determines that the user is relaxed, it adds apples to the cart and recommends a relaxing music CD.

[1415] Examples of prompt statements

[1416] User: "I want to buy 3 apples."

[1417] Application: "I've added 3 apples to your cart. Would you like a relaxing music CD as well?"

[1418] In this way, we can create a smart shopping assistant that automatically adds products to the cart based on the user's voice input and makes product recommendations according to their emotional state.

[1419] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1420] Step 1:

[1421] A user opens an application on their smartphone and says, "I want to buy three apples."

[1422] Input: User's voice input

[1423] Output: Audio data

[1424] Specific behavior: The user launches the application and speaks their request into the microphone.

[1425] Step 2:

[1426] The voice recognition means of the terminal recognizes the user's voice in real time and converts the voice into text data.

[1427] Input: Audio data

[1428] Output: Text data

[1429] Specific behavior: Uses the speech_recognition library to convert audio data into text data.

[1430] Step 3:

[1431] The terminal's analysis means analyzes the user's request from the text data and extracts the request "three apples."

[1432] Input: Text data

[1433] Output: User's request (3 apples)

[1434] Specific operation: Uses the TextBlob library to parse text data and extract user requests.

[1435] Step 4:

[1436] An emotion recognition means of the terminal recognizes the emotional state of the user from the tone, pitch, volume, etc. of the user's voice and determines that the user is relaxed.

[1437] Input: Audio data

[1438] Output: Emotional state (Relaxed)

[1439] What it does: Analyzes the tone, pitch, and volume of audio data and uses a simple emotion recognition algorithm to determine emotional state.

[1440] Step 5:

[1441] Based on the analysis, the server's mail order solution adds three apples to the cart and completes the purchase.

[1442] Input: User request (3 apples)

[1443] Output: Items added to cart (3 apples)

[1444] What it does: Uses the requests library to send an API request to add a product to the cart.

[1445] Step 6:

[1446] The server's recommendation means recommends products such as relaxing music CDs based on the emotion recognition results.

[1447] Input: Emotional state (relaxed)

[1448] Output: Recommended Products (Relaxing Music CD)

[1449] Specific operation: Based on the emotion recognition results, select relaxing products and recommend them to the user.

[1450] In this way, we can create a smart shopping assistant that automatically adds products to the cart based on the user's voice input and makes product recommendations according to their emotional state.

[1451] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1452] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with a prompt including an instruction, and also with voice data indicating a voice, a text message, and the like.

[1453] Inference data such as text data indicating a text and image data indicating an image are input. The data generation model 58 performs inference based on the input inference data in accordance with instructions indicated by a prompt, and outputs the inference result in a data format such as voice data and text data. Inference here refers to, for example, analysis, classification, prediction, and / or summarization.

[1454] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1455] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1456] [Fourth embodiment]

[1457] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1458] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1459] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1460] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1461] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1462] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1463] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1464] The control target 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1465] Fig. 8 shows an example of main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1466] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1467] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1468] In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1469] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1470] "Example 1"

[1471] The present invention is embodied as a system including a voice recognition means, an analysis means, and a mail order means. The voice recognition means recognizes voice input from a user in real time. For example, when a user says, "I would like to buy three apples," the voice recognition means converts the voice into text data. Next, the analysis means analyzes the user's request from the text data. In this example, the request "three apples" is analyzed. Finally, the mail order means automatically adds the product to the cart based on the analysis result and completes the purchase. In this example, three apples are added to the cart, and the user is prompted to complete the purchase.

[1472] "Example 2"

[1473] The present invention is embodied as a system including a voice recognition means, an analysis means, and a mail order means. The voice recognition means recognizes voice input from a user in real time. For example, when a user says, "I would like to buy three apples," the voice recognition means converts the voice into text data. Next, the analysis means analyzes the user's request from the text data. In this example, the request "three apples" is analyzed. Finally, the mail order means automatically adds the product to the cart based on the analysis result and completes the purchase. In this example, three apples are added to the cart, and the user is prompted to complete the purchase.

[1474] The process flow of each embodiment will be described below.

[1475] "Example 1"

[1476] Step 1: The user opens the application and says, "I want to buy three apples."

[1477] Step 2: The speech recognition means recognizes the speech input from the user in real time and converts the speech into text data. In this example, the speech "I want to buy three apples" is converted into text data.

[1478] Step 3: The analysis means analyzes the user's request from the text data. In this example, the request "3 apples" is analyzed.

[1479] Step 4: The online retailer automatically adds the products to the cart based on the analysis and checks out. In this example, it adds 3 apples to the cart and prompts the user to check out.

[1480] Example 1

[1481] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1482] Conventional voice recognition systems have had difficulty in accurately recognizing and analyzing users' voice inputs and conducting e-commerce based on the results. In addition, the accuracy and speed of the process of converting voice data into text data and analyzing the text data was insufficient, making it impossible to quickly and accurately reflect users' requests. This resulted in a decrease in user convenience and a decrease in the efficiency of e-commerce.

[1483] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1484] In this invention, the server includes a voice recognition means, an analysis means, an electronic commerce means, a conversion means for converting voice data into text data, a natural language processing means for analyzing the text data, a cart addition means for adding products to a cart based on the analysis result, and a procedure promotion means for promoting the purchase procedure. This makes it possible to recognize and analyze a user's voice input in real time, and to quickly and accurately add products to the cart based on the result, and promote the purchase procedure.

[1485] A "voice recognition means" is a device or software that recognizes voice input from a user in real time and acquires it as voice data.

[1486] The "analysis means" is a device or software that analyzes the user's request based on the voice data acquired by the voice recognition means.

[1487] The "electronic commerce means" refers to a device or software that provides electronic commerce services based on the user's requests analyzed by the analysis means.

[1488] A "conversion means" is a device or software that converts voice data into text data.

[1489] A "natural language processing means" is a device or software that uses natural language processing technology to analyze text data and extract user needs.

[1490] The "cart addition means" refers to a device or software that automatically adds products to a cart based on the analysis results.

[1491] The "procedure promotion means" refers to a device or software that displays a message to the user to encourage the user to complete the purchase procedure, and advances the purchase procedure.

[1492] The present invention is a system including a voice recognition unit, an analysis unit, an electronic commerce unit, a conversion unit, a natural language processing unit, an add-to-cart unit, and a procedure promotion unit. A specific embodiment of this system will be described below.

[1493] Hardware and Software Used

[1494] Speech recognition method: Use a speech recognition API (e.g. Google Cloud Speech-to-Text API) to recognize voice input in real time.

[1495] Analysis methods: Use Python scripts and natural language processing libraries (e.g. spaCy) to analyze the audio data.

[1496] E-commerce tools: Use the API of an e-commerce site (e.g. Amazon API) to provide e-commerce services.

[1497] Conversion method: A speech recognition API is used to convert voice data into text data.

[1498] Natural language processing tools: Use natural language processing libraries to analyze text data.

[1499] Method for adding to cart: Use the API of the e-commerce site to add products to the cart.

[1500] Checkout Facilitators: Use the user interface to facilitate checkout.

[1501] System Operation

[1502] 1. Acquiring voice input

[1503] The user says to the terminal, "I want to buy three apples."

[1504] The device uses a built-in microphone to capture and temporarily store audio data.

[1505] 2. Converting voice data to text

[1506] The device sends the saved voice data to the voice recognition API.

[1507] The server receives the voice data and converts it into text data in real time.

[1508] The server returns the converted text data "I want to buy three apples" to the terminal.

[1509] 3. Analysis of text data

[1510] The terminal passes the received text data to the Python script.

[1511] The server runs a Python script and uses natural language processing libraries to analyze the text data.

[1512] The server extracts the request, such as "three apples," and returns the analysis results to the terminal.

[1513] 4. Add to Cart

[1514] The terminal sends the analysis results to the EC site's API.

[1515] The server adds "3 apples" to the cart via the API.

[1516] The server confirms that the item has been added to the cart and notifies the terminal.

[1517] 5. Facilitating checkout

[1518] The terminal displays a message prompting the user to complete the purchase procedure.

[1519] The user confirms the displayed message and proceeds with the purchase procedure.

[1520] The terminal receives the user's input and completes the final purchase transaction.

[1521] Examples

[1522] User dictation: "I want to buy 3 apples."

[1523] Example prompt for a generative AI model: “If the user says they want to buy three apples, explain how you would proceed.”

[1524] In this way, the system specifically executes a series of processes from the user's voice input to product purchase.

[1525] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1526] Step 1: Getting voice input

[1527] The user says to the terminal, "I want to buy three apples." The terminal uses the built-in microphone to capture the voice data and temporarily stores it. The input is the user's voice, and the output is the stored voice data. Specifically, the terminal displays "Recording..." and captures the voice data.

[1528] Step 2: Convert audio to text

[1529] The device sends the saved voice data to the voice recognition API. The server receives the voice data and converts it into text data in real time. The input is voice data, and the output is the text data "I would like to buy three apples." In concrete terms, the device displays "Recognizing voice..." and the server converts the voice data into text data.

[1530] Step 3: Analyze the text data

[1531] The terminal passes the received text data to the Python script. The server executes the Python script and analyzes the text data using a natural language processing library. The input is the text data "I want to buy three apples," and the output is the analysis result "three apples." In concrete terms, the terminal displays "Analyzing text..." and the server extracts the request "three apples."

[1532] Step 4: Adding products to cart

[1533] The terminal sends the analysis result to the API of the EC site. The server adds "3 apples" to the cart via the API. The input is the analysis result "3 apples" and the output is information about the product added to the cart. In concrete terms, the terminal displays "Adding product to cart..." and the server adds the product to the cart.

[1534] Step 5: Facilitate checkout

[1535] The terminal displays a message encouraging the user to complete the purchase. The user confirms the displayed message and proceeds with the purchase. The input is information about the products added to the cart, and the output is a notification that the purchase has been completed. Specifically, the terminal displays the message "You have added three apples to your cart. Would you like to proceed with the purchase?" and the user selects "Yes" to complete the purchase.

[1536] (Application example 1)

[1537] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1538] Conventional online shopping systems have the problem that it takes a lot of time and effort for users to search for products, add them to their cart, and complete the purchase procedure. In addition, these operations are often difficult, especially for elderly people and people with visual impairments. Furthermore, even in systems that use voice recognition technology, it can be difficult to accurately recognize voice input and properly analyze the user's requests. This can result in a poor user experience and discourage users from using online shopping sites.

[1539] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1540] In this invention, the server includes a voice recognition means, an analysis means, and a mail order means. This allows a user to search for products through voice input, add them to a cart, and complete a purchase procedure. Specifically, the voice recognition means recognizes voice input from the user in real time, and the analysis means analyzes the user's request based on the recognized voice data. Furthermore, the mail order means automatically adds the product to the cart based on the analysis result, and completes the purchase procedure. This allows the user to easily purchase products using only voice, and provides a system that is particularly easy to use for the elderly and visually impaired. In addition, by using a generative AI model to analyze the user's request and generate a prompt sentence, more accurate analysis is possible, improving the user experience.

[1541] A "voice recognition means" is a device or software that recognizes voice input from a user in real time and converts the voice data into text data.

[1542] The "analysis means" is a device or software that analyzes the text data generated by the voice recognition means and extracts the user's request.

[1543] The "mail order means" refers to a device or software that automatically adds products to a cart and completes the purchase procedure based on the user's desires analyzed by the analysis means.

[1544] A "smartphone" is a portable information and communication terminal that implements voice recognition means, analysis means, and mail order means.

[1545] A "generative AI model" is an artificial intelligence model that analyzes user requests and generates prompt sentences.

[1546] A "prompt sentence" is text that specifically expresses the user's request, generated by a generative AI model.

[1547] "API" stands for Application Program Interface, an interface for exchanging data and functions between different software programs.

[1548] A "cart" is a virtual shopping basket on an online shopping site that temporarily stores products that a user wishes to purchase.

[1549] A "checkout" is a series of operations and processes that a user performs to actually purchase the items that they have added to their cart.

[1550] As an embodiment of the present invention, a system including a voice recognition unit, an analysis unit, and a mail order unit is implemented as an application installed on a smartphone. A specific embodiment of this system will be described below.

[1551] Voice Recognition Means

[1552] The speech recognition means recognizes voice input from the user in real time and converts the voice data into text data. Specifically, it collects voice using a microphone built into the smartphone and converts the voice data into text data using the SpeechRecognition library. When a user says, "I want to buy three apples," this speech recognition means converts the voice into text data.

[1553] Analysis means

[1554] The analysis means analyzes the text data generated by the voice recognition means and extracts the user's request. Specifically, the analysis means analyzes the text data using a generative AI model and identifies the user's request. The analysis means extracts the request "3 apples."

[1555] Mail order methods

[1556] The mail order means automatically adds products to the cart and completes the purchase procedure based on the user's request analyzed by the analysis means. Specifically, it sends a request to the API of the mail order site to add the specified products to the cart. The mail order means adds three apples to the cart and prompts the user to complete the purchase procedure.

[1557] Hardware and Software Used

[1558] Hardware: Smartphone, microphone

[1559] Software: Python, SpeechRecognition library, requests library, generative AI model

[1560] Examples

[1561] When a user says to their smartphone, "I want to buy three apples," the speech recognition means converts the speech into text data. Next, the analysis means uses a generative AI model to analyze the text data and extracts the request, "three apples." Finally, the mail order means sends a request to the API of the mail order site to add three apples to the cart and prompt the user to complete the purchase.

[1562] Examples of prompt statements

[1563] "When a user says to their smartphone, 'I want to buy three apples,' the voice recognition system converts the speech into text, and the analysis system extracts the request, 'three apples.' Then it sends a request to the API of the online shopping site to add three apples to the cart."

[1564] In this way, users can easily purchase products using only their voice, providing a system that is particularly easy to use for the elderly and visually impaired.

[1565] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1566] Step 1:

[1567] The user inputs voice into the smartphone.

[1568] Input: User speech (e.g. "I want to buy 3 apples")

[1569] Output: Audio data

[1570] Specific operation: Voice data is collected when the user speaks into the smartphone's microphone.

[1571] Step 2:

[1572] The terminal converts the voice data into text data using a voice recognition means.

[1573] Input: Audio data

[1574] Output: Text data (e.g. "I want to buy 3 apples")

[1575] Specific operation: The SpeechRecognition library installed on the smartphone analyzes the voice data and converts it into text data.

[1576] Step 3:

[1577] The terminal uses an analysis means to analyze the text data and extract the user's request.

[1578] Input: Text data (e.g. "I want to buy 3 apples")

[1579] Output: Demand data (e.g. "3 apples")

[1580] How it works: The generative AI model analyzes the text data and identifies the user's request. For example, the request "three apples" is extracted.

[1581] Step 4:

[1582] The terminal uses the mail order means to send a request to the API of the mail order site based on the analyzed request data.

[1583] Input: Request data (e.g. "3 apples")

[1584] Output: The result of the API request (e.g., a response indicating that a product was added to the cart)

[1585] Specific operation: The smartphone uses the requests library to send a request to the online shopping site's API to "add three apples to the cart."

[1586] Step 5:

[1587] The server receives the API request, adds the specified product to the cart, and completes the purchase.

[1588] Input: API request (e.g. "Add 3 apples to cart")

[1589] Output: A confirmation that the purchase has been completed (e.g. "3 apples added to cart")

[1590] Specific operation: The mail-order site's server processes the API request, adds the specified product to the cart, and completes the purchase. Once the process is complete, a notification of the completion of the purchase is sent to the smartphone.

[1591] Step 6:

[1592] The terminal displays a notification to the user that the purchase process is complete.

[1593] Input: Checkout completion notification (e.g. "3 apples added to cart")

[1594] Output: Display a notification to the user

[1595] Specific behavior: The smartphone receives a notification that the purchase procedure has been completed and displays the notification to the user, allowing the user to confirm that the purchase procedure has been completed.

[1596] Example 2

[1597] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1598] In conventional e-commerce systems, when users wanted to purchase a product, they had to manually search for the product, add it to their cart, and then complete the purchase procedure, which was time-consuming and inconvenient. Even in systems that used voice input, the accuracy and real-time nature of voice recognition was insufficient, and users' requests were sometimes not accurately reflected. This resulted in a poor user experience and made users avoid using the system.

[1599] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1600] In this invention, the server includes a voice recognition means, an analysis means, and an electronic commerce means, so that when a user purchases a product through voice input, the voice can be converted into text in real time, the text can be analyzed to understand the user's request, and the product can be automatically added to a cart based on the analysis results, and the user can be prompted to complete the purchase procedure.

[1601] A "voice recognition means" is a device or software that has the function of recognizing voice input from a user in real time and converting the voice data into text data.

[1602] The "analysis means" refers to a device or software having a function for analyzing the text data converted by the voice recognition means and understanding the user's request.

[1603] An "electronic commerce means" is a device or software having a function of automatically adding products to a cart and carrying out a purchasing procedure based on the user's desires analyzed by an analysis means.

[1604] "User" means an individual or organization that uses the system through voice input to purchase goods.

[1605] A "server" is a computer system that manages the entire system including the speech recognition means, analysis means, and electronic commerce means, and executes various processes.

[1606] "Voice input" refers to voice data generated by a user through an input device such as a microphone.

[1607] "Text data" refers to character string data converted from voice data by a voice recognition means.

[1608] A "cart" is a virtual shopping basket in an electronic commerce means for temporarily storing products that a user wishes to purchase.

[1609] A "checkout" is a series of operations and processes that a user performs to actually purchase an item that has been added to their cart.

[1610] The present invention relates to a system including a speech recognition unit, an analysis unit, and an electronic commerce unit. A specific embodiment of the system will be described below.

[1611] Hardware and Software Used

[1612] Hardware: Microphone, server, user device (smartphone, PC, etc.)

[1613] Software: voice recognition software (e.g. voice recognition APIs), analytics software (e.g. natural language processing libraries), e-commerce platforms (e.g. APIs for e-commerce sites)

[1614] System Operation

[1615] 1. Receiving voice input

[1616] The user provides voice input through a microphone.

[1617] For example, a user might say, "I want to buy three apples."

[1618] 2. Converting voice data to text

[1619] The server uses voice recognition software (voice recognition API) to convert the user's voice into text data in real time.

[1620] Specifically, the server sends voice data to the API, which converts the voice into text and returns it.

[1621] 3. Analysis of text data

[1622] The server passes the converted text data to analysis software (natural language processing library).

[1623] Analysis software analyzes user requests from text data.

[1624] Specifically, the product name "apple" and the quantity "3" are extracted from the text "three apples."

[1625] 4. Add to Cart

[1626] Based on the analysis results, the server calls the API of the e-commerce platform.

[1627] Specifically, the server sends the product name and quantity to the API, and the API adds the specified product to the cart.

[1628] 5. Prompt for purchase

[1629] The server sends a notification to the user's terminal prompting them to complete the purchase procedure.

[1630] Specifically, a message is displayed on the user's device stating, "Three apples have been added to your cart. Please complete the purchase."

[1631] Examples

[1632] User dictation: "I want to buy 3 apples."

[1633] Speech recognition result: "I want to buy three apples."

[1634] Analysis result: Product name: "Apple", quantity: "3"

[1635] E-commerce transaction action: Uses the API of the e-commerce site to add three apples to the cart and prompts the user to complete the purchase.

[1636] Example prompts for generative AI models

[1637] If a user says, "I want to buy three apples," the server uses speech recognition software to convert the speech to text, analyzes the text to understand what the user wants, and then calls the e-commerce platform's API to add three apples to the cart and prompt the user to check out.

[1638] In this way, the system automates the entire process from user voice input to product purchase.

[1639] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1640] Step 1:

[1641] Receiving voice input

[1642] The user provides voice input through a microphone.

[1643] For example, a user might say, "I want to buy three apples."

[1644] Input: User's voice data

[1645] Output: Audio data captured from the microphone.

[1646] Step 2:

[1647] Converting audio data to text

[1648] The server uses voice recognition software (voice recognition API) to convert the user's voice into text data in real time.

[1649] Specifically, the server sends voice data to the API, which converts the voice into text and returns it.

[1650] Input: Audio data

[1651] Data processing: Convert voice data into text data using speech recognition API

[1652] Output: Text data (e.g. "I want to buy 3 apples")

[1653] Step 3:

[1654] Analyzing Text Data

[1655] The server passes the converted text data to analysis software (natural language processing library).

[1656] Analysis software analyzes user requests from text data.

[1657] Specifically, the product name "apple" and the quantity "3" are extracted from the text "three apples."

[1658] Input: Text data

[1659] Data Calculation: Analyze text data using natural language processing libraries to extract product names and quantities

[1660] Output: Analysis result (product name "Apple", quantity "3")

[1661] Step 4:

[1662] Add to Cart

[1663] Based on the analysis results, the server calls the API of the e-commerce platform.

[1664] Specifically, the server sends the product name and quantity to the API, and the API adds the specified product to the cart.

[1665] Input: Analysis result (product name "Apple", quantity "3")

[1666] Data processing: Add products to cart using API of e-commerce platform

[1667] Output: Product information added to cart

[1668] Step 5:

[1669] Prompt for purchase

[1670] The server sends a notification to the user's terminal prompting them to complete the purchase procedure.

[1671] Specifically, a message is displayed on the user's device stating, "Three apples have been added to your cart. Please complete the purchase."

[1672] Input: Product information added to cart

[1673] Data processing: generating notification messages

[1674] Output: A notification message displayed on the user's terminal.

[1675] In this way, the system automates the entire process from user voice input to product purchase.

[1676] (Application example 2)

[1677] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1678] Conventional online shopping systems require users to search for products, add them to their cart, and complete the purchase process in multiple steps, which is time-consuming. Even in systems that use voice input, there are issues with the accuracy of voice recognition and real-time performance, which means the user experience does not improve.

[1679] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1680] In this invention, the server includes a voice recognition means, an analysis means, a mail order means, and a means including an application to be installed on a smartphone, which allows a user to search for products through voice input, add them to a cart, and complete the purchase procedure in real time.

[1681] A "voice recognition means" is a device or software that recognizes voice input from a user in real time and converts the voice data into text data.

[1682] The "analysis means" refers to a device or software that analyzes the text data obtained by the voice recognition means and extracts the user's request.

[1683] The "mail order means" refers to a device or software that selects products based on the user's needs analyzed by the analysis means, adds them to a cart, and completes the purchase procedure.

[1684] "Means including an application installed on a smartphone" refers to an application that runs on a smartphone and integrates speech recognition means, analysis means, and mail order means to enable a user to purchase products through voice input.

[1685] The system for carrying out the present invention includes a voice recognition unit, an analysis unit, a mail order unit, and a unit including an application installed on a smartphone. Specific embodiments of each unit will be described below.

[1686] First, the speech recognition means recognizes the voice input from the user in real time and converts the voice data into text data. This is done using the smartphone's microphone and a speech recognition library (for example, the speech_recognition library). When the user says, "I want to buy three apples," the speech recognition means converts the voice into text data.

[1687] Next, the analysis means analyzes the text data obtained by the voice recognition means and extracts the user's request. Specifically, an analysis logic is used to extract the product name and quantity from the text data. For example, from the text data "I want to buy three apples," "apples" and "three" are extracted.

[1688] The mail order means then selects products based on the user's needs analyzed by the analysis means, adds them to the cart, and completes the purchase procedure. This means uses the API of the mail order site to add the specified products and quantities to the cart and proceeds with the purchase procedure. For example, it adds three apples to the cart based on the analysis results, and prompts the user to complete the purchase procedure.

[1689] The means including the application installed on the smartphone integrates the voice recognition means, analysis means, and mail order means, and enables the user to purchase products through voice input. The user can search for products, add them to a cart, and complete the purchase process simply by speaking to the smartphone.

[1690] As a specific example, when a user says, "I would like to buy two bottles of milk," the voice recognition means converts the voice into text data, the analysis means extracts "milk" and "two bottles," and the mail order means adds the two bottles of milk to the cart and prompts the user to proceed with the purchase.

[1691] Example of a prompt to be input to a generative AI model:

[1692] If a user says, "I would like to buy two bottles of milk," generate a Python program that performs speech recognition, converts the speech to text, analyzes it, and adds two bottles of milk to the shopping cart on an online shopping site.

[1693] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1694] Step 1:

[1695] A user speaks to a smartphone, for example, "I want to buy two bottles of milk." This voice input is captured by the smartphone's microphone.

[1696] Step 2:

[1697] The smartphone's voice recognition means converts the captured voice data into text data. Specifically, it uses the speech_recognition library to analyze the voice data and generate text data such as "I would like to buy two bottles of milk." The input is voice data, and the output is text data.

[1698] Step 3:

[1699] The smartphone's analysis means analyzes the text data generated by the voice recognition means. Specifically, it uses analysis logic to extract product names and quantities from the text data. For example, from the text data "I would like to buy two bottles of milk," it extracts "milk" and "two bottles." The input is text data, and the output is a pair of product name and quantity.

[1700] Step 4:

[1701] The mail-order means of the smartphone calls the API of the mail-order site based on the product name and quantity extracted by the analysis means. Specifically, the requests library is used to send the product name and quantity to the API endpoint of the mail-order site and add it to the cart. For example, an API request is sent to add "Milk" and "2 bottles" to the cart. The input is the product name and quantity, and the output is the response to the API request.

[1702] Step 5:

[1703] The smartphone shopping tool receives the response to the API request and displays a message to the user urging them to complete the purchase. For example, it displays a message saying, "You have added two bottles of milk to your cart. Would you like to complete the purchase?" The input is the response to the API request, and the output is the message to the user.

[1704] Step 6:

[1705] The user performs an operation to confirm the purchase procedure, for example by tapping a "Purchase" button on the smartphone screen. This operation is captured by the smartphone's mail order processing means.

[1706] Step 7:

[1707] The smartphone mail order tool sends an API request to complete the purchase procedure based on the user's confirmation operation. Specifically, it uses the requests library to send a request to an API endpoint for completing the purchase procedure. The input is the user's confirmation operation, and the output is the API request response indicating the completion of the purchase procedure.

[1708] Step 8:

[1709] The smartphone mail order tool receives the response to the API request indicating the completion of the purchase procedure and displays a purchase completion message to the user. For example, it displays a message saying "The purchase procedure has been completed. Thank you." The input is the response to the API request indicating the completion of the purchase procedure, and the output is the purchase completion message to the user.

[1710] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1711] "Example 1"

[1712] As one embodiment of the present invention, a system incorporating an emotion recognition means is provided. This system includes an emotion recognition means for recognizing emotions from the user's voice, in addition to a voice recognition means, an analysis means, and a mail order means. Specifically, when a user opens an application and speaks "I want to buy three apples," the voice recognition means recognizes the voice input from the user in real time and converts the voice into text data. In this example, the voice "I want to buy three apples" is converted into text data. The analysis means analyzes the user's request from this text data. In this example, the request "three apples" is analyzed. The emotion recognition means recognizes the user's emotional state from the tone, pitch, volume, etc. of the user's voice. In this example, if the user speaks in a calm tone about wanting to buy three apples, the user feels relaxed. The mail order means automatically adds the product to the cart based on the analysis result and the emotion recognition result, and performs the purchase procedure. In this example, three apples are added to the cart, and the user is prompted to perform the purchase procedure. In addition, if the user feels relaxed, a product such as a relaxing music CD is recommended.

[1713] "Example 2"

[1714] As one embodiment of the present invention, a system incorporating an emotion recognition means is provided. This system includes an emotion recognition means for recognizing emotions from the user's voice, in addition to a voice recognition means, an analysis means, and a mail order means. Specifically, when a user opens an application and speaks "I want to buy three apples," the voice recognition means recognizes the voice input from the user in real time and converts the voice into text data. In this example, the voice "I want to buy three apples" is converted into text data. The analysis means analyzes the user's request from this text data. In this example, the request "three apples" is analyzed. The emotion recognition means recognizes the user's emotional state from the tone, pitch, volume, etc. of the user's voice. In this example, if the user speaks in a calm tone about wanting to buy three apples, the user feels relaxed. The mail order means automatically adds the product to the cart based on the analysis result and the emotion recognition result, and performs the purchase procedure. In this example, three apples are added to the cart, and the user is prompted to perform the purchase procedure. In addition, if the user feels relaxed, a product such as a relaxing music CD is recommended.

[1715] The process flow of each embodiment will be described below.

[1716] "Example 1"

[1717] Step 1: The user opens the application and says, "I want to buy 3 apples."

[1718] Step 2: The speech recognition means recognizes the speech input from the user in real time and converts the speech into text data. In this example, the speech "I want to buy three apples" is converted into text data.

[1719] Step 3: The analysis means analyzes the user's request from the text data. In this example, the request "3 apples" is analyzed.

[1720] Step 4: The emotion recognition means recognizes the user's emotional state from the tone, pitch, volume, etc. of the user's voice. In this example, if the user speaks in a calm tone about his / her desire to buy three apples, the user is perceived as relaxed.

[1721] Step 5: The online retailer automatically adds the products to the cart based on the analysis and emotion recognition results, and checks out. In this example, it adds three apples to the cart and prompts the user to check out.

[1722] Step 6: If the user feels relaxed, recommend a product, such as a relaxing music CD.

[1723] Example 1

[1724] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1725] Conventional voice recognition systems were able to analyze users' requests and provide online shopping services, but they did not recommend products that took into account the user's emotional state. As a result, they were unable to recommend appropriate products according to the user's emotions, which resulted in a lack of improvement in the user experience.

[1726] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1727] In this invention, the server includes a voice recognition means, a prompt sentence for analyzing a user's request based on the voice data recognized by the voice recognition means, and adding a product selected based on the analysis result of the user's request to a shopping cart on a shopping site, and a means for carrying out a purchase procedure for a product according to the user's request using a generative AI model, an emotion recognition means for recognizing emotion from the user's voice, a prompt sentence for instructing the user to select a product on the shopping site based on the emotional state recognized by the emotion recognition means, and a recommendation means for selecting a product on the shopping site using the generative AI model and presenting it to the user as an additional recommended product. In the server, the means for carrying out the purchase procedure carries out the purchase procedure by sending a request for the purchase procedure to the API of the online shopping site. In the server, the recommendation means includes, when the emotional state recognized by the emotion recognition means is negative, the prompt sentence for instructing the user to select a product that suppresses the emotional state, and the generative AI model, and selecting a product on the shopping site and presenting it to the user as an additional recommended product. This makes it possible to not only recognize the user's voice input in real time and analyze the request, but also to recommend appropriate products based on the user's emotional state.

[1728] A "voice recognition means" is a means for recognizing voice input from a user in real time and converting the voice data into text data.

[1729] The "analysis means" is a means for analyzing a user's request based on the text data recognized by the voice recognition means.

[1730] The "mail order means" is a means for automatically adding products to a shopping cart and carrying out a purchasing procedure based on the user's desires analyzed by the analysis means.

[1731] An "emotion recognition means" is a means for analyzing the tone, pitch, volume, etc. of a user's voice and recognizing the user's emotional state.

[1732] The "recommendation means" is a means for presenting appropriate additional recommended products to the user based on the emotional state recognized by the emotion recognition means.

[1733] The present invention is a system including a voice recognition unit, an analysis unit, a mail order sales unit, an emotion recognition unit, and a recommendation unit. A specific embodiment of the system will be described below.

[1734] Hardware and Software Used

[1735] Speech recognition method: General speech recognition API (e.g., speech recognition cloud service)

[1736] Generative AI model: GPT-4 Analysis method: Natural language processing library (e.g. Python natural language processing library)

[1737] Mail order method: EC site API (e.g. online shopping API)

[1738] Emotion recognition means: Emotion analysis service (e.g. emotion analysis cloud service)

[1739] Recommendation method: Recommendation engine (e.g., recommendation system API)

[1740] System operation explanation

[1741] 1. Voice Recognition Method

[1742] The user opens the application and says, "I want to buy three apples."

[1743] The device captures the user's voice with a microphone and transmits the voice data to the server.

[1744] The server uses a voice recognition cloud service to convert the voice data into text data.

[1745] For example, speech saying "I want to buy three apples" is converted into text data saying "I want to buy three apples."

[1746] 2. Analysis method

[1747] The server uses a natural language processing library to analyze the user's request from the text data.

[1748] In this example, the request "3 apples" is extracted.

[1749] The server stores the analysis results in an internal database.

[1750] 3. Emotion recognition means

[1751] The server calls an emotion analysis cloud service to recognize the user's emotional state from the tone, pitch, volume, etc. of their voice.

[1752] For example, if the user speaks in a calm tone, the server will recognize that the user is relaxed.

[1753] The server stores the emotion recognition results in an internal database.

[1754] 4. Mail Order Methods

[1755] The server analyzes the user's request based on the recognized voice data, and adds the product selected based on the analysis result of the user's request to a shopping cart on the online shopping site, and performs the purchase procedure of the product according to the user's request using a prompt sentence and the generative AI model that instructs the user to complete the purchase procedure. Specifically, the server calls an online shopping API based on the analysis result and adds three apples to the cart. The server then sends a message to the terminal to prompt the user to complete the purchase procedure.

[1756] The terminal displays to the user, "You have added three apples to your cart. Would you like to complete the purchase?" If the user answers "Yes," the server executes the purchase procedure on the online shopping site. At this time, the purchase procedure is carried out by sending a request for the purchase procedure to the API of the online shopping site.

[1757] 5. Recommendation method

[1758] The server uses a prompt sentence instructing the user to select a product from the mail-order site based on the emotional state recognized by the emotion recognition means and a generative AI model to select a product from the mail-order site and presents the product to the user as an additional recommended product. At this time, if the recognized emotional state is negative, the server uses a prompt sentence instructing the user to select a product that suppresses the emotional state and a generative AI model to select a product from the mail-order site and presents the product to the user as an additional recommended product. For example, if the emotion recognition result is negative, the server selects a relaxing product (e.g., a music CD).

[1759] The server uses the recommendation system API to obtain information about the recommended products.

[1760] The server transmits information about the recommended products to the terminal.

[1761] The terminal displays to the user, "Would you like to listen to some relaxing music on CD?"

[1762] Examples

[1763] Example 1 of a prompt to check out a product "The user says, 'I want to buy three apples.' Analyze the user's request, select the product that best suits the user's request from the products on the online shopping site AA, and add it to the cart. Ask the user if they want to purchase the product, and then proceed with the purchase."

[1764] Example 2 of a prompt to check out a product "The user says, 'I'm looking for a relaxing music CD.' Analyze the user's request, select the product that best suits the user's request from the products on the online shopping site AA, and add it to the cart. Ask the user if they want to purchase the product, and then proceed with the purchase."

[1765] Example 2 of a prompt to recommend additional products "The user's emotion has been recognized as 'anger.' Please select a product from the AA online shopping site that can suppress the user's emotion and add it to the cart. Please present it to the user as an additional recommended product, ask if they would like to purchase it, and then proceed with the purchase." In this way, it is possible to not only recognize the user's voice input in real time and analyze their needs, but also to recommend appropriate products based on the user's emotional state.

[1766] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1767] Step 1:

[1768] The user performs voice input.

[1769] Input: The user says, "I would like to buy three apples."

[1770] What happens: The device captures the user's voice with the microphone.

[1771] Output: The captured audio data.

[1772] Step 2:

[1773] The server converts the audio into text data.

[1774] Input: Audio data sent from the device.

[1775] Specific operation: The server calls a voice recognition cloud service and converts the voice data into text data.

[1776] Output: The text "I want to buy three apples".

[1777] Step 3:

[1778] The server analyzes the text data.

[1779] Input: Text data generated by a speech recognition tool.

[1780] How it works: The server uses the generative AI model to analyze the user's request from the text data.

[1781] Output: The request "3 apples".

[1782] Step 4:

[1783] The server recognizes the emotion.

[1784] Input: Audio and text data.

[1785] Specific operation: The server calls the emotion analysis cloud service and recognizes the user's emotional state from the tone, pitch, volume, etc. of their voice.

[1786] Output: Emotion recognition result that the user is relaxed.

[1787] Step 5:

[1788] The server uses the generative AI model to add products selected based on the analysis results of the user's requests to a cart on the mail-order site, and prompts the user to complete the purchase process, and using the generative AI model, adds products according to the user's requests to the cart and prompts the user to complete the purchase process.

[1789] Input: User request as analysis result.

[1790] Specific actions: The server calls an online shopping API to add three apples to the cart. The server then sends a message to the device prompting the user to complete the purchase.

[1791] Output: The message "3 apples added to cart. Would you like to checkout?"

[1792] Step 6:

[1793] The server uses a prompt sentence instructing the user to select a product from the online shopping site based on the recognized emotional state and a generative AI model to select a product from the online shopping site and presents it to the user as an additional recommended product.

[1794] Input: Emotion recognition results.

[1795] Specific operation: The server uses the generated AI model and the API of the shopping site to select relaxing products (e.g. music CDs) and obtains information about the recommended products. The server then sends the information about the recommended products to the device.

[1796] Output: The message "Would you like a relaxing music CD?"

[1797] (Application example 1)

[1798] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1799] In conventional mail-order systems, when a user orders a product by voice, product recommendations do not take into account the user's emotional state, limiting the user experience. In addition, it is difficult to improve user satisfaction because the system is unable to recommend appropriate products according to the user's emotional state, such as when the user is relaxed or stressed.

[1800] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1801] In this invention, the server includes a voice recognition means, an analysis means, a mail order means, an emotion recognition means, and a recommendation means, which makes it possible to recognize a user's voice input in real time, analyze the user's request based on the voice data, and further recognize the user's emotional state to recommend products according to the emotional state.

[1802] A "voice recognition means" is a device or software that recognizes voice input from a user in real time and converts the voice into text data.

[1803] The "analysis means" refers to a device or software that analyzes the user's request from the text data recognized by the voice recognition means.

[1804] The "mail order means" refers to a device or software that automatically adds products to a cart based on the user's desires analyzed by the analysis means and carries out the purchasing procedure.

[1805] An "emotion recognition means" is a device or software that analyzes the tone, pitch, volume, etc. of a user's voice and recognizes the user's emotional state.

[1806] The "recommendation means" is a device or software that recommends products suitable for the user based on the emotional state recognized by the emotion recognition means.

[1807] As an embodiment of the present invention, a system including a voice recognition unit, an analysis unit, a mail order sales unit, an emotion recognition unit, and a recommendation unit is constructed. A specific embodiment of this system will be described below.

[1808] First, the server uses a voice recognition means to recognize voice input from the user in real time and convert the voice into text data. For this voice recognition means, for example, the "speech_recognition" library can be used.

[1809] Next, the server uses an analysis means to analyze the user's request from the text data recognized by the voice recognition means, which may use, for example, the "textblob" library.

[1810] The server then uses the mail order means to automatically add products to the cart based on the user's needs analyzed by the analysis means and to carry out the purchase procedure. The mail order means can send API requests to the mail order means using, for example, the "requests" library.

[1811] In addition, the server uses emotion recognition means to analyze the tone, pitch, volume, etc., of the user's voice to recognize the user's emotional state. For example, the emotion recognition means may use the "textblob" library.

[1812] Finally, the server uses a recommendation means to recommend products suitable for the user based on the emotional state recognized by the emotion recognition means. The recommendation means recommends products according to the emotional state of the user, for example, recommending relaxing products when the user is relaxed.

[1813] As a specific example, when a user says "I want to buy three apples," the speech recognition means converts the speech into text data, and the analysi...

Claims

1. A voice recognition means; a means for analyzing a user's request based on the voice data recognized by the voice recognition means, and adding a product selected based on the analysis result of the user's request to a cart on an online shopping site and carrying out a purchase procedure using a prompt sentence and a generation AI model to instruct the user to carry out a purchase procedure; An emotion recognition means for recognizing emotions from a user's voice; a recommendation means for selecting a product from the online shopping site based on the emotional state recognized by the emotion recognition means, using a prompt sentence instructing the user to select a product from the online shopping site and the generative AI model, and presenting the product to the user as an additional recommended product; A system including:

2. 2. The system according to claim 1, wherein the means for carrying out the purchase procedure carries out the purchase procedure by sending a request for the purchase procedure to an API of the online shopping site.

3. The system described in claim 1, wherein the recommendation means selects a product from the mail-order site using the prompt sentence instructing the user to select a product that will suppress the emotional state when the emotional state recognized by the emotion recognition means is negative, and the generative AI model, and presents the product to the user as an additional recommended product.

Images