System

Abstract

A system is provided.SOLUTION: The system includes a voice input means, a text conversion means for converting voice data into text, a natural language processing means for analyzing text data and extracting a keyword, a commodity retrieval means for acquiring commodity information based on the extracted keyword, a display means for displaying the acquired commodity information, a voice selection means for receiving voice input for commodity selection again, a purchase procedure means for advancing the selected commodity to a purchase procedure, and a payment means for presenting a payment method and executing settlement.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In addition to the problem that elderly people living in rural areas have difficulty operating smartphones and are therefore reluctant to shop online, there are also other issues, such as the physical burden of shopping, feelings of loneliness due to fewer opportunities to talk to people, and an increased risk of fraud. These current situations are factors that reduce the quality of life for the elderly. For this reason, there is a need for a system that allows elderly people to easily enjoy online shopping, alleviates feelings of loneliness, and reduces the risk of fraud. [Means for solving the problem]

[0005] The present invention provides a system including a voice input means, a text conversion means for converting voice data into text, a natural language processing means for analyzing text data and extracting keywords, a product search means for acquiring product information based on the extracted keywords, a display means for displaying the acquired product information, a voice selection means for re-accepting voice input for product selection, a purchase procedure means for proceeding with the purchase procedure for the selected product, and a payment means for presenting payment methods and executing payment. Furthermore, the system includes a product recognition means for analyzing product images acquired by a camera and presenting corresponding product information, and a data sharing means for a voice assistant to converse with a user, grasp changes in the user's physical condition and any problems the user may have, and share the information with a medical institution as necessary, thereby providing an environment in which the elderly can live with greater peace of mind.

[0006] The "voice input means" is a mechanism for capturing the user's voice and generating voice data.

[0007] The "text conversion means" is a mechanism for converting voice data into text data.

[0008] The "natural language processing means" is a mechanism for analyzing text data and extracting keywords.

[0009] The "product search means" is a mechanism for acquiring product information based on the extracted keywords.

[0010] The "display means" is a mechanism for visually displaying the acquired product information.

[0011] The "voice selection means" is a mechanism for accepting voice input for product selection again.

[0012] The "purchase procedure means" is a mechanism for proceeding with the purchase procedure for the selected product.

[0013] A "payment method" is a mechanism for presenting a payment method and executing a payment.

[0014] The "product recognition means" is a mechanism for analyzing product images acquired by a camera and presenting corresponding product information.

[0015] The "data sharing means" is a mechanism that allows the voice assistant to understand changes in the user's physical condition and any problems they may have, and share the information with medical institutions as necessary. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The embodiment of the present invention follows the following configuration and processing flow.

[0038] System Configuration

[0039] The system consists of the following main components:

[0040] 1. Voice input method

[0041] It has the function of capturing the user's voice and generating the voice data.

[0042] 2. Text Conversion Methods

[0043] It has the function of receiving voice data and converting it into text data using a voice recognition engine.

[0044] 3. Natural Language Processing Methods

[0045] It has the ability to analyze text data and extract user intent and keywords.

[0046] 4. Product search methods

[0047] It has the function of retrieving relevant product information from a product database based on the extracted keywords.

[0048] 5. Display means

[0049] It has the function of displaying the acquired product information on a display device such as a television screen.

[0050] 6. Audio Selection Method

[0051] It has the function of accepting additional voice input to select a product.

[0052] 7. Purchasing Procedures

[0053] It has the function of proceeding to the purchase procedure for the selected product.

[0054] 8. Payment Methods

[0055] It has the function of presenting payment methods and executing payments.

[0056] 9. Product recognition means

[0057] It has the function of analyzing product images captured by the camera and presenting corresponding products.

[0058] 10. Data sharing methods

[0059] The voice assistant will monitor the user's condition and share information with medical institutions if necessary.

[0060] Program processing

[0061] Acquiring and analyzing voice input

[0062] The user speaks to a television or dedicated terminal, saying, "Find the Mito Komon DVD." The terminal captures this as voice data and converts it into text data using a text conversion means. The server receives the text data and uses natural language processing means to extract keywords such as "Mito Komon," "DVD," and "search."

[0063] Obtaining and displaying product information

[0064] The server executes a search query against the product database using these keywords. Product information is obtained as a search result and displayed on the television screen through the display means. Specifically, a list of "Mito Komon DVDs" is visually displayed on the user's screen.

[0065] Product selection and purchase process

[0066] The user looks at the displayed product list and again says, "I'd like to buy this Mito Komon DVD." The device again captures the voice data and analyzes it using the procedure described above. The server adds the product to the shopping cart and presents options for confirming the payment method (for example, "I'd like to pay by credit card").

[0067] The user selects "I'll pay by credit card" by voice, and the payment is executed by the payment method. If the payment is successful, a confirmation message is sent to the user.

[0068] Product search using camera

[0069] The user points the camera at the product they want to purchase. The device acquires the image data from the camera and analyzes the image using product recognition means. As a result of the analysis, the relevant product information is displayed. For example, if the user holds a "white shirt" up to the camera, a list of that product will be displayed on the TV screen.

[0070] Voice assistant support

[0071] The voice assistant will understand changes in the user's physical condition and any problems they may have through dialogue with the user, and if necessary, will share the information with medical institutions through data sharing means to support the user's health management.

[0072] With these components and processing flow, the system of the present invention enables elderly people to easily enjoy online shopping, eliminating feelings of loneliness and reducing the risk of fraud.

[0073] The processing flow will be explained below.

[0074] Acquiring and analyzing voice input

[0075] Step 1:

[0076] User: Speak to the TV or dedicated device and say, "Find the Mito Komon DVD."

[0077] Step 2:

[0078] Device: Uses the built-in microphone to capture the user's voice and generate audio data.

[0079] Step 3:

[0080] Terminal: Sends the generated voice data to the server via the network.

[0081] Step 4:

[0082] Server: Converts voice data into text data using a speech recognition engine.

[0083] Step 5:

[0084] Server: The converted text data is sent to a natural language processing module, which extracts keywords ("Mito Komon," "DVD," "search").

[0085] Obtaining and displaying product information

[0086] Step 1:

[0087] Server: Executes a search query to the product database based on the extracted keywords and retrieves the corresponding product information.

[0088] Step 2:

[0089] Server: Sends the acquired product information to the terminal via the network.

[0090] Step 3:

[0091] Terminal: Analyzes the received product information and creates a layout for display on the TV screen.

[0092] Step 4:

[0093] Device: Display a list of "Mito Komon DVDs" on the TV screen.

[0094] Product selection and purchase process

[0095] Step 1:

[0096] User: While looking at the displayed product list, say, "I'd like to buy this Mito Komon DVD."

[0097] Step 2:

[0098] Terminal: Capture the audio again and generate the audio data.

[0099] Step 3:

[0100] Terminal: Sends the generated voice data to the server.

[0101] Step 4:

[0102] Server: Analyzes the voice data and identifies the product selected by the user.

[0103] Step 5:

[0104] Server: Confirms the user's selection and adds the selected items to the shopping cart.

[0105] Step 6:

[0106] Server: Generates payment options and presents them to the user through the voice assistant.

[0107] Step 7:

[0108] User: Select "Pay with credit card" by voice.

[0109] Step 8:

[0110] Terminal: Captures the user's voice and sends the voice data to the server.

[0111] Payment Processing

[0112] Step 1:

[0113] Server: Analyzes the voice data and confirms the user's payment method selection.

[0114] Step 2:

[0115] Server: Processes the payment using credit card information.

[0116] Step 3:

[0117] Server: Generates a confirmation message of payment completion and notifies the user via the voice assistant.

[0118] Product search using camera

[0119] Step 1:

[0120] User: Point the camera at the item they want to purchase.

[0121] Step 2:

[0122] Terminal: Uses a camera to capture an image of the product and generate image data.

[0123] Step 3:

[0124] Terminal: Sends the generated image data to the server.

[0125] Step 4:

[0126] Server: Analyzes the image using image recognition algorithms to identify relevant product information.

[0127] Step 5:

[0128] Server: Sends the relevant product information to the terminal.

[0129] Step 6:

[0130] Terminal: Displays the product on the TV screen based on the received product information.

[0131] Voice assistant support

[0132] Step 1:

[0133] Voice assistant: Understands changes in the user's physical condition and concerns through dialogue with the user.

[0134] Step 2:

[0135] Voice assistants: Use data sharing methods as needed to share acquired information with healthcare providers.

[0136] These processing steps enable users to easily shop online using voice and images, and receive support in their daily lives through a voice assistant.

[0137] Example 1

[0138] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0139] In today's world, users, especially the elderly and those with little technical knowledge, can find it difficult to select and purchase products over the Internet due to the complex operations and procedures involved. Furthermore, there are issues such as the risk of selecting the wrong product during the product selection process and the lack of means to accurately share health management information.

[0140] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0141] In this invention, the server includes an input means for accepting voice input, a conversion means for converting voice data into text, an analysis means for analyzing the text data and extracting keywords, a search means for acquiring information based on the extracted keywords, a display means for displaying the acquired information, an input means for accepting further voice input, a processing means for advancing the selected information to processing, and a payment means for presenting payment methods and executing payment. This allows users to easily search, select, and purchase products via voice input, and also provides a means for appropriately sharing health management information.

[0142] "Voice input" means a device or function for capturing a user's voice and inputting it into the system.

[0143] The "conversion" means is software or hardware for converting the captured voice data into text data.

[0144] The "analysis" means software or algorithms that analyze the converted text data and extract the user's intent and specific keywords.

[0145] The "search" means is a function for retrieving related information from databases and external information sources based on the extracted keywords.

[0146] "Display" means refers to a device or function for visually presenting acquired information to a user, and includes a television screen, monitor, and the like.

[0147] The "procedure" means is a function for proceeding to the next processing step, particularly the purchase procedure, based on the information selected by the user.

[0148] The "payment" means is a function for presenting a payment method to a user and processing the actual payment.

[0149] "Recognition" means analyzing image data acquired using a camera or other device, and is a function for identifying and recognizing specific products or objects.

[0150] The "sharing" means is a function for sharing the user's health information and status in cooperation with external institutions such as medical institutions.

[0151] A "voice assistant" is an interactive artificial intelligence that provides various services through voice interaction with users.

[0152] In the embodiment of the present invention, a system configured using the following main hardware and software will be described.

[0153] System Configuration

[0154] 1. Voice input method

[0155] The device is equipped with a microphone to capture the user's voice. Specifically, a dedicated voice capture device (such as a smart speaker) is used.

[0156] 2. A means of converting voice data into text

[0157] The device converts the captured voice data into text data in real time using voice recognition software such as the Google® Cloud Speech-to-Text API.

[0158] 3. Analysis method for analyzing text data and extracting keywords

[0159] The server receives the text data and analyzes it using a natural language processing engine (e.g., AWS (registered trademark) Comprehend). As a result of the analysis, the user's intent and keywords are extracted.

[0160] 4. Search methods to obtain information based on extracted keywords

[0161] The server uses the extracted keywords to query a database that contains product information and other related information.

[0162] 5. Display method for displaying acquired information

[0163] The terminal displays the information obtained from the server on a television screen or a dedicated display, for example, visually displaying a product list to the user.

[0164] 6. A voice selection means for accepting voice input again

[0165] The user looks at the displayed information and again makes selections or commands through voice input, which the device captures and converts again into text data.

[0166] 7. Procedural means for forwarding selected information for processing

[0167] The server then uses the selected information to complete the purchase, adding the product to the shopping cart and prompting for payment.

[0168] 8. Payment methods for submitting payment methods and executing payments

[0169] The server presents the user with multiple payment options (credit card, bank transfer, etc.) and executes the payment using the selected payment method, using a payment service such as Stripe or PayPal.

[0170] 9. A recognition method for analyzing product images captured by a camera and presenting relevant product information

[0171] The device is equipped with a camera that captures image data of the product the user points the camera at, which is then analyzed using image recognition software such as Google Cloud Vision API to search for relevant product information.

[0172] 10. Means for sharing user health information and status with medical institutions

[0173] The voice assistant will constantly monitor the user's health status through daily conversations with them, and if necessary, share information with medical institutions through data sharing methods to support the user's health management.

[0174] Specific examples

[0175] Let's take the example of a user speaking into a TV, saying, "Find a Mito Komon DVD." The device (e.g., a smart speaker) captures the user's voice and converts it into text data using commercially available voice recognition software. The text data is sent to the server in the format "Find a Mito Komon DVD." The server analyzes this text data using a natural language processing engine and extracts the keywords "Mito Komon," "DVD," and "search." It then queries a product database using a search tool to obtain relevant product information. The obtained information is then displayed on the TV screen via the device. The user then uses the displayed information to issue a purchase instruction again using voice input, and the server then proceeds with the purchase process based on this, ultimately completing the payment.

[0176] Prompt Sentence Examples

[0177] "Please explain the process of your voice search system. Please provide a detailed explanation of the steps a user takes to search for, select, and purchase a product using voice. Please also specify what software and hardware are used."

[0178] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0179] Step 1:

[0180] The user speaks to a TV or dedicated terminal, saying, "Find the Mito Komon DVD." The input voice is recognized by the system when the user presses a button on the remote control to switch to voice input mode, which generates voice data.

[0181] Step 2:

[0182] The device captures the user's voice data using a microphone. Specifically, the voice is recorded using a dedicated voice capture device (such as a smart speaker) and saved as digital data. The input is voice data, and the output is digital voice data.

[0183] Step 3:

[0184] The device sends the captured voice data in real time to a conversion mechanism using the Google Cloud Speech-to-Text API, which takes digital voice data as input and outputs text data.

[0185] Step 4:

[0186] The server receives the text data sent from the text conversion means. The input is text data, and this text data is in the format "Looking for a Mito Komon DVD." The output is the same text data.

[0187] Step 5:

[0188] The server runs the text data through an analysis tool and analyzes it using a natural language processing engine (e.g., AWS Comprehend). The input is text data, and through this analysis, the keywords "Mito Komon," "DVD," and "search for" are extracted. The output is the extracted keywords.

[0189] Step 6:

[0190] The server executes a search query to a database using a search tool based on the extracted keywords. Specifically, it executes an SQL query to a product database (e.g., a MySQL (registered trademark) database). The input is the extracted keywords, and the output is the corresponding product information.

[0191] Step 7:

[0192] The server sends the product information obtained as a search result to the display means. The obtained product information is sent in HTML format and converted into a data format for display on the screen. The input is the product information, and the output is the display data.

[0193] Step 8:

[0194] The terminal uses the received information to display a product list on the TV screen. Specifically, the information is visualized using a display device connected to the TV or a dedicated display. The input is the data to be displayed, and the output is the displayed product list.

[0195] Step 9:

[0196] The user looks at the displayed product list and again inputs by voice, "I'll buy this Mito Komon DVD." Specific actions include the user specifying a specific product number or name. The input is voice data, and the output is a selection instruction.

[0197] Step 10:

[0198] The terminal captures the voice data again and converts it into text using the conversion means described above. The input is voice data and the output is text data.

[0199] Step 11:

[0200] The server receives the text data regarding the product selection and analyzes the content. The analysis identifies the intention to purchase, such as "I will buy," and the selected product information. The input is the text data, and the output is the analysis result.

[0201] Step 12:

[0202] The server adds the selected items to the shopping cart via a procedure and presents payment options, such as "Do you want to pay by credit card?". The input is the analysis result, and the output is the payment options.

[0203] Step 13:

[0204] The user selects the desired payment method by voice, specifically by saying, "I'll pay by credit card."

[0205] Step 14:

[0206] The device captures the voice data again, converts it into text data, and sends it to the server. The input is voice data, and the output is text data.

[0207] Step 15:

[0208] The server executes the payment via a payment method such as Stripe or PayPal. The input is text data (payment instructions) and the output is payment confirmation.

[0209] Step 16:

[0210] The server sends a confirmation message of successful payment to the display means to notify the user. The input is the payment confirmation, and the output is the display data.

[0211] Step 17:

[0212] The terminal displays the received confirmation message on the TV screen. Specifically, the message "Purchase completed" is displayed on the screen. The input is the data to be displayed, and the output is the display of the confirmation message.

[0213] Step 18:

[0214] The user points the camera at the product they want to purchase. Specifically, they use a webcam connected to a TV. The input is a product image, and the output is image data.

[0215] Step 19:

[0216] The device acquires camera image data and sends it to the recognition means. The Google Cloud Vision API is used for recognition. The input is image data, and the output is analysis data.

[0217] Step 20:

[0218] The server analyzes the received image data and identifies the corresponding product. Product information is searched based on the analysis results. The input is the analyzed data, and the output is product information.

[0219] Step 21:

[0220] The server uses a display unit to display product information based on the analysis results on a television screen. Specifically, a list of "white shirt" products is displayed on the television screen. The input is product information, and the output is display data.

[0221] Step 22:

[0222] The voice assistant periodically asks the user about their health status and understands the situation through dialogue. The input is the user's voice data, and the output is analysis data.

[0223] Step 23:

[0224] The voice assistant analyzes the user's responses using a shared method and shares the information with medical institutions as needed. The input is the analyzed data, and the output is the shared data.

[0225] (Application example 1)

[0226] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0227] There is a need to reduce the complexity and anxiety that seniors face when shopping online, particularly by enabling them to search for and purchase products through intuitive voice commands. Camera-based product recognition and easy voice-based purchasing procedures are also needed. Furthermore, it is also important for voice assistants to connect appropriate services and information to support the health and needs of seniors.

[0228] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0229] In this invention, the server includes a voice input unit, a text conversion unit that converts voice data into text, a natural language processing unit that analyzes the text data and extracts keywords, a product search unit that acquires product information based on the extracted keywords, a display unit that displays the acquired product information, a voice selection unit that accepts voice input for product selection again, a purchase procedure unit that proceeds with the purchase of the selected product, a payment unit that presents a payment method and executes the payment, an image input unit for product recognition, and a product recognition unit that analyzes product images and presents corresponding product information. This allows elderly people to intuitively search for and purchase products using voice control or image recognition, providing ease of operation and safety. Furthermore, support using a voice assistant or data sharing unit provides appropriate responses according to the user's health condition and requests.

[0230] The "voice input means" is a device that has the function of capturing the user's voice and generating voice data.

[0231] The "text conversion means" is a means for converting voice data into text, and is a device that uses voice recognition technology.

[0232] "Natural language processing means" is a technical means for analyzing text data and extracting user intent and keywords.

[0233] The "product search means" is a means for searching and acquiring product information based on the extracted keywords.

[0234] "Display means" refers to a device for visually displaying acquired product information, including the screen of a smartphone and the HUD of smart glasses.

[0235] The "voice selection means" is a means having a function of accepting voice input again to select a product.

[0236] The "purchase procedure means" is a means for proceeding with the purchase procedure for the selected product.

[0237] A "payment method" is a method for presenting a payment method and actually making a payment.

[0238] "Image input means" refers to a means for acquiring product images using a camera or other sensor.

[0239] The "product recognition means" is a means for analyzing the product image acquired by the image input means and presenting the corresponding product information.

[0240] A "voice assistant" is software that understands the user's requests and status through dialogue with the user, and responds and provides information as needed.

[0241] "Data sharing means" refers to means that has the function of sharing user information with other service providers as needed.

[0242] The system embodying this invention performs a comprehensive process from voice input to text conversion, natural language processing, product search, display, product recognition, voice selection, purchase procedure, payment, and data sharing. Each component and its processing method are described in detail below.

[0243] System Configuration

[0244] The system includes the following main components:

[0245] 1. Voice input means: A means for capturing the user's voice and generating the voice data. Specifically, it uses a microphone built into a smartphone or smart glasses.

[0246] 2. Text conversion method: A method for converting voice data into text using voice recognition technology. Google Cloud Speech-to-Text API is used.

[0247] 3. Natural language processing: A means of analyzing text data and extracting keywords and user intent. Google Cloud Natural Language API is used.

[0248] 4. Product search method: This is a method to obtain product information based on extracted keywords. Firebase is used as the product database.

[0249] 5. Display means: A means for visually displaying the acquired product information. This includes the screen of a smartphone and the HUD of smart glasses.

[0250] 6. Voice selection means: A means for accepting voice input again to select a product.

[0251] 7. Purchasing procedure: A procedure for proceeding to purchase the selected product.

[0252] 8. Payment Method: A method for submitting a payment method and executing a payment. Uses the Stripe API.

[0253] 9. Image input means: A means of acquiring product images using a camera or other sensor.

[0254] 10. Product recognition means: A means for analyzing product images acquired by the image input means and presenting corresponding product information.

[0255] 11. Data Sharing Measures: These measures have the ability to share user information with other service providers as needed.

[0256] Program processing

[0257] The data processing and calculation flow at each step of this system will be explained below.

[0258] 1. Acquiring and parsing voice input:

[0259] A user speaks to a smartphone or smart glasses, saying, "I want to buy milk." The device captures this as audio data and converts it into text using the Google Cloud Speech-to-Text API. The server receives the text data and extracts keywords such as "milk" and "I want to buy" using the Google Cloud Natural Language API.

[0260] 2. Obtaining and displaying product information:

[0261] The server executes a search query against the product database (Firebase) using these keywords. Product information is retrieved as a search result and displayed on the smartphone screen or smart glasses HUD through a display means. For example, a list of "Milk" products is visually displayed on the user's screen.

[0262] 3. Product selection and purchase procedure:

[0263] While looking at the displayed list of products, the user again commands by voice, "Add the third milk to the cart." The device again captures the voice data and analyzes it using the procedure described above. The server adds the product to the shopping cart and presents options to confirm the payment method. The user selects by voice, "Pay with credit card," and the payment is executed via the Stripe API.

[0264] 4. Search for products using your camera:

[0265] The user points the camera at the product they want to purchase. The device acquires the camera's image data and analyzes the image using product recognition means. As a result of the analysis, the relevant product information is displayed. For example, if the user holds a "white shirt" up to the camera, a list of that product will be displayed on the smartphone screen or smart glasses' HUD.

[0266] 5. Voice assistant support:

[0267] Voice assistants will understand changes in the user's physical condition and requests through dialogue with the user, and share information with other service providers as needed through data sharing means, enabling them to manage the user's health and respond to their requests.

[0268] Examples of concrete examples and prompts

[0269] Examples:

[0270] When an elderly person says to their smartphone, "I want to buy milk," the system retrieves a list of milk products from Firebase and displays it on the screen. Then, when they say, "Add the third milk to my cart," the product is added to their cart. Finally, when they say, "Pay with credit card," the payment is completed using the Stripe API.

[0271] Example prompt sentence:

[0272] I want to buy milk.

[0273] Add a third milk to your cart.

[0274] Pay by credit card.

[0275] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0276] Step 1:

[0277] The user utters "I want to buy milk" via a voice input means. The device (smartphone or smart glasses) captures the voice data through a microphone.

[0278] Input: User's voice

[0279] Output: Audio data

[0280] Step 2:

[0281] The captured voice data is converted to text data by a text conversion means on the device using the Google Cloud Speech-to-Text API.

[0282] Input: Audio data

[0283] Output: Text data (e.g. "I want to buy milk")

[0284] Step 3:

[0285] The server receives the text data and analyzes it using natural language processing means with the Google Cloud Natural Language API to extract the keyword "milk" and the intent "want to buy."

[0286] Input: Text data

[0287] Output: Keywords (e.g., "milk"), Intent (e.g., "I want to buy")

[0288] Step 4:

[0289] Based on the extracted keywords, the server executes a search query for the relevant product information from the Firebase database via the product search means, and obtains a list of milk products as search results.

[0290] Input: Keyword (e.g. "milk")

[0291] Output: List of product information (e.g., list of milk products)

[0292] Step 5:

[0293] The acquired product information is displayed on the smartphone screen or smart glasses HUD through the device's display means, allowing the user to visually check the list of milk options.

[0294] Input: List of product information

[0295] Output: Visually displayed product information

[0296] Step 6:

[0297] The user instructs via voice input means to "add the third milk to the cart." The terminal again captures the voice data and converts it into text data.

[0298] Input: User's voice

[0299] Output: New audio data

[0300] Step 7:

[0301] The add-to-cart instruction is similarly converted to text, and the server parses the instruction and adds the third milk to the shopping cart.

[0302] Input: Speech data (e.g., "Add the third milk to my cart.")

[0303] Output: Updated shopping cart state

[0304] Step 8:

[0305] The server presents a payment method via the payment method (e.g., credit card). The user then voice-instructs "Pay with credit card," and the payment is executed using the Stripe API.

[0306] Input: Product information, payment method selection

[0307] Output: Payment confirmation, payment completion notification

[0308] Step 9:

[0309] The user takes a photo of a product with a camera as an additional product search. The terminal acquires the product image with the image input means and analyzes the corresponding product information using the product recognition means.

[0310] Input: Product image

[0311] Output: Parsed product information

[0312] Step 10:

[0313] The analyzed product information is displayed on the smartphone screen or smart glasses HUD through the terminal's display means.

[0314] Input: Parsed product information

[0315] Output: Visually displayed product information

[0316] Step 11:

[0317] In terms of health and request support, the voice assistant will understand the user's requests and status through dialogue with the user, and share information with other service providers through data sharing means as needed.

[0318] Input: Interaction data

[0319] Output: Information shared, suggested actions

[0320] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0321] The embodiment of the present invention follows the following configuration and processing flow.

[0322] System Configuration

[0323] The system consists of the following main components:

[0324] 1. Voice input method

[0325] It has the function of capturing the user's voice and generating the voice data.

[0326] 2. Text Conversion Methods

[0327] It has the function of receiving voice data and converting it into text data using a voice recognition engine.

[0328] 3. Natural Language Processing Methods

[0329] It has the ability to analyze text data and extract user intent and keywords.

[0330] 4. Product search methods

[0331] It has the function of retrieving relevant product information from a product database based on the extracted keywords.

[0332] 5. Display means

[0333] It has the function of displaying the acquired product information on a display device such as a television screen.

[0334] 6. Audio Selection Method

[0335] It has the function of accepting additional voice input to select a product.

[0336] 7. Purchasing Procedures

[0337] It has the function of proceeding to the purchase procedure for the selected product.

[0338] 8. Payment Methods

[0339] It has the function of presenting payment methods and executing payments.

[0340] 9. Product recognition means

[0341] It has the function of analyzing product images captured by the camera and presenting corresponding products.

[0342] 10. Data sharing methods

[0343] The voice assistant will monitor the user's condition and share information with medical institutions if necessary.

[0344] 11. Emotion Engine

[0345] It has the ability to recognize the user's emotions and respond appropriately based on them.

[0346] Program processing

[0347] Acquiring and analyzing voice input

[0348] The user speaks to a television or dedicated terminal, saying, "Find the Mito Komon DVD." The terminal captures this as voice data and converts it into text data using a text conversion means. The server receives the text data and uses natural language processing means to extract keywords such as "Mito Komon," "DVD," and "search."

[0349] Obtaining and displaying product information

[0350] The server executes a search query against the product database using these keywords. Product information is obtained as a search result and displayed on the television screen through the display means. Specifically, a list of "Mito Komon DVDs" is visually displayed on the user's screen.

[0351] Product selection and purchase process

[0352] The user looks at the displayed product list and again says, "I'd like to buy this Mito Komon DVD." The device again captures the voice data and analyzes it using the procedure described above. The server adds the product to the shopping cart and presents options for confirming the payment method (for example, "I'd like to pay by credit card").

[0353] The user selects "I'll pay by credit card" by voice, and the payment is executed by the payment method. If the payment is successful, a confirmation message is sent to the user.

[0354] Product search using camera

[0355] The user points the camera at the product they want to purchase. The device acquires the image data from the camera and analyzes the image using product recognition means. As a result of the analysis, the relevant product information is displayed. For example, if the user holds a "white shirt" up to the camera, a list of that product will be displayed on the TV screen.

[0356] Voice assistant support

[0357] The voice assistant will understand changes in the user's physical condition and any problems they may have through dialogue with the user, and if necessary, will share the information with medical institutions through data sharing means to support the user's health management.

[0358] Use of emotion engine

[0359] The emotion engine recognizes the user's emotions from their voice, facial expressions, and choice of words. For example, if a user says, "I feel tired today," the emotion engine recognizes the user's sense of fatigue. Based on this information, the display unit suggests products that will help them relax (for example, aroma oils or massage machines).

[0360] The emotion engine also allows the voice assistant to adapt its dialogue based on the user's emotions. For example, if the user is feeling down, the voice assistant will offer encouraging and kind words. Conversely, if the user seems happy, it will suggest topics that will pique the user's interest.

[0361] With these components and processing flow, the system of the present invention allows seniors to easily enjoy online shopping, alleviating feelings of loneliness and reducing the risk of fraud. It also provides a more sophisticated user experience by taking user emotions into consideration.

[0362] The processing flow will be explained below.

[0363] Acquiring and analyzing voice input

[0364] Step 1:

[0365] User: Speak to the TV or dedicated device and say, "Find the Mito Komon DVD."

[0366] Step 2:

[0367] Device: Uses the built-in microphone to capture the user's voice and generate audio data.

[0368] Step 3:

[0369] Terminal: Sends the generated voice data to the server via the network.

[0370] Step 4:

[0371] Server: Converts voice data into text data using a speech recognition engine.

[0372] Step 5:

[0373] Server: The converted text data is sent to a natural language processing module, which extracts keywords ("Mito Komon," "DVD," "search").

[0374] Obtaining and displaying product information

[0375] Step 1:

[0376] Server: Executes a search query to the product database based on the extracted keywords and retrieves the corresponding product information.

[0377] Step 2:

[0378] Server: Sends the acquired product information to the terminal via the network.

[0379] Step 3:

[0380] Terminal: Analyzes the received product information and creates a layout for display on the TV screen.

[0381] Step 4:

[0382] Device: Display a list of "Mito Komon DVDs" on the TV screen.

[0383] Product selection and purchase process

[0384] Step 1:

[0385] User: While looking at the displayed product list, say, "I'd like to buy this Mito Komon DVD."

[0386] Step 2:

[0387] Terminal: Captures audio and generates audio data.

[0388] Step 3:

[0389] Terminal: Sends the generated voice data to the server.

[0390] Step 4:

[0391] Server: Analyzes the voice data and identifies the product selected by the user.

[0392] Step 5:

[0393] Server: Confirms the user's selection and adds the selected items to the shopping cart.

[0394] Step 6:

[0395] Server: Generates payment options and presents them to the user through the voice assistant.

[0396] Step 7:

[0397] User: Select "Pay with credit card" by voice.

[0398] Step 8:

[0399] Terminal: Captures the user's voice and sends the voice data to the server.

[0400] Payment Processing

[0401] Step 1:

[0402] Server: Analyzes the voice data and confirms the user's payment method selection.

[0403] Step 2:

[0404] Server: Processes the payment using credit card information.

[0405] Step 3:

[0406] Server: Generates a confirmation message of payment completion and notifies the user via the voice assistant.

[0407] Product search using camera

[0408] Step 1:

[0409] User: Point the camera at the item they want to purchase.

[0410] Step 2:

[0411] Terminal: Uses a camera to capture an image of the product and generate image data.

[0412] Step 3:

[0413] Terminal: Sends the generated image data to the server.

[0414] Step 4:

[0415] Server: Analyzes the image using image recognition algorithms to identify relevant product information.

[0416] Step 5:

[0417] Server: Sends the relevant product information to the terminal.

[0418] Step 6:

[0419] Terminal: Displays the product on the TV screen based on the received product information.

[0420] Voice assistant support

[0421] Step 1:

[0422] Voice assistant: Understands changes in the user's physical condition and concerns through dialogue with the user.

[0423] Step 2:

[0424] Voice assistants: Use data sharing methods as needed to share acquired information with healthcare providers.

[0425] Use of emotion engine

[0426] Step 1:

[0427] User: Say "I'm tired today" aloud.

[0428] Step 2:

[0429] Terminal: Captures the user's voice and generates voice data.

[0430] Step 3:

[0431] Terminal: Sends the generated voice data to the server.

[0432] Step 4:

[0433] Server: Analyzes the voice data and recognizes the user's emotions (in this case, fatigue).

[0434] Step 5:

[0435] Emotion engine: Based on the recognized emotions, it generates recommendation information that suggests appropriate products to the user (e.g., relaxing aroma oils or massage machines).

[0436] Step 6:

[0437] Server: Sends the generated recommendation information to the terminal.

[0438] Step 7:

[0439] Device: The received recommendation information is displayed on the TV screen.

[0440] Step 8:

[0441] Voice assistant: Based on the recognized emotion, it will communicate with the user, saying, "I'd like to introduce some products that will help you relax."

[0442] Step 9:

[0443] Voice assistant: Changes the dialogue content according to the recognized emotions, offering gentle words if the user is tired and lively words and topics if they are energetic.

[0444] These processing steps allow users to easily shop online using voice and images, and receive support in daily life through a voice assistant.Furthermore, by utilizing an emotion engine, it is possible to provide optimal services according to the user's mood and emotions.

[0445] Example 2

[0446] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0447] Today's elderly often find it difficult to shop online due to the complexity of the process. Furthermore, existing systems are unable to address other needs, such as health management and reducing feelings of loneliness, which must also be met. In particular, intuitive operation using voice input and natural language processing, as well as personalized user support using emotion recognition, are needed.

[0448] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a voice input means, a text conversion means for converting voice data into text, a natural language processing means for analyzing the text data and extracting keywords, a product search means for acquiring product information based on the extracted keywords, a display means for displaying the acquired product information, a voice selection means for re-accepting voice input for product selection, a purchase procedure means for proceeding with the purchase of the selected product, a payment means for presenting a payment method and executing the payment, a product recognition means for pointing a camera at the product to be purchased, acquiring image data, and presenting corresponding product information, a data sharing means for the voice assistant to communicate with the user, grasp changes in the user's physical condition and problems, and share the information with an external organization as necessary, and an emotion recognition means for recognizing emotions from the user's voice and facial expressions and taking appropriate action. This allows elderly people to enjoy online shopping intuitively and easily, while at the same time satisfying multiple needs such as health management and reducing loneliness.

[0449] The "voice input means" is a device that has the function of capturing the user's speech as voice data.

[0450] The "text conversion means" is a device that has the function of converting captured voice data into text data.

[0451] A "natural language processing means" is a device that has the function of analyzing text data and extracting user intent and keywords.

[0452] The "product search means" is a device that has the function of retrieving relevant product information from a product database based on the extracted keywords.

[0453] The "display means" is a device that has the function of visually displaying the acquired product information.

[0454] The "voice selection means" is a device that has the function of accepting a second voice input from the user for product selection.

[0455] The "purchase procedure means" is a device that has the function of proceeding with the purchase procedure for the selected product.

[0456] A "payment method" is a device that has the function of presenting a payment method and actually carrying out the payment.

[0457] The "product recognition means" is a device that has the function of analyzing image data acquired by a camera and presenting corresponding product information.

[0458] A "data sharing means" is a device that allows the voice assistant to monitor the user's condition and share information with external medical institutions as necessary.

[0459] The "emotion recognition means" is a device that has the function of recognizing the user's emotions from their voice and facial expressions, and taking appropriate action based on that information.

[0460] This invention is an online shopping system primarily targeted at seniors, which combines voice input and natural language processing to enable intuitive and easy operation. Furthermore, by incorporating emotion recognition and health information sharing functions, it improves the overall user experience, helping to manage health and alleviate feelings of loneliness.

[0461] System configuration

[0462] The system consists of the following main components:

[0463] 1. Voice input method

[0464] 2. Text Conversion Methods

[0465] 3. Natural Language Processing Methods

[0466] 4. Product search methods

[0467] 5. Display means

[0468] 6. Audio Selection Method

[0469] 7. Purchasing Procedures

[0470] 8. Payment Methods

[0471] 9. Product recognition means

[0472] 10. Data sharing methods

[0473] 11. Emotion recognition means

[0474] Operation overview

[0475] Voice input and text conversion methods

[0476] The user speaks to the dedicated device or TV, for example, saying, "Look for a Mito Komon DVD." The device uses a built-in microphone to capture the voice data and converts it into text data using the Google Speech-to-Text API. This text data contains the content, "Look for a Mito Komon DVD."

[0477] Natural language processing tools

[0478] The server receives the text data and analyzes it using a natural language processing engine such as spaCy. As a result of the analysis, keywords such as "Mito Komon," "DVD," and "search for" are extracted.

[0479] Product search and display methods

[0480] The server uses the extracted keywords to execute a search query against a product database via the API of Amazon or Rakuten. The retrieved product information is displayed on the TV screen using HTML5 and JavaScript (registered trademark). For example, a list of "Mito Komon DVDs" is displayed visually to the user.

[0481] Voice selection method, purchase procedure method and payment method

[0482] When the user again says "I'd like to buy this Mito Komon DVD," the device captures the voice and converts it back into text data. The server analyzes this text data and adds the item to the shopping cart. Payment methods are then presented, and if the user selects by voice, for example, "I'd like to pay by credit card," the payment is executed using the Stripe API. If the payment is successful, a confirmation message is sent to the user.

[0483] Product recognition means

[0484] When a user points the camera at a product they want to purchase, the device uses the camera to capture image data. The server then analyzes the image using image recognition services such as Google Cloud Vision and retrieves the corresponding product information. This information is then displayed on the TV screen.

[0485] Data sharing methods

[0486] Through dialogue with the user, the voice assistant will understand changes in the user's physical condition and any problems they may have. If necessary, it will use FHIR (Fast Healthcare Interoperability Resources) to share information with external medical institutions and support the user's health management.

[0487] emotion recognition means

[0488] The emotion recognition means recognizes the user's emotions from their voice and facial expressions. For example, if the user says, "I'm tired today," the emotion recognition means recognizes the user's sense of fatigue, and based on this information, the display means suggests products that will help them relax (such as aroma oils or massage machines). The voice assistant also provides words of encouragement or interesting topics depending on the user's emotional state.

[0489] Examples of concrete examples and prompts

[0490] As a specific example, consider a situation where a user says to the TV, "Look for aroma oil." In this case, the device captures the voice and converts it into text data using the Google Speech-to-Text API. The server extracts the keyword "aroma oil," searches the product database, and displays the results on the TV screen.

[0491] Example prompt sentence:

[0492] "When a user says to the TV, 'I'm looking for aroma oil,' the voice data is captured and converted into text data, and the server uses natural language processing to extract keywords, retrieves the relevant product information from the product database, and displays it on the TV screen."

[0493] This system allows seniors to enjoy online shopping intuitively and easily, while also providing health and emotional care at the same time.

[0494] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0495] Step 1: Getting voice input

[0496] The user speaks to a TV or dedicated terminal, saying, "Find the Mito Komon DVD." The terminal uses a built-in microphone to capture the user's speech as audio data. The input is the user's voice, and the output is the captured audio data.

[0497] Step 2: Text conversion

[0498] The device generates the captured voice data using the Google Speech-to-Text API. The voice data is converted into text data. The input here is the voice data, and the output is the text data, "Looking for the Mito Komon DVD."

[0499] Step 3: Natural Language Processing

[0500] The server receives the text data and analyzes it using tools such as spaCy. During the analysis, keywords are extracted to understand the user's search intent. The input is text data, and the output is keywords such as "Mito Komon," "DVD," and "search."

[0501] Step 4: Product Search

[0502] The server uses the extracted keywords to execute a search query against the product database via the API of Amazon or Rakuten. Here, the process of retrieving related product information based on the keywords is performed. The input is the keyword, and the output is the corresponding product information (for example, a list of Mito Komon DVDs).

[0503] Step 5: View the results

[0504] The server displays the retrieved product information on the TV screen. It uses HTML5 and JavaScript to process the visual display. The input is the product information, and the output is a screen displaying a list of "Mito Komon DVDs."

[0505] Step 6: Product Selection

[0506] The user looks at the displayed product list and says verbally, "I'll buy this Mito Komon DVD." The device again captures the voice and converts the voice data into text data, just like in step 2. The input is the user's voice, and the output is the text data, "I'll buy this Mito Komon DVD."

[0507] Step 7: Checkout

[0508] The server adds the selected items to the shopping cart based on the analysis results. It then displays payment options and prepares the payment. The input is text data, and the output is product information added to the shopping cart and payment options.

[0509] Step 8: Payment

[0510] When the user selects "I'd like to pay with a credit card," the terminal captures the voice and converts it back into text data. The server then executes the payment using the Stripe API. The input is the user's voice and text data, and the output is a message that the payment was successful.

[0511] Step 9: Product Recognition

[0512] When a user points the camera at a product they want to purchase, the device uses the camera to capture image data. The server then analyzes the image using the Google Cloud Vision API and retrieves the corresponding product information. The input is image data, and the output is the corresponding product information.

[0513] Step 10: Data sharing

[0514] Voice assistants communicate with users to understand changes in their physical condition and any concerns they may have. They use FHIR to share information with external medical institutions as needed. The input is the user's voice and situational information, and the output is health information shared with medical institutions.

[0515] Step 11: Emotion Recognition

[0516] Emotion recognition is based on the user's voice and facial expressions. For example, if a user says, "I'm tired today," the system recognizes the user's emotion and suggests an appropriate response. The input is voice and facial expression data, and the output is emotional information and a suggested response based on that information.

[0517] (Application example 2)

[0518] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0519] The traditional product search and purchasing process in physical stores required time-consuming walking around the store to find products, resulting in low usability. It was also difficult to instantly check detailed product information and reviews, and product suggestions based on emotions or physical condition were not provided. Furthermore, there was a lack of a way to provide the same easy payment procedures in physical stores as online. This made shopping a burden, especially for elderly customers and those in poor health.

[0520] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0521] In this invention, the server includes a voice input means, a text conversion means for converting voice data into text, a natural language processing means for analyzing the text data and extracting keywords, a product search means for acquiring product information based on the extracted keywords, a display means for displaying the acquired product information, a voice selection means for re-accepting voice input for product selection, a purchase procedure means for proceeding with the purchase of the selected product, a payment means for presenting payment methods and executing the payment, an emotion recognition means for analyzing user emotions, a data feedback means for analyzing the user's expressions of intent and adjusting the content presented, and an augmented reality display means for providing a user interface using an augmented reality function. This enables an interactive product search and purchase process in a physical store, reduces the burden on users, and provides a more comfortable shopping experience.

[0522] The "voice input means" is a device that has the function of capturing the user's voice and generating voice data.

[0523] The "text conversion means" is a device that has the function of receiving voice data and converting it into text data using a voice recognition engine.

[0524] A "natural language processing means" is a device that has the function of analyzing text data and extracting user intent and keywords.

[0525] The "product search means" is a device that has the function of acquiring relevant product information from a product database based on the extracted keywords.

[0526] The "display means" is a display device for visualizing the acquired product information.

[0527] The "voice selection means" is a device that has the function of accepting voice input again to select a product.

[0528] The "purchase procedure means" is a device that has the function of proceeding with the purchase procedure for the selected product.

[0529] A "payment method" is a device that has the function of presenting a payment method and executing a payment.

[0530] The "emotion recognition means for analyzing user emotions" is a device that has the function of analyzing emotions from the user's voice, facial expressions, and content of statements.

[0531] The "data feedback means for adjusting the content of presentation" is a device that has the function of adapting the display and content of suggestions based on the user's intentions and emotions.

[0532] An "augmented reality display means" is a device that has the function of providing a user interface using augmented reality (AR) technology.

[0533] The "product recognition means" is a device that has the function of analyzing product images acquired by a camera and presenting corresponding products.

[0534] A "data sharing means" is a device that allows the voice assistant to monitor the user's condition and share information with medical institutions as necessary.

[0535] An embodiment of the present invention is a smart shopping assistant system for realizing an interactive product search and purchasing process in a physical store. This system is composed of a voice input means, a voice recognition means, a natural language processing means, a product search means, an augmented reality display means, an emotion recognition means, a data feedback means, and a payment means.

[0536] System Configuration

[0537] Voice Input and Recognition

[0538] The user inputs voice into a device such as smart glasses or a head-mounted display. For example, the user might say, "I'm looking for a red dress." The voice input means captures this voice data and converts it into text data using the text conversion means. Specifically, it uses the Google Speech Recognition API.

[0539] Natural Language Processing and Product Search

[0540] The server receives the text data and analyzes it using an NLP processor (e.g., TextBlob or AnyMLibrary NLP) to extract keywords. Based on these keywords, the product search means queries the database to obtain relevant product information. For example, the keyword "red dress" is used to search the store's inventory database.

[0541] Augmented reality display

[0542] The acquired product information is superimposed on the screen of the smart glasses or head-mounted display by the augmented reality display means, allowing the user to visually check the product information. For example, an image of a "red dress," price information, and reviews are visually displayed.

[0543] Emotion Recognition and Data Feedback

[0544] When displaying retrieved product information, the system uses emotion recognition to analyze the user's emotions from their voice and facial expressions. For example, if a user says, "I'm tired today," the system uses TextBlob to analyze their emotions and adjust the product suggestions. Data feedback is used to optimize product suggestions and displays based on the user's intentions and emotions.

[0545] Purchase process and payment

[0546] The user checks the visually displayed product information and again voice-inputs "I would like to purchase this dress." The system again converts the voice input into text data and confirms the user's selection through natural language processing. The system then proceeds to purchase the selected product using the checkout means, presents the payment method through the payment means, and executes the payment.

[0547] Hardware and software used

[0548] Hardware: smart glasses, head-mounted displays, cameras

[0549] Speech recognition software: Google Speech Recognition API

[0550] Natural Language Processing Software: TextBlob, AnyMLibrary NLP

[0551] Database Connection: Custom Database Module

[0552] Payment Software: Payment Processor Custom Module

[0553] Display software: AR display function

[0554] Specific examples

[0555] For example, in a scenario where a user puts on smart glasses and says, "I'm looking for a red dress," the system converts the speech into text data and uses an NLP processor to extract the keyword "red dress." It then searches for the corresponding product in the store's inventory database and displays the product information on the smart glasses' AR display. The user then says, "I'd like to purchase this dress," and the system proceeds with the purchase process and payment. The system also analyzes the user's emotions from their speech and facial expressions and adjusts the suggestions based on the results.

[0556] Prompt Sentence Examples

[0557] Preprocess the following speech input and extract keywords for product search:

[0558] Dictation: "I'm looking for a red dress."

[0559] Example output: ['red dress', 'search', 'looking for']

[0560] This enables the smart shopping assistant system to realize an interactive product search and purchasing process in a physical store, providing users with a comfortable shopping experience.

[0561] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0562] Step 1:

[0563] Acquiring voice input

[0564] The user inputs a voice command to search for a product into the smart glasses or head-mounted display (e.g., "I'm looking for a red dress"). The device captures this voice data and prepares for voice recognition. The input is voice data, and the output is also voice data. Specifically, the device records the voice through a microphone.

[0565] Step 2:

[0566] Converting audio data to text

[0567] The device converts the captured audio data into text data using the Google Speech Recognition API. The input is audio data and the output is text data. This process involves analyzing the audio waveform and converting it into a corresponding string of characters.

[0568] Step 3:

[0569] Natural language processing of text data

[0570] The server receives the text data and analyzes it using an NLP processor (for example, TextBlob or AnyMLibrary NLP). The input is the text data, and the output is a set of extracted keywords. Specifically, the text data is tokenized and important keywords and phrases are extracted.

[0571] Step 4:

[0572] Search for products

[0573] The server then searches the product database using the extracted keywords. The input is a set of keywords, and the output is the corresponding product information. Specifically, the server generates a database query to retrieve product inventory information and detailed information.

[0574] Step 5:

[0575] Displaying product information

[0576] The device displays the acquired product information using the augmented reality function of smart glasses or a head-mounted display. The input is product information, and the output is a visual presentation of the product information. Specifically, the device uses an AR overlay to display product images, prices, reviews, and more in the user's field of view.

[0577] Step 6:

[0578] Emotion recognition

[0579] The server analyzes the user's voice and facial expression data and recognizes emotions using tools such as TextBlob. The input is the user's voice and facial expression data, and the output is the analysis result of the emotional state. Specifically, it analyzes the voice and facial expression data and evaluates the emotion.

[0580] Step 7:

[0581] Data feedback adjustment

[0582] The server then adjusts the display and recommendations based on the recognized emotions. The input is the emotion analysis results, and the output is adjusted product information and recommendations. Specifically, the server adapts the display content based on the emotion data.

[0583] Step 8:

[0584] Product selection and purchase process

[0585] The user checks the displayed product information and again voice-inputs "I'd like to purchase this dress." The device again captures the voice data and confirms the user's selection through text conversion and natural language processing. The input is the re-captured voice data, and the output is the analysis result of the text data.

[0586] Step 9:

[0587] Purchase process and payment

[0588] The server then checks out the selected product, presents the payment method, and executes the transaction. The input is the selected product information, and the output is confirmation of the transaction. Specific operations include inputting and processing payment information and sending a purchase confirmation.

[0589] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0590] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0591] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0592] [Second embodiment]

[0593] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0594] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0595] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0596] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0597] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0598] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0599] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0600] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0601] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0602] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0603] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0604] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0605] The embodiment of the present invention follows the following configuration and processing flow.

[0606] System Configuration

[0607] The system consists of the following main components:

[0608] 1. Voice input method

[0609] It has the function of capturing the user's voice and generating the voice data.

[0610] 2. Text Conversion Methods

[0611] It has the function of receiving voice data and converting it into text data using a voice recognition engine.

[0612] 3. Natural Language Processing Methods

[0613] It has the ability to analyze text data and extract user intent and keywords.

[0614] 4. Product search methods

[0615] It has the function of retrieving relevant product information from a product database based on the extracted keywords.

[0616] 5. Display means

[0617] It has the function of displaying the acquired product information on a display device such as a television screen.

[0618] 6. Audio Selection Method

[0619] It has the function of accepting additional voice input to select a product.

[0620] 7. Purchasing Procedures

[0621] It has the function of proceeding to the purchase procedure for the selected product.

[0622] 8. Payment Methods

[0623] It has the function of presenting payment methods and executing payments.

[0624] 9. Product recognition means

[0625] It has the function of analyzing product images captured by the camera and presenting corresponding products.

[0626] 10. Data sharing methods

[0627] The voice assistant will monitor the user's condition and share information with medical institutions if necessary.

[0628] Program processing

[0629] Acquiring and analyzing voice input

[0630] The user speaks to a television or dedicated terminal, saying, "Find the Mito Komon DVD." The terminal captures this as voice data and converts it into text data using a text conversion means. The server receives the text data and uses natural language processing means to extract keywords such as "Mito Komon," "DVD," and "search."

[0631] Obtaining and displaying product information

[0632] The server executes a search query against the product database using these keywords. Product information is obtained as a search result and displayed on the television screen through the display means. Specifically, a list of "Mito Komon DVDs" is visually displayed on the user's screen.

[0633] Product selection and purchase process

[0634] The user looks at the displayed product list and again says, "I'd like to buy this Mito Komon DVD." The device again captures the voice data and analyzes it using the procedure described above. The server adds the product to the shopping cart and presents options for confirming the payment method (for example, "I'd like to pay by credit card").

[0635] The user selects "I'll pay by credit card" by voice, and the payment is executed by the payment method. If the payment is successful, a confirmation message is sent to the user.

[0636] Product search using camera

[0637] The user points the camera at the product they want to purchase. The device acquires the image data from the camera and analyzes the image using product recognition means. As a result of the analysis, the relevant product information is displayed. For example, if the user holds a "white shirt" up to the camera, a list of that product will be displayed on the TV screen.

[0638] Voice assistant support

[0639] The voice assistant will understand changes in the user's physical condition and any problems they may have through dialogue with the user, and if necessary, will share the information with medical institutions through data sharing means to support the user's health management.

[0640] With these components and processing flow, the system of the present invention enables elderly people to easily enjoy online shopping, eliminating feelings of loneliness and reducing the risk of fraud.

[0641] The processing flow will be explained below.

[0642] Acquiring and analyzing voice input

[0643] Step 1:

[0644] User: Speak to the TV or dedicated device and say, "Find the Mito Komon DVD."

[0645] Step 2:

[0646] Device: Uses the built-in microphone to capture the user's voice and generate audio data.

[0647] Step 3:

[0648] Terminal: Sends the generated voice data to the server via the network.

[0649] Step 4:

[0650] Server: Converts voice data into text data using a speech recognition engine.

[0651] Step 5:

[0652] Server: The converted text data is sent to a natural language processing module, which extracts keywords ("Mito Komon," "DVD," "search").

[0653] Obtaining and displaying product information

[0654] Step 1:

[0655] Server: Executes a search query to the product database based on the extracted keywords and retrieves the corresponding product information.

[0656] Step 2:

[0657] Server: Sends the acquired product information to the terminal via the network.

[0658] Step 3:

[0659] Terminal: Analyzes the received product information and creates a layout for display on the TV screen.

[0660] Step 4:

[0661] Device: Display a list of "Mito Komon DVDs" on the TV screen.

[0662] Product selection and purchase process

[0663] Step 1:

[0664] User: While looking at the displayed product list, say, "I'd like to buy this Mito Komon DVD."

[0665] Step 2:

[0666] Terminal: Capture the audio again and generate the audio data.

[0667] Step 3:

[0668] Terminal: Sends the generated voice data to the server.

[0669] Step 4:

[0670] Server: Analyzes the voice data and identifies the product selected by the user.

[0671] Step 5:

[0672] Server: Confirms the user's selection and adds the selected items to the shopping cart.

[0673] Step 6:

[0674] Server: Generates payment options and presents them to the user through the voice assistant.

[0675] Step 7:

[0676] User: Select "Pay with credit card" by voice.

[0677] Step 8:

[0678] Terminal: Captures the user's voice and sends the voice data to the server.

[0679] Payment Processing

[0680] Step 1:

[0681] Server: Analyzes the voice data and confirms the user's payment method selection.

[0682] Step 2:

[0683] Server: Processes the payment using credit card information.

[0684] Step 3:

[0685] Server: Generates a confirmation message of payment completion and notifies the user via the voice assistant.

[0686] Product search using camera

[0687] Step 1:

[0688] User: Point the camera at the item they want to purchase.

[0689] Step 2:

[0690] Terminal: Uses a camera to capture an image of the product and generate image data.

[0691] Step 3:

[0692] Terminal: Sends the generated image data to the server.

[0693] Step 4:

[0694] Server: Analyzes the image using image recognition algorithms to identify relevant product information.

[0695] Step 5:

[0696] Server: Sends the relevant product information to the terminal.

[0697] Step 6:

[0698] Terminal: Displays the product on the TV screen based on the received product information.

[0699] Voice assistant support

[0700] Step 1:

[0701] Voice assistant: Understands changes in the user's physical condition and concerns through dialogue with the user.

[0702] Step 2:

[0703] Voice assistants: Use data sharing methods as needed to share acquired information with healthcare providers.

[0704] These processing steps enable users to easily shop online using voice and images, and receive support in their daily lives through a voice assistant.

[0705] Example 1

[0706] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0707] In today's world, users, especially the elderly and those with little technical knowledge, can find it difficult to select and purchase products over the Internet due to the complex operations and procedures involved. Furthermore, there are issues such as the risk of selecting the wrong product during the product selection process and the lack of means to accurately share health management information.

[0708] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0709] In this invention, the server includes an input means for accepting voice input, a conversion means for converting voice data into text, an analysis means for analyzing the text data and extracting keywords, a search means for acquiring information based on the extracted keywords, a display means for displaying the acquired information, an input means for accepting further voice input, a processing means for advancing the selected information to processing, and a payment means for presenting payment methods and executing payment. This allows users to easily search, select, and purchase products via voice input, and also provides a means for appropriately sharing health management information.

[0710] "Voice input" means a device or function for capturing a user's voice and inputting it into the system.

[0711] The "conversion" means is software or hardware for converting the captured voice data into text data.

[0712] The "analysis" means software or algorithms that analyze the converted text data and extract the user's intent and specific keywords.

[0713] The "search" means is a function for retrieving related information from databases and external information sources based on the extracted keywords.

[0714] "Display" means refers to a device or function for visually presenting acquired information to a user, and includes a television screen, monitor, and the like.

[0715] The "procedure" means is a function for proceeding to the next processing step, particularly the purchase procedure, based on the information selected by the user.

[0716] The "payment" means is a function for presenting a payment method to a user and processing the actual payment.

[0717] "Recognition" means analyzing image data acquired using a camera or other device, and is a function for identifying and recognizing specific products or objects.

[0718] The "sharing" means is a function for sharing the user's health information and status in cooperation with external institutions such as medical institutions.

[0719] A "voice assistant" is an interactive artificial intelligence that provides various services through voice interaction with users.

[0720] In the embodiment of the present invention, a system configured using the following main hardware and software will be described.

[0721] System Configuration

[0722] 1. Voice input method

[0723] The device is equipped with a microphone to capture the user's voice. Specifically, a dedicated voice capture device (such as a smart speaker) is used.

[0724] 2. A means of converting voice data into text

[0725] The device converts the captured audio data into text in real time using voice recognition software such as the Google Cloud Speech-to-Text API.

[0726] 3. Analysis method for analyzing text data and extracting keywords

[0727] The server receives the text data and analyzes it using a natural language processing engine (e.g., AWS Comprehend). As a result of the analysis, the user's intent and keywords are extracted.

[0728] 4. Search methods to obtain information based on extracted keywords

[0729] The server uses the extracted keywords to query a database that contains product information and other related information.

[0730] 5. Display method for displaying acquired information

[0731] The terminal displays the information obtained from the server on a television screen or a dedicated display, for example, visually displaying a product list to the user.

[0732] 6. A voice selection means for accepting voice input again

[0733] The user looks at the displayed information and again makes selections or commands through voice input, which the device captures and converts again into text data.

[0734] 7. Procedural means for forwarding selected information for processing

[0735] The server then uses the selected information to complete the purchase, adding the product to the shopping cart and prompting for payment.

[0736] 8. Payment methods for submitting payment methods and executing payments

[0737] The server presents the user with multiple payment options (credit card, bank transfer, etc.) and executes the payment using the selected payment method, using a payment service such as Stripe or PayPal.

[0738] 9. A recognition method for analyzing product images captured by a camera and presenting relevant product information

[0739] The device is equipped with a camera that captures image data of the product the user points the camera at, which is then analyzed using image recognition software such as Google Cloud Vision API to search for relevant product information.

[0740] 10. Means for sharing user health information and status with medical institutions

[0741] The voice assistant will constantly monitor the user's health status through daily conversations with them, and if necessary, share information with medical institutions through data sharing methods to support the user's health management.

[0742] Specific examples

[0743] Let's take the example of a user speaking into a TV, saying, "Find a Mito Komon DVD." The device (e.g., a smart speaker) captures the user's voice and converts it into text data using commercially available voice recognition software. The text data is sent to the server in the format "Find a Mito Komon DVD." The server analyzes this text data using a natural language processing engine and extracts the keywords "Mito Komon," "DVD," and "search." It then queries a product database using a search tool to obtain relevant product information. The obtained information is then displayed on the TV screen via the device. The user then uses the displayed information to issue a purchase instruction again using voice input, and the server then proceeds with the purchase process based on this, ultimately completing the payment.

[0744] Prompt Sentence Examples

[0745] "Please explain the process of your voice search system. Please provide a detailed explanation of the steps a user takes to search for, select, and purchase a product using voice. Please also specify what software and hardware are used."

[0746] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0747] Step 1:

[0748] The user speaks to a TV or dedicated terminal, saying, "Find the Mito Komon DVD." The input voice is recognized by the system when the user presses a button on the remote control to switch to voice input mode, which generates voice data.

[0749] Step 2:

[0750] The device captures the user's voice data using a microphone. Specifically, the voice is recorded using a dedicated voice capture device (such as a smart speaker) and saved as digital data. The input is voice data, and the output is digital voice data.

[0751] Step 3:

[0752] The device sends the captured voice data in real time to a conversion mechanism using the Google Cloud Speech-to-Text API, which takes digital voice data as input and outputs text data.

[0753] Step 4:

[0754] The server receives the text data sent from the text conversion means. The input is text data, and this text data is in the format "Looking for a Mito Komon DVD." The output is the same text data.

[0755] Step 5:

[0756] The server runs the text data through an analysis tool and analyzes it using a natural language processing engine (e.g., AWS Comprehend). The input is text data, and through this analysis, the keywords "Mito Komon," "DVD," and "search for" are extracted. The output is the extracted keywords.

[0757] Step 6:

[0758] The server executes a search query to a database using a search engine based on the extracted keywords. Specifically, it executes an SQL query to a product database (e.g., a MySQL database). The input is the extracted keywords, and the output is the corresponding product information.

[0759] Step 7:

[0760] The server sends the product information obtained as a search result to the display means. The obtained product information is sent in HTML format and converted into a data format for display on the screen. The input is the product information, and the output is the display data.

[0761] Step 8:

[0762] The terminal uses the received information to display a product list on the TV screen. Specifically, the information is visualized using a display device connected to the TV or a dedicated display. The input is the data to be displayed, and the output is the displayed product list.

[0763] Step 9:

[0764] The user looks at the displayed product list and again inputs by voice, "I'll buy this Mito Komon DVD." Specific actions include the user specifying a specific product number or name. The input is voice data, and the output is a selection instruction.

[0765] Step 10:

[0766] The terminal captures the voice data again and converts it into text using the conversion means described above. The input is voice data and the output is text data.

[0767] Step 11:

[0768] The server receives the text data regarding the product selection and analyzes the content. The analysis identifies the intention to purchase, such as "I will buy," and the selected product information. The input is the text data, and the output is the analysis result.

[0769] Step 12:

[0770] The server adds the selected items to the shopping cart via a procedure and presents payment options, such as "Do you want to pay by credit card?". The input is the analysis result, and the output is the payment options.

[0771] Step 13:

[0772] The user selects the desired payment method by voice, specifically by saying, "I'll pay by credit card."

[0773] Step 14:

[0774] The device captures the voice data again, converts it into text data, and sends it to the server. The input is voice data, and the output is text data.

[0775] Step 15:

[0776] The server executes the payment via a payment method such as Stripe or PayPal. The input is text data (payment instructions) and the output is payment confirmation.

[0777] Step 16:

[0778] The server sends a confirmation message of successful payment to the display means to notify the user. The input is the payment confirmation, and the output is the display data.

[0779] Step 17:

[0780] The terminal displays the received confirmation message on the TV screen. Specifically, the message "Purchase completed" is displayed on the screen. The input is the data to be displayed, and the output is the display of the confirmation message.

[0781] Step 18:

[0782] The user points the camera at the product they want to purchase. Specifically, they use a webcam connected to a TV. The input is a product image, and the output is image data.

[0783] Step 19:

[0784] The device acquires camera image data and sends it to the recognition means. The Google Cloud Vision API is used for recognition. The input is image data, and the output is analysis data.

[0785] Step 20:

[0786] The server analyzes the received image data and identifies the corresponding product. Product information is searched based on the analysis results. The input is the analyzed data, and the output is product information.

[0787] Step 21:

[0788] The server uses a display unit to display product information based on the analysis results on a television screen. Specifically, a list of "white shirt" products is displayed on the television screen. The input is product information, and the output is display data.

[0789] Step 22:

[0790] The voice assistant periodically asks the user about their health status and understands the situation through dialogue. The input is the user's voice data, and the output is analysis data.

[0791] Step 23:

[0792] The voice assistant analyzes the user's responses using a shared method and shares the information with medical institutions as needed. The input is the analyzed data, and the output is the shared data.

[0793] (Application example 1)

[0794] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0795] There is a need to reduce the complexity and anxiety that seniors face when shopping online, particularly by enabling them to search for and purchase products through intuitive voice commands. Camera-based product recognition and easy voice-based purchasing procedures are also needed. Furthermore, it is also important for voice assistants to connect appropriate services and information to support the health and needs of seniors.

[0796] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0797] In this invention, the server includes a voice input unit, a text conversion unit that converts voice data into text, a natural language processing unit that analyzes the text data and extracts keywords, a product search unit that acquires product information based on the extracted keywords, a display unit that displays the acquired product information, a voice selection unit that accepts voice input for product selection again, a purchase procedure unit that proceeds with the purchase of the selected product, a payment unit that presents a payment method and executes the payment, an image input unit for product recognition, and a product recognition unit that analyzes product images and presents corresponding product information. This allows elderly people to intuitively search for and purchase products using voice control or image recognition, providing ease of operation and safety. Furthermore, support using a voice assistant or data sharing unit provides appropriate responses according to the user's health condition and requests.

[0798] The "voice input means" is a device that has the function of capturing the user's voice and generating voice data.

[0799] The "text conversion means" is a means for converting voice data into text, and is a device that uses voice recognition technology.

[0800] "Natural language processing means" is a technical means for analyzing text data and extracting user intent and keywords.

[0801] The "product search means" is a means for searching and acquiring product information based on the extracted keywords.

[0802] "Display means" refers to a device for visually displaying acquired product information, including the screen of a smartphone and the HUD of smart glasses.

[0803] The "voice selection means" is a means having a function of accepting voice input again to select a product.

[0804] The "purchase procedure means" is a means for proceeding with the purchase procedure for the selected product.

[0805] A "payment method" is a method for presenting a payment method and actually making a payment.

[0806] "Image input means" refers to a means for acquiring product images using a camera or other sensor.

[0807] The "product recognition means" is a means for analyzing the product image acquired by the image input means and presenting the corresponding product information.

[0808] A "voice assistant" is software that understands the user's requests and status through dialogue with the user, and responds and provides information as needed.

[0809] "Data sharing means" refers to means that has the function of sharing user information with other service providers as needed.

[0810] The system embodying this invention performs a comprehensive process from voice input to text conversion, natural language processing, product search, display, product recognition, voice selection, purchase procedure, payment, and data sharing. Each component and its processing method are described in detail below.

[0811] System Configuration

[0812] The system includes the following main components:

[0813] 1. Voice input means: A means for capturing the user's voice and generating the voice data. Specifically, it uses a microphone built into a smartphone or smart glasses.

[0814] 2. Text conversion method: A method for converting voice data into text using voice recognition technology. Google Cloud Speech-to-Text API is used.

[0815] 3. Natural language processing: A means of analyzing text data and extracting keywords and user intent. Google Cloud Natural Language API is used.

[0816] 4. Product search method: This is a method to obtain product information based on extracted keywords. Firebase is used as the product database.

[0817] 5. Display means: A means for visually displaying the acquired product information. This includes the screen of a smartphone and the HUD of smart glasses.

[0818] 6. Voice selection means: A means for accepting voice input again to select a product.

[0819] 7. Purchasing procedure: A procedure for proceeding to purchase the selected product.

[0820] 8. Payment Method: A method for submitting a payment method and executing a payment. Uses the Stripe API.

[0821] 9. Image input means: A means of acquiring product images using a camera or other sensor.

[0822] 10. Product recognition means: A means for analyzing product images acquired by the image input means and presenting corresponding product information.

[0823] 11. Data Sharing Measures: These measures have the ability to share user information with other service providers as needed.

[0824] Program processing

[0825] The data processing and calculation flow at each step of this system will be explained below.

[0826] 1. Acquiring and parsing voice input:

[0827] A user speaks to a smartphone or smart glasses, saying, "I want to buy milk." The device captures this as audio data and converts it into text using the Google Cloud Speech-to-Text API. The server receives the text data and extracts keywords such as "milk" and "I want to buy" using the Google Cloud Natural Language API.

[0828] 2. Obtaining and displaying product information:

[0829] The server executes a search query against the product database (Firebase) using these keywords. Product information is retrieved as a search result and displayed on the smartphone screen or smart glasses HUD through a display means. For example, a list of "Milk" products is visually displayed on the user's screen.

[0830] 3. Product selection and purchase procedure:

[0831] While looking at the displayed list of products, the user again commands by voice, "Add the third milk to the cart." The device again captures the voice data and analyzes it using the procedure described above. The server adds the product to the shopping cart and presents options to confirm the payment method. The user selects by voice, "Pay with credit card," and the payment is executed via the Stripe API.

[0832] 4. Search for products using your camera:

[0833] The user points the camera at the product they want to purchase. The device acquires the camera's image data and analyzes the image using product recognition means. As a result of the analysis, the relevant product information is displayed. For example, if the user holds a "white shirt" up to the camera, a list of that product will be displayed on the smartphone screen or smart glasses' HUD.

[0834] 5. Voice assistant support:

[0835] Voice assistants will understand changes in the user's physical condition and requests through dialogue with the user, and share information with other service providers as needed through data sharing means, enabling them to manage the user's health and respond to their requests.

[0836] Examples of concrete examples and prompts

[0837] Examples:

[0838] When an elderly person says to their smartphone, "I want to buy milk," the system retrieves a list of milk products from Firebase and displays it on the screen. Then, when they say, "Add the third milk to my cart," the product is added to their cart. Finally, when they say, "Pay with credit card," the payment is completed using the Stripe API.

[0839] Example prompt sentence:

[0840] I want to buy milk.

[0841] Add a third milk to your cart.

[0842] Pay by credit card.

[0843] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0844] Step 1:

[0845] The user utters "I want to buy milk" via a voice input means. The device (smartphone or smart glasses) captures the voice data through a microphone.

[0846] Input: User's voice

[0847] Output: Audio data

[0848] Step 2:

[0849] The captured voice data is converted to text data by a text conversion means on the device using the Google Cloud Speech-to-Text API.

[0850] Input: Audio data

[0851] Output: Text data (e.g. "I want to buy milk")

[0852] Step 3:

[0853] The server receives the text data and analyzes it using natural language processing means with the Google Cloud Natural Language API to extract the keyword "milk" and the intent "want to buy."

[0854] Input: Text data

[0855] Output: Keywords (e.g., "milk"), Intent (e.g., "I want to buy")

[0856] Step 4:

[0857] Based on the extracted keywords, the server executes a search query for the relevant product information from the Firebase database via the product search means, and obtains a list of milk products as search results.

[0858] Input: Keyword (e.g. "milk")

[0859] Output: List of product information (e.g., list of milk products)

[0860] Step 5:

[0861] The acquired product information is displayed on the smartphone screen or smart glasses HUD through the device's display means, allowing the user to visually check the list of milk options.

[0862] Input: List of product information

[0863] Output: Visually displayed product information

[0864] Step 6:

[0865] The user instructs via voice input means to "add the third milk to the cart." The terminal again captures the voice data and converts it into text data.

[0866] Input: User's voice

[0867] Output: New audio data

[0868] Step 7:

[0869] The add-to-cart instruction is similarly converted to text, and the server parses the instruction and adds the third milk to the shopping cart.

[0870] Input: Speech data (e.g., "Add the third milk to my cart.")

[0871] Output: Updated shopping cart state

[0872] Step 8:

[0873] The server presents a payment method via the payment method (e.g., credit card). The user then voice-instructs "Pay with credit card," and the payment is executed using the Stripe API.

[0874] Input: Product information, payment method selection

[0875] Output: Payment confirmation, payment completion notification

[0876] Step 9:

[0877] The user takes a photo of a product with a camera as an additional product search. The terminal acquires the product image with the image input means and analyzes the corresponding product information using the product recognition means.

[0878] Input: Product image

[0879] Output: Parsed product information

[0880] Step 10:

[0881] The analyzed product information is displayed on the smartphone screen or smart glasses HUD through the terminal's display means.

[0882] Input: Parsed product information

[0883] Output: Visually displayed product information

[0884] Step 11:

[0885] In terms of health and request support, the voice assistant will understand the user's requests and status through dialogue with the user, and share information with other service providers through data sharing means as needed.

[0886] Input: Interaction data

[0887] Output: Information shared, suggested actions

[0888] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0889] The embodiment of the present invention follows the following configuration and processing flow.

[0890] System Configuration

[0891] The system consists of the following main components:

[0892] 1. Voice input method

[0893] It has the function of capturing the user's voice and generating the voice data.

[0894] 2. Text Conversion Methods

[0895] It has the function of receiving voice data and converting it into text data using a voice recognition engine.

[0896] 3. Natural Language Processing Methods

[0897] It has the ability to analyze text data and extract user intent and keywords.

[0898] 4. Product search methods

[0899] It has the function of retrieving relevant product information from a product database based on the extracted keywords.

[0900] 5. Display means

[0901] It has the function of displaying the acquired product information on a display device such as a television screen.

[0902] 6. Audio Selection Method

[0903] It has the function of accepting additional voice input to select a product.

[0904] 7. Purchasing Procedures

[0905] It has the function of proceeding to the purchase procedure for the selected product.

[0906] 8. Payment Methods

[0907] It has the function of presenting payment methods and executing payments.

[0908] 9. Product recognition means

[0909] It has the function of analyzing product images captured by the camera and presenting corresponding products.

[0910] 10. Data sharing methods

[0911] The voice assistant will monitor the user's condition and share information with medical institutions if necessary.

[0912] 11. Emotion Engine

[0913] It has the ability to recognize the user's emotions and respond appropriately based on them.

[0914] Program processing

[0915] Acquiring and analyzing voice input

[0916] The user speaks to a television or dedicated terminal, saying, "Find the Mito Komon DVD." The terminal captures this as voice data and converts it into text data using a text conversion means. The server receives the text data and uses natural language processing means to extract keywords such as "Mito Komon," "DVD," and "search."

[0917] Obtaining and displaying product information

[0918] The server executes a search query against the product database using these keywords. Product information is obtained as a search result and displayed on the television screen through the display means. Specifically, a list of "Mito Komon DVDs" is visually displayed on the user's screen.

[0919] Product selection and purchase process

[0920] The user looks at the displayed product list and again says, "I'd like to buy this Mito Komon DVD." The device again captures the voice data and analyzes it using the procedure described above. The server adds the product to the shopping cart and presents options for confirming the payment method (for example, "I'd like to pay by credit card").

[0921] The user selects "I'll pay by credit card" by voice, and the payment is executed by the payment method. If the payment is successful, a confirmation message is sent to the user.

[0922] Product search using camera

[0923] The user points the camera at the product they want to purchase. The device acquires the image data from the camera and analyzes the image using product recognition means. As a result of the analysis, the relevant product information is displayed. For example, if the user holds a "white shirt" up to the camera, a list of that product will be displayed on the TV screen.

[0924] Voice assistant support

[0925] The voice assistant will understand changes in the user's physical condition and any problems they may have through dialogue with the user, and if necessary, will share the information with medical institutions through data sharing means to support the user's health management.

[0926] Use of emotion engine

[0927] The emotion engine recognizes the user's emotions from their voice, facial expressions, and choice of words. For example, if a user says, "I feel tired today," the emotion engine recognizes the user's sense of fatigue. Based on this information, the display unit suggests products that will help them relax (for example, aroma oils or massage machines).

[0928] The emotion engine also allows the voice assistant to adapt its dialogue based on the user's emotions. For example, if the user is feeling down, the voice assistant will offer encouraging and kind words. Conversely, if the user seems happy, it will suggest topics that will pique the user's interest.

[0929] With these components and processing flow, the system of the present invention allows seniors to easily enjoy online shopping, alleviating feelings of loneliness and reducing the risk of fraud. It also provides a more sophisticated user experience by taking user emotions into consideration.

[0930] The processing flow will be explained below.

[0931] Acquiring and analyzing voice input

[0932] Step 1:

[0933] User: Speak to the TV or dedicated device and say, "Find the Mito Komon DVD."

[0934] Step 2:

[0935] Device: Uses the built-in microphone to capture the user's voice and generate audio data.

[0936] Step 3:

[0937] Terminal: Sends the generated voice data to the server via the network.

[0938] Step 4:

[0939] Server: Converts voice data into text data using a speech recognition engine.

[0940] Step 5:

[0941] Server: The converted text data is sent to a natural language processing module, which extracts keywords ("Mito Komon," "DVD," "search").

[0942] Obtaining and displaying product information

[0943] Step 1:

[0944] Server: Executes a search query to the product database based on the extracted keywords and retrieves the corresponding product information.

[0945] Step 2:

[0946] Server: Sends the acquired product information to the terminal via the network.

[0947] Step 3:

[0948] Terminal: Analyzes the received product information and creates a layout for display on the TV screen.

[0949] Step 4:

[0950] Device: Display a list of "Mito Komon DVDs" on the TV screen.

[0951] Product selection and purchase process

[0952] Step 1:

[0953] User: While looking at the displayed product list, say, "I'd like to buy this Mito Komon DVD."

[0954] Step 2:

[0955] Terminal: Captures audio and generates audio data.

[0956] Step 3:

[0957] Terminal: Sends the generated voice data to the server.

[0958] Step 4:

[0959] Server: Analyzes the voice data and identifies the product selected by the user.

[0960] Step 5:

[0961] Server: Confirms the user's selection and adds the selected items to the shopping cart.

[0962] Step 6:

[0963] Server: Generates payment options and presents them to the user through the voice assistant.

[0964] Step 7:

[0965] User: Select "Pay with credit card" by voice.

[0966] Step 8:

[0967] Terminal: Captures the user's voice and sends the voice data to the server.

[0968] Payment Processing

[0969] Step 1:

[0970] Server: Analyzes the voice data and confirms the user's payment method selection.

[0971] Step 2:

[0972] Server: Processes the payment using credit card information.

[0973] Step 3:

[0974] Server: Generates a confirmation message of payment completion and notifies the user via the voice assistant.

[0975] Product search using camera

[0976] Step 1:

[0977] User: Point the camera at the item they want to purchase.

[0978] Step 2:

[0979] Terminal: Uses a camera to capture an image of the product and generate image data.

[0980] Step 3:

[0981] Terminal: Sends the generated image data to the server.

[0982] Step 4:

[0983] Server: Analyzes the image using image recognition algorithms to identify relevant product information.

[0984] Step 5:

[0985] Server: Sends the relevant product information to the terminal.

[0986] Step 6:

[0987] Terminal: Displays the product on the TV screen based on the received product information.

[0988] Voice assistant support

[0989] Step 1:

[0990] Voice assistant: Understands changes in the user's physical condition and concerns through dialogue with the user.

[0991] Step 2:

[0992] Voice assistants: Use data sharing methods as needed to share acquired information with healthcare providers.

[0993] Use of emotion engine

[0994] Step 1:

[0995] User: Say "I'm tired today" aloud.

[0996] Step 2:

[0997] Terminal: Captures the user's voice and generates voice data.

[0998] Step 3:

[0999] Terminal: Sends the generated voice data to the server.

[1000] Step 4:

[1001] Server: Analyzes the voice data and recognizes the user's emotions (in this case, fatigue).

[1002] Step 5:

[1003] Emotion engine: Based on the recognized emotions, it generates recommendation information that suggests appropriate products to the user (e.g., relaxing aroma oils or massage machines).

[1004] Step 6:

[1005] Server: Sends the generated recommendation information to the terminal.

[1006] Step 7:

[1007] Device: The received recommendation information is displayed on the TV screen.

[1008] Step 8:

[1009] Voice assistant: Based on the recognized emotion, it will communicate with the user, saying, "I'd like to introduce some products that will help you relax."

[1010] Step 9:

[1011] Voice assistant: Changes the dialogue content according to the recognized emotions, offering gentle words if the user is tired and lively words and topics if they are energetic.

[1012] These processing steps allow users to easily shop online using voice and images, and receive support in daily life through a voice assistant.Furthermore, by utilizing an emotion engine, it is possible to provide optimal services according to the user's mood and emotions.

[1013] Example 2

[1014] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1015] Today's elderly often find it difficult to shop online due to the complexity of the process. Furthermore, existing systems are unable to address other needs, such as health management and reducing feelings of loneliness, which must also be met. In particular, intuitive operation using voice input and natural language processing, as well as personalized user support using emotion recognition, are needed.

[1016] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a voice input means, a text conversion means for converting voice data into text, a natural language processing means for analyzing the text data and extracting keywords, a product search means for acquiring product information based on the extracted keywords, a display means for displaying the acquired product information, a voice selection means for re-accepting voice input for product selection, a purchase procedure means for proceeding with the purchase of the selected product, a payment means for presenting a payment method and executing the payment, a product recognition means for pointing a camera at the product to be purchased, acquiring image data, and presenting corresponding product information, a data sharing means for the voice assistant to communicate with the user, grasp changes in the user's physical condition and problems, and share the information with an external organization as necessary, and an emotion recognition means for recognizing emotions from the user's voice and facial expressions and taking appropriate action. This allows elderly people to enjoy online shopping intuitively and easily, while at the same time satisfying multiple needs such as health management and reducing loneliness.

[1017] The "voice input means" is a device that has the function of capturing the user's speech as voice data.

[1018] The "text conversion means" is a device that has the function of converting captured voice data into text data.

[1019] A "natural language processing means" is a device that has the function of analyzing text data and extracting user intent and keywords.

[1020] The "product search means" is a device that has the function of retrieving relevant product information from a product database based on the extracted keywords.

[1021] The "display means" is a device that has the function of visually displaying the acquired product information.

[1022] The "voice selection means" is a device that has the function of accepting a second voice input from the user for product selection.

[1023] The "purchase procedure means" is a device that has the function of proceeding with the purchase procedure for the selected product.

[1024] A "payment method" is a device that has the function of presenting a payment method and actually carrying out the payment.

[1025] The "product recognition means" is a device that has the function of analyzing image data acquired by a camera and presenting corresponding product information.

[1026] A "data sharing means" is a device that allows the voice assistant to monitor the user's condition and share information with external medical institutions as necessary.

[1027] The "emotion recognition means" is a device that has the function of recognizing the user's emotions from their voice and facial expressions, and taking appropriate action based on that information.

[1028] This invention is an online shopping system primarily targeted at seniors, which combines voice input and natural language processing to enable intuitive and easy operation. Furthermore, by incorporating emotion recognition and health information sharing functions, it improves the overall user experience, helping to manage health and alleviate feelings of loneliness.

[1029] System configuration

[1030] The system consists of the following main components:

[1031] 1. Voice input method

[1032] 2. Text Conversion Methods

[1033] 3. Natural Language Processing Methods

[1034] 4. Product search methods

[1035] 5. Display means

[1036] 6. Audio Selection Method

[1037] 7. Purchasing Procedures

[1038] 8. Payment Methods

[1039] 9. Product recognition means

[1040] 10. Data sharing methods

[1041] 11. Emotion recognition means

[1042] Operation overview

[1043] Voice input and text conversion methods

[1044] The user speaks to the dedicated device or TV, for example, saying, "Look for a Mito Komon DVD." The device uses a built-in microphone to capture the voice data and converts it into text data using the Google Speech-to-Text API. This text data contains the content, "Look for a Mito Komon DVD."

[1045] Natural language processing tools

[1046] The server receives the text data and analyzes it using a natural language processing engine such as spaCy. As a result of the analysis, keywords such as "Mito Komon," "DVD," and "search for" are extracted.

[1047] Product search and display methods

[1048] The server uses the extracted keywords to execute a search query against the product database via the API of Amazon or Rakuten. The retrieved product information is displayed on the TV screen using HTML5 and JavaScript. For example, a list of "Mito Komon DVDs" is displayed visually to the user.

[1049] Voice selection method, purchase procedure method and payment method

[1050] When the user again says "I'd like to buy this Mito Komon DVD," the device captures the voice and converts it back into text data. The server analyzes this text data and adds the item to the shopping cart. Payment methods are then presented, and if the user selects by voice, for example, "I'd like to pay by credit card," the payment is executed using the Stripe API. If the payment is successful, a confirmation message is sent to the user.

[1051] Product recognition means

[1052] When a user points the camera at a product they want to purchase, the device uses the camera to capture image data. The server then analyzes the image using image recognition services such as Google Cloud Vision and retrieves the corresponding product information. This information is then displayed on the TV screen.

[1053] Data sharing methods

[1054] Through dialogue with the user, the voice assistant will understand changes in the user's physical condition and any problems they may have. If necessary, it will use FHIR (Fast Healthcare Interoperability Resources) to share information with external medical institutions and support the user's health management.

[1055] emotion recognition means

[1056] The emotion recognition means recognizes the user's emotions from their voice and facial expressions. For example, if the user says, "I'm tired today," the emotion recognition means recognizes the user's sense of fatigue, and based on this information, the display means suggests products that will help them relax (such as aroma oils or massage machines). The voice assistant also provides words of encouragement or interesting topics depending on the user's emotional state.

[1057] Examples of concrete examples and prompts

[1058] As a specific example, consider a situation where a user says to the TV, "Look for aroma oil." In this case, the device captures the voice and converts it into text data using the Google Speech-to-Text API. The server extracts the keyword "aroma oil," searches the product database, and displays the results on the TV screen.

[1059] Example prompt sentence:

[1060] "When a user says to the TV, 'I'm looking for aroma oil,' the voice data is captured and converted into text data, and the server uses natural language processing to extract keywords, retrieves the relevant product information from the product database, and displays it on the TV screen."

[1061] This system allows seniors to enjoy online shopping intuitively and easily, while also providing health and emotional care at the same time.

[1062] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1063] Step 1: Getting voice input

[1064] The user speaks to a TV or dedicated terminal, saying, "Find the Mito Komon DVD." The terminal uses a built-in microphone to capture the user's speech as audio data. The input is the user's voice, and the output is the captured audio data.

[1065] Step 2: Text conversion

[1066] The device generates the captured voice data using the Google Speech-to-Text API. The voice data is converted into text data. The input here is the voice data, and the output is the text data, "Looking for the Mito Komon DVD."

[1067] Step 3: Natural Language Processing

[1068] The server receives the text data and analyzes it using tools such as spaCy. During the analysis, keywords are extracted to understand the user's search intent. The input is text data, and the output is keywords such as "Mito Komon," "DVD," and "search."

[1069] Step 4: Product Search

[1070] The server uses the extracted keywords to execute a search query against the product database via the API of Amazon or Rakuten. Here, the process of retrieving related product information based on the keywords is performed. The input is the keyword, and the output is the corresponding product information (for example, a list of Mito Komon DVDs).

[1071] Step 5: View the results

[1072] The server displays the retrieved product information on the TV screen. It uses HTML5 and JavaScript to process the visual display. The input is the product information, and the output is a screen displaying a list of "Mito Komon DVDs."

[1073] Step 6: Product Selection

[1074] The user looks at the displayed product list and says verbally, "I'll buy this Mito Komon DVD." The device again captures the voice and converts the voice data into text data, just like in step 2. The input is the user's voice, and the output is the text data, "I'll buy this Mito Komon DVD."

[1075] Step 7: Checkout

[1076] The server adds the selected items to the shopping cart based on the analysis results. It then displays payment options and prepares the payment. The input is text data, and the output is product information added to the shopping cart and payment options.

[1077] Step 8: Payment

[1078] When the user selects "I'd like to pay with a credit card," the terminal captures the voice and converts it back into text data. The server then executes the payment using the Stripe API. The input is the user's voice and text data, and the output is a message that the payment was successful.

[1079] Step 9: Product Recognition

[1080] When a user points the camera at a product they want to purchase, the device uses the camera to capture image data. The server then analyzes the image using the Google Cloud Vision API and retrieves the corresponding product information. The input is image data, and the output is the corresponding product information.

[1081] Step 10: Data sharing

[1082] Voice assistants communicate with users to understand changes in their physical condition and any concerns they may have. They use FHIR to share information with external medical institutions as needed. The input is the user's voice and situational information, and the output is health information shared with medical institutions.

[1083] Step 11: Emotion Recognition

[1084] Emotion recognition is based on the user's voice and facial expressions. For example, if a user says, "I'm tired today," the system recognizes the user's emotion and suggests an appropriate response. The input is voice and facial expression data, and the output is emotional information and a suggested response based on that information.

[1085] (Application example 2)

[1086] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1087] The traditional product search and purchasing process in physical stores required time-consuming walking around the store to find products, resulting in low usability. It was also difficult to instantly check detailed product information and reviews, and product suggestions based on emotions or physical condition were not provided. Furthermore, there was a lack of a way to provide the same easy payment procedures in physical stores as online. This made shopping a burden, especially for elderly customers and those in poor health.

[1088] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1089] In this invention, the server includes a voice input means, a text conversion means for converting voice data into text, a natural language processing means for analyzing the text data and extracting keywords, a product search means for acquiring product information based on the extracted keywords, a display means for displaying the acquired product information, a voice selection means for re-accepting voice input for product selection, a purchase procedure means for proceeding with the purchase of the selected product, a payment means for presenting payment methods and executing the payment, an emotion recognition means for analyzing user emotions, a data feedback means for analyzing the user's expressions of intent and adjusting the content presented, and an augmented reality display means for providing a user interface using an augmented reality function. This enables an interactive product search and purchase process in a physical store, reduces the burden on users, and provides a more comfortable shopping experience.

[1090] The "voice input means" is a device that has the function of capturing the user's voice and generating voice data.

[1091] The "text conversion means" is a device that has the function of receiving voice data and converting it into text data using a voice recognition engine.

[1092] A "natural language processing means" is a device that has the function of analyzing text data and extracting user intent and keywords.

[1093] The "product search means" is a device that has the function of acquiring relevant product information from a product database based on the extracted keywords.

[1094] The "display means" is a display device for visualizing the acquired product information.

[1095] The "voice selection means" is a device that has the function of accepting voice input again to select a product.

[1096] The "purchase procedure means" is a device that has the function of proceeding with the purchase procedure for the selected product.

[1097] A "payment method" is a device that has the function of presenting a payment method and executing a payment.

[1098] The "emotion recognition means for analyzing user emotions" is a device that has the function of analyzing emotions from the user's voice, facial expressions, and content of statements.

[1099] The "data feedback means for adjusting the content of presentation" is a device that has the function of adapting the display and content of suggestions based on the user's intentions and emotions.

[1100] An "augmented reality display means" is a device that has the function of providing a user interface using augmented reality (AR) technology.

[1101] The "product recognition means" is a device that has the function of analyzing product images acquired by a camera and presenting corresponding products.

[1102] A "data sharing means" is a device that allows the voice assistant to monitor the user's condition and share information with medical institutions as necessary.

[1103] An embodiment of the present invention is a smart shopping assistant system for realizing an interactive product search and purchasing process in a physical store. This system is composed of a voice input means, a voice recognition means, a natural language processing means, a product search means, an augmented reality display means, an emotion recognition means, a data feedback means, and a payment means.

[1104] System Configuration

[1105] Voice Input and Recognition

[1106] The user inputs voice into a device such as smart glasses or a head-mounted display. For example, the user might say, "I'm looking for a red dress." The voice input means captures this voice data and converts it into text data using the text conversion means. Specifically, it uses the Google Speech Recognition API.

[1107] Natural Language Processing and Product Search

[1108] The server receives the text data and analyzes it using an NLP processor (e.g., TextBlob or AnyMLibrary NLP) to extract keywords. Based on these keywords, the product search means queries the database to obtain relevant product information. For example, the keyword "red dress" is used to search the store's inventory database.

[1109] Augmented reality display

[1110] The acquired product information is superimposed on the screen of the smart glasses or head-mounted display by the augmented reality display means, allowing the user to visually check the product information. For example, an image of a "red dress," price information, and reviews are visually displayed.

[1111] Emotion Recognition and Data Feedback

[1112] When displaying retrieved product information, the system uses emotion recognition to analyze the user's emotions from their voice and facial expressions. For example, if a user says, "I'm tired today," the system uses TextBlob to analyze their emotions and adjust the product suggestions. Data feedback is used to optimize product suggestions and displays based on the user's intentions and emotions.

[1113] Purchase process and payment

[1114] The user checks the visually displayed product information and again voice-inputs "I would like to purchase this dress." The system again converts the voice input into text data and confirms the user's selection through natural language processing. The system then proceeds to purchase the selected product using the checkout means, presents the payment method through the payment means, and executes the payment.

[1115] Hardware and software used

[1116] Hardware: smart glasses, head-mounted displays, cameras

[1117] Speech recognition software: Google Speech Recognition API

[1118] Natural Language Processing Software: TextBlob, AnyMLibrary NLP

[1119] Database Connection: Custom Database Module

[1120] Payment Software: Payment Processor Custom Module

[1121] Display software: AR display function

[1122] Specific examples

[1123] For example, in a scenario where a user puts on smart glasses and says, "I'm looking for a red dress," the system converts the speech into text data and uses an NLP processor to extract the keyword "red dress." It then searches for the corresponding product in the store's inventory database and displays the product information on the smart glasses' AR display. The user then says, "I'd like to purchase this dress," and the system proceeds with the purchase process and payment. The system also analyzes the user's emotions from their speech and facial expressions and adjusts the suggestions based on the results.

[1124] Prompt Sentence Examples

[1125] Preprocess the following speech input and extract keywords for product search:

[1126] Dictation: "I'm looking for a red dress."

[1127] Example output: ['red dress', 'search', 'looking for']

[1128] This enables the smart shopping assistant system to realize an interactive product search and purchasing process in a physical store, providing users with a comfortable shopping experience.

[1129] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1130] Step 1:

[1131] Acquiring voice input

[1132] The user inputs a voice command to search for a product into the smart glasses or head-mounted display (e.g., "I'm looking for a red dress"). The device captures this voice data and prepares for voice recognition. The input is voice data, and the output is also voice data. Specifically, the device records the voice through a microphone.

[1133] Step 2:

[1134] Converting audio data to text

[1135] The device converts the captured audio data into text data using the Google Speech Recognition API. The input is audio data and the output is text data. This process involves analyzing the audio waveform and converting it into a corresponding string of characters.

[1136] Step 3:

[1137] Natural language processing of text data

[1138] The server receives the text data and analyzes it using an NLP processor (for example, TextBlob or AnyMLibrary NLP). The input is the text data, and the output is a set of extracted keywords. Specifically, the text data is tokenized and important keywords and phrases are extracted.

[1139] Step 4:

[1140] Search for products

[1141] The server then searches the product database using the extracted keywords. The input is a set of keywords, and the output is the corresponding product information. Specifically, the server generates a database query to retrieve product inventory information and detailed information.

[1142] Step 5:

[1143] Displaying product information

[1144] The device displays the acquired product information using the augmented reality function of smart glasses or a head-mounted display. The input is product information, and the output is a visual presentation of the product information. Specifically, the device uses an AR overlay to display product images, prices, reviews, and more in the user's field of view.

[1145] Step 6:

[1146] Emotion recognition

[1147] The server analyzes the user's voice and facial expression data and recognizes emotions using tools such as TextBlob. The input is the user's voice and facial expression data, and the output is the analysis result of the emotional state. Specifically, it analyzes the voice and facial expression data and evaluates the emotion.

[1148] Step 7:

[1149] Data feedback adjustment

[1150] The server then adjusts the display and recommendations based on the recognized emotions. The input is the emotion analysis results, and the output is adjusted product information and recommendations. Specifically, the server adapts the display content based on the emotion data.

[1151] Step 8:

[1152] Product selection and purchase process

[1153] The user checks the displayed product information and again voice-inputs "I'd like to purchase this dress." The device again captures the voice data and confirms the user's selection through text conversion and natural language processing. The input is the re-captured voice data, and the output is the analysis result of the text data.

[1154] Step 9:

[1155] Purchase process and payment

[1156] The server then checks out the selected product, presents the payment method, and executes the transaction. The input is the selected product information, and the output is confirmation of the transaction. Specific operations include inputting and processing payment information and sending a purchase confirmation.

[1157] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1158] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1159] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1160] [Third embodiment]

[1161] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1162] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1163] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1164] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1165] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1166] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1167] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1168] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1169] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1170] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1171] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1172] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1173] The embodiment of the present invention follows the following configuration and processing flow.

[1174] System Configuration

[1175] The system consists of the following main components:

[1176] 1. Voice input method

[1177] It has the function of capturing the user's voice and generating the voice data.

[1178] 2. Text Conversion Methods

[1179] It has the function of receiving voice data and converting it into text data using a voice recognition engine.

[1180] 3. Natural Language Processing Methods

[1181] It has the ability to analyze text data and extract user intent and keywords.

[1182] 4. Product search methods

[1183] It has the function of retrieving relevant product information from a product database based on the extracted keywords.

[1184] 5. Display means

[1185] It has the function of displaying the acquired product information on a display device such as a television screen.

[1186] 6. Audio Selection Method

[1187] It has the function of accepting additional voice input to select a product.

[1188] 7. Purchasing Procedures

[1189] It has the function of proceeding to the purchase procedure for the selected product.

[1190] 8. Payment Methods

[1191] It has the function of presenting payment methods and executing payments.

[1192] 9. Product recognition means

[1193] It has the function of analyzing product images captured by the camera and presenting corresponding products.

[1194] 10. Data sharing methods

[1195] The voice assistant will monitor the user's condition and share information with medical institutions if necessary.

[1196] Program processing

[1197] Acquiring and analyzing voice input

[1198] The user speaks to a television or dedicated terminal, saying, "Find the Mito Komon DVD." The terminal captures this as voice data and converts it into text data using a text conversion means. The server receives the text data and uses natural language processing means to extract keywords such as "Mito Komon," "DVD," and "search."

[1199] Obtaining and displaying product information

[1200] The server executes a search query against the product database using these keywords. Product information is obtained as a search result and displayed on the television screen through the display means. Specifically, a list of "Mito Komon DVDs" is visually displayed on the user's screen.

[1201] Product selection and purchase process

[1202] The user looks at the displayed product list and again says, "I'd like to buy this Mito Komon DVD." The device again captures the voice data and analyzes it using the procedure described above. The server adds the product to the shopping cart and presents options for confirming the payment method (for example, "I'd like to pay by credit card").

[1203] The user selects "I'll pay by credit card" by voice, and the payment is executed by the payment method. If the payment is successful, a confirmation message is sent to the user.

[1204] Product search using camera

[1205] The user points the camera at the product they want to purchase. The device acquires the image data from the camera and analyzes the image using product recognition means. As a result of the analysis, the relevant product information is displayed. For example, if the user holds a "white shirt" up to the camera, a list of that product will be displayed on the TV screen.

[1206] Voice assistant support

[1207] The voice assistant will understand changes in the user's physical condition and any problems they may have through dialogue with the user, and if necessary, will share the information with medical institutions through data sharing means to support the user's health management.

[1208] With these components and processing flow, the system of the present invention enables elderly people to easily enjoy online shopping, eliminating feelings of loneliness and reducing the risk of fraud.

[1209] The processing flow will be explained below.

[1210] Acquiring and analyzing voice input

[1211] Step 1:

[1212] User: Speak to the TV or dedicated device and say, "Find the Mito Komon DVD."

[1213] Step 2:

[1214] Device: Uses the built-in microphone to capture the user's voice and generate audio data.

[1215] Step 3:

[1216] Terminal: Sends the generated voice data to the server via the network.

[1217] Step 4:

[1218] Server: Converts voice data into text data using a speech recognition engine.

[1219] Step 5:

[1220] Server: The converted text data is sent to a natural language processing module, which extracts keywords ("Mito Komon," "DVD," "search").

[1221] Obtaining and displaying product information

[1222] Step 1:

[1223] Server: Executes a search query to the product database based on the extracted keywords and retrieves the corresponding product information.

[1224] Step 2:

[1225] Server: Sends the acquired product information to the terminal via the network.

[1226] Step 3:

[1227] Terminal: Analyzes the received product information and creates a layout for display on the TV screen.

[1228] Step 4:

[1229] Device: Display a list of "Mito Komon DVDs" on the TV screen.

[1230] Product selection and purchase process

[1231] Step 1:

[1232] User: While looking at the displayed product list, say, "I'd like to buy this Mito Komon DVD."

[1233] Step 2:

[1234] Terminal: Capture the audio again and generate the audio data.

[1235] Step 3:

[1236] Terminal: Sends the generated voice data to the server.

[1237] Step 4:

[1238] Server: Analyzes the voice data and identifies the product selected by the user.

[1239] Step 5:

[1240] Server: Confirms the user's selection and adds the selected items to the shopping cart.

[1241] Step 6:

[1242] Server: Generates payment options and presents them to the user through the voice assistant.

[1243] Step 7:

[1244] User: Select "Pay with credit card" by voice.

[1245] Step 8:

[1246] Terminal: Captures the user's voice and sends the voice data to the server.

[1247] Payment Processing

[1248] Step 1:

[1249] Server: Analyzes the voice data and confirms the user's payment method selection.

[1250] Step 2:

[1251] Server: Processes the payment using credit card information.

[1252] Step 3:

[1253] Server: Generates a confirmation message of payment completion and notifies the user via the voice assistant.

[1254] Product search using camera

[1255] Step 1:

[1256] User: Point the camera at the item they want to purchase.

[1257] Step 2:

[1258] Terminal: Uses a camera to capture an image of the product and generate image data.

[1259] Step 3:

[1260] Terminal: Sends the generated image data to the server.

[1261] Step 4:

[1262] Server: Analyzes the image using image recognition algorithms to identify relevant product information.

[1263] Step 5:

[1264] Server: Sends the relevant product information to the terminal.

[1265] Step 6:

[1266] Terminal: Displays the product on the TV screen based on the received product information.

[1267] Voice assistant support

[1268] Step 1:

[1269] Voice assistant: Understands changes in the user's physical condition and concerns through dialogue with the user.

[1270] Step 2:

[1271] Voice assistants: Use data sharing methods as needed to share acquired information with healthcare providers.

[1272] These processing steps enable users to easily shop online using voice and images, and receive support in their daily lives through a voice assistant.

[1273] Example 1

[1274] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1275] In today's world, users, especially the elderly and those with little technical knowledge, can find it difficult to select and purchase products over the Internet due to the complex operations and procedures involved. Furthermore, there are issues such as the risk of selecting the wrong product during the product selection process and the lack of means to accurately share health management information.

[1276] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1277] In this invention, the server includes an input means for accepting voice input, a conversion means for converting voice data into text, an analysis means for analyzing the text data and extracting keywords, a search means for acquiring information based on the extracted keywords, a display means for displaying the acquired information, an input means for accepting further voice input, a processing means for advancing the selected information to processing, and a payment means for presenting payment methods and executing payment. This allows users to easily search, select, and purchase products via voice input, and also provides a means for appropriately sharing health management information.

[1278] "Voice input" means a device or function for capturing a user's voice and inputting it into the system.

[1279] The "conversion" means is software or hardware for converting the captured voice data into text data.

[1280] The "analysis" means software or algorithms that analyze the converted text data and extract the user's intent and specific keywords.

[1281] The "search" means is a function for retrieving related information from databases and external information sources based on the extracted keywords.

[1282] "Display" means refers to a device or function for visually presenting acquired information to a user, and includes a television screen, monitor, and the like.

[1283] The "procedure" means is a function for proceeding to the next processing step, particularly the purchase procedure, based on the information selected by the user.

[1284] The "payment" means is a function for presenting a payment method to a user and processing the actual payment.

[1285] "Recognition" means analyzing image data acquired using a camera or other device, and is a function for identifying and recognizing specific products or objects.

[1286] The "sharing" means is a function for sharing the user's health information and status in cooperation with external institutions such as medical institutions.

[1287] A "voice assistant" is an interactive artificial intelligence that provides various services through voice interaction with users.

[1288] In the embodiment of the present invention, a system configured using the following main hardware and software will be described.

[1289] System Configuration

[1290] 1. Voice input method

[1291] The device is equipped with a microphone to capture the user's voice. Specifically, a dedicated voice capture device (such as a smart speaker) is used.

[1292] 2. A means of converting voice data into text

[1293] The device converts the captured audio data into text in real time using voice recognition software such as the Google Cloud Speech-to-Text API.

[1294] 3. Analysis method for analyzing text data and extracting keywords

[1295] The server receives the text data and analyzes it using a natural language processing engine (e.g., AWS Comprehend). As a result of the analysis, the user's intent and keywords are extracted.

[1296] 4. Search methods to obtain information based on extracted keywords

[1297] The server uses the extracted keywords to query a database that contains product information and other related information.

[1298] 5. Display method for displaying acquired information

[1299] The terminal displays the information obtained from the server on a television screen or a dedicated display, for example, visually displaying a product list to the user.

[1300] 6. A voice selection means for accepting voice input again

[1301] The user looks at the displayed information and again makes selections or commands through voice input, which the device captures and converts again into text data.

[1302] 7. Procedural means for forwarding selected information for processing

[1303] The server then uses the selected information to complete the purchase, adding the product to the shopping cart and prompting for payment.

[1304] 8. Payment methods for submitting payment methods and executing payments

[1305] The server presents the user with multiple payment options (credit card, bank transfer, etc.) and executes the payment using the selected payment method, using a payment service such as Stripe or PayPal.

[1306] 9. A recognition method for analyzing product images captured by a camera and presenting relevant product information

[1307] The device is equipped with a camera that captures image data of the product the user points the camera at, which is then analyzed using image recognition software such as Google Cloud Vision API to search for relevant product information.

[1308] 10. Means for sharing user health information and status with medical institutions

[1309] The voice assistant will constantly monitor the user's health status through daily conversations with them, and if necessary, share information with medical institutions through data sharing methods to support the user's health management.

[1310] Specific examples

[1311] Let's take the example of a user speaking into a TV, saying, "Find a Mito Komon DVD." The device (e.g., a smart speaker) captures the user's voice and converts it into text data using commercially available voice recognition software. The text data is sent to the server in the format "Find a Mito Komon DVD." The server analyzes this text data using a natural language processing engine and extracts the keywords "Mito Komon," "DVD," and "search." It then queries a product database using a search tool to obtain relevant product information. The obtained information is then displayed on the TV screen via the device. The user then uses the displayed information to issue a purchase instruction again using voice input, and the server then proceeds with the purchase process based on this, ultimately completing the payment.

[1312] Prompt Sentence Examples

[1313] "Please explain the process of your voice search system. Please provide a detailed explanation of the steps a user takes to search for, select, and purchase a product using voice. Please also specify what software and hardware are used."

[1314] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1315] Step 1:

[1316] The user speaks to a TV or dedicated terminal, saying, "Find the Mito Komon DVD." The input voice is recognized by the system when the user presses a button on the remote control to switch to voice input mode, which generates voice data.

[1317] Step 2:

[1318] The device captures the user's voice data using a microphone. Specifically, the voice is recorded using a dedicated voice capture device (such as a smart speaker) and saved as digital data. The input is voice data, and the output is digital voice data.

[1319] Step 3:

[1320] The device sends the captured voice data in real time to a conversion mechanism using the Google Cloud Speech-to-Text API, which takes digital voice data as input and outputs text data.

[1321] Step 4:

[1322] The server receives the text data sent from the text conversion means. The input is text data, and this text data is in the format "Looking for a Mito Komon DVD." The output is the same text data.

[1323] Step 5:

[1324] The server runs the text data through an analysis tool and analyzes it using a natural language processing engine (e.g., AWS Comprehend). The input is text data, and through this analysis, the keywords "Mito Komon," "DVD," and "search for" are extracted. The output is the extracted keywords.

[1325] Step 6:

[1326] The server executes a search query to a database using a search engine based on the extracted keywords. Specifically, it executes an SQL query to a product database (e.g., a MySQL database). The input is the extracted keywords, and the output is the corresponding product information.

[1327] Step 7:

[1328] The server sends the product information obtained as a search result to the display means. The obtained product information is sent in HTML format and converted into a data format for display on the screen. The input is the product information, and the output is the display data.

[1329] Step 8:

[1330] The terminal uses the received information to display a product list on the TV screen. Specifically, the information is visualized using a display device connected to the TV or a dedicated display. The input is the data to be displayed, and the output is the displayed product list.

[1331] Step 9:

[1332] The user looks at the displayed product list and again inputs by voice, "I'll buy this Mito Komon DVD." Specific actions include the user specifying a specific product number or name. The input is voice data, and the output is a selection instruction.

[1333] Step 10:

[1334] The terminal captures the voice data again and converts it into text using the conversion means described above. The input is voice data and the output is text data.

[1335] Step 11:

[1336] The server receives the text data regarding the product selection and analyzes the content. The analysis identifies the intention to purchase, such as "I will buy," and the selected product information. The input is the text data, and the output is the analysis result.

[1337] Step 12:

[1338] The server adds the selected items to the shopping cart via a procedure and presents payment options, such as "Do you want to pay by credit card?". The input is the analysis result, and the output is the payment options.

[1339] Step 13:

[1340] The user selects the desired payment method by voice, specifically by saying, "I'll pay by credit card."

[1341] Step 14:

[1342] The device captures the voice data again, converts it into text data, and sends it to the server. The input is voice data, and the output is text data.

[1343] Step 15:

[1344] The server executes the payment via a payment method such as Stripe or PayPal. The input is text data (payment instructions) and the output is payment confirmation.

[1345] Step 16:

[1346] The server sends a confirmation message of successful payment to the display means to notify the user. The input is the payment confirmation, and the output is the display data.

[1347] Step 17:

[1348] The terminal displays the received confirmation message on the TV screen. Specifically, the message "Purchase completed" is displayed on the screen. The input is the data to be displayed, and the output is the display of the confirmation message.

[1349] Step 18:

[1350] The user points the camera at the product they want to purchase. Specifically, they use a webcam connected to a TV. The input is a product image, and the output is image data.

[1351] Step 19:

[1352] The device acquires camera image data and sends it to the recognition means. The Google Cloud Vision API is used for recognition. The input is image data, and the output is analysis data.

[1353] Step 20:

[1354] The server analyzes the received image data and identifies the corresponding product. Product information is searched based on the analysis results. The input is the analyzed data, and the output is product information.

[1355] Step 21:

[1356] The server uses a display unit to display product information based on the analysis results on a television screen. Specifically, a list of "white shirt" products is displayed on the television screen. The input is product information, and the output is display data.

[1357] Step 22:

[1358] The voice assistant periodically asks the user about their health status and understands the situation through dialogue. The input is the user's voice data, and the output is analysis data.

[1359] Step 23:

[1360] The voice assistant analyzes the user's responses using a shared method and shares the information with medical institutions as needed. The input is the analyzed data, and the output is the shared data.

[1361] (Application example 1)

[1362] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1363] There is a need to reduce the complexity and anxiety that seniors face when shopping online, particularly by enabling them to search for and purchase products through intuitive voice commands. Camera-based product recognition and easy voice-based purchasing procedures are also needed. Furthermore, it is also important for voice assistants to connect appropriate services and information to support the health and needs of seniors.

[1364] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1365] In this invention, the server includes a voice input unit, a text conversion unit that converts voice data into text, a natural language processing unit that analyzes the text data and extracts keywords, a product search unit that acquires product information based on the extracted keywords, a display unit that displays the acquired product information, a voice selection unit that accepts voice input for product selection again, a purchase procedure unit that proceeds with the purchase of the selected product, a payment unit that presents a payment method and executes the payment, an image input unit for product recognition, and a product recognition unit that analyzes product images and presents corresponding product information. This allows elderly people to intuitively search for and purchase products using voice control or image recognition, providing ease of operation and safety. Furthermore, support using a voice assistant or data sharing unit provides appropriate responses according to the user's health condition and requests.

[1366] The "voice input means" is a device that has the function of capturing the user's voice and generating voice data.

[1367] The "text conversion means" is a means for converting voice data into text, and is a device that uses voice recognition technology.

[1368] "Natural language processing means" is a technical means for analyzing text data and extracting user intent and keywords.

[1369] The "product search means" is a means for searching and acquiring product information based on the extracted keywords.

[1370] "Display means" refers to a device for visually displaying acquired product information, including the screen of a smartphone and the HUD of smart glasses.

[1371] The "voice selection means" is a means having a function of accepting voice input again to select a product.

[1372] The "purchase procedure means" is a means for proceeding with the purchase procedure for the selected product.

[1373] A "payment method" is a method for presenting a payment method and actually making a payment.

[1374] "Image input means" refers to a means for acquiring product images using a camera or other sensor.

[1375] The "product recognition means" is a means for analyzing the product image acquired by the image input means and presenting the corresponding product information.

[1376] A "voice assistant" is software that understands the user's requests and status through dialogue with the user, and responds and provides information as needed.

[1377] "Data sharing means" refers to means that has the function of sharing user information with other service providers as needed.

[1378] The system embodying this invention performs a comprehensive process from voice input to text conversion, natural language processing, product search, display, product recognition, voice selection, purchase procedure, payment, and data sharing. Each component and its processing method are described in detail below.

[1379] System Configuration

[1380] The system includes the following main components:

[1381] 1. Voice input means: A means for capturing the user's voice and generating the voice data. Specifically, it uses a microphone built into a smartphone or smart glasses.

[1382] 2. Text conversion method: A method for converting voice data into text using voice recognition technology. Google Cloud Speech-to-Text API is used.

[1383] 3. Natural language processing: A means of analyzing text data and extracting keywords and user intent. Google Cloud Natural Language API is used.

[1384] 4. Product search method: This is a method to obtain product information based on extracted keywords. Firebase is used as the product database.

[1385] 5. Display means: A means for visually displaying the acquired product information. This includes the screen of a smartphone and the HUD of smart glasses.

[1386] 6. Voice selection means: A means for accepting voice input again to select a product.

[1387] 7. Purchasing procedure: A procedure for proceeding to purchase the selected product.

[1388] 8. Payment Method: A method for submitting a payment method and executing a payment. Uses the Stripe API.

[1389] 9. Image input means: A means of acquiring product images using a camera or other sensor.

[1390] 10. Product recognition means: A means for analyzing product images acquired by the image input means and presenting corresponding product information.

[1391] 11. Data Sharing Measures: These measures have the ability to share user information with other service providers as needed.

[1392] Program processing

[1393] The data processing and calculation flow at each step of this system will be explained below.

[1394] 1. Acquiring and parsing voice input:

[1395] A user speaks to a smartphone or smart glasses, saying, "I want to buy milk." The device captures this as audio data and converts it into text using the Google Cloud Speech-to-Text API. The server receives the text data and extracts keywords such as "milk" and "I want to buy" using the Google Cloud Natural Language API.

[1396] 2. Obtaining and displaying product information:

[1397] The server executes a search query against the product database (Firebase) using these keywords. Product information is retrieved as a search result and displayed on the smartphone screen or smart glasses HUD through a display means. For example, a list of "Milk" products is visually displayed on the user's screen.

[1398] 3. Product selection and purchase procedure:

[1399] While looking at the displayed list of products, the user again commands by voice, "Add the third milk to the cart." The device again captures the voice data and analyzes it using the procedure described above. The server adds the product to the shopping cart and presents options to confirm the payment method. The user selects by voice, "Pay with credit card," and the payment is executed via the Stripe API.

[1400] 4. Search for products using your camera:

[1401] The user points the camera at the product they want to purchase. The device acquires the camera's image data and analyzes the image using product recognition means. As a result of the analysis, the relevant product information is displayed. For example, if the user holds a "white shirt" up to the camera, a list of that product will be displayed on the smartphone screen or smart glasses' HUD.

[1402] 5. Voice assistant support:

[1403] Voice assistants will understand changes in the user's physical condition and requests through dialogue with the user, and share information with other service providers as needed through data sharing means, enabling them to manage the user's health and respond to their requests.

[1404] Examples of concrete examples and prompts

[1405] Examples:

[1406] When an elderly person says to their smartphone, "I want to buy milk," the system retrieves a list of milk products from Firebase and displays it on the screen. Then, when they say, "Add the third milk to my cart," the product is added to their cart. Finally, when they say, "Pay with credit card," the payment is completed using the Stripe API.

[1407] Example prompt sentence:

[1408] I want to buy milk.

[1409] Add a third milk to your cart.

[1410] Pay by credit card.

[1411] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1412] Step 1:

[1413] The user utters "I want to buy milk" via a voice input means. The device (smartphone or smart glasses) captures the voice data through a microphone.

[1414] Input: User's voice

[1415] Output: Audio data

[1416] Step 2:

[1417] The captured voice data is converted to text data by a text conversion means on the device using the Google Cloud Speech-to-Text API.

[1418] Input: Audio data

[1419] Output: Text data (e.g. "I want to buy milk")

[1420] Step 3:

[1421] The server receives the text data and analyzes it using natural language processing means with the Google Cloud Natural Language API to extract the keyword "milk" and the intent "want to buy."

[1422] Input: Text data

[1423] Output: Keywords (e.g., "milk"), Intent (e.g., "I want to buy")

[1424] Step 4:

[1425] Based on the extracted keywords, the server executes a search query for the relevant product information from the Firebase database via the product search means, and obtains a list of milk products as search results.

[1426] Input: Keyword (e.g. "milk")

[1427] Output: List of product information (e.g., list of milk products)

[1428] Step 5:

[1429] The acquired product information is displayed on the smartphone screen or smart glasses HUD through the device's display means, allowing the user to visually check the list of milk options.

[1430] Input: List of product information

[1431] Output: Visually displayed product information

[1432] Step 6:

[1433] The user instructs via voice input means to "add the third milk to the cart." The terminal again captures the voice data and converts it into text data.

[1434] Input: User's voice

[1435] Output: New audio data

[1436] Step 7:

[1437] The add-to-cart instruction is similarly converted to text, and the server parses the instruction and adds the third milk to the shopping cart.

[1438] Input: Speech data (e.g., "Add the third milk to my cart.")

[1439] Output: Updated shopping cart state

[1440] Step 8:

[1441] The server presents a payment method via the payment method (e.g., credit card). The user then voice-instructs "Pay with credit card," and the payment is executed using the Stripe API.

[1442] Input: Product information, payment method selection

[1443] Output: Payment confirmation, payment completion notification

[1444] Step 9:

[1445] The user takes a photo of a product with a camera as an additional product search. The terminal acquires the product image with the image input means and analyzes the corresponding product information using the product recognition means.

[1446] Input: Product image

[1447] Output: Parsed product information

[1448] Step 10:

[1449] The analyzed product information is displayed on the smartphone screen or smart glasses HUD through the terminal's display means.

[1450] Input: Parsed product information

[1451] Output: Visually displayed product information

[1452] Step 11:

[1453] In terms of health and request support, the voice assistant will understand the user's requests and status through dialogue with the user, and share information with other service providers through data sharing means as needed.

[1454] Input: Interaction data

[1455] Output: Information shared, suggested actions

[1456] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1457] The embodiment of the present invention follows the following configuration and processing flow.

[1458] System Configuration

[1459] The system consists of the following main components:

[1460] 1. Voice input method

[1461] It has the function of capturing the user's voice and generating the voice data.

[1462] 2. Text Conversion Methods

[1463] It has the function of receiving voice data and converting it into text data using a voice recognition engine.

[1464] 3. Natural Language Processing Methods

[1465] It has the ability to analyze text data and extract user intent and keywords.

[1466] 4. Product search methods

[1467] It has the function of retrieving relevant product information from a product database based on the extracted keywords.

[1468] 5. Display means

[1469] It has the function of displaying the acquired product information on a display device such as a television screen.

[1470] 6. Audio Selection Method

[1471] It has the function of accepting additional voice input to select a product.

[1472] 7. Purchasing Procedures

[1473] It has the function of proceeding to the purchase procedure for the selected product.

[1474] 8. Payment Methods

[1475] It has the function of presenting payment methods and executing payments.

[1476] 9. Product recognition means

[1477] It has the function of analyzing product images captured by the camera and presenting corresponding products.

[1478] 10. Data sharing methods

[1479] The voice assistant will monitor the user's condition and share information with medical institutions if necessary.

[1480] 11. Emotion Engine

[1481] It has the ability to recognize the user's emotions and respond appropriately based on them.

[1482] Program processing

[1483] Acquiring and analyzing voice input

[1484] The user speaks to a television or dedicated terminal, saying, "Find the Mito Komon DVD." The terminal captures this as voice data and converts it into text data using a text conversion means. The server receives the text data and uses natural language processing means to extract keywords such as "Mito Komon," "DVD," and "search."

[1485] Obtaining and displaying product information

[1486] The server executes a search query against the product database using these keywords. Product information is obtained as a search result and displayed on the television screen through the display means. Specifically, a list of "Mito Komon DVDs" is visually displayed on the user's screen.

[1487] Product selection and purchase process

[1488] The user looks at the displayed product list and again says, "I'd like to buy this Mito Komon DVD." The device again captures the voice data and analyzes it using the procedure described above. The server adds the product to the shopping cart and presents options for confirming the payment method (for example, "I'd like to pay by credit card").

[1489] The user selects "I'll pay by credit card" by voice, and the payment is executed by the payment method. If the payment is successful, a confirmation message is sent to the user.

[1490] Product search using camera

[1491] The user points the camera at the product they want to purchase. The device acquires the image data from the camera and analyzes the image using product recognition means. As a result of the analysis, the relevant product information is displayed. For example, if the user holds a "white shirt" up to the camera, a list of that product will be displayed on the TV screen.

[1492] Voice assistant support

[1493] The voice assistant will understand changes in the user's physical condition and any problems they may have through dialogue with the user, and if necessary, will share the information with medical institutions through data sharing means to support the user's health management.

[1494] Use of emotion engine

[1495] The emotion engine recognizes the user's emotions from their voice, facial expressions, and choice of words. For example, if a user says, "I feel tired today," the emotion engine recognizes the user's sense of fatigue. Based on this information, the display unit suggests products that will help them relax (for example, aroma oils or massage machines).

[1496] The emotion engine also allows the voice assistant to adapt its dialogue based on the user's emotions. For example, if the user is feeling down, the voice assistant will offer encouraging and kind words. Conversely, if the user seems happy, it will suggest topics that will pique the user's interest.

[1497] With these components and processing flow, the system of the present invention allows seniors to easily enjoy online shopping, alleviating feelings of loneliness and reducing the risk of fraud. It also provides a more sophisticated user experience by taking user emotions into consideration.

[1498] The processing flow will be explained below.

[1499] Acquiring and analyzing voice input

[1500] Step 1:

[1501] User: Speak to the TV or dedicated device and say, "Find the Mito Komon DVD."

[1502] Step 2:

[1503] Device: Uses the built-in microphone to capture the user's voice and generate audio data.

[1504] Step 3:

[1505] Terminal: Sends the generated voice data to the server via the network.

[1506] Step 4:

[1507] Server: Converts voice data into text data using a speech recognition engine.

[1508] Step 5:

[1509] Server: The converted text data is sent to a natural language processing module, which extracts keywords ("Mito Komon," "DVD," "search").

[1510] Obtaining and displaying product information

[1511] Step 1:

[1512] Server: Executes a search query to the product database based on the extracted keywords and retrieves the corresponding product information.

[1513] Step 2:

[1514] Server: Sends the acquired product information to the terminal via the network.

[1515] Step 3:

[1516] Terminal: Analyzes the received product information and creates a layout for display on the TV screen.

[1517] Step 4:

[1518] Device: Display a list of "Mito Komon DVDs" on the TV screen.

[1519] Product selection and purchase process

[1520] Step 1:

[1521] User: While looking at the displayed product list, say, "I'd like to buy this Mito Komon DVD."

[1522] Step 2:

[1523] Terminal: Captures audio and generates audio data.

[1524] Step 3:

[1525] Terminal: Sends the generated voice data to the server.

[1526] Step 4:

[1527] Server: Analyzes the voice data and identifies the product selected by the user.

[1528] Step 5:

[1529] Server: Confirms the user's selection and adds the selected items to the shopping cart.

[1530] Step 6:

[1531] Server: Generates payment options and presents them to the user through the voice assistant.

[1532] Step 7:

[1533] User: Select "Pay with credit card" by voice.

[1534] Step 8:

[1535] Terminal: Captures the user's voice and sends the voice data to the server.

[1536] Payment Processing

[1537] Step 1:

[1538] Server: Analyzes the voice data and confirms the user's payment method selection.

[1539] Step 2:

[1540] Server: Processes the payment using credit card information.

[1541] Step 3:

[1542] Server: Generates a confirmation message of payment completion and notifies the user via the voice assistant.

[1543] Product search using camera

[1544] Step 1:

[1545] User: Point the camera at the item they want to purchase.

[1546] Step 2:

[1547] Terminal: Uses a camera to capture an image of the product and generate image data.

[1548] Step 3:

[1549] Terminal: Sends the generated image data to the server.

[1550] Step 4:

[1551] Server: Analyzes the image using image recognition algorithms to identify relevant product information.

[1552] Step 5:

[1553] Server: Sends the relevant product information to the terminal.

[1554] Step 6:

[1555] Terminal: Displays the product on the TV screen based on the received product information.

[1556] Voice assistant support

[1557] Step 1:

[1558] Voice assistant: Understands changes in the user's physical condition and concerns through dialogue with the user.

[1559] Step 2:

[1560] Voice assistants: Use data sharing methods as needed to share acquired information with healthcare providers.

[1561] Use of emotion engine

[1562] Step 1:

[1563] User: Say "I'm tired today" aloud.

[1564] Step 2:

[1565] Terminal: Captures the user's voice and generates voice data.

[1566] Step 3:

[1567] Terminal: Sends the generated voice data to the server.

[1568] Step 4:

[1569] Server: Analyzes the voice data and recognizes the user's emotions (in this case, fatigue).

[1570] Step 5:

[1571] Emotion engine: Based on the recognized emotions, it generates recommendation information that suggests appropriate products to the user (e.g., relaxing aroma oils or massage machines).

[1572] Step 6:

[1573] Server: Sends the generated recommendation information to the terminal.

[1574] Step 7:

[1575] Device: The received recommendation information is displayed on the TV screen.

[1576] Step 8:

[1577] Voice assistant: Based on the recognized emotion, it will communicate with the user, saying, "I'd like to introduce some products that will help you relax."

[1578] Step 9:

[1579] Voice assistant: Changes the dialogue content according to the recognized emotions, offering gentle words if the user is tired and lively words and topics if they are energetic.

[1580] These processing steps allow users to easily shop online using voice and images, and receive support in daily life through a voice assistant.Furthermore, by utilizing an emotion engine, it is possible to provide optimal services according to the user's mood and emotions.

[1581] Example 2

[1582] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1583] Today's elderly often find it difficult to shop online due to the complexity of the process. Furthermore, existing systems are unable to address other needs, such as health management and reducing feelings of loneliness, which must also be met. In particular, intuitive operation using voice input and natural language processing, as well as personalized user support using emotion recognition, are needed.

[1584] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a voice input means, a text conversion means for converting voice data into text, a natural language processing means for analyzing the text data and extracting keywords, a product search means for acquiring product information based on the extracted keywords, a display means for displaying the acquired product information, a voice selection means for re-accepting voice input for product selection, a purchase procedure means for proceeding with the purchase of the selected product, a payment means for presenting a payment method and executing the payment, a product recognition means for pointing a camera at the product to be purchased, acquiring image data, and presenting corresponding product information, a data sharing means for the voice assistant to communicate with the user, grasp changes in the user's physical condition and problems, and share the information with an external organization as necessary, and an emotion recognition means for recognizing emotions from the user's voice and facial expressions and taking appropriate action. This allows elderly people to enjoy online shopping intuitively and easily, while at the same time satisfying multiple needs such as health management and reducing loneliness.

[1585] The "voice input means" is a device that has the function of capturing the user's speech as voice data.

[1586] The "text conversion means" is a device that has the function of converting captured voice data into text data.

[1587] A "natural language processing means" is a device that has the function of analyzing text data and extracting user intent and keywords.

[1588] The "product search means" is a device that has the function of retrieving relevant product information from a product database based on the extracted keywords.

[1589] The "display means" is a device that has the function of visually displaying the acquired product information.

[1590] The "voice selection means" is a device that has the function of accepting a second voice input from the user for product selection.

[1591] The "purchase procedure means" is a device that has the function of proceeding with the purchase procedure for the selected product.

[1592] A "payment method" is a device that has the function of presenting a payment method and actually carrying out the payment.

[1593] The "product recognition means" is a device that has the function of analyzing image data acquired by a camera and presenting corresponding product information.

[1594] A "data sharing means" is a device that allows the voice assistant to monitor the user's condition and share information with external medical institutions as necessary.

[1595] The "emotion recognition means" is a device that has the function of recognizing the user's emotions from their voice and facial expressions, and taking appropriate action based on that information.

[1596] This invention is an online shopping system primarily targeted at seniors, which combines voice input and natural language processing to enable intuitive and easy operation. Furthermore, by incorporating emotion recognition and health information sharing functions, it improves the overall user experience, helping to manage health and alleviate feelings of loneliness.

[1597] System configuration

[1598] The system consists of the following main components:

[1599] 1. Voice input method

[1600] 2. Text Conversion Methods

[1601] 3. Natural Language Processing Methods

[1602] 4. Product search methods

[1603] 5. Display means

[1604] 6. Audio Selection Method

[1605] 7. Purchasing Procedures

[1606] 8. Payment Methods

[1607] 9. Product recognition means

[1608] 10. Data sharing methods

[1609] 11. Emotion recognition means

[1610] Operation overview

[1611] Voice input and text conversion methods

[1612] The user speaks to the dedicated device or TV, for example, saying, "Look for a Mito Komon DVD." The device uses a built-in microphone to capture the voice data and converts it into text data using the Google Speech-to-Text API. This text data contains the content, "Look for a Mito Komon DVD."

[1613] Natural language processing tools

[1614] The server receives the text data and analyzes it using a natural language processing engine such as spaCy. As a result of the analysis, keywords such as "Mito Komon," "DVD," and "search for" are extracted.

[1615] Product search and display methods

[1616] The server uses the extracted keywords to execute a search query against the product database via the API of Amazon or Rakuten. The retrieved product information is displayed on the TV screen using HTML5 and JavaScript. For example, a list of "Mito Komon DVDs" is displayed visually to the user.

[1617] Voice selection method, purchase procedure method and payment method

[1618] When the user again says "I'd like to buy this Mito Komon DVD," the device captures the voice and converts it back into text data. The server analyzes this text data and adds the item to the shopping cart. Payment methods are then presented, and if the user selects by voice, for example, "I'd like to pay by credit card," the payment is executed using the Stripe API. If the payment is successful, a confirmation message is sent to the user.

[1619] Product recognition means

[1620] When a user points the camera at a product they want to purchase, the device uses the camera to capture image data. The server then analyzes the image using image recognition services such as Google Cloud Vision and retrieves the corresponding product information. This information is then displayed on the TV screen.

[1621] Data sharing methods

[1622] Through dialogue with the user, the voice assistant will understand changes in the user's physical condition and any problems they may have. If necessary, it will use FHIR (Fast Healthcare Interoperability Resources) to share information with external medical institutions and support the user's health management.

[1623] emotion recognition means

[1624] The emotion recognition means recognizes the user's emotions from their voice and facial expressions. For example, if the user says, "I'm tired today," the emotion recognition means recognizes the user's sense of fatigue, and based on this information, the display means suggests products that will help them relax (such as aroma oils or massage machines). The voice assistant also provides words of encouragement or interesting topics depending on the user's emotional state.

[1625] Examples of concrete examples and prompts

[1626] As a specific example, consider a situation where a user says to the TV, "Look for aroma oil." In this case, the device captures the voice and converts it into text data using the Google Speech-to-Text API. The server extracts the keyword "aroma oil," searches the product database, and displays the results on the TV screen.

[1627] Example prompt sentence:

[1628] "When a user says to the TV, 'I'm looking for aroma oil,' the voice data is captured and converted into text data, and the server uses natural language processing to extract keywords, retrieves the relevant product information from the product database, and displays it on the TV screen."

[1629] This system allows seniors to enjoy online shopping intuitively and easily, while also providing health and emotional care at the same time.

[1630] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1631] Step 1: Getting voice input

[1632] The user speaks to a TV or dedicated terminal, saying, "Find the Mito Komon DVD." The terminal uses a built-in microphone to capture the user's speech as audio data. The input is the user's voice, and the output is the captured audio data.

[1633] Step 2: Text conversion

[1634] The device generates the captured voice data using the Google Speech-to-Text API. The voice data is converted into text data. The input here is the voice data, and the output is the text data, "Looking for the Mito Komon DVD."

[1635] Step 3: Natural Language Processing

[1636] The server receives the text data and analyzes it using tools such as spaCy. During the analysis, keywords are extracted to understand the user's search intent. The input is text data, and the output is keywords such as "Mito Komon," "DVD," and "search."

[1637] Step 4: Product Search

[1638] The server uses the extracted keywords to execute a search query against the product database via the API of Amazon or Rakuten. Here, the process of retrieving related product information based on the keywords is performed. The input is the keyword, and the output is the corresponding product information (for example, a list of Mito Komon DVDs).

[1639] Step 5: View the results

[1640] The server displays the retrieved product information on the TV screen. It uses HTML5 and JavaScript to process the visual display. The input is the product information, and the output is a screen displaying a list of "Mito Komon DVDs."

[1641] Step 6: Product Selection

[1642] The user looks at the displayed product list and says verbally, "I'll buy this Mito Komon DVD." The device again captures the voice and converts the voice data into text data, just like in step 2. The input is the user's voice, and the output is the text data, "I'll buy this Mito Komon DVD."

[1643] Step 7: Checkout

[1644] The server adds the selected items to the shopping cart based on the analysis results. It then displays payment options and prepares the payment. The input is text data, and the output is product information added to the shopping cart and payment options.

[1645] Step 8: Payment

[1646] When the user selects "I'd like to pay with a credit card," the terminal captures the voice and converts it back into text data. The server then executes the payment using the Stripe API. The input is the user's voice and text data, and the output is a message that the payment was successful.

[1647] Step 9: Product Recognition

[1648] When a user points the camera at a product they want to purchase, the device uses the camera to capture image data. The server then analyzes the image using the Google Cloud Vision API and retrieves the corresponding product information. The input is image data, and the output is the corresponding product information.

[1649] Step 10: Data sharing

[1650] Voice assistants communicate with users to understand changes in their physical condition and any concerns they may have. They use FHIR to share information with external medical institutions as needed. The input is the user's voice and situational information, and the output is health information shared with medical institutions.

[1651] Step 11: Emotion Recognition

[1652] Emotion recognition is based on the user's voice and facial expressions. For example, if a user says, "I'm tired today," the system recognizes the user's emotion and suggests an appropriate response. The input is voice and facial expression data, and the output is emotional information and a suggested response based on that information.

[1653] (Application example 2)

[1654] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1655] The traditional product search and purchasing process in physical stores required time-consuming walking around the store to find products, resulting in low usability. It was also difficult to instantly check detailed product information and reviews, and product suggestions based on emotions or physical condition were not provided. Furthermore, there was a lack of a way to provide the same easy payment procedures in physical stores as online. This made shopping a burden, especially for elderly customers and those in poor health.

[1656] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1657] In this invention, the server includes a voice input means, a text conversion means for converting voice data into text, a natural language processing means for analyzing the text data and extracting keywords, a product search means for acquiring product information based on the extracted keywords, a display means for displaying the acquired product information, a voice selection means for re-accepting voice input for product selection, a purchase procedure means for proceeding with the purchase of the selected product, a payment means for presenting payment methods and executing the payment, an emotion recognition means for analyzing user emotions, a data feedback means for analyzing the user's expressions of intent and adjusting the content presented, and an augmented reality display means for providing a user interface using an augmented reality function. This enables an interactive product search and purchase process in a physical store, reduces the burden on users, and provides a more comfortable shopping experience.

[1658] The "voice input means" is a device that has the function of capturing the user's voice and generating voice data.

[1659] The "text conversion means" is a device that has the function of receiving voice data and converting it into text data using a voice recognition engine.

[1660] A "natural language processing means" is a device that has the function of analyzing text data and extracting user intent and keywords.

[1661] The "product search means" is a device that has the function of acquiring relevant product information from a product database based on the extracted keywords.

[1662] The "display means" is a display device for visualizing the acquired product information.

[1663] The "voice selection means" is a device that has the function of accepting voice input again to select a product.

[1664] The "purchase procedure means" is a device that has the function of proceeding with the purchase procedure for the selected product.

[1665] A "payment method" is a device that has the function of presenting a payment method and executing a payment.

[1666] The "emotion recognition means for analyzing user emotions" is a device that has the function of analyzing emotions from the user's voice, facial expressions, and content of statements.

[1667] The "data feedback means for adjusting the content of presentation" is a device that has the function of adapting the display and content of suggestions based on the user's intentions and emotions.

[1668] An "augmented reality display means" is a device that has the function of providing a user interface using augmented reality (AR) technology.

[1669] The "product recognition means" is a device that has the function of analyzing product images acquired by a camera and presenting corresponding products.

[1670] A "data sharing means" is a device that allows the voice assistant to monitor the user's condition and share information with medical institutions as necessary.

[1671] An embodiment of the present invention is a smart shopping assistant system for realizing an interactive product search and purchasing process in a physical store. This system is composed of a voice input means, a voice recognition means, a natural language processing means, a product search means, an augmented reality display means, an emotion recognition means, a data feedback means, and a payment means.

[1672] System Configuration

[1673] Voice Input and Recognition

[1674] The user inputs voice into a device such as smart glasses or a head-mounted display. For example, the user might say, "I'm looking for a red dress." The voice input means captures this voice data and converts it into text data using the text conversion means. Specifically, it uses the Google Speech Recognition API.

[1675] Natural Language Processing and Product Search

[1676] The server receives the text data and analyzes it using an NLP processor (e.g., TextBlob or AnyMLibrary NLP) to extract keywords. Based on these keywords, the product search means queries the database to obtain relevant product information. For example, the keyword "red dress" is used to search the store's inventory database.

[1677] Augmented reality display

[1678] The acquired product information is superimposed on the screen of the smart glasses or head-mounted display by the augmented reality display means, allowing the user to visually check the product information. For example, an image of a "red dress," price information, and reviews are visually displayed.

[1679] Emotion Recognition and Data Feedback

[1680] When displaying retrieved product information, the system uses emotion recognition to analyze the user's emotions from their voice and facial expressions. For example, if a user says, "I'm tired today," the system uses TextBlob to analyze their emotions and adjust the product suggestions. Data feedback is used to optimize product suggestions and displays based on the user's intentions and emotions.

[1681] Purchase process and payment

[1682] The user checks the visually displayed product information and again voice-inputs "I would like to purchase this dress." The system again converts the voice input into text data and confirms the user's selection through natural language processing. The system then proceeds to purchase the selected product using the checkout means, presents the payment method through the payment means, and executes the payment.

[1683] Hardware and software used

[1684] Hardware: smart glasses, head-mounted displays, cameras

[1685] Speech recognition software: Google Speech Recognition API

[1686] Natural Language Processing Software: TextBlob, AnyMLibrary NLP

[1687] Database Connection: Custom Database Module

[1688] Payment Software: Payment Processor Custom Module

[1689] Display software: AR display function

[1690] Specific examples

[1691] For example, in a scenario where a user puts on smart glasses and says, "I'm looking for a red dress," the system converts the speech into text data and uses an NLP processor to extract the keyword "red dress." It then searches for the corresponding product in the store's inventory database and displays the product information on the smart glasses' AR display. The user then says, "I'd like to purchase this dress," and the system proceeds with the purchase process and payment. The system also analyzes the user's emotions from their speech and facial expressions and adjusts the suggestions based on the results.

[1692] Prompt Sentence Examples

[1693] Preprocess the following speech input and extract keywords for product search:

[1694] Dictation: "I'm looking for a red dress."

[1695] Example output: ['red dress', 'search', 'looking for']

[1696] This enables the smart shopping assistant system to realize an interactive product search and purchasing process in a physical store, providing users with a comfortable shopping experience.

[1697] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1698] Step 1:

[1699] Acquiring voice input

[1700] The user inputs a voice command to search for a product into the smart glasses or head-mounted display (e.g., "I'm looking for a red dress"). The device captures this voice data and prepares for voice recognition. The input is voice data, and the output is also voice data. Specifically, the device records the voice through a microphone.

[1701] Step 2:

[1702] Converting audio data to text

[1703] The device converts the captured audio data into text data using the Google Speech Recognition API. The input is audio data and the output is text data. This process involves analyzing the audio waveform and converting it into a corresponding string of characters.

[1704] Step 3:

[1705] Natural language processing of text data

[1706] The server receives the text data and analyzes it using an NLP processor (for example, TextBlob or AnyMLibrary NLP). The input is the text data, and the output is a set of extracted keywords. Specifically, the text data is tokenized and important keywords and phrases are extracted.

[1707] Step 4:

[1708] Search for products

[1709] The server then searches the product database using the extracted keywords. The input is a set of keywords, and the output is the corresponding product information. Specifically, the server generates a database query to retrieve product inventory information and detailed information.

[1710] Step 5:

[1711] Displaying product information

[1712] The device displays the acquired product information using the augmented reality function of smart glasses or a head-mounted display. The input is product information, and the output is a visual presentation of the product information. Specifically, the device uses an AR overlay to display product images, prices, reviews, and more in the user's field of view.

[1713] Step 6:

[1714] Emotion recognition

[1715] The server analyzes the user's voice and facial expression data and recognizes emotions using tools such as TextBlob. The input is the user's voice and facial expression data, and the output is the analysis result of the emotional state. Specifically, it analyzes the voice and facial expression data and evaluates the emotion.

[1716] Step 7:

[1717] Data feedback adjustment

[1718] The server then adjusts the display and recommendations based on the recognized emotions. The input is the emotion analysis results, and the output is adjusted product information and recommendations. Specifically, the server adapts the display content based on the emotion data.

[1719] Step 8:

[1720] Product selection and purchase process

[1721] The user checks the displayed product information and again voice-inputs "I'd like to purchase this dress." The device again captures the voice data and confirms the user's selection through text conversion and natural language processing. The input is the re-captured voice data, and the output is the analysis result of the text data.

[1722] Step 9:

[1723] Purchase process and payment

[1724] The server then checks out the selected product, presents the payment method, and executes the transaction. The input is the selected product information, and the output is confirmation of the transaction. Specific operations include inputting and processing payment information and sending a purchase confirmation.

[1725] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1726] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1727] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1728] [Fourth embodiment]

[1729] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1730] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1731] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1732] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1733] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1734] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1735] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1736] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1737] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1738] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1739] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1740] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1741] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1742] The embodiment of the present invention follows the following configuration and processing flow.

[1743] System Configuration

[1744] The system consists of the following main components:

[1745] 1. Voice input method

[1746] It has the function of capturing the user's voice and generating the voice data.

[1747] 2. Text Conversion Methods

[1748] It has the function of receiving voice data and converting it into text data using a voice recognition engine.

[1749] 3. Natural Language Processing Methods

[1750] It has the ability to analyze text data and extract user intent and keywords.

[1751] 4. Product search methods

[1752] It has the function of retrieving relevant product information from a product database based on the extracted keywords.

[1753] 5. Display means

[1754] It has the function of displaying the acquired product information on a display device such as a television screen.

[1755] 6. Audio Selection Method

[1756] It has the function of accepting additional voice input to select a product.

[1757] 7. Purchasing Procedures

[1758] It has the function of proceeding to the purchase procedure for the selected product.

[1759] 8. Payment Methods

[1760] It has the function of presenting payment methods and executing payments.

[1761] 9. Product recognition means

[1762] It has the function of analyzing product images captured by the camera and presenting corresponding products.

[1763] 10. Data sharing methods

[1764] The voice assistant will monitor the user's condition and share information with medical institutions if necessary.

[1765] Program processing

[1766] Acquiring and analyzing voice input

[1767] The user speaks to a television or dedicated terminal, saying, "Find the Mito Komon DVD." The terminal captures this as voice data and converts it into text data using a text conversion means. The server receives the text data and uses natural language processing means to extract keywords such as "Mito Komon," "DVD," and "search."

[1768] Obtaining and displaying product information

[1769] The server executes a search query against the product database using these keywords. Product information is obtained as a search result and displayed on the television screen through the display means. Specifically, a list of "Mito Komon DVDs" is visually displayed on the user's screen.

[1770] Product selection and purchase process

[1771] The user looks at the displayed product list and again says, "I'd like to buy this Mito Komon DVD." The device again captures the voice data and analyzes it using the procedure described above. The server adds the product to the shopping cart and presents options for confirming the payment method (for example, "I'd like to pay by credit card").

[1772] The user selects "I'll pay by credit card" by voice, and the payment is executed by the payment method. If the payment is successful, a confirmation message is sent to the user.

[1773] Product search using camera

[1774] The user points the camera at the product they want to purchase. The device acquires the image data from the camera and analyzes the image using product recognition means. As a result of the analysis, the relevant product information is displayed. For example, if the user holds a "white shirt" up to the camera, a list of that product will be displayed on the TV screen.

[1775] Voice assistant support

[1776] The voice assistant will understand changes in the user's physical condition and any problems they may have through dialogue with the user, and if necessary, will share the information with medical institutions through data sharing means to support the user's health management.

[1777] With these components and processing flow, the system of the present invention enables elderly people to easily enjoy online shopping, eliminating feelings of loneliness and reducing the risk of fraud.

[1778] The processing flow will be explained below.

[1779] Acquiring and analyzing voice input

[1780] Step 1:

[1781] User: Speak to the TV or dedicated device and say, "Find the Mito Komon DVD."

[1782] Step 2:

[1783] Device: Uses the built-in microphone to capture the user's voice and generate audio data.

[1784] Step 3:

[1785] Terminal: Sends the generated voice data to the server via the network.

[1786] Step 4:

[1787] Server: Converts voice data into text data using a speech recognition engine.

[1788] Step 5:

[1789] Server: The converted text data is sent to a natural language processing module, which extracts keywords ("Mito Komon," "DVD," "search").

[1790] Obtaining and displaying product information

[1791] Step 1:

[1792] Server: Executes a search query to the product database based on the extracted keywords and retrieves the corresponding product information.

[1793] Step 2:

[1794] Server: Sends the acquired product information to the terminal via the network.

[1795] Step 3:

[1796] Terminal: Analyzes the received product information and creates a layout for display on the TV screen.

[1797] Step 4:

[1798] Device: Display a list of "Mito Komon DVDs" on the TV screen.

[1799] Product selection and purchase process

[1800] Step 1:

[1801] User: While looking at the displayed product list, say, "I'd like to buy this Mito Komon DVD."

[1802] Step 2:

[1803] Terminal: Capture the audio again and generate the audio data.

[1804] Step 3:

[1805] Terminal: Sends the generated voice data to the server.

[1806] Step 4:

[1807] Server: Analyzes the voice data and identifies the product selected by the user.

[1808] Step 5:

[1809] Server: Confirms the user's selection and adds the selected items to the shopping cart.

[1810] Step 6:

[1811] Server: Generates payment options and presents them to the user through the voice assistant.

[1812] Step 7:

[1813] User: Select "Pay with credit card" by voice.

[1814] Step 8:

[1815] Terminal: Captures the user's voice and sends the voice data to the server.

[1816] Payment Processing

[1817] Step 1:

[1818] Server: Analyzes the voice data and confirms the user's payment method selection.

[1819] Step 2:

[1820] Server: Processes the payment using credit card information.

[1821] Step 3:

[1822] Server: Generates a confirmation message of payment completion and notifies the user via the voice assistant.

[1823] Product search using camera

[1824] Step 1:

[1825] User: Point the camera at the item they want to purchase.

[1826] Step 2:

[1827] Terminal: Uses a camera to capture an image of the product and generate image data.

[1828] Step 3:

[1829] Terminal: Sends the generated image data to the server.

[1830] Step 4:

[1831] Server: Analyzes the image using image recognition algorithms to identify relevant product information.

[1832] Step 5:

[1833] Server: Sends the relevant product information to the terminal.

[1834] Step 6:

[1835] Terminal: Displays the product on the TV screen based on the received product information.

[1836] Voice assistant support

[1837] Step 1:

[1838] Voice assistant: Understands changes in the user's physical condition and concerns through dialogue with the user.

[1839] Step 2:

[1840] Voice assistants: Use data sharing methods as needed to share acquired information with healthcare providers.

[1841] These processing steps enable users to easily shop online using voice and images, and receive support in their daily lives through a voice assistant.

[1842] Example 1

[1843] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1844] In today's world, users, especially the elderly and those with little technical knowledge, can find it difficult to select and purchase products over the Internet due to the complex operations and procedures involved. Furthermore, there are issues such as the risk of selecting the wrong product during the product selection process and the lack of means to accurately share health management information.

[1845] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1846] In this invention, the server includes an input means for accepting voice input, a conversion means for converting voice data into text, an analysis means for analyzing the text data and extracting keywords, a search means for acquiring information based on the extracted keywords, a display means for displaying the acquired information, an input means for accepting further voice input, a processing means for advancing the selected information to processing, and a payment means for presenting payment methods and executing payment. This allows users to easily search, select, and purchase products via voice input, and also provides a means for appropriately sharing health management information.

[1847] "Voice input" means a device or function for capturing a user's voice and inputting it into the system.

[1848] The "conversion" means is software or hardware for converting the captured voice data into text data.

[1849] The "analysis" means software or algorithms that analyze the converted text data and extract the user's intent and specific keywords.

[1850] The "search" means is a function for retrieving related information from databases and external information sources based on the extracted keywords.

[1851] "Display" means refers to a device or function for visually presenting acquired information to a user, and includes a television screen, monitor, and the like.

[1852] The "procedure" means is a function for proceeding to the next processing step, particularly the purchase procedure, based on the information selected by the user.

[1853] The "payment" means is a function for presenting a payment method to a user and processing the actual payment.

[1854] "Recognition" means analyzing image data acquired using a camera or other device, and is a function for identifying and recognizing specific products or objects.

[1855] The "sharing" means is a function for sharing the user's health information and status in cooperation with external institutions such as medical institutions.

[1856] A "voice assistant" is an interactive artificial intelligence that provides various services through voice interaction with users.

[1857] In the embodiment of the present invention, a system configured using the following main hardware and software will be described.

[1858] System Configuration

[1859] 1. Voice input method

[1860] The device is equipped with a microphone to capture the user's voice. Specifically, a dedicated voice capture device (such as a smart speaker) is used.

[1861] 2. A means of converting voice data into text

[1862] The device converts the captured audio data into text in real time using voice recognition software such as the Google Cloud Speech-to-Text API.

[1863] 3. Analysis method for analyzing text data and extracting keywords

[1864] The server receives the text data and analyzes it using a natural language processing engine (e.g., AWS Comprehend). As a result of the analysis, the user's intent and keywords are extracted.

[1865] 4. Search methods to obtain information based on extracted keywords

[1866] The server uses the extracted keywords to query a database that contains product information and other related information.

[1867] 5. Display method for displaying acquired information

[1868] The terminal displays the information obtained from the server on a television screen or a dedicated display, for example, visually displaying a product list to the user.

[1869] 6. A voice selection means for accepting voice input again

[1870] The user looks at the displayed information and again makes selections or commands through voice input, which the device captures and converts again into text data.

[1871] 7. Procedural means for forwarding selected information for processing

[1872] The server then uses the selected information to complete the purchase, adding the product to the shopping cart and prompting for payment.

[1873] 8. Payment methods for submitting payment methods and executing payments

[1874] The server presents the user with multiple payment options (credit card, bank transfer, etc.) and executes the payment using the selected payment method, using a payment service such as Stripe or PayPal.

[1875] 9. A recognition method for analyzing product images captured by a camera and presenting relevant product information

[1876] The device is equipped with a camera that captures image data of the product the user points the camera at, which is then analyzed using image recognition software such as Google Cloud Vision API to search for relevant product information.

[1877] 10. Means for sharing user health information and status with medical institutions

[1878] The voice assistant will constantly monitor the user's health status through daily conversations with them, and if necessary, share information with medical institutions through data sharing methods to support the user's health management.

[1879] Specific examples

[1880] Let's take the example of a user speaking into a TV, saying, "Find a Mito Komon DVD." The device (e.g., a smart speaker) captures the user's voice and converts it into text data using commercially available voice recognition software. The text data is sent to the server in the format "Find a Mito Komon DVD." The server analyzes this text data using a natural language processing engine and extracts the keywords "Mito Komon," "DVD," and "search." It then queries a product database using a search tool to obtain relevant product information. The obtained information is then displayed on the TV screen via the device. The user then uses the displayed information to issue a purchase instruction again using voice input, and the server then proceeds with the purchase process based on this, ultimately completing the payment.

[1881] Prompt Sentence Examples

[1882] "Please explain the process of your voice search system. Please provide a detailed explanation of the steps a user takes to search for, select, and purchase a product using voice. Please also specify what software and hardware are used."

[1883] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1884] Step 1:

[1885] The user speaks to a TV or dedicated terminal, saying, "Find the Mito Komon DVD." The input voice is recognized by the system when the user presses a button on the remote control to switch to voice input mode, which generates voice data.

[1886] Step 2:

[1887] The device captures the user's voice data using a microphone. Specifically, the voice is recorded using a dedicated voice capture device (such as a smart speaker) and saved as digital data. The input is voice data, and the output is digital voice data.

[1888] Step 3:

[1889] The device sends the captured voice data in real time to a conversion mechanism using the Google Cloud Speech-to-Text API, which takes digital voice data as input and outputs text data.

[1890] Step 4:

[1891] The server receives the text data sent from the text conversion means. The input is text data, and this text data is in the format "Looking for a Mito Komon DVD." The output is the same text data.

[1892] Step 5:

[1893] The server runs the text data through an analysis tool and analyzes it using a natural language processing engine (e.g., AWS Comprehend). The input is text data, and through this analysis, the keywords "Mito Komon," "DVD," and "search for" are extracted. The output is the extracted keywords.

[1894] Step 6:

[1895] The server executes a search query to a database using a search engine based on the extracted keywords. Specifically, it executes an SQL query to a product database (e.g., a MySQL database). The input is the extracted keywords, and the output is the corresponding product information.

[1896] Step 7:

[1897] The server sends the product information obtained as a search result to the display means. The obtained product information is sent in HTML format and converted into a data format for display on the screen. The input is the product information, and the output is the display data.

[1898] Step 8:

[1899] The terminal uses the received information to display a product list on the TV screen. Specifically, the information is visualized using a display device connected to the TV or a dedicated display. The input is the data to be displayed, and the output is the displayed product list.

[1900] Step 9:

[1901] The user looks at the displayed product list and again inputs by voice, "I'll buy this Mito Komon DVD." Specific actions include the user specifying a specific product number or name. The input is voice data, and the output is a selection instruction.

[1902] Step 10:

[1903] The terminal captures the voice data again and converts it into text using the conversion means described above. The input is voice data and the output is text data.

[1904] Step 11:

[1905] The server receives the text data regarding the product selection and analyzes the content. The analysis identifies the intention to purchase, such as "I will buy," and the selected product information. The input is the text data, and the output is the analysis result.

[1906] Step 12:

[1907] The server adds the selected items to the shopping cart via a procedure and presents payment options, such as "Do you want to pay by credit card?". The input is the analysis result, and the output is the payment options.

[1908] Step 13:

[1909] The user selects the desired payment method by voice, specifically by saying, "I'll pay by credit card."

[1910] Step 14:

[1911] The device captures the voice data again, converts it into text data, and sends it to the server. The input is voice data, and the output is text data.

[1912] Step 15:

[1913] The server executes the payment via a payment method such as Stripe or PayPal. The input is text data (payment instructions) and the output is payment confirmation.

[1914] Step 16:

[1915] The server sends a confirmation message of successful payment to the display means to notify the user. The input is the payment confirmation, and the output is the display data.

[1916] Step 17:

[1917] The terminal displays the received confirmation message on the TV screen. Specifically, the message "Purchase completed" is displayed on the screen. The input is the data to be displayed, and the output is the display of the confirmation message.

[1918] Step 18:

[1919] The user points the camera at the product they want to purchase. Specifically, they use a webcam connected to a TV. The input is a product image, and the output is image data.

[1920] Step 19:

[1921] The device acquires camera image data and sends it to the recognition means. The Google Cloud Vision API is used for recognition. The input is image data, and the output is analysis data.

[1922] Step 20:

[1923] The server analyzes the received image data and identifies the corresponding product. Product information is searched based on the analysis results. The input is the analyzed data, and the output is product information.

[1924] Step 21:

[1925] The server uses a display unit to display product information based on the analysis results on a television screen. Specifically, a list of "white shirt" products is displayed on the television screen. The input is product information, and the output is display data.

[1926] Step 22:

[1927] The voice assistant periodically asks the user about their health status and understands the situation through dialogue. The input is the user's voice data, and the output is analysis data.

[1928] Step 23:

[1929] The voice assistant analyzes the user's responses using a shared method and shares the information with medical institutions as needed. The input is the analyzed data, and the output is the shared data.

[1930] (Application example 1)

[1931] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1932] There is a need to reduce the complexity and anxiety that seniors face when shopping online, particularly by enabling them to search for and purchase products through intuitive voice commands. Camera-based product recognition and easy voice-based purchasing procedures are also needed. Furthermore, it is also important for voice assistants to connect appropriate services and information to support the health and needs of seniors.

[1933] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1934] In this invention, the server includes a voice input unit, a text conversion unit that converts voice data into text, a natural language processing unit that analyzes the text data and extracts keywords, a product search unit that acquires product information based on the extracted keywords, a display unit that displays the acquired product information, a voice selection unit that accepts voice input for product selection again, a purchase procedure unit that proceeds with the purchase of the selected product, a payment unit that presents a payment method and executes the payment, an image input unit for product recognition, and a product recognition unit that analyzes product images and presents corresponding product information. This allows elderly people to intuitively search for and purchase products using voice control or image recognition, providing ease of operation and safety. Furthermore, support using a voice assistant or data sharing unit provides appropriate responses according to the user's health condition and requests.

[1935] The "voice input means" is a device that has the function of capturing the user's voice and generating voice data.

[1936] The "text conversion means" is a means for converting voice data into text, and is a device that uses voice recognition technology.

[1937] "Natural language processing means" is a technical means for analyzing text data and extracting user intent and keywords.

[1938] The "product search means" is a means for searching and acquiring product information based on the extracted keywords.

[1939] "Display means" refers to a device for visually displaying acquired product information, including the screen of a smartphone and the HUD of smart glasses.

[1940] The "voice selection means" is a means having a function of accepting voice input again to select a product.

[1941] The "purchase procedure means" is a means for proceeding with the purchase procedure for the selected product.

[1942] A "payment method" is a method for presenting a payment method and actually making a payment.

[1943] "Image input means" refers to a means for acquiring product images using a camera or other sensor.

[1944] The "product recognition means" is a means for analyzing the product image acquired by the image input means and presenting the corresponding product information.

[1945] A "voice assistant" is software that understands the user's requests and status through dialogue with the user, and responds and provides information as needed.

[1946] "Data sharing means" refers to means that has the function of sharing user information with other service providers as needed.

[1947] The system embodying this invention performs a comprehensive process from voice input to text conversion, natural language processing, product search, display, product recognition, voice selection, purchase procedure, payment, and data sharing. Each component and its processing method are described in detail below.

[1948] System Configuration

[1949] The system includes the following main components:

[1950] 1. Voice input means: A means for capturing the user's voice and generating the voice data. Specifically, it uses a microphone built into a smartphone or smart glasses.

[1951] 2. Text conversion method: A method for converting voice data into text using voice recognition technology. Google Cloud Speech-to-Text API is used.

[1952] 3. Natural language processing: A means of analyzing text data and extracting keywords and user intent. Google Cloud Natural Language API is used.

[1953] 4. Product search method: This is a method to obtain product information based on extracted keywords. Firebase is used as the product database.

[1954] 5. Display means: A means for visually displaying the acquired product information. This includes the screen of a smartphone and the HUD of smart glasses.

[1955] 6. Voice selection means: A means for accepting voice input again to select a product.

[1956] 7. Purchasing procedure: A procedure for proceeding to purchase the selected product.

[1957] 8. Payment Method: A method for submitting a payment method and executing a payment. Uses the Stripe API.

[1958] 9. Image input means: A means of acquiring product images using a camera or other sensor.

[1959] 10. Product recognition means: A means for analyzing product images acquired by the image input means and presenting corresponding product information.

[1960] 11. Data Sharing Measures: These measures have the ability to share user information with other service providers as needed.

[1961] Program processing

[1962] The data processing and calculation flow at each step of this system will be explained below.

[1963] 1. Acquiring and parsing voice input:

[1964] A user speaks to a smartphone or smart glasses, saying, "I want to buy milk." The device captures this as audio data and converts it into text using the Google Cloud Speech-to-Text API. The server receives the text data and extracts keywords such as "milk" and "I want to buy" using the Google Cloud Natural Language API.

[1965] 2. Obtaining and displaying product information:

[1966] The server executes a search query against the product database (Firebase) using these keywords. Product information is retrieved as a search result and displayed on the smartphone screen or smart glasses HUD through a display means. For example, a list of "Milk" products is visually displayed on the user's screen.

[1967] 3. Product selection and purchase procedure:

[1968] While looking at the displayed list of products, the user again commands by voice, "Add the third milk to the cart." The device again captures the voice data and analyzes it using the procedure described above. The server adds the product to the shopping cart and presents options to confirm the payment method. The user selects by voice, "Pay with credit card," and the payment is executed via the Stripe API.

[1969] 4. Search for products using your camera:

[1970] The user points the camera at the product they want to purchase. The device acquires the camera's image data and analyzes the image using product recognition means. As a result of the analysis, the relevant product information is displayed. For example, if the user holds a "white shirt" up to the camera, a list of that product will be displayed on the smartphone screen or smart glasses' HUD.

[1971] 5. Voice assistant support:

[1972] Voice assistants will understand changes in the user's physical condition and requests through dialogue with the user, and share information with other service providers as needed through data sharing means, enabling them to manage the user's health and respond to their requests.

[1973] Examples of concrete examples and prompts

[1974] Examples:

[1975] When an elderly person says to their smartphone, "I want to buy milk," the system retrieves a list of milk products from Firebase and displays it on the screen. Then, when they say, "Add the third milk to my cart," the product is added to their cart. Finally, when they say, "Pay with credit card," the payment is completed using the Stripe API.

[1976] Example prompt sentence:

[1977] I want to buy milk.

[1978] Add a third milk to your cart.

[1979] Pay by credit card.

[1980] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1981] Step 1:

[1982] The user utters "I want to buy milk" via a voice input means. The device (smartphone or smart glasses) captures the voice data through a microphone.

[1983] Input: User's voice

[1984] Output: Audio data

[1985] Step 2:

[1986] The captured voice data is converted to text data by a text conversion means on the device using the Google Cloud Speech-to-Text API.

[1987] Input: Audio data

[1988] Output: Text data (e.g. "I want to buy milk")

[1989] Step 3:

[1990] The server receives the text data and analyzes it using natural language processing means with the Google Cloud Natural Language API to extract the keyword "milk" and the intent "want to buy."

[1991] Input: Text data

[1992] Output: Keywords (e.g., "milk"), Intent (e.g., "I want to buy")

[1993] Step 4:

[1994] Based on the extracted keywords, the server executes a search query for the relevant product information from the Firebase database via the product search means, and obtains a list of milk products as search results.

[1995] Input: Keyword (e.g. "milk")

[1996] Output: List of product information (e.g., list of milk products)

[1997] Step 5:

[1998] The acquired product information is displayed on the smartphone screen or smart glasses HUD through the device's display means, allowing the user to visually check the list of milk options.

[1999] Input: List of product information

[2000] Output: Visually displayed product information

[2001] Step 6:

[2002] The user instructs via voice input means to "add the third milk to the cart." The terminal again captures the voice data and converts it into text data.

[2003] Input: User's voice

[2004] Output: New audio data

[2005] Step 7:

[2006] The add-to-cart instruction is similarly converted to text, and the server parses the instruction and adds the third milk to the shopping cart.

[2007] Input: Speech data (e.g., "Add the third milk to my cart.")

[2008] Output: Updated shopping cart state

[2009] Step 8:

[2010] The server presents a payment method via the payment method (e.g., credit card). The user then voice-instructs "Pay with credit card," and the payment is executed using the Stripe API.

[2011] Input: Product information, payment method selection

[2012] Output: Payment confirmation, payment completion notification

[2013] Step 9:

[2014] The user takes a photo of a product with a camera as an additional product search. The terminal acquires the product image with the image input means and analyzes the corresponding product information using the product recognition means.

[2015] Input: Product image

[2016] Output: Parsed product information

[2017] Step 10:

[2018] The analyzed product information is displayed on the smartphone screen or smart glasses HUD through the terminal's display means.

[2019] Input: Parsed product information

[2020] Output: Visually displayed product information

[2021] Step 11:

[2022] In terms of health and request support, the voice assistant will understand the user's requests and status through dialogue with the user, and share information with other service providers through data sharing means as needed.

[2023] Input: Interaction data

[2024] Output: Information shared, suggested actions

[2025] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2026] The embodiment of the present invention follows the following configuration and processing flow.

[2027] System Configuration

[2028] The system consists of the following main components:

[2029] 1. Voice input method

[2030] It has the function of capturing the user's voice and generating the voice data.

[2031] 2. Text Conversion Methods

[2032] It has the function of receiving voice data and converting it into text data using a voice recognition engine.

[2033] 3. Natural Language Processing Methods

[2034] It has the ability to analyze text data and extract user intent and keywords.

[2035] 4. Product search methods

[2036] It has the function of retrieving relevant product information from a product database based on the extracted keywords.

[2037] 5. Display means

[2038] It has the function of displaying the acquired product information on a display device such as a television screen.

[2039] 6. Audio Selection Method

[2040] It has the function of accepting additional voice input to select a product.

[2041] 7. Purchasing Procedures

[2042] It has the function of proceeding to the purchase procedure for the selected product.

[2043] 8. Payment Methods

[2044] It has the function of presenting payment methods and executing payments.

[2045] 9. Product recognition means

[2046] It has the function of analyzing product images captured by the camera and presenting corresponding products.

[2047] 10. Data sharing methods

[2048] The voice assistant will monitor the user's condition and share information with medical institutions if necessary.

[2049] 11. Emotion Engine

[2050] It has the ability to recognize the user's emotions and respond appropriately based on them.

[2051] Program processing

[2052] Acquiring and analyzing voice input

[2053] The user speaks to a television or dedicated terminal, saying, "Find the Mito Komon DVD." The terminal captures this as voice data and converts it into text data using a text conversion means. The server receives the text data and uses natural language processing means to extract keywords such as "Mito Komon," "DVD," and "search."

[2054] Obtaining and displaying product information

[2055] The server executes a search query against the product database using these keywords. Product information is obtained as a search result and displayed on the television screen through the display means. Specifically, a list of "Mito Komon DVDs" is visually displayed on the user's screen.

[2056] Product selection and purchase process

[2057] The user looks at the displayed product list and again says, "I'd like to buy this Mito Komon DVD." The device again captures the voice data and analyzes it using the procedure described above. The server adds the product to the shopping cart and presents options for confirming the payment method (for example, "I'd like to pay by credit card").

[2058] The user selects "I'll pay by credit card" by voice, and the payment is executed by the payment method. If the payment is successful, a confirmation message is sent to the user.

[2059] Product search using camera

[2060] The user points the camera at the product they want to purchase. The device acquires the image data from the camera and analyzes the image using product recognition means. As a result of the analysis, the relevant product information is displayed. For example, if the user holds a "white shirt" up to the camera, a list of that product will be displayed on the TV screen.

[2061] Voice assistant support

[2062] The voice assistant will understand changes in the user's physical condition and any problems they may have through dialogue with the user, and if necessary, will share the information with medical institutions through data sharing means to support the user's health management.

[2063] Use of emotion engine

[2064] The emotion engine recognizes the user's emotions from their voice, facial expressions, and choice of words. For example, if a user says, "I feel tired today," the emotion engine recognizes the user's sense of fatigue. Based on this information, the display unit suggests products that will help them relax (for example, aroma oils or massage machines).

[2065] The emotion engine also allows the voice assistant to adapt its dialogue based on the user's emotions. For example, if the user is feeling down, the voice assistant will offer encouraging and kind words. Conversely, if the user seems happy, it will suggest topics that will pique the user's interest.

[2066] With these components and processing flow, the system of the present invention allows seniors to easily enjoy online shopping, alleviating feelings of loneliness and reducing the risk of fraud. It also provides a more sophisticated user experience by taking user emotions into consideration.

[2067] The processing flow will be explained below.

[2068] Acquiring and analyzing voice input

[2069] Step 1:

[2070] User: Speak to the TV or dedicated device and say, "Find the Mito Komon DVD."

[2071] Step 2:

[2072] Device: Uses the built-in microphone to capture the user's voice and generate audio data.

[2073] Step 3:

[2074] Terminal: Sends the generated voice data to the server via the network.

[2075] Step 4:

[2076] Server: Converts voice data into text data using a speech recognition engine.

[2077] Step 5:

[2078] Server: The converted text data is sent to a natural language processing module, which extracts keywords ("Mito Komon," "DVD," "search").

[2079] Obtaining and displaying product information

[2080] Step 1:

[2081] Server: Executes a search query to the product database based on the extracted keywords and retrieves the corresponding product information.

[2082] Step 2:

[2083] Server: Sends the acquired product information to the terminal via the network.

[2084] Step 3:

[2085] Terminal: Analyzes the received product information and creates a layout for display on the TV screen.

[2086] Step 4:

[2087] Device: Display a list of "Mito Komon DVDs" on the TV screen.

[2088] Product selection and purchase process

[2089] Step 1:

[2090] User: While looking at the displayed product list, say, "I'd like to buy this Mito Komon DVD."

[2091] Step 2:

[2092] Terminal: Captures audio and generates audio data.

[2093] Step 3:

[2094] Terminal: Sends the generated voice data to the server.

[2095] Step 4:

[2096] Server: Analyzes the voice data and identifies the product selected by the user.

[2097] Step 5:

[2098] Server: Confirms the user's selection and adds the selected items to the shopping cart.

[2099] Step 6:

[2100] Server: Generates payment options and presents them to the user through the voice assistant.

[2101] Step 7:

[2102] User: Select "Pay with credit card" by voice.

[2103] Step 8:

[2104] Terminal: Captures the user's voice and sends the voice data to the server.

[2105] Payment Processing

[2106] Step 1:

[2107] Server: Analyzes the voice data and confirms the user's payment method selection.

[2108] Step 2:

[2109] Server: Processes the payment using credit card information.

[2110] Step 3:

[2111] Server: Generates a confirmation message of payment completion and notifies the user via the voice assistant.

[2112] Product search using camera

[2113] Step 1:

[2114] User: Point the camera at the item they want to purchase.

[2115] Step 2:

[2116] Terminal: Uses a camera to capture an image of the product and generate image data.

[2117] Step 3:

[2118] Terminal: Sends the generated image data to the server.

[2119] Step 4:

[2120] Server: Analyzes the image using image recognition algorithms to identify relevant product information.

[2121] Step 5:

[2122] Server: Sends the relevant product information to the terminal.

[2123] Step 6:

[2124] Terminal: Displays the product on the TV screen based on the received product information.

[2125] Voice assistant support

[2126] Step 1:

[2127] Voice assistant: Understands changes in the user's physical condition and concerns through dialogue with the user.

[2128] Step 2:

[2129] Voice assistants: Use data sharing methods as needed to share acquired information with healthcare providers.

[2130] Use of emotion engine

[2131] Step 1:

[2132] User: Say "I'm tired today" aloud.

[2133] Step 2:

[2134] Terminal: Captures the user's voice and generates voice data.

[2135] Step 3:

[2136] Terminal: Sends the generated voice data to the server.

[2137] Step 4:

[2138] Server: Analyzes the voice data and recognizes the user's emotions (in this case, fatigue).

[2139] Step 5:

[2140] Emotion engine: Based on the recognized emotions, it generates recommendation information that suggests appropriate products to the user (e.g., relaxing aroma oils or massage machines).

[2141] Step 6:

[2142] Server: Sends the generated recommendation information to the terminal.

[2143] Step 7:

[2144] Device: The received recommendation information is displayed on the TV screen.

[2145] Step 8:

[2146] Voice assistant: Based on the recognized emotion, it will communicate with the user, saying, "I'd like to introduce some products that will help you relax."

[2147] Step 9:

[2148] Voice assistant: Changes the dialogue content according to the recognized emotions, offering gentle words if the user is tired and lively words and topics if they are energetic.

[2149] These processing steps allow users to easily shop online using voice and images, and receive support in daily life through a voice assistant.Furthermore, by utilizing an emotion engine, it is possible to provide optimal services according to the user's mood and emotions.

[2150] Example 2

[2151] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2152] Today's elderly often find it difficult to shop online due to the complexity of the process. Furthermore, existing systems are unable to address other needs, such as health management and reducing feelings of loneliness, which must also be met. In particular, intuitive operation using voice input and natural language processing, as well as personalized user support using emotion recognition, are needed.

[2153] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a voice input means, a text conversion means for converting voice data into text, a natural language processing means for analyzing the text data and extracting keywords, a product search means for acquiring product information based on the extracted keywords, a display means for displaying the acquired product information, a voice selection means for re-accepting voice input for product selection, a purchase procedure means for proceeding with the purchase of the selected product, a payment means for presenting a payment method and executing the payment, a product recognition means for pointing a camera at the product to be purchased, acquiring image data, and presenting corresponding product information, a data sharing means for the voice assistant to communicate with the user, grasp changes in the user's physical condition and problems, and share the information with an external organization as necessary, and an emotion recognition means for recognizing emotions from the user's voice and facial expressions and taking appropriate action. This allows elderly people to enjoy online shopping intuitively and easily, while at the same time satisfying multiple needs such as health management and reducing loneliness.

[2154] The "voice input means" is a device that has the function of capturing the user's speech as voice data.

[2155] The "text conversion means" is a device that has the function of converting captured voice data into text data.

[2156] A "natural language processing means" is a device that has the function of analyzing text data and extracting user intent and keywords.

[2157] The "product search means" is a device that has the function of retrieving relevant product information from a product database based on the extracted keywords.

[2158] The "display means" is a device that has the function of visually displaying the acquired product information.

[2159] The "voice selection means" is a device that has the function of accepting a second voice input from the user for product selection.

[2160] The "purchase procedure means" is a device that has the function of proceeding with the purchase procedure for the selected product.

[2161] A "payment method" is a device that has the function of presenting a payment method and actually carrying out the payment.

[2162] The "product recognition means" is a device that has the function of analyzing image data acquired by a camera and presenting corresponding product information.

[2163] A "data sharing means" is a device that allows the voice assistant to monitor the user's condition and share information with external medical institutions as necessary.

[2164] The "emotion recognition means" is a device that has the function of recognizing the user's emotions from their voice and facial expressions, and taking appropriate action based on that information.

[2165] This invention is an online shopping system primarily targeted at seniors, which combines voice input and natural language processing to enable intuitive and easy operation. Furthermore, by incorporating emotion recognition and health information sharing functions, it improves the overall user experience, helping to manage health and alleviate feelings of loneliness.

[2166] System configuration

[2167] The system consists of the following main components:

[2168] 1. Voice input method

[2169] 2. Text Conversion Methods

[2170] 3. Natural Language Processing Methods

[2171] 4. Product search methods

[2172] 5. Display means

[2173] 6. Audio Selection Method

[2174] 7. Purchasing Procedures

[2175] 8. Payment Methods

[2176] 9. Product recognition means

[2177] 10. Data sharing methods

[2178] 11. Emotion recognition means

[2179] Operation overview

[2180] Voice input and text conversion methods

[2181] The user speaks to the dedicated device or TV, for example, saying, "Look for a Mito Komon DVD." The device uses a built-in microphone to capture the voice data and converts it into text data using the Google Speech-to-Text API. This text data contains the content, "Look for a Mito Komon DVD."

[2182] Natural language processing tools

[2183] The server receives the text data and analyzes it using a natural language processing engine such as spaCy. As a result of the analysis, keywords such as "Mito Komon," "DVD," and "search for" are extracted.

[2184] Product search and display methods

[2185] The server uses the extracted keywords to execute a search query against the product database via the API of Amazon or Rakuten. The retrieved product information is displayed on the TV screen using HTML5 and JavaScript. For example, a list of "Mito Komon DVDs" is displayed visually to the user.

[2186] Voice selection method, purchase procedure method and payment method

[2187] When the user again says "I'd like to buy this Mito Komon DVD," the device captures the voice and converts it back into text data. The server analyzes this text data and adds the item to the shopping cart. Payment methods are then presented, and if the user selects by voice, for example, "I'd like to pay by credit card," the payment is executed using the Stripe API. If the payment is successful, a confirmation message is sent to the user.

[2188] Product recognition means

[2189] When a user points the camera at a product they want to purchase, the device uses the camera to capture image data. The server then analyzes the image using image recognition services such as Google Cloud Vision and retrieves the corresponding product information. This information is then displayed on the TV screen.

[2190] Data sharing methods

[2191] Through dialogue with the user, the voice assistant will understand changes in the user's physical condition and any problems they may have. If necessary, it will use FHIR (Fast Healthcare Interoperability Resources) to share information with external medical institutions and support the user's health management.

[2192] emotion recognition means

[2193] The emotion recognition means recognizes the user's emotions from their voice and facial expressions. For example, if the user says, "I'm tired today," the emotion recognition means recognizes the user's sense of fatigue, and based on this information, the display means suggests products that will help them relax (such as aroma oils or massage machines). The voice assistant also provides words of encouragement or interesting topics depending on the user's emotional state.

[2194] Examples of concrete examples and prompts

[2195] As a specific example, consider a situation where a user says to the TV, "Look for aroma oil." In this case, the device captures the voice and converts it into text data using the Google Speech-to-Text API. The server extracts the keyword "aroma oil," searches the product database, and displays the results on the TV screen.

[2196] Example prompt sentence:

[2197] "When a user says to the TV, 'I'm looking for aroma oil,' the voice data is captured and converted into text data, and the server uses natural language processing to extract keywords, retrieves the relevant product information from the product database, and displays it on the TV screen."

[2198] This system allows seniors to enjoy online shopping intuitively and easily, while also providing health and emotional care at the same time.

[2199] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2200] Step 1: Getting voice input

[2201] The user speaks to a TV or dedicated terminal, saying, "Find the Mito Komon DVD." The terminal uses a built-in microphone to capture the user's speech as audio data. The input is the user's voice, and the output is the captured audio data.

[2202] Step 2: Text conversion

[2203] The device generates the captured voice data using the Google Speech-to-Text API. The voice data is converted into text data. The input here is the voice data, and the output is the text data, "Looking for the Mito Komon DVD."

[2204] Step 3: Natural Language Processing

[2205] The server receives the text data and analyzes it using tools such as spaCy. During the analysis, keywords are extracted to understand the user's search intent. The input is text data, and the output is keywords such as "Mito Komon," "DVD," and "search."

[2206] Step 4: Product Search

[2207] The server uses the extracted keywords to execute a search query against the product database via the API of Amazon or Rakuten. Here, the process of retrieving related product information based on the keywords is performed. The input is the keyword, and the output is the corresponding product information (for example, a list of Mito Komon DVDs).

[2208] Step 5: View the results

[2209] The server displays the retrieved product information on the TV screen. It uses HTML5 and JavaScript to process the visual display. The input is the product information, and the output is a screen displaying a list of "Mito Komon DVDs."

[2210] Step 6: Product Selection

[2211] The user looks at the displayed product list and says verbally, "I'll buy this Mito Komon DVD." The device again captures the voice and converts the voice data into text data, just like in step 2. The input is the user's voice, and the output is the text data, "I'll buy this Mito Komon DVD."

[2212] Step 7: Checkout

[2213] The server adds the selected items to the shopping cart based on the analysis results. It then displays payment options and prepares the payment. The input is text data, and the output is product information added to the shopping cart and payment options.

[2214] Step 8: Payment

[2215] When the user selects "I'd like to pay with a credit card," the terminal captures the voice and converts it back into text data. The server then executes the payment using the Stripe API. The input is the user's voice and text data, and the output is a message that the payment was successful.

[2216] Step 9: Product Recognition

[2217] When a user points the camera at a product they want to purchase, the device uses the camera to capture image data. The server then analyzes the image using the Google Cloud Vision API and retrieves the corresponding product information. The input is image data, and the output is the corresponding product information.

[2218] Step 10: Data sharing

[2219] Voice assistants communicate with users to understand changes in their physical condition and any concerns they may have. They use FHIR to share information with external medical institutions as needed. The input is the user's voice and situational information, and the output is health information shared with medical institutions.

[2220] Step 11: Emotion Recognition

[2221] Emotion recognition is based on the user's voice and facial expressions. For example, if a user says, "I'm tired today," the system recognizes the user's emotion and suggests an appropriate response. The input is voice and facial expression data, and the output is emotional information and a suggested response based on that information.

[2222] (Application example 2)

[2223] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2224] The traditional product search and purchasing process in physical stores required time-consuming walking around the store to find products, resulting in low usability. It was also difficult to instantly check detailed product information and reviews, and product suggestions based on emotions or physical condition were not provided. Furthermore, there was a lack of a way to provide the same easy payment procedures in physical stores as online. This made shopping a burden, especially for elderly customers and those in poor health.

[2225] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2226] In this invention, the server includes a voice input means, a text conversion means for converting voice data into text, a natural language processing means for analyzing the text data and extracting keywords, a product search means for acquiring product information based on the extracted keywords, a display means for displaying the acquired product information, a voice selection means for re-accepting voice input for product selection, a purchase procedure means for proceeding with the purchase of the selected product, a payment means for presenting payment methods and executing the payment, an emotion recognition means for analyzing user emotions, a data feedback means for analyzing the user's expressions of intent and adjusting the content presented, and an augmented reality display means for providing a user interface using an augmented reality function. This enables an interactive product search and purchase process in a physical store, reduces the burden on users, and provides a more comfortable shopping experience.

[2227] The "voice input means" is a device that has the function of capturing the user's voice and generating voice data.

[2228] The "text conversion means" is a device that has the function of receiving voice data and converting it into text data using a voice recognition engine.

[2229] A "natural language processing means" is a device that has the function of analyzing text data and extracting user intent and keywords.

[2230] The "product search means" is a device that has the function of acquiring relevant product information from a product database based on the extracted keywords.

[2231] The "display means" is a display device for visualizing the acquired product information.

[2232] The "voice selection means" is a device that has the function of accepting voice input again to select a product.

[2233] The "purchase procedure means" is a device that has the function of proceeding with the purchase procedure for the selected product.

[2234] A "payment method" is a device that has the function of presenting a payment method and executing a payment.

[2235] The "emotion recognition means for analyzing user emotions" is a device that has the function of analyzing emotions from the user's voice, facial expressions, and content of statements.

[2236] The "data feedback means for adjusting the content of presentation" is a device that has the function of adapting the display and content of suggestions based on the user's intentions and emotions.

[2237] An "augmented reality display means" is a device that has the function of providing a user interface using augmented reality (AR) technology.

[2238] The "product recognition means" is a device that has the function of analyzing product images acquired by a camera and presenting corresponding products.

[2239] A "data sharing means" is a device that allows the voice assistant to monitor the user's condition and share information with medical institutions as necessary.

[2240] An embodiment of the present invention is a smart shopping assistant system for realizing an interactive product search and purchasing process in a physical store. This system is composed of a voice input means, a voice recognition means, a natural language processing means, a product search means, an augmented reality display means, an emotion recognition means, a data feedback means, and a payment means.

[2241] System Configuration

[2242] Voice Input and Recognition

[2243] The user inputs voice into a device such as smart glasses or a head-mounted display. For example, the user might say, "I'm looking for a red dress." The voice input means captures this voice data and converts it into text data using the text conversion means. Specifically, it uses the Google Speech Recognition API.

[2244] Natural Language Processing and Product Search

[2245] The server receives the text data and analyzes it using an NLP processor (e.g., TextBlob or AnyMLibrary NLP) to extract keywords. Based on these keywords, the product search means queries the database to obtain relevant product information. For example, the keyword "red dress" is used to search the store's inventory database.

[2246] Augmented reality display

[2247] The acquired product information is superimposed on the screen of the smart glasses or head-mounted display by the augmented reality display means, allowing the user to visually check the product information. For example, an image of a "red dress," price information, and reviews are visually displayed.

[2248] Emotion Recognition and Data Feedback

[2249] When displaying retrieved product information, the system uses emotion recognition to analyze the user's emotions from their voice and facial expressions. For example, if a user says, "I'm tired today," the system uses TextBlob to analyze their emotions and adjust the product suggestions. Data feedback is used to optimize product suggestions and displays based on the user's intentions and emotions.

[2250] Purchase process and payment

[2251] The user checks the visually displayed product information and again voice-inputs "I would like to purchase this dress." The system again converts the voice input into text data and confirms the user's selection through natural language processing. The system then proceeds to purchase the selected product using the checkout means, presents the payment method through the payment means, and executes the payment.

[2252] Hardware and software used

[2253] Hardware: smart glasses, head-mounted displays, cameras

[2254] Speech recognition software: Google Speech Recognition API

[2255] Natural Language Processing Software: TextBlob, AnyMLibrary NLP

[2256] Database Connection: Custom Database Module

[2257] Payment Software: Payment Processor Custom Module

[2258] Display software: AR display function

[2259] Specific examples

[2260] For example, in a scenario where a user puts on smart glasses and says, "I'm looking for a red dress," the system converts the speech into text data and uses an NLP processor to extract the keyword "red dress." It then searches for the corresponding product in the store's inventory database and displays the product information on the smart glasses' AR display. The user then says, "I'd like to purchase this dress," and the system proceeds with the purchase process and payment. The system also analyzes the user's emotions from their speech and facial expressions and adjusts the suggestions based on the results.

[2261] Prompt Sentence Examples

[2262] Preprocess the following speech input and extract keywords for product search:

[2263] Dictation: "I'm looking for a red dress."

[2264] Example output: ['red dress', 'search', 'looking for']

[2265] This enables the smart shopping assistant system to realize an interactive product search and purchasing process in a physical store, providing users with a comfortable shopping experience.

[2266] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2267] Step 1:

[2268] Acquiring voice input

[2269] The user inputs a voice command to search for a product into the smart glasses or head-mounted display (e.g., "I'm looking for a red dress"). The device captures this voice data and prepares for voice recognition. The input is voice data, and the output is also voice data. Specifically, the device records the voice through a microphone.

[2270] Step 2:

[2271] Converting audio data to text

[2272] The device converts the captured audio data into text data using the Google Speech Recognition API. The input is audio data and the output is text data. This process involves analyzing the audio waveform and converting it into a corresponding string of charac...

Claims

1. A voice input means; a text conversion means for converting voice data into text; natural language processing means for analyzing text data and extracting keywords; a product search means for acquiring product information based on the extracted keywords; a display means for displaying the acquired product information; a voice selection means for accepting voice input for product selection again; A purchasing procedure means for proceeding to purchase the selected product; a payment instrument that presents a payment method and executes a payment; A system including:

2. The system according to claim 1 , further comprising a product recognition means for analyzing product images acquired by the camera and presenting corresponding product information.

3. The system according to claim 1 , further comprising a data sharing means for the voice assistant to interact with the user, grasp changes in the user's physical condition and concerns, and share information with a medical institution as necessary.

Images