System

Abstract

A system is provided.SOLUTION: The system includes a means for receiving an input from a user and transmitting the input to a server, a means for analyzing the input received by the server and retrieving a corresponding commodity from a commodity database, and a means for selecting an optimum commodity on the basis of an analysis result and providing an appropriate retrieval result to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When shopping online, users often have difficulty searching for products. Without the right input, it's difficult to get good search results. Furthermore, comparing multiple products and service conditions takes time and effort. This invention aims to solve these problems for users by utilizing AI technology to improve search efficiency and accuracy. It improves the user experience by enabling appropriate search results even with ambiguous input and supporting various input methods, such as voice and images. [Means for solving the problem]

[0005] The present invention solves the problems by using the following means: A system including means for receiving input from a user and sending the input to a server, means for the server to analyze the received input and search for corresponding products from a product database, means for selecting optimal products based on the analysis results and providing appropriate search results to the user. Furthermore, the system includes natural language processing means for generating appropriate search queries even when ambiguous input is made, and means for receiving voice input and image input, analyzing the input data, and using it as search conditions.

[0006] This system allows users to efficiently search for products and allows input using a variety of methods, such as voice and images, thereby improving search accuracy and significantly reducing the burden on users.

[0007] "User" means an individual or organization that uses the System to search for and purchase products.

[0008] "Input" refers to information provided by a user to a system through an interface, and may include text, voice, images, etc.

[0009] The "server" is the central device that receives and analyzes input from users and interacts with the product database to generate search results.

[0010] A "product database" is a collection of data in which various types of product information are centrally stored, and is the subject of search processing.

[0011] "Analysis" is the process by which the server understands user input using natural language processing and converts it into an appropriate search query.

[0012] "Natural language processing means" is a technology that analyzes ambiguous input from users, accurately understands their intent, and generates appropriate search queries.

[0013] "Voice input" refers to the way in which a user provides information to a system using their voice, which is converted into text using speech recognition technology.

[0014] "Image input" refers to the way in which a user provides image data to the system, and image analysis techniques are used to extract information.

[0015] "Search results" are a list of products extracted by the server from the product database and optimized based on the user's input and conditions.

[0016] "Selection" is the process in which the server identifies the best product for the user from multiple product candidates based on the analysis results. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention provides a system that utilizes AI technology to improve the product search experience in order to make product searches more efficient in online shopping. Specific embodiments are described below.

[0039] User Interface Design

[0040] Users access the system through a dedicated application or web browser on their device (smartphone or PC). The user interface is simple and easy to use, with a search bar, voice input button, and image upload button.

[0041] Processing User Input

[0042] Users can enter text into the search bar, press the microphone button to speak, or press the camera button to upload an image, for example, "I'm looking for black shoes for the office" or upload a photo of their feet.

[0043] Sending and parsing input

[0044] The device captures the user's input data and transmits it over the network to a server, either as text, audio, or images.

[0045] The server analyzes the received data. For text data, it uses natural language processing technology to generate search queries. For voice data, it uses voice recognition technology to convert it into text and analyze it. For image data, it uses image analysis technology to extract features of the uploaded image and generate search criteria based on them.

[0046] Linking and searching with product databases

[0047] The server uses the generated search query to search for relevant products in a product database, which contains regularly updated product information from multiple vendors, and returns a list of products that match the search query.

[0048] Product selection and ranking

[0049] The server uses an AI model to rank the product list obtained as a search result, in an optimal order based on the user's past purchase history, ratings, and current search criteria.

[0050] Displaying search results

[0051] The terminal displays the ranked product list sent from the server to the user, and the search results include detailed information such as product images, prices, user reviews, and availability.

[0052] Feedback and interactive search tuning

[0053] Users are also provided with an interface to provide feedback on the search results and further refine their search criteria, for example by lowering the price range or selecting a specific brand.

[0054] The server receives feedback from the user, updates the search criteria in real time, and searches again, resulting in more accurate search results.

[0055] Start of purchase flow

[0056] When a user finds a product they want, they are taken to the product detail page and begin the checkout process, which includes entering their shipping address and payment information.

[0057] The server receives the purchase request, updates the relevant database, and works with an external service that handles payment processing to complete the transaction.

[0058] As described above, this invention uses AI technology to provide appropriate search results even when the user provides vague input, and also supports voice and image input, thereby significantly improving the online shopping experience.

[0059] The processing flow will be explained below.

[0060] Step 1:

[0061] Using a device, a user accesses a search screen. The user can choose to enter text, speak, or upload an image. For example, the user enters "I'm looking for black shoes for the office" into the search bar.

[0062] Step 2:

[0063] The device captures input from the user. In the case of text, it is captured as text data, in the case of voice, the voice data is converted to text, and in the case of images, the image data is captured as is.

[0064] Step 3:

[0065] The device sends the captured input data to the server using a network communication method such as an HTTP POST request, with the data being sent in a properly formatted format such as JSON or XML.

[0066] Step 4:

[0067] The server analyzes the received data. In the case of text data, natural language processing (NLP) technology is used to extract search intent and generate appropriate search queries. In the case of audio data, speech recognition technology is used to convert it into text and analyze it in the same way. In the case of image data, image analysis technology is used to extract important features and generate appropriate search terms.

[0068] Step 5:

[0069] The server uses the generated search query to search a product database, for example by issuing an SQL query, to retrieve a list of matching products. This database contains up-to-date product information from multiple vendors.

[0070] Step 6:

[0071] The server then passes the search results to an AI model for ranking, which uses the user's past purchase history, ratings, and other relevant data to rank the most relevant products.

[0072] Step 7:

[0073] The server then sends the ranked search results to the device, which includes detailed information such as product images, prices, user reviews, and availability.

[0074] Step 8:

[0075] The device receives search results from the server and displays them to the user, along with filtering and sorting options to further refine the search criteria.

[0076] Step 9:

[0077] Users can review the search results and then perform new searches by further narrowing the search criteria, such as selecting a price range or a specific brand.

[0078] Step 10:

[0079] The server receives new feedback from the user and performs the product search again, updating the search criteria in real time to generate optimal results.

[0080] Step 11:

[0081] Once the user selects the desired product, the terminal displays a checkout screen where the user enters their shipping address and payment information.

[0082] Step 12:

[0083] The server receives the purchase request, updates the relevant database, coordinates with external services that handle payment processing, and sends a purchase confirmation to the user once the transaction is complete.

[0084] This step allows users to search for products and complete the purchasing process efficiently and conveniently.

[0085] Example 1

[0086] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0087] Conventional online shopping product search systems have difficulty providing accurate search results when user input is ambiguous or when they need to accept a variety of input formats (text, voice, images). Furthermore, there are still challenges in providing a function that reflects user feedback in real time to update search results, and in providing personalized search results based on past purchase history and ratings.

[0088] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0089] In this invention, the server includes means for receiving input from a user and transmitting the input to the server, means for analyzing the input received by the server and searching for corresponding products from a product database, means for selecting optimal products based on the analysis results using a ranking means and providing appropriate search results to the user, means for receiving feedback from the user and updating search conditions in real time, and means for taking into account the user's past purchase history and ratings in subsequent searches. This makes it possible to provide appropriate search results even when the user has entered ambiguous input, support a variety of input formats, and provide personalized search results.

[0090] "User" refers to an individual or corporation that uses the online shopping system.

[0091] "Input" refers to information provided by a user as a search query, and may be in the form of text, audio, images, or the like.

[0092] "Server" refers to a computing resource on a network that receives and analyzes input data from users and executes product searches.

[0093] "Parsing" refers to the process by which the server interprets the input data it receives, understands its meaning, and generates appropriate search queries.

[0094] "Product database" refers to a data storage device that stores product information provided by multiple vendors.

[0095] "Ranking method" refers to an algorithm that prioritizes search results based on a user's past purchasing history, ratings, and current search criteria.

[0096] "Search Results" refers to a list of products selected by the server based on user input.

[0097] "Feedback" refers to additional instructions or ratings given by a user in response to a search result.

[0098] "Real-time update" means that feedback from users is reflected immediately, and product searches are performed again to provide the latest search results.

[0099] "Purchase history" refers to a record of products a user has purchased in the past.

[0100] "Personalized" refers to providing search results that are customized based on the individual user's preferences and behavioral history.

[0101] The present invention is directed to a system for improving the efficiency of product searches in online shopping. Specific embodiments will be described below.

[0102] Users access the system through a dedicated application or a web browser using devices such as smartphones or PCs. The user interface is designed to be easy to use, with a search bar, voice input button, and image upload button located on the screen.

[0103] Users can enter keywords into the search bar, use voice input by pressing the microphone button, or upload an image by pressing the camera icon. For example, they can enter text such as "I'm looking for black shoes that go well with the office," say "I'm looking for casual sneakers," or upload a photo of the product they're looking for.

[0104] The device encodes the captured input data and securely transmits it to the server. The server then analyzes the received data. Specifically, for text data, natural language processing techniques (e.g., BERT or GPT-3) are used to generate search queries. For audio data, speech recognition techniques (e.g., DeepSpeech or Google Speech-to-Text) are used to convert the data into text and analyze that text. For image data, image analysis techniques (e.g., YOLO or ResNet) are used to extract image features and generate search queries based on them.

[0105] The server uses the analyzed search query to search for relevant products in a product database. The product database contains the latest product information provided by numerous vendors. The server then ranks the retrieved product list using an AI model (e.g., random forest or deep learning model). This ranking is based on the user's past purchase history, ratings, and current search criteria.

[0106] The terminal displays the ranked product list sent from the server to the user. The search results include detailed information such as product images, prices, user reviews, and availability. The user is provided with an interface to provide feedback on the displayed search results and further refine the search criteria. For example, the user can change the price range or select a specific brand.

[0107] The server receives feedback from the user, updates the search criteria in real time, and searches again, resulting in more accurate search results. When the user finds the desired product, they are redirected to the product detail page and begin the purchase process, which includes entering shipping address and payment information. The server receives the purchase request, updates the relevant database, and completes the transaction by interacting with external payment processing services (e.g., Stripe or PayPal).

[0108] Specific examples

[0109] A user types "I want black shoes suitable for the office" into the search bar. The device sends this text to the server, which uses natural language processing technology to generate a search query and search a product database. Results such as "black leather shoes suitable for the office" are returned and displayed on the device.

[0110] The user presses the microphone button and says, "I want casual sneakers." The device captures the voice data and sends it to the server. The server uses speech recognition technology to convert the speech into text and generate a search query. Search results for casual sneakers are returned and displayed on the device.

[0111] Example prompt: "If a user is looking for a simple bag, how can you use AI technology to provide the best search results?"

[0112] As described above, the present invention is a system that uses AI technology to provide appropriate search results even when the user enters ambiguous information, and also supports voice and image input, in order to significantly improve the online shopping experience.

[0113] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0114] Step 1: Launch the user interface

[0115] The user launches a dedicated application or web browser on a device such as a smartphone or PC, and the user interface displays a search bar, a voice input button, and an image upload button.

[0116] Input: A user action (launching an application or web browser)

[0117] Output: Display of the user interface

[0118] Step 2: Receiving user input

[0119] Users can enter text keywords into the search bar, press the microphone button to input using their voice, or press the image upload button to upload an image, for example, by typing "black shoes," saying "looking for casual t-shirts," or uploading a photo of a product.

[0120] Input: Text input, voice input, or image upload

[0121] Output: Captured input data (text, audio, images)

[0122] Step 3: Capturing and sending input

[0123] The terminal encodes the input data captured from the user and transmits it over the network to the server.

[0124] Input: Captured input data (text, audio, images)

[0125] Output: Sends the encoded input data to the server

[0126] Step 4: Data analysis by the server

[0127] The server analyzes the received data using appropriate techniques: text data is analyzed using natural language processing techniques (e.g., GPT-3) to generate search queries; voice data is converted to text using speech recognition techniques (e.g., Google Speech-to-Text) and analyzed; and image data is analyzed using image analysis techniques (e.g., YOLO) to extract features and generate search queries.

[0128] Input: Transmitted encoded data (text, audio, image)

[0129] Output: Parsed search query

[0130] Step 5: Product database search

[0131] The server uses the generated search query to search for the relevant product in a product database, which stores up-to-date product information provided by multiple vendors.

[0132] Input: Parsed search query

[0133] Output: Product list as search results

[0134] Step 6: Product Listing Selection and Ranking

[0135] The server then uses an AI model (e.g., a deep learning model) to rank the retrieved product list, based on the user's past purchase history, ratings, and current search criteria.

[0136] Input: Product list as search results

[0137] Output: Ranked product list

[0138] Step 7: Viewing the ranking results

[0139] The terminal receives the ranked product list sent from the server and displays it on the user interface. The search results include information such as product images, prices, user reviews, and stock status.

[0140] Input: Ranked product list

[0141] Output: Display of search results with detailed information

[0142] Step 8: Receive feedback and refine your search criteria

[0143] Users can provide feedback on the displayed search results and enter new criteria, such as a specific price range or brand.

[0144] Input: User feedback (new conditions)

[0145] Output: Updated feedback data

[0146] Step 9: Re-search based on feedback

[0147] The server receives feedback from the user, updates the search criteria in real time, and performs the search again, resulting in more accurate search results.

[0148] Input: Updated feedback data

[0149] Output: Updated product list searched again

[0150] Step 10: Start Checkout

[0151] When the user selects the desired product, they are taken to the product details page and begin the purchase process. They enter their shipping address and payment information to confirm the purchase.

[0152] Input: User's selected product, shipping address, payment information

[0153] Output: Purchase request completed

[0154] Step 11: Complete the purchase process

[0155] The server receives the purchase request, updates the relevant database, and coordinates with an external service (e.g., payment processing software) that handles the payment process and completes the transaction.

[0156] Input: Purchase request and payment information

[0157] Output: Transaction completed and database updated

[0158] (Application example 1)

[0159] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0160] Conventional online shopping systems make it difficult for users to efficiently find the products they are looking for, and the interfaces are cumbersome. Furthermore, especially for use in autonomous vehicles, voice-only operation and image input are important, but there is a lack of technology to properly process these. Therefore, there is a demand for a system that allows users to enjoy a comfortable and efficient online shopping experience inside the car.

[0161] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0162] In this invention, the server includes a means for receiving input from a user and transmitting the input to the server, a means for the server to analyze the received input and search for corresponding products from a product database, and a means for selecting optimal products based on the analysis results and providing appropriate search results to the user. This makes it possible to provide a system that combines a means for capturing a user's voice input using an in-vehicle voice recognition device and a means for visually displaying search results using an in-vehicle infotainment system and allowing the user to select products via a touch screen or voice control. This enables users to efficiently and comfortably shop online even while in an autonomous vehicle.

[0163] The "means for receiving input from a user and transmitting the input to a server" is a part of a system that has the function of capturing input data, such as voice or images, from a user and transmitting the data to a server.

[0164] "Means for the server to analyze the input received and search for the relevant product in the product database" is part of a system that has the processing power to process the text, voice, or image data received by the server from the user to find the relevant product.

[0165] "Means for selecting the most suitable product based on the analysis results and providing the user with appropriate search results" refers to part of a system that has the function of allowing the server to select the most suitable product based on the analysis results of the input data and display that information on the user's device.

[0166] The "means for capturing user voice input using an in-vehicle voice recognition device" is part of a system that receives user voice commands within an autonomous vehicle and captures them as data.

[0167] "Means of using the in-vehicle infotainment system to visually display search results and allow the user to select products via touchscreen or voice control" refers to a part of a system that displays search results on a display inside an autonomous vehicle and allows the user to select products via touchscreen or voice command.

[0168] "Natural language processing means that generates appropriate search queries even when ambiguous input is given" refers to technology that properly interprets unclear expressions or ambiguous words used by users and generates optimal search queries.

[0169] "Means for receiving voice input and image input, analyzing the input data, and using it as search conditions" refers to part of a system that has the function of receiving voice and image data, analyzing the data, and using it as product search conditions.

[0170] The present invention provides a system that allows a user to comfortably and efficiently perform online shopping while inside an autonomous vehicle. Specific embodiments will be described below.

[0171] The system of the present invention provides a means for users to search for and purchase products using voice and images, and includes the following main components:

[0172] 1. Capture audio and video inputs

[0173] A voice recognition device is used to capture voice inputs from users inside the autonomous vehicle. The device detects the user's voice through a microphone installed in the vehicle and converts it into digital data. Also, if the user uploads an image from their smartphone, it will be captured by the vehicle's infotainment system.

[0174] 2. Data submission and analysis

[0175] The voice and image data captured from the user is sent via the infotainment system to a server, which analyzes the data using the following techniques:

[0176] Speech Recognition: Converts speech data into text using Python and TensorFlow.

[0177] Image analysis: Using OpenCV, features of uploaded images are extracted and search criteria are generated based on them.

[0178] 3. Product Search and Query Generation

[0179] The server uses natural language processing technology to generate a search query based on the received text data and image analysis data, and uses this query to search for matching products in a product database.

[0180] 4. Displaying search results

[0181] Search results are displayed on the infotainment system display, which can be used to select products using a touchscreen or voice control. Search results include detailed information such as product images, prices, and user reviews.

[0182] 5. Purchase Procedure

[0183] A checkout interface is provided to purchase the selected items immediately, with shipping address and payment information securely processed within the system.

[0184] Specific examples

[0185] For example, if a user were to say, "I'm looking for black shoes for the office, especially leather ones," the voice recognition device would capture this speech, convert it into text data, and send it to the server. The server would then use natural language processing technology to generate a search query and search for matching products in a product database. The search results would then be displayed on the infotainment system's display.

[0186] A specific example of a prompt sentence is a voice input such as "I'm looking for black shoes that can be used in the office. I'd especially like leather shoes."

[0187] The system of the present invention allows users to shop online efficiently and comfortably even while inside an autonomous vehicle.

[0188] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0189] Step 1:

[0190] The user inputs voice or uploads images from a smartphone while in an autonomous vehicle. The voice recognition device captures the user's voice through a microphone in the car and converts it into digital voice data. Images are captured through the infotainment system. Input: User's voice or image. Output: Digital voice data or image data.

[0191] Step 2:

[0192] The device sends the captured digital audio or image data to the server. The data is transferred to the server via the infotainment system and over the network. Input: Digital audio or image data. Output: Data sent to the server.

[0193] Step 3:

[0194] The server analyzes the digital voice data it receives and converts it into text data using voice recognition technology. Speech recognition is performed using Python and TensorFlow. In the case of image data, features are extracted using OpenCV. Input: Digital voice data or image data. Output: Text data or image feature data.

[0195] Step 4:

[0196] The server generates search queries using natural language processing technology based on text data and image feature data. A generative AI model is used to generate appropriate search queries even with ambiguous input. Input: Text data or image feature data. Output: Search query.

[0197] Step 5:

[0198] The server uses the generated search query to search for matching products in a product database that contains up-to-date product information from multiple vendors. Input: Search query. Output: Search result list.

[0199] Step 6:

[0200] The server ranks the search results and sends the top results to the terminal. The ranking is done in the optimal order based on the user's past purchase history, ratings, and current search conditions. Input: Search result list. Output: Ranked product list.

[0201] Step 7:

[0202] The terminal displays the ranked product list sent from the server on the display of the infotainment system in the car. The user can select products using the touch screen or voice control. Input: Ranked product list. Output: Product list displayed on the display.

[0203] Step 8:

[0204] The user selects the desired product on the display and begins the purchase process. The terminal prompts the user to enter the shipping address and payment information, which are then sent to the server. Input: User-selected product and purchase information. Output: Purchase request sent to the server.

[0205] Step 9:

[0206] The server receives the purchase request, updates the relevant database, and completes the transaction by contacting an external service responsible for payment processing. The item is then shipped to the specified delivery address. Input: Purchase request. Output: Completed transaction and shipping instructions.

[0207] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0208] The present invention provides a system for providing more personalized search results by combining product search and user emotion recognition in online shopping. This system improves the user experience by recognizing the user's emotions and making appropriate product suggestions based on those emotions. Specific embodiments are described below.

[0209] User Interface Design

[0210] Users access the system via a dedicated application or web browser on their smartphone or PC. The user interface includes a search bar, a voice input button, an image upload button, and a camera icon for emotion recognition.

[0211] Processing User Input

[0212] Users can enter text into the search bar, or use voice input by pressing the microphone button. They can also upload an image by pressing the camera button or click the emotion recognition camera icon. For example, they can turn on emotion recognition by typing, "I want black shoes that go with the office."

[0213] Sending and parsing input

[0214] The device captures input data from the user and transmits it over the network to a server, which can be text, voice, images, or emotion recognition data.

[0215] The server analyzes the received data. For text data, it uses natural language processing technology to generate search queries. For voice data, it uses voice recognition technology to convert it into text and analyze it in the same way. For image data, it uses image analysis technology to extract important features and generate relevant search terms. Furthermore, for emotion recognition data, an emotion engine analyzes facial expressions and voice tone to identify the user's emotional state.

[0216] Linking and searching with product databases

[0217] The server uses the generated search query to search a product database, for example by issuing an SQL query, to retrieve a list of matching products. This database contains up-to-date product information from multiple vendors.

[0218] Product selection and ranking

[0219] The server passes the product list obtained as a search result to the AI ​​model, which then ranks the products. The AI ​​model generates rankings by taking into account the user's past purchase history, ratings, current search criteria, and even their emotional state. For example, if the AI ​​model detects that the user is "tired," it will prioritize products with a relaxing effect.

[0220] Displaying search results

[0221] The device displays a ranked list of products sent from the server to the user, and the search results include detailed information such as product images, prices, user reviews, availability, and even recommendations based on emotional state.

[0222] Feedback and Interaction

[0223] Users can provide feedback on the search results displayed, and the device will send the feedback information to the server, which will then further optimize the search results. In this process, the system will update the search results in real time, even if the user indicates a change in sentiment.

[0224] Start of purchase flow

[0225] When the user finds the product they want, the device displays a checkout screen where they can enter their shipping address and payment information.

[0226] The server receives the purchase request, updates the relevant database, and completes the transaction by connecting with an external service responsible for payment processing. Based on emotion recognition, the user's purchasing experience is personalized.

[0227] This embodiment allows users to obtain search results that reflect their emotional state, allowing them to enjoy more satisfying online shopping. Furthermore, by combining this with voice and image input, the system can accommodate a wider variety of search conditions.

[0228] The processing flow will be explained below.

[0229] Step 1:

[0230] Users access the system through a dedicated application or web browser on their device by entering text into the search bar, pressing the microphone button for voice input, pressing the camera button to upload an image, or clicking the camera icon for emotion recognition.

[0231] Step 2:

[0232] The device captures the user's input data. In the case of text input, it is saved as text data. In the case of voice input, it is converted into text using voice recognition technology. In the case of image input, the image data is saved. It also captures the user's facial expressions and tone of voice using a camera and microphone dedicated to emotion recognition.

[0233] Step 3:

[0234] The device sends the captured input data to the server using a network communication method such as an HTTP POST request, and the data is sent as properly formatted JSON or XML.

[0235] Step 4:

[0236] The server analyzes the received data. In the case of text data, natural language processing (NLP) technology is used to extract the user's intent and convert it into an appropriate search query. In the case of voice data, voice recognition technology is used to convert it into text, which is then analyzed as text data. In the case of image data, image analysis technology is used to extract important features to use as search criteria. Furthermore, in the case of emotion recognition data, an emotion engine analyzes facial expressions and voice tone to identify the user's emotional state.

[0237] Step 5:

[0238] The server uses the generated search query to search the product database using an SQL query or API request to retrieve a list of matching products based on the product information in the database.

[0239] Step 6:

[0240] The server passes the returned product list to the AI ​​model for ranking. This ranking is based on the user's past purchase history, ratings, and current search criteria. In addition, the model takes into account the user's emotional state and prioritizes the most appropriate products based on that emotional state.

[0241] Step 7:

[0242] The server sends the ranked search results to the device, including detailed information such as product images, prices, user reviews, availability, and recommendations (including explanations of the reasons for recommendations if they are based on emotional states).

[0243] Step 8:

[0244] The device will then display the search results received from the server to the user, who can then review the displayed results and use filter options and sorting functions to further refine the search criteria.

[0245] Step 9:

[0246] Users can provide feedback on the search results they see, for example, by setting more specific criteria (price, brand, etc.) based on a particular emotional state (e.g., wanting to relax).

[0247] Step 10:

[0248] The server receives feedback from users, updates the search criteria in real time, and searches again, re-ranking the search results and generating the best product list based on the latest information.

[0249] Step 11:

[0250] When the user finds the product they want, the device displays a checkout screen where they can enter their shipping address and payment information to complete the purchase.

[0251] Step 12:

[0252] The server receives the purchase request, updates the relevant database, and completes the transaction by contacting an external payment processing service. Once the purchase is complete, it sends a confirmation to the user.

[0253] This step allows users to efficiently and personalizedly search and purchase products.

[0254] Example 2

[0255] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0256] Current online shopping systems generate search results based solely on input data provided by the user, making it difficult to provide product suggestions that reflect the user's emotional state or current mood. This can result in a lack of optimal suggestions for the products the user is looking for, which can lead to low satisfaction. Furthermore, they are unable to effectively utilize various input formats, such as voice input or image input, preventing flexible searches. Furthermore, they lack a mechanism for recognizing user emotions in real time and reflecting them in search results, making it difficult to provide more personalized search results for each individual user.

[0257] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0258] In this invention, the server includes a means for receiving input from a user and transmitting the input to the server, a means for analyzing the input received by the server and searching for corresponding products from a product database, a means for selecting optimal products based on the analysis results and providing the user with appropriate search results, a means for recognizing the user's emotions and making product suggestions based on the emotions, and a means for analyzing text, voice, and images provided by the user to generate search criteria. This enables personalized product suggestions that reflect the user's emotional state, improving the user experience. Furthermore, flexible search criteria generation is realized by effectively utilizing various input formats, including voice input and image input.

[0259] "User" refers to an individual or organization that uses the System to search for and purchase products.

[0260] "Input" means data provided by a user to the system, and may include any one or more of text, voice, image, and / or emotion recognition data.

[0261] "Server" refers to a central processing unit that analyzes input data received from users and searches the product database and makes product suggestions based on the results.

[0262] "Product Database" refers to a data storage system that stores up-to-date product information provided by multiple vendors.

[0263] "Analysis" refers to the process by which the server processes input data received from the user and extracts meaningful information.

[0264] "Product suggestion" refers to the server presenting suitable products to the user based on the analysis results.

[0265] "Emotion recognition" refers to the process of analyzing a user's facial expressions and tone of voice to determine their current emotional state.

[0266] "Natural language processing" refers to technology that enables computers to understand and analyze the language that humans use on a daily basis.

[0267] "Speech recognition" refers to the technology of converting voice data into text data.

[0268] "Image analysis" refers to the technology of extracting important features and information from image data.

[0269] "Ranking" refers to the process of prioritizing products based on a user's purchasing history, ratings, current search criteria, and emotional state.

[0270] "Feedback" refers to ratings and comments provided by users that are used to improve the performance of the system and the quality of suggestions.

[0271] "Checkout" refers to the process by which a User enters the necessary information to purchase a selected Product and completes the transaction.

[0272] The present invention provides a system for providing more personalized search results by combining product search and user emotion recognition in online shopping. This system improves the user experience by recognizing the user's emotions and suggesting appropriate products based on those emotions. Specific embodiments are described below.

[0273] User Interface Design

[0274] Users access the system through a dedicated application or a web browser on their smartphone or PC. The user interface includes a search bar, a voice input button, an image upload button, and a camera icon for emotion recognition, allowing users to input search requests in a variety of ways.

[0275] Processing User Input

[0276] Users can enter specific search criteria using text input, voice input, image upload, and emotion recognition. For example, a user may enter text such as "I'm looking for black shoes for the office" and enable emotion recognition. In this case, the user can also use voice input and image upload.

[0277] Sending and parsing input

[0278] The device captures the user's input data and transmits it to a server over a network, including text, voice, images, and emotion recognition data.

[0279] The server analyzes the received data, specifically:

[0280] The text data is analyzed using natural language processing techniques to generate search queries.

[0281] The voice data is converted into text using voice recognition technology, and the text is analyzed.

[0282] Image data is subjected to image analysis techniques to extract features.

[0283] The emotion recognition data is analyzed by an emotion engine to identify the user's emotional state.

[0284] Linking and searching with product databases

[0285] The server uses the generated search query to search a product database, which contains up-to-date product information and data provided by multiple vendors. For example, it issues an SQL query to retrieve a list of matching products.

[0286] Product Selection and Ranking

[0287] The server passes the retrieved product list to an AI model for ranking. The AI ​​model generates rankings by taking into account the user's past purchase history and ratings, current search criteria, and emotional state. For example, if the user's emotion is recognized as "tired," products with a relaxing effect may be displayed preferentially.

[0288] Displaying search results

[0289] The terminal displays a ranked list of products sent from the server to the user, and the search results include detailed information such as product images, prices, user reviews, availability, and reasons for recommendation.

[0290] Feedback and Interaction

[0291] The user can provide feedback on the displayed search results, for example, by clicking a rating button or entering a comment. The device sends the feedback information to the server, and the server further optimizes the search results based on the feedback.

[0292] Start of purchase flow

[0293] Once the user selects the desired product, the terminal displays the checkout screen, where the user enters their shipping address and payment information. The server processes the purchase request, updates the relevant database, and completes the transaction by contacting an external service that handles payment processing.

[0294] Prompt Sentence Examples

[0295] "How can I use natural language processing techniques to turn text into a search query?"

[0296] This system allows users to obtain search results that reflect their emotional state, enabling them to enjoy a more satisfying online shopping experience. In addition, by combining it with voice and image input, the system can accommodate a wider variety of search conditions.

[0297] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0298] Step 1:

[0299] Users access the system by opening a dedicated application or web browser on their smartphone or PC. The user interface displays a search bar, a voice input button, an image upload button, and a camera icon for emotion recognition, allowing users to select which form of input they want.

[0300] Step 2:

[0301] Input is provided in the form of your choice. For example, you can type "I want black shoes for the office" into the search bar, or press the voice input button to search by voice. You can also click the camera icon to turn on emotion recognition. Input data can include text, voice, image data, and emotion data.

[0302] Step 3:

[0303] The device captures the user's input data and sends it to the server via the network. The data sent can be text, audio, images, emotion recognition data, etc. The most common method of transmission is via HTTP request.

[0304] Step 4:

[0305] The server analyzes the received data. In the case of text data, it is analyzed using natural language processing (NLP) technology and converted into a search query. Voice data is converted into text using voice recognition technology, and image data is used to extract important features using image analysis technology. Emotion recognition data is analyzed by an emotion engine to determine the user's emotional state, such as whether they are tired or happy.

[0306] Input: Text, audio, image, and emotion data from users

[0307] Data processing: natural language processing, speech recognition, image analysis, emotion recognition

[0308] Output: Parsed search queries and sentiment data

[0309] Step 5:

[0310] The server uses the generated search query to search a product database, e.g., using an SQL query, to retrieve a list of relevant products. This database contains up-to-date product information from multiple vendors.

[0311] Input: Search query

[0312] Data Calculation: SQL Query Issue

[0313] Output: List of related products

[0314] Step 6:

[0315] The server passes the retrieved product list to an AI model for ranking. The AI ​​model generates rankings based on the user's past purchase history and ratings, current search criteria, and emotional state. For example, if a user is perceived as "tired," products with a relaxing effect will be displayed preferentially.

[0316] Input: product list, user emotional state

[0317] Data Calculation: Ranking by AI Model

[0318] Output: Ranked product list

[0319] Step 7:

[0320] The device displays a ranked list of products sent from the server to the user, and the search results include product images, prices, user reviews, availability, and recommendations based on emotional state.

[0321] Input: Ranked product list

[0322] Data processing: Display processing on HTML or application

[0323] Output: Search results displayed on the user's screen

[0324] Step 8:

[0325] Users can provide feedback on the search results they see by clicking the rating button or entering comments. The device sends this feedback information to the server, which then uses it to optimize search results and rankings.

[0326] Input: User feedback data

[0327] Data processing: Analysis of feedback information and learning

[0328] Output: Optimized ranking and search results

[0329] Step 9:

[0330] If the user decides to purchase the desired product, the device displays a checkout screen, where the user enters the required shipping address and payment information. The server processes the purchase request, updates the product database, and completes the transaction by contacting an external service that handles payment processing.

[0331] Input: User purchase information, payment information

[0332] Data calculation: Processing purchase information and linking with external services

[0333] Output: Confirmation of successful purchase and notification to the user

[0334] (Application example 2)

[0335] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0336] In conventional online shopping systems, when users select from a large number of products, they are unable to receive suggestions for products that match their current emotions or mood, resulting in a uniform, unpersonalized purchasing experience. Furthermore, the payment process does not take into account the user's emotional state, making it difficult to improve user satisfaction. This raises concerns that users' purchasing motivation may decrease and the conversion rate of e-commerce sites may fall.

[0337] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0338] In this invention, the server includes means for receiving input from a user and transmitting the input to the server, means for the server to analyze the input received and search for corresponding products from a product database, means for selecting optimal products based on the analysis results and providing appropriate search results to the user, and means for recognizing the emotional state of the user and adjusting the search results provided based on the emotional state, thereby enabling the provision of personalized search results that take into account the emotional state of the user and the optimization of the payment interface according to the user's emotions.

[0339] "User" means an individual or legal entity that uses the System to search for and purchase Products.

[0340] "Input" is information a user provides to a system, which may be text, audio, images, or a combination of these.

[0341] A "server" is a computer system that receives and analyzes input submitted by a user.

[0342] The "product database" is a database that stores information about products to be searched.

[0343] "Analysis" refers to the means of processing and understanding received input data, and includes natural language processing, speech recognition, image analysis, emotion recognition, etc.

[0344] "Search Results" are lists of relevant products provided by the server based on the user's input and emotional state.

[0345] "Emotional state" refers to the user's psychological and emotional state, as analyzed from the user's facial expressions, tone of voice, etc.

[0346] An "emotion recognizer" is a combination of software and hardware for analyzing a user's emotional state.

[0347] The "payment interface" is a user interface that allows a user to confirm the purchase of a product and carry out payment procedures.

[0348] "Optimization" is the process of improving the user experience by adjusting payment interfaces and search results based on the user's emotional state.

[0349] MODE FOR CARRYING OUT THE INVENTION

[0350] The present invention is an electronic payment system that uses emotion recognition to provide personalized search results and payment interfaces. The system basically consists of a user terminal, a server, an emotion recognition engine, a natural language processing engine, a voice recognition engine, an image analysis engine, and a product database.

[0351] User Interface Design

[0352] Users access the system using a smartphone or personal computer through a dedicated application or a web browser. The user interface includes a search bar, a voice input button, an image upload button, and a camera icon for emotion recognition, allowing users to provide various forms of input to the system.

[0353] Processing User Input

[0354] Users can enter text into the search bar or use voice input by pressing the voice input button. They can also upload an image or click the emotion recognition camera icon to capture natural facial expressions. For example, they can turn on emotion recognition while typing, "I want black shoes that go with the office."

[0355] Data transmission to the server and analysis

[0356] The device captures input data from the user and sends it over the network to a server. The data sent can be text, voice, images, or emotion recognition data. The server analyzes the received data and generates a search query using appropriate technologies, such as a natural language processing engine, voice recognition engine, image analysis engine, or emotion recognition engine.

[0357] Linking and searching with product databases

[0358] The server uses the generated search query to search the product database, which stores the latest product information, and retrieves a list of matching products. The results are retrieved in the form of a database query, such as an SQL query.

[0359] Product selection and ranking

[0360] The server passes the product list obtained as a search result to an artificial intelligence (AI) model for ranking. The AI ​​model generates rankings by taking into account the user's past purchase history, ratings, current search criteria, and emotional state. For example, if the user's emotion is recognized as "tired," products with a relaxing effect will be displayed preferentially.

[0361] Displaying search results

[0362] The device displays a ranked list of products sent from the server to the user, and the search results include detailed information such as product images, prices, user reviews, availability, and even recommendations based on emotional state.

[0363] Optimizing payment interfaces based on emotion recognition

[0364] Once the user finds the product they want, the terminal will prompt them for payment. The payment interface is optimized based on the user's emotional state. For example, if the user is feeling stressed, the terminal will offer the option to complete the payment with one click.

[0365] Feedback and Interaction

[0366] Users can provide feedback on the search results displayed, and the device will send the feedback information to the server and further optimize the search results based on it. If the user indicates a change in their sentiment, the system will update the search results in real time.

[0367] Examples of concrete examples and prompts

[0368] For example, if a user's camera image indicates that they look tired, a simple interface will be provided to complete payment with one click. Additionally, promotions for products that have a relaxing effect will be displayed based on the user's past purchase history and current emotional state.

[0369] Prompt Sentence Examples

[0370] If the user feels tired, generate a prompt to introduce relaxation items, such as massagers, aroma candles, etc., and provide an interface for one-click payment.

[0371] This allows users to easily find the perfect product based on their emotional state and expedite the checkout process.

[0372] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0373] Program processing steps

[0374] Step 1:

[0375] Users can use their device to enter text into the search bar or use voice input by pressing the voice input button, upload an image, or capture facial expressions by clicking the emotion recognition camera icon.

[0376] Input: text, audio, images

[0377] Output: Input data is captured to the terminal

[0378] Step 2:

[0379] The device transmits the captured input data to a server over a network, including text data, voice data, image data, and emotion recognition data.

[0380] Input: Captured data

[0381] Output: The input data is sent to the server

[0382] Step 3:

[0383] The server analyzes the data it receives: for text data, a natural language processing engine generates search queries; for audio data, a speech recognition engine converts it into text; for image data, an image analysis engine extracts key features; for emotion data, an emotion recognition engine analyzes facial expressions and voice tone to identify emotional states.

[0384] Input: Data sent to the server

[0385] Output: search query, emotional state

[0386] Step 4:

[0387] The server uses the generated search query to search a product database, which contains up-to-date product information, to retrieve a list of suitable products.

[0388] Input: search query

[0389] Output: Product list

[0390] Step 5:

[0391] The server passes the search results to an AI model that ranks the products, taking into account the user's past purchase history, ratings, current search criteria, and emotional state.

[0392] Input: Product list, past purchase history, emotional state

[0393] Output: Ranked product list

[0394] Step 6:

[0395] The server sends the ranked product list to the device, which is also based on the user's emotional state, resulting in personalized results.

[0396] Input: Ranked product list

[0397] Output: Ranked product list sent to device

[0398] Step 7:

[0399] The device displays a ranked list of products sent from the server to the user, and the displayed search results include product images, prices, user reviews, availability, and recommendations based on emotional state.

[0400] Input: Ranked product list

[0401] Output: Product information displayed to the user

[0402] Step 8:

[0403] Once the user finds the product they want, the terminal will prompt them for payment. The payment interface is optimized based on the user's emotional state. For example, if the user is feeling stressed, the terminal will offer the option to complete the payment with one click.

[0404] Input: Emotional state, product selection data

[0405] Output: Payment interface

[0406] Step 9:

[0407] Users can provide feedback on the displayed search results, and the terminal will send the feedback information to the server, which will then further optimize the search results.

[0408] Input: User feedback

[0409] Output: Optimized search results

[0410] Specific example of operation

[0411] For example, the following shows how to generate a prompt sentence that determines that the user is "tired" and displays a promotion for a product that has a relaxing effect.

[0412] Example prompts

[0413] If the user feels tired, generate a prompt to introduce relaxation items, such as massagers, aroma candles, etc., and provide an interface for one-click payment.

[0414] This allows users to easily find the perfect product based on their emotional state and expedite the checkout process.

[0415] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0416] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0417] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0418] [Second embodiment]

[0419] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0420] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0421] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0422] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0423] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0424] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0425] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0426] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0427] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0428] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0429] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0430] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0431] The present invention provides a system that utilizes AI technology to improve the product search experience in order to make product searches more efficient in online shopping. Specific embodiments are described below.

[0432] User Interface Design

[0433] Users access the system through a dedicated application or web browser on their device (smartphone or PC). The user interface is simple and easy to use, with a search bar, voice input button, and image upload button.

[0434] Processing User Input

[0435] Users can enter text into the search bar, press the microphone button to speak, or press the camera button to upload an image, for example, "I'm looking for black shoes for the office" or upload a photo of their feet.

[0436] Sending and parsing input

[0437] The device captures the user's input data and transmits it over the network to a server, either as text, audio, or images.

[0438] The server analyzes the received data. For text data, it uses natural language processing technology to generate search queries. For voice data, it uses voice recognition technology to convert it into text and analyze it. For image data, it uses image analysis technology to extract features of the uploaded image and generate search criteria based on them.

[0439] Linking and searching with product databases

[0440] The server uses the generated search query to search for relevant products in a product database, which contains regularly updated product information from multiple vendors, and returns a list of products that match the search query.

[0441] Product selection and ranking

[0442] The server uses an AI model to rank the product list obtained as a search result, in an optimal order based on the user's past purchase history, ratings, and current search criteria.

[0443] Displaying search results

[0444] The terminal displays the ranked product list sent from the server to the user, and the search results include detailed information such as product images, prices, user reviews, and availability.

[0445] Feedback and interactive search tuning

[0446] Users are also provided with an interface to provide feedback on the search results and further refine their search criteria, for example by lowering the price range or selecting a specific brand.

[0447] The server receives feedback from the user, updates the search criteria in real time, and searches again, resulting in more accurate search results.

[0448] Start of purchase flow

[0449] When a user finds a product they want, they are taken to the product detail page and begin the checkout process, which includes entering their shipping address and payment information.

[0450] The server receives the purchase request, updates the relevant database, and works with an external service that handles payment processing to complete the transaction.

[0451] As described above, this invention uses AI technology to provide appropriate search results even when the user provides vague input, and also supports voice and image input, thereby significantly improving the online shopping experience.

[0452] The processing flow will be explained below.

[0453] Step 1:

[0454] Using a device, a user accesses a search screen. The user can choose to enter text, speak, or upload an image. For example, the user enters "I'm looking for black shoes for the office" into the search bar.

[0455] Step 2:

[0456] The device captures input from the user. In the case of text, it is captured as text data, in the case of voice, the voice data is converted to text, and in the case of images, the image data is captured as is.

[0457] Step 3:

[0458] The device sends the captured input data to the server using a network communication method such as an HTTP POST request, with the data being sent in a properly formatted format such as JSON or XML.

[0459] Step 4:

[0460] The server analyzes the received data. In the case of text data, natural language processing (NLP) technology is used to extract search intent and generate appropriate search queries. In the case of audio data, speech recognition technology is used to convert it into text and analyze it in the same way. In the case of image data, image analysis technology is used to extract important features and generate appropriate search terms.

[0461] Step 5:

[0462] The server uses the generated search query to search a product database, for example by issuing an SQL query, to retrieve a list of matching products. This database contains up-to-date product information from multiple vendors.

[0463] Step 6:

[0464] The server then passes the search results to an AI model for ranking, which uses the user's past purchase history, ratings, and other relevant data to rank the most relevant products.

[0465] Step 7:

[0466] The server then sends the ranked search results to the device, which includes detailed information such as product images, prices, user reviews, and availability.

[0467] Step 8:

[0468] The device receives search results from the server and displays them to the user, along with filtering and sorting options to further refine the search criteria.

[0469] Step 9:

[0470] Users can review the search results and then perform new searches by further narrowing the search criteria, such as selecting a price range or a specific brand.

[0471] Step 10:

[0472] The server receives new feedback from the user and performs the product search again, updating the search criteria in real time to generate optimal results.

[0473] Step 11:

[0474] Once the user selects the desired product, the terminal displays a checkout screen where the user enters their shipping address and payment information.

[0475] Step 12:

[0476] The server receives the purchase request, updates the relevant database, coordinates with external services that handle payment processing, and sends a purchase confirmation to the user once the transaction is complete.

[0477] This step allows users to search for products and complete the purchasing process efficiently and conveniently.

[0478] Example 1

[0479] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0480] Conventional online shopping product search systems have difficulty providing accurate search results when user input is ambiguous or when they need to accept a variety of input formats (text, voice, images). Furthermore, there are still challenges in providing a function that reflects user feedback in real time to update search results, and in providing personalized search results based on past purchase history and ratings.

[0481] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0482] In this invention, the server includes means for receiving input from a user and transmitting the input to the server, means for analyzing the input received by the server and searching for corresponding products from a product database, means for selecting optimal products based on the analysis results using a ranking means and providing appropriate search results to the user, means for receiving feedback from the user and updating search conditions in real time, and means for taking into account the user's past purchase history and ratings in subsequent searches. This makes it possible to provide appropriate search results even when the user has entered ambiguous input, support a variety of input formats, and provide personalized search results.

[0483] "User" refers to an individual or corporation that uses the online shopping system.

[0484] "Input" refers to information provided by a user as a search query, and may be in the form of text, audio, images, or the like.

[0485] "Server" refers to a computing resource on a network that receives and analyzes input data from users and executes product searches.

[0486] "Parsing" refers to the process by which the server interprets the input data it receives, understands its meaning, and generates appropriate search queries.

[0487] "Product database" refers to a data storage device that stores product information provided by multiple vendors.

[0488] "Ranking method" refers to an algorithm that prioritizes search results based on a user's past purchasing history, ratings, and current search criteria.

[0489] "Search Results" refers to a list of products selected by the server based on user input.

[0490] "Feedback" refers to additional instructions or ratings given by a user in response to a search result.

[0491] "Real-time update" means that feedback from users is reflected immediately, and product searches are performed again to provide the latest search results.

[0492] "Purchase history" refers to a record of products a user has purchased in the past.

[0493] "Personalized" refers to providing search results that are customized based on the individual user's preferences and behavioral history.

[0494] The present invention is directed to a system for improving the efficiency of product searches in online shopping. Specific embodiments will be described below.

[0495] Users access the system through a dedicated application or a web browser using devices such as smartphones or PCs. The user interface is designed to be easy to use, with a search bar, voice input button, and image upload button located on the screen.

[0496] Users can enter keywords into the search bar, use voice input by pressing the microphone button, or upload an image by pressing the camera icon. For example, they can enter text such as "I'm looking for black shoes that go well with the office," say "I'm looking for casual sneakers," or upload a photo of the product they're looking for.

[0497] The device encodes the captured input data and securely transmits it to the server. The server then analyzes the received data. Specifically, for text data, natural language processing techniques (e.g., BERT or GPT-3) are used to generate search queries. For audio data, speech recognition techniques (e.g., DeepSpeech or Google Speech-to-Text) are used to convert the data into text and analyze that text. For image data, image analysis techniques (e.g., YOLO or ResNet) are used to extract image features and generate search queries based on them.

[0498] The server uses the analyzed search query to search for relevant products in a product database. The product database contains the latest product information provided by numerous vendors. The server then ranks the retrieved product list using an AI model (e.g., random forest or deep learning model). This ranking is based on the user's past purchase history, ratings, and current search criteria.

[0499] The terminal displays the ranked product list sent from the server to the user. The search results include detailed information such as product images, prices, user reviews, and availability. The user is provided with an interface to provide feedback on the displayed search results and further refine the search criteria. For example, the user can change the price range or select a specific brand.

[0500] The server receives feedback from the user, updates the search criteria in real time, and searches again, resulting in more accurate search results. When the user finds the desired product, they are redirected to the product detail page and begin the purchase process, which includes entering shipping address and payment information. The server receives the purchase request, updates the relevant database, and completes the transaction by interacting with external payment processing services (e.g., Stripe or PayPal).

[0501] Specific examples

[0502] A user types "I want black shoes suitable for the office" into the search bar. The device sends this text to the server, which uses natural language processing technology to generate a search query and search a product database. Results such as "black leather shoes suitable for the office" are returned and displayed on the device.

[0503] The user presses the microphone button and says, "I want casual sneakers." The device captures the voice data and sends it to the server. The server uses speech recognition technology to convert the speech into text and generate a search query. Search results for casual sneakers are returned and displayed on the device.

[0504] Example prompt: "If a user is looking for a simple bag, how can you use AI technology to provide the best search results?"

[0505] As described above, the present invention is a system that uses AI technology to provide appropriate search results even when the user enters ambiguous information, and also supports voice and image input, in order to significantly improve the online shopping experience.

[0506] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0507] Step 1: Launch the user interface

[0508] The user launches a dedicated application or web browser on a device such as a smartphone or PC, and the user interface displays a search bar, a voice input button, and an image upload button.

[0509] Input: A user action (launching an application or web browser)

[0510] Output: Display of the user interface

[0511] Step 2: Receiving user input

[0512] Users can enter text keywords into the search bar, press the microphone button to input using their voice, or press the image upload button to upload an image, for example, by typing "black shoes," saying "looking for casual t-shirts," or uploading a photo of a product.

[0513] Input: Text input, voice input, or image upload

[0514] Output: Captured input data (text, audio, images)

[0515] Step 3: Capturing and sending input

[0516] The terminal encodes the input data captured from the user and transmits it over the network to the server.

[0517] Input: Captured input data (text, audio, images)

[0518] Output: Sends the encoded input data to the server

[0519] Step 4: Data analysis by the server

[0520] The server analyzes the received data using appropriate techniques: text data is analyzed using natural language processing techniques (e.g., GPT-3) to generate search queries; voice data is converted to text using speech recognition techniques (e.g., Google Speech-to-Text) and analyzed; and image data is analyzed using image analysis techniques (e.g., YOLO) to extract features and generate search queries.

[0521] Input: Transmitted encoded data (text, audio, image)

[0522] Output: Parsed search query

[0523] Step 5: Product database search

[0524] The server uses the generated search query to search for the relevant product in a product database, which stores up-to-date product information provided by multiple vendors.

[0525] Input: Parsed search query

[0526] Output: Product list as search results

[0527] Step 6: Product Listing Selection and Ranking

[0528] The server then uses an AI model (e.g., a deep learning model) to rank the retrieved product list, based on the user's past purchase history, ratings, and current search criteria.

[0529] Input: Product list as search results

[0530] Output: Ranked product list

[0531] Step 7: Viewing the ranking results

[0532] The terminal receives the ranked product list sent from the server and displays it on the user interface. The search results include information such as product images, prices, user reviews, and stock status.

[0533] Input: Ranked product list

[0534] Output: Display of search results with detailed information

[0535] Step 8: Receive feedback and refine your search criteria

[0536] Users can provide feedback on the displayed search results and enter new criteria, such as a specific price range or brand.

[0537] Input: User feedback (new conditions)

[0538] Output: Updated feedback data

[0539] Step 9: Re-search based on feedback

[0540] The server receives feedback from the user, updates the search criteria in real time, and performs the search again, resulting in more accurate search results.

[0541] Input: Updated feedback data

[0542] Output: Updated product list searched again

[0543] Step 10: Start Checkout

[0544] When the user selects the desired product, they are taken to the product details page and begin the purchase process. They enter their shipping address and payment information to confirm the purchase.

[0545] Input: User's selected product, shipping address, payment information

[0546] Output: Purchase request completed

[0547] Step 11: Complete the purchase process

[0548] The server receives the purchase request, updates the relevant database, and coordinates with an external service (e.g., payment processing software) that handles the payment process and completes the transaction.

[0549] Input: Purchase request and payment information

[0550] Output: Transaction completed and database updated

[0551] (Application example 1)

[0552] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0553] Conventional online shopping systems make it difficult for users to efficiently find the products they are looking for, and the interfaces are cumbersome. Furthermore, especially for use in autonomous vehicles, voice-only operation and image input are important, but there is a lack of technology to properly process these. Therefore, there is a demand for a system that allows users to enjoy a comfortable and efficient online shopping experience inside the car.

[0554] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0555] In this invention, the server includes a means for receiving input from a user and transmitting the input to the server, a means for the server to analyze the received input and search for corresponding products from a product database, and a means for selecting optimal products based on the analysis results and providing appropriate search results to the user. This makes it possible to provide a system that combines a means for capturing a user's voice input using an in-vehicle voice recognition device and a means for visually displaying search results using an in-vehicle infotainment system and allowing the user to select products via a touch screen or voice control. This enables users to efficiently and comfortably shop online even while in an autonomous vehicle.

[0556] The "means for receiving input from a user and transmitting the input to a server" is a part of a system that has the function of capturing input data, such as voice or images, from a user and transmitting the data to a server.

[0557] "Means for the server to analyze the input received and search for the relevant product in the product database" is part of a system that has the processing power to process the text, voice, or image data received by the server from the user to find the relevant product.

[0558] "Means for selecting the most suitable product based on the analysis results and providing the user with appropriate search results" refers to part of a system that has the function of allowing the server to select the most suitable product based on the analysis results of the input data and display that information on the user's device.

[0559] The "means for capturing user voice input using an in-vehicle voice recognition device" is part of a system that receives user voice commands within an autonomous vehicle and captures them as data.

[0560] "Means of using the in-vehicle infotainment system to visually display search results and allow the user to select products via touchscreen or voice control" refers to a part of a system that displays search results on a display inside an autonomous vehicle and allows the user to select products via touchscreen or voice command.

[0561] "Natural language processing means that generates appropriate search queries even when ambiguous input is given" refers to technology that properly interprets unclear expressions or ambiguous words used by users and generates optimal search queries.

[0562] "Means for receiving voice input and image input, analyzing the input data, and using it as search conditions" refers to part of a system that has the function of receiving voice and image data, analyzing the data, and using it as product search conditions.

[0563] The present invention provides a system that allows a user to comfortably and efficiently perform online shopping while inside an autonomous vehicle. Specific embodiments will be described below.

[0564] The system of the present invention provides a means for users to search for and purchase products using voice and images, and includes the following main components:

[0565] 1. Capture audio and video inputs

[0566] A voice recognition device is used to capture voice inputs from users inside the autonomous vehicle. The device detects the user's voice through a microphone installed in the vehicle and converts it into digital data. Also, if the user uploads an image from their smartphone, it will be captured by the vehicle's infotainment system.

[0567] 2. Data submission and analysis

[0568] The voice and image data captured from the user is sent via the infotainment system to a server, which analyzes the data using the following techniques:

[0569] Speech Recognition: Converts speech data into text using Python and TensorFlow.

[0570] Image analysis: Using OpenCV, features of uploaded images are extracted and search criteria are generated based on them.

[0571] 3. Product Search and Query Generation

[0572] The server uses natural language processing technology to generate a search query based on the received text data and image analysis data, and uses this query to search for matching products in a product database.

[0573] 4. Displaying search results

[0574] Search results are displayed on the infotainment system display, which can be used to select products using a touchscreen or voice control. Search results include detailed information such as product images, prices, and user reviews.

[0575] 5. Purchase Procedure

[0576] A checkout interface is provided to purchase the selected items immediately, with shipping address and payment information securely processed within the system.

[0577] Specific examples

[0578] For example, if a user were to say, "I'm looking for black shoes for the office, especially leather ones," the voice recognition device would capture this speech, convert it into text data, and send it to the server. The server would then use natural language processing technology to generate a search query and search for matching products in a product database. The search results would then be displayed on the infotainment system's display.

[0579] A specific example of a prompt sentence is a voice input such as "I'm looking for black shoes that can be used in the office. I'd especially like leather shoes."

[0580] The system of the present invention allows users to shop online efficiently and comfortably even while inside an autonomous vehicle.

[0581] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0582] Step 1:

[0583] The user inputs voice or uploads images from a smartphone while in an autonomous vehicle. The voice recognition device captures the user's voice through a microphone in the car and converts it into digital voice data. Images are captured through the infotainment system. Input: User's voice or image. Output: Digital voice data or image data.

[0584] Step 2:

[0585] The device sends the captured digital audio or image data to the server. The data is transferred to the server via the infotainment system and over the network. Input: Digital audio or image data. Output: Data sent to the server.

[0586] Step 3:

[0587] The server analyzes the digital voice data it receives and converts it into text data using voice recognition technology. Speech recognition is performed using Python and TensorFlow. In the case of image data, features are extracted using OpenCV. Input: Digital voice data or image data. Output: Text data or image feature data.

[0588] Step 4:

[0589] The server generates search queries using natural language processing technology based on text data and image feature data. A generative AI model is used to generate appropriate search queries even with ambiguous input. Input: Text data or image feature data. Output: Search query.

[0590] Step 5:

[0591] The server uses the generated search query to search for matching products in a product database that contains up-to-date product information from multiple vendors. Input: Search query. Output: Search result list.

[0592] Step 6:

[0593] The server ranks the search results and sends the top results to the terminal. The ranking is done in the optimal order based on the user's past purchase history, ratings, and current search conditions. Input: Search result list. Output: Ranked product list.

[0594] Step 7:

[0595] The terminal displays the ranked product list sent from the server on the display of the infotainment system in the car. The user can select products using the touch screen or voice control. Input: Ranked product list. Output: Product list displayed on the display.

[0596] Step 8:

[0597] The user selects the desired product on the display and begins the purchase process. The terminal prompts the user to enter the shipping address and payment information, which are then sent to the server. Input: User-selected product and purchase information. Output: Purchase request sent to the server.

[0598] Step 9:

[0599] The server receives the purchase request, updates the relevant database, and completes the transaction by contacting an external service responsible for payment processing. The item is then shipped to the specified delivery address. Input: Purchase request. Output: Completed transaction and shipping instructions.

[0600] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0601] The present invention provides a system for providing more personalized search results by combining product search and user emotion recognition in online shopping. This system improves the user experience by recognizing the user's emotions and making appropriate product suggestions based on those emotions. Specific embodiments are described below.

[0602] User Interface Design

[0603] Users access the system via a dedicated application or web browser on their smartphone or PC. The user interface includes a search bar, a voice input button, an image upload button, and a camera icon for emotion recognition.

[0604] Processing User Input

[0605] Users can enter text into the search bar, or use voice input by pressing the microphone button. They can also upload an image by pressing the camera button or click the emotion recognition camera icon. For example, they can turn on emotion recognition by typing, "I want black shoes that go with the office."

[0606] Sending and parsing input

[0607] The device captures input data from the user and transmits it over the network to a server, which can be text, voice, images, or emotion recognition data.

[0608] The server analyzes the received data. For text data, it uses natural language processing technology to generate search queries. For voice data, it uses voice recognition technology to convert it into text and analyze it in the same way. For image data, it uses image analysis technology to extract important features and generate relevant search terms. Furthermore, for emotion recognition data, an emotion engine analyzes facial expressions and voice tone to identify the user's emotional state.

[0609] Linking and searching with product databases

[0610] The server uses the generated search query to search a product database, for example by issuing an SQL query, to retrieve a list of matching products. This database contains up-to-date product information from multiple vendors.

[0611] Product selection and ranking

[0612] The server passes the product list obtained as a search result to the AI ​​model, which then ranks the products. The AI ​​model generates rankings by taking into account the user's past purchase history, ratings, current search criteria, and even their emotional state. For example, if the AI ​​model detects that the user is "tired," it will prioritize products with a relaxing effect.

[0613] Displaying search results

[0614] The device displays a ranked list of products sent from the server to the user, and the search results include detailed information such as product images, prices, user reviews, availability, and even recommendations based on emotional state.

[0615] Feedback and Interaction

[0616] Users can provide feedback on the search results displayed, and the device will send the feedback information to the server, which will then further optimize the search results. In this process, the system will update the search results in real time, even if the user indicates a change in sentiment.

[0617] Start of purchase flow

[0618] When the user finds the product they want, the device displays a checkout screen where they can enter their shipping address and payment information.

[0619] The server receives the purchase request, updates the relevant database, and completes the transaction by connecting with an external service responsible for payment processing. Based on emotion recognition, the user's purchasing experience is personalized.

[0620] This embodiment allows users to obtain search results that reflect their emotional state, allowing them to enjoy more satisfying online shopping. Furthermore, by combining this with voice and image input, the system can accommodate a wider variety of search conditions.

[0621] The processing flow will be explained below.

[0622] Step 1:

[0623] Users access the system through a dedicated application or web browser on their device by entering text into the search bar, pressing the microphone button for voice input, pressing the camera button to upload an image, or clicking the camera icon for emotion recognition.

[0624] Step 2:

[0625] The device captures the user's input data. In the case of text input, it is saved as text data. In the case of voice input, it is converted into text using voice recognition technology. In the case of image input, the image data is saved. It also captures the user's facial expressions and tone of voice using a camera and microphone dedicated to emotion recognition.

[0626] Step 3:

[0627] The device sends the captured input data to the server using a network communication method such as an HTTP POST request, and the data is sent as properly formatted JSON or XML.

[0628] Step 4:

[0629] The server analyzes the received data. In the case of text data, natural language processing (NLP) technology is used to extract the user's intent and convert it into an appropriate search query. In the case of voice data, voice recognition technology is used to convert it into text, which is then analyzed as text data. In the case of image data, image analysis technology is used to extract important features to use as search criteria. Furthermore, in the case of emotion recognition data, an emotion engine analyzes facial expressions and voice tone to identify the user's emotional state.

[0630] Step 5:

[0631] The server uses the generated search query to search the product database using an SQL query or API request to retrieve a list of matching products based on the product information in the database.

[0632] Step 6:

[0633] The server passes the returned product list to the AI ​​model for ranking. This ranking is based on the user's past purchase history, ratings, and current search criteria. In addition, the model takes into account the user's emotional state and prioritizes the most appropriate products based on that emotional state.

[0634] Step 7:

[0635] The server sends the ranked search results to the device, including detailed information such as product images, prices, user reviews, availability, and recommendations (including explanations of the reasons for recommendations if they are based on emotional states).

[0636] Step 8:

[0637] The device will then display the search results received from the server to the user, who can then review the displayed results and use filter options and sorting functions to further refine the search criteria.

[0638] Step 9:

[0639] Users can provide feedback on the search results they see, for example, by setting more specific criteria (price, brand, etc.) based on a particular emotional state (e.g., wanting to relax).

[0640] Step 10:

[0641] The server receives feedback from users, updates the search criteria in real time, and searches again, re-ranking the search results and generating the best product list based on the latest information.

[0642] Step 11:

[0643] When the user finds the product they want, the device displays a checkout screen where they can enter their shipping address and payment information to complete the purchase.

[0644] Step 12:

[0645] The server receives the purchase request, updates the relevant database, and completes the transaction by contacting an external payment processing service. Once the purchase is complete, it sends a confirmation to the user.

[0646] This step allows users to efficiently and personalizedly search and purchase products.

[0647] Example 2

[0648] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0649] Current online shopping systems generate search results based solely on input data provided by the user, making it difficult to provide product suggestions that reflect the user's emotional state or current mood. This can result in a lack of optimal suggestions for the products the user is looking for, which can lead to low satisfaction. Furthermore, they are unable to effectively utilize various input formats, such as voice input or image input, preventing flexible searches. Furthermore, they lack a mechanism for recognizing user emotions in real time and reflecting them in search results, making it difficult to provide more personalized search results for each individual user.

[0650] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0651] In this invention, the server includes a means for receiving input from a user and transmitting the input to the server, a means for analyzing the input received by the server and searching for corresponding products from a product database, a means for selecting optimal products based on the analysis results and providing the user with appropriate search results, a means for recognizing the user's emotions and making product suggestions based on the emotions, and a means for analyzing text, voice, and images provided by the user to generate search criteria. This enables personalized product suggestions that reflect the user's emotional state, improving the user experience. Furthermore, flexible search criteria generation is realized by effectively utilizing various input formats, including voice input and image input.

[0652] "User" refers to an individual or organization that uses the System to search for and purchase products.

[0653] "Input" means data provided by a user to the system, and may include any one or more of text, voice, image, and / or emotion recognition data.

[0654] "Server" refers to a central processing unit that analyzes input data received from users and searches the product database and makes product suggestions based on the results.

[0655] "Product Database" refers to a data storage system that stores up-to-date product information provided by multiple vendors.

[0656] "Analysis" refers to the process by which the server processes input data received from the user and extracts meaningful information.

[0657] "Product suggestion" refers to the server presenting suitable products to the user based on the analysis results.

[0658] "Emotion recognition" refers to the process of analyzing a user's facial expressions and tone of voice to determine their current emotional state.

[0659] "Natural language processing" refers to technology that enables computers to understand and analyze the language that humans use on a daily basis.

[0660] "Speech recognition" refers to the technology of converting voice data into text data.

[0661] "Image analysis" refers to the technology of extracting important features and information from image data.

[0662] "Ranking" refers to the process of prioritizing products based on a user's purchasing history, ratings, current search criteria, and emotional state.

[0663] "Feedback" refers to ratings and comments provided by users that are used to improve the performance of the system and the quality of suggestions.

[0664] "Checkout" refers to the process by which a User enters the necessary information to purchase a selected Product and completes the transaction.

[0665] The present invention provides a system for providing more personalized search results by combining product search and user emotion recognition in online shopping. This system improves the user experience by recognizing the user's emotions and suggesting appropriate products based on those emotions. Specific embodiments are described below.

[0666] User Interface Design

[0667] Users access the system through a dedicated application or a web browser on their smartphone or PC. The user interface includes a search bar, a voice input button, an image upload button, and a camera icon for emotion recognition, allowing users to input search requests in a variety of ways.

[0668] Processing User Input

[0669] Users can enter specific search criteria using text input, voice input, image upload, and emotion recognition. For example, a user may enter text such as "I'm looking for black shoes for the office" and enable emotion recognition. In this case, the user can also use voice input and image upload.

[0670] Sending and parsing input

[0671] The device captures the user's input data and transmits it to a server over a network, including text, voice, images, and emotion recognition data.

[0672] The server analyzes the received data, specifically:

[0673] The text data is analyzed using natural language processing techniques to generate search queries.

[0674] The voice data is converted into text using voice recognition technology, and the text is analyzed.

[0675] Image data is subjected to image analysis techniques to extract features.

[0676] The emotion recognition data is analyzed by an emotion engine to identify the user's emotional state.

[0677] Linking and searching with product databases

[0678] The server uses the generated search query to search a product database, which contains up-to-date product information and data provided by multiple vendors. For example, it issues an SQL query to retrieve a list of matching products.

[0679] Product Selection and Ranking

[0680] The server passes the retrieved product list to an AI model for ranking. The AI ​​model generates rankings by taking into account the user's past purchase history and ratings, current search criteria, and emotional state. For example, if the user's emotion is recognized as "tired," products with a relaxing effect may be displayed preferentially.

[0681] Displaying search results

[0682] The terminal displays a ranked list of products sent from the server to the user, and the search results include detailed information such as product images, prices, user reviews, availability, and reasons for recommendation.

[0683] Feedback and Interaction

[0684] The user can provide feedback on the displayed search results, for example, by clicking a rating button or entering a comment. The device sends the feedback information to the server, and the server further optimizes the search results based on the feedback.

[0685] Start of purchase flow

[0686] Once the user selects the desired product, the terminal displays the checkout screen, where the user enters their shipping address and payment information. The server processes the purchase request, updates the relevant database, and completes the transaction by contacting an external service that handles payment processing.

[0687] Prompt Sentence Examples

[0688] "How can I use natural language processing techniques to turn text into a search query?"

[0689] This system allows users to obtain search results that reflect their emotional state, enabling them to enjoy a more satisfying online shopping experience. In addition, by combining it with voice and image input, the system can accommodate a wider variety of search conditions.

[0690] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0691] Step 1:

[0692] Users access the system by opening a dedicated application or web browser on their smartphone or PC. The user interface displays a search bar, a voice input button, an image upload button, and a camera icon for emotion recognition, allowing users to select which form of input they want.

[0693] Step 2:

[0694] Input is provided in the form of your choice. For example, you can type "I want black shoes for the office" into the search bar, or press the voice input button to search by voice. You can also click the camera icon to turn on emotion recognition. Input data can include text, voice, image data, and emotion data.

[0695] Step 3:

[0696] The device captures the user's input data and sends it to the server via the network. The data sent can be text, audio, images, emotion recognition data, etc. The most common method of transmission is via HTTP request.

[0697] Step 4:

[0698] The server analyzes the received data. In the case of text data, it is analyzed using natural language processing (NLP) technology and converted into a search query. Voice data is converted into text using voice recognition technology, and image data is used to extract important features using image analysis technology. Emotion recognition data is analyzed by an emotion engine to determine the user's emotional state, such as whether they are tired or happy.

[0699] Input: Text, audio, image, and emotion data from users

[0700] Data processing: natural language processing, speech recognition, image analysis, emotion recognition

[0701] Output: Parsed search queries and sentiment data

[0702] Step 5:

[0703] The server uses the generated search query to search a product database, e.g., using an SQL query, to retrieve a list of relevant products. This database contains up-to-date product information from multiple vendors.

[0704] Input: Search query

[0705] Data Calculation: SQL Query Issue

[0706] Output: List of related products

[0707] Step 6:

[0708] The server passes the retrieved product list to an AI model for ranking. The AI ​​model generates rankings based on the user's past purchase history and ratings, current search criteria, and emotional state. For example, if a user is perceived as "tired," products with a relaxing effect will be displayed preferentially.

[0709] Input: product list, user emotional state

[0710] Data Calculation: Ranking by AI Model

[0711] Output: Ranked product list

[0712] Step 7:

[0713] The device displays a ranked list of products sent from the server to the user, and the search results include product images, prices, user reviews, availability, and recommendations based on emotional state.

[0714] Input: Ranked product list

[0715] Data processing: Display processing on HTML or application

[0716] Output: Search results displayed on the user's screen

[0717] Step 8:

[0718] Users can provide feedback on the search results they see by clicking the rating button or entering comments. The device sends this feedback information to the server, which then uses it to optimize search results and rankings.

[0719] Input: User feedback data

[0720] Data processing: Analysis of feedback information and learning

[0721] Output: Optimized ranking and search results

[0722] Step 9:

[0723] If the user decides to purchase the desired product, the device displays a checkout screen, where the user enters the required shipping address and payment information. The server processes the purchase request, updates the product database, and completes the transaction by contacting an external service that handles payment processing.

[0724] Input: User purchase information, payment information

[0725] Data calculation: Processing purchase information and linking with external services

[0726] Output: Confirmation of successful purchase and notification to the user

[0727] (Application example 2)

[0728] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0729] In conventional online shopping systems, when users select from a large number of products, they are unable to receive suggestions for products that match their current emotions or mood, resulting in a uniform, unpersonalized purchasing experience. Furthermore, the payment process does not take into account the user's emotional state, making it difficult to improve user satisfaction. This raises concerns that users' purchasing motivation may decrease and the conversion rate of e-commerce sites may fall.

[0730] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0731] In this invention, the server includes means for receiving input from a user and transmitting the input to the server, means for the server to analyze the input received and search for corresponding products from a product database, means for selecting optimal products based on the analysis results and providing appropriate search results to the user, and means for recognizing the emotional state of the user and adjusting the search results provided based on the emotional state, thereby enabling the provision of personalized search results that take into account the emotional state of the user and the optimization of the payment interface according to the user's emotions.

[0732] "User" means an individual or legal entity that uses the System to search for and purchase Products.

[0733] "Input" is information a user provides to a system, which may be text, audio, images, or a combination of these.

[0734] A "server" is a computer system that receives and analyzes input submitted by a user.

[0735] The "product database" is a database that stores information about products to be searched.

[0736] "Analysis" refers to the means of processing and understanding received input data, and includes natural language processing, speech recognition, image analysis, emotion recognition, etc.

[0737] "Search Results" are lists of relevant products provided by the server based on the user's input and emotional state.

[0738] "Emotional state" refers to the user's psychological and emotional state, as analyzed from the user's facial expressions, tone of voice, etc.

[0739] An "emotion recognizer" is a combination of software and hardware for analyzing a user's emotional state.

[0740] The "payment interface" is a user interface that allows a user to confirm the purchase of a product and carry out payment procedures.

[0741] "Optimization" is the process of improving the user experience by adjusting payment interfaces and search results based on the user's emotional state.

[0742] MODE FOR CARRYING OUT THE INVENTION

[0743] The present invention is an electronic payment system that uses emotion recognition to provide personalized search results and payment interfaces. The system basically consists of a user terminal, a server, an emotion recognition engine, a natural language processing engine, a voice recognition engine, an image analysis engine, and a product database.

[0744] User Interface Design

[0745] Users access the system using a smartphone or personal computer through a dedicated application or a web browser. The user interface includes a search bar, a voice input button, an image upload button, and a camera icon for emotion recognition, allowing users to provide various forms of input to the system.

[0746] Processing User Input

[0747] Users can enter text into the search bar or use voice input by pressing the voice input button. They can also upload an image or click the emotion recognition camera icon to capture natural facial expressions. For example, they can turn on emotion recognition while typing, "I want black shoes that go with the office."

[0748] Data transmission to the server and analysis

[0749] The device captures input data from the user and sends it over the network to a server. The data sent can be text, voice, images, or emotion recognition data. The server analyzes the received data and generates a search query using appropriate technologies, such as a natural language processing engine, voice recognition engine, image analysis engine, or emotion recognition engine.

[0750] Linking and searching with product databases

[0751] The server uses the generated search query to search the product database, which stores the latest product information, and retrieves a list of matching products. The results are retrieved in the form of a database query, such as an SQL query.

[0752] Product selection and ranking

[0753] The server passes the product list obtained as a search result to an artificial intelligence (AI) model for ranking. The AI ​​model generates rankings by taking into account the user's past purchase history, ratings, current search criteria, and emotional state. For example, if the user's emotion is recognized as "tired," products with a relaxing effect will be displayed preferentially.

[0754] Displaying search results

[0755] The device displays a ranked list of products sent from the server to the user, and the search results include detailed information such as product images, prices, user reviews, availability, and even recommendations based on emotional state.

[0756] Optimizing payment interfaces based on emotion recognition

[0757] Once the user finds the product they want, the terminal will prompt them for payment. The payment interface is optimized based on the user's emotional state. For example, if the user is feeling stressed, the terminal will offer the option to complete the payment with one click.

[0758] Feedback and Interaction

[0759] Users can provide feedback on the search results displayed, and the device will send the feedback information to the server and further optimize the search results based on it. If the user indicates a change in their sentiment, the system will update the search results in real time.

[0760] Examples of concrete examples and prompts

[0761] For example, if a user's camera image indicates that they look tired, a simple interface will be provided to complete payment with one click. Additionally, promotions for products that have a relaxing effect will be displayed based on the user's past purchase history and current emotional state.

[0762] Prompt Sentence Examples

[0763] If the user feels tired, generate a prompt to introduce relaxation items, such as massagers, aroma candles, etc., and provide an interface for one-click payment.

[0764] This allows users to easily find the perfect product based on their emotional state and expedite the checkout process.

[0765] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0766] Program processing steps

[0767] Step 1:

[0768] Users can use their device to enter text into the search bar or use voice input by pressing the voice input button, upload an image, or capture facial expressions by clicking the emotion recognition camera icon.

[0769] Input: text, audio, images

[0770] Output: Input data is captured to the terminal

[0771] Step 2:

[0772] The device transmits the captured input data to a server over a network, including text data, voice data, image data, and emotion recognition data.

[0773] Input: Captured data

[0774] Output: The input data is sent to the server

[0775] Step 3:

[0776] The server analyzes the data it receives: for text data, a natural language processing engine generates search queries; for audio data, a speech recognition engine converts it into text; for image data, an image analysis engine extracts key features; for emotion data, an emotion recognition engine analyzes facial expressions and voice tone to identify emotional states.

[0777] Input: Data sent to the server

[0778] Output: search query, emotional state

[0779] Step 4:

[0780] The server uses the generated search query to search a product database, which contains up-to-date product information, to retrieve a list of suitable products.

[0781] Input: search query

[0782] Output: Product list

[0783] Step 5:

[0784] The server passes the search results to an AI model that ranks the products, taking into account the user's past purchase history, ratings, current search criteria, and emotional state.

[0785] Input: Product list, past purchase history, emotional state

[0786] Output: Ranked product list

[0787] Step 6:

[0788] The server sends the ranked product list to the device, which is also based on the user's emotional state, resulting in personalized results.

[0789] Input: Ranked product list

[0790] Output: Ranked product list sent to device

[0791] Step 7:

[0792] The device displays a ranked list of products sent from the server to the user, and the displayed search results include product images, prices, user reviews, availability, and recommendations based on emotional state.

[0793] Input: Ranked product list

[0794] Output: Product information displayed to the user

[0795] Step 8:

[0796] Once the user finds the product they want, the terminal will prompt them for payment. The payment interface is optimized based on the user's emotional state. For example, if the user is feeling stressed, the terminal will offer the option to complete the payment with one click.

[0797] Input: Emotional state, product selection data

[0798] Output: Payment interface

[0799] Step 9:

[0800] Users can provide feedback on the displayed search results, and the terminal will send the feedback information to the server, which will then further optimize the search results.

[0801] Input: User feedback

[0802] Output: Optimized search results

[0803] Specific example of operation

[0804] For example, the following shows how to generate a prompt sentence that determines that the user is "tired" and displays a promotion for a product that has a relaxing effect.

[0805] Example prompts

[0806] If the user feels tired, generate a prompt to introduce relaxation items, such as massagers, aroma candles, etc., and provide an interface for one-click payment.

[0807] This allows users to easily find the perfect product based on their emotional state and expedite the checkout process.

[0808] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0809] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0810] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0811] [Third embodiment]

[0812] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0813] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0814] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0815] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0816] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0817] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0818] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0819] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0820] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0821] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0822] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0823] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0824] The present invention provides a system that utilizes AI technology to improve the product search experience in order to make product searches more efficient in online shopping. Specific embodiments are described below.

[0825] User Interface Design

[0826] Users access the system through a dedicated application or web browser on their device (smartphone or PC). The user interface is simple and easy to use, with a search bar, voice input button, and image upload button.

[0827] Processing User Input

[0828] Users can enter text into the search bar, press the microphone button to speak, or press the camera button to upload an image, for example, "I'm looking for black shoes for the office" or upload a photo of their feet.

[0829] Sending and parsing input

[0830] The device captures the user's input data and transmits it over the network to a server, either as text, audio, or images.

[0831] The server analyzes the received data. For text data, it uses natural language processing technology to generate search queries. For voice data, it uses voice recognition technology to convert it into text and analyze it. For image data, it uses image analysis technology to extract features of the uploaded image and generate search criteria based on them.

[0832] Linking and searching with product databases

[0833] The server uses the generated search query to search for relevant products in a product database, which contains regularly updated product information from multiple vendors, and returns a list of products that match the search query.

[0834] Product selection and ranking

[0835] The server uses an AI model to rank the product list obtained as a search result, in an optimal order based on the user's past purchase history, ratings, and current search criteria.

[0836] Displaying search results

[0837] The terminal displays the ranked product list sent from the server to the user, and the search results include detailed information such as product images, prices, user reviews, and availability.

[0838] Feedback and interactive search tuning

[0839] Users are also provided with an interface to provide feedback on the search results and further refine their search criteria, for example by lowering the price range or selecting a specific brand.

[0840] The server receives feedback from the user, updates the search criteria in real time, and searches again, resulting in more accurate search results.

[0841] Start of purchase flow

[0842] When a user finds a product they want, they are taken to the product detail page and begin the checkout process, which includes entering their shipping address and payment information.

[0843] The server receives the purchase request, updates the relevant database, and works with an external service that handles payment processing to complete the transaction.

[0844] As described above, this invention uses AI technology to provide appropriate search results even when the user provides vague input, and also supports voice and image input, thereby significantly improving the online shopping experience.

[0845] The processing flow will be explained below.

[0846] Step 1:

[0847] Using a device, a user accesses a search screen. The user can choose to enter text, speak, or upload an image. For example, the user enters "I'm looking for black shoes for the office" into the search bar.

[0848] Step 2:

[0849] The device captures input from the user. In the case of text, it is captured as text data, in the case of voice, the voice data is converted to text, and in the case of images, the image data is captured as is.

[0850] Step 3:

[0851] The device sends the captured input data to the server using a network communication method such as an HTTP POST request, with the data being sent in a properly formatted format such as JSON or XML.

[0852] Step 4:

[0853] The server analyzes the received data. In the case of text data, natural language processing (NLP) technology is used to extract search intent and generate appropriate search queries. In the case of audio data, speech recognition technology is used to convert it into text and analyze it in the same way. In the case of image data, image analysis technology is used to extract important features and generate appropriate search terms.

[0854] Step 5:

[0855] The server uses the generated search query to search a product database, for example by issuing an SQL query, to retrieve a list of matching products. This database contains up-to-date product information from multiple vendors.

[0856] Step 6:

[0857] The server then passes the search results to an AI model for ranking, which uses the user's past purchase history, ratings, and other relevant data to rank the most relevant products.

[0858] Step 7:

[0859] The server then sends the ranked search results to the device, which includes detailed information such as product images, prices, user reviews, and availability.

[0860] Step 8:

[0861] The device receives search results from the server and displays them to the user, along with filtering and sorting options to further refine the search criteria.

[0862] Step 9:

[0863] Users can review the search results and then perform new searches by further narrowing the search criteria, such as selecting a price range or a specific brand.

[0864] Step 10:

[0865] The server receives new feedback from the user and performs the product search again, updating the search criteria in real time to generate optimal results.

[0866] Step 11:

[0867] Once the user selects the desired product, the terminal displays a checkout screen where the user enters their shipping address and payment information.

[0868] Step 12:

[0869] The server receives the purchase request, updates the relevant database, coordinates with external services that handle payment processing, and sends a purchase confirmation to the user once the transaction is complete.

[0870] This step allows users to search for products and complete the purchasing process efficiently and conveniently.

[0871] Example 1

[0872] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0873] Conventional online shopping product search systems have difficulty providing accurate search results when user input is ambiguous or when they need to accept a variety of input formats (text, voice, images). Furthermore, there are still challenges in providing a function that reflects user feedback in real time to update search results, and in providing personalized search results based on past purchase history and ratings.

[0874] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0875] In this invention, the server includes means for receiving input from a user and transmitting the input to the server, means for analyzing the input received by the server and searching for corresponding products from a product database, means for selecting optimal products based on the analysis results using a ranking means and providing appropriate search results to the user, means for receiving feedback from the user and updating search conditions in real time, and means for taking into account the user's past purchase history and ratings in subsequent searches. This makes it possible to provide appropriate search results even when the user has entered ambiguous input, support a variety of input formats, and provide personalized search results.

[0876] "User" refers to an individual or corporation that uses the online shopping system.

[0877] "Input" refers to information provided by a user as a search query, and may be in the form of text, audio, images, or the like.

[0878] "Server" refers to a computing resource on a network that receives and analyzes input data from users and executes product searches.

[0879] "Parsing" refers to the process by which the server interprets the input data it receives, understands its meaning, and generates appropriate search queries.

[0880] "Product database" refers to a data storage device that stores product information provided by multiple vendors.

[0881] "Ranking method" refers to an algorithm that prioritizes search results based on a user's past purchasing history, ratings, and current search criteria.

[0882] "Search Results" refers to a list of products selected by the server based on user input.

[0883] "Feedback" refers to additional instructions or ratings given by a user in response to a search result.

[0884] "Real-time update" means that feedback from users is reflected immediately, and product searches are performed again to provide the latest search results.

[0885] "Purchase history" refers to a record of products a user has purchased in the past.

[0886] "Personalized" refers to providing search results that are customized based on the individual user's preferences and behavioral history.

[0887] The present invention is directed to a system for improving the efficiency of product searches in online shopping. Specific embodiments will be described below.

[0888] Users access the system through a dedicated application or a web browser using devices such as smartphones or PCs. The user interface is designed to be easy to use, with a search bar, voice input button, and image upload button located on the screen.

[0889] Users can enter keywords into the search bar, use voice input by pressing the microphone button, or upload an image by pressing the camera icon. For example, they can enter text such as "I'm looking for black shoes that go well with the office," say "I'm looking for casual sneakers," or upload a photo of the product they're looking for.

[0890] The device encodes the captured input data and securely transmits it to the server. The server then analyzes the received data. Specifically, for text data, natural language processing techniques (e.g., BERT or GPT-3) are used to generate search queries. For audio data, speech recognition techniques (e.g., DeepSpeech or Google Speech-to-Text) are used to convert the data into text and analyze that text. For image data, image analysis techniques (e.g., YOLO or ResNet) are used to extract image features and generate search queries based on them.

[0891] The server uses the analyzed search query to search for relevant products in a product database. The product database contains the latest product information provided by numerous vendors. The server then ranks the retrieved product list using an AI model (e.g., random forest or deep learning model). This ranking is based on the user's past purchase history, ratings, and current search criteria.

[0892] The terminal displays the ranked product list sent from the server to the user. The search results include detailed information such as product images, prices, user reviews, and availability. The user is provided with an interface to provide feedback on the displayed search results and further refine the search criteria. For example, the user can change the price range or select a specific brand.

[0893] The server receives feedback from the user, updates the search criteria in real time, and searches again, resulting in more accurate search results. When the user finds the desired product, they are redirected to the product detail page and begin the purchase process, which includes entering shipping address and payment information. The server receives the purchase request, updates the relevant database, and completes the transaction by interacting with external payment processing services (e.g., Stripe or PayPal).

[0894] Specific examples

[0895] A user types "I want black shoes suitable for the office" into the search bar. The device sends this text to the server, which uses natural language processing technology to generate a search query and search a product database. Results such as "black leather shoes suitable for the office" are returned and displayed on the device.

[0896] The user presses the microphone button and says, "I want casual sneakers." The device captures the voice data and sends it to the server. The server uses speech recognition technology to convert the speech into text and generate a search query. Search results for casual sneakers are returned and displayed on the device.

[0897] Example prompt: "If a user is looking for a simple bag, how can you use AI technology to provide the best search results?"

[0898] As described above, the present invention is a system that uses AI technology to provide appropriate search results even when the user enters ambiguous information, and also supports voice and image input, in order to significantly improve the online shopping experience.

[0899] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0900] Step 1: Launch the user interface

[0901] The user launches a dedicated application or web browser on a device such as a smartphone or PC, and the user interface displays a search bar, a voice input button, and an image upload button.

[0902] Input: A user action (launching an application or web browser)

[0903] Output: Display of the user interface

[0904] Step 2: Receiving user input

[0905] Users can enter text keywords into the search bar, press the microphone button to input using their voice, or press the image upload button to upload an image, for example, by typing "black shoes," saying "looking for casual t-shirts," or uploading a photo of a product.

[0906] Input: Text input, voice input, or image upload

[0907] Output: Captured input data (text, audio, images)

[0908] Step 3: Capturing and sending input

[0909] The terminal encodes the input data captured from the user and transmits it over the network to the server.

[0910] Input: Captured input data (text, audio, images)

[0911] Output: Sends the encoded input data to the server

[0912] Step 4: Data analysis by the server

[0913] The server analyzes the received data using appropriate techniques: text data is analyzed using natural language processing techniques (e.g., GPT-3) to generate search queries; voice data is converted to text using speech recognition techniques (e.g., Google Speech-to-Text) and analyzed; and image data is analyzed using image analysis techniques (e.g., YOLO) to extract features and generate search queries.

[0914] Input: Transmitted encoded data (text, audio, image)

[0915] Output: Parsed search query

[0916] Step 5: Product database search

[0917] The server uses the generated search query to search for the relevant product in a product database, which stores up-to-date product information provided by multiple vendors.

[0918] Input: Parsed search query

[0919] Output: Product list as search results

[0920] Step 6: Product Listing Selection and Ranking

[0921] The server then uses an AI model (e.g., a deep learning model) to rank the retrieved product list, based on the user's past purchase history, ratings, and current search criteria.

[0922] Input: Product list as search results

[0923] Output: Ranked product list

[0924] Step 7: Viewing the ranking results

[0925] The terminal receives the ranked product list sent from the server and displays it on the user interface. The search results include information such as product images, prices, user reviews, and stock status.

[0926] Input: Ranked product list

[0927] Output: Display of search results with detailed information

[0928] Step 8: Receive feedback and refine your search criteria

[0929] Users can provide feedback on the displayed search results and enter new criteria, such as a specific price range or brand.

[0930] Input: User feedback (new conditions)

[0931] Output: Updated feedback data

[0932] Step 9: Re-search based on feedback

[0933] The server receives feedback from the user, updates the search criteria in real time, and performs the search again, resulting in more accurate search results.

[0934] Input: Updated feedback data

[0935] Output: Updated product list searched again

[0936] Step 10: Start Checkout

[0937] When the user selects the desired product, they are taken to the product details page and begin the purchase process. They enter their shipping address and payment information to confirm the purchase.

[0938] Input: User's selected product, shipping address, payment information

[0939] Output: Purchase request completed

[0940] Step 11: Complete the purchase process

[0941] The server receives the purchase request, updates the relevant database, and coordinates with an external service (e.g., payment processing software) that handles the payment process and completes the transaction.

[0942] Input: Purchase request and payment information

[0943] Output: Transaction completed and database updated

[0944] (Application example 1)

[0945] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0946] Conventional online shopping systems make it difficult for users to efficiently find the products they are looking for, and the interfaces are cumbersome. Furthermore, especially for use in autonomous vehicles, voice-only operation and image input are important, but there is a lack of technology to properly process these. Therefore, there is a demand for a system that allows users to enjoy a comfortable and efficient online shopping experience inside the car.

[0947] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0948] In this invention, the server includes a means for receiving input from a user and transmitting the input to the server, a means for the server to analyze the received input and search for corresponding products from a product database, and a means for selecting optimal products based on the analysis results and providing appropriate search results to the user. This makes it possible to provide a system that combines a means for capturing a user's voice input using an in-vehicle voice recognition device and a means for visually displaying search results using an in-vehicle infotainment system and allowing the user to select products via a touch screen or voice control. This enables users to efficiently and comfortably shop online even while in an autonomous vehicle.

[0949] The "means for receiving input from a user and transmitting the input to a server" is a part of a system that has the function of capturing input data, such as voice or images, from a user and transmitting the data to a server.

[0950] "Means for the server to analyze the input received and search for the relevant product in the product database" is part of a system that has the processing power to process the text, voice, or image data received by the server from the user to find the relevant product.

[0951] "Means for selecting the most suitable product based on the analysis results and providing the user with appropriate search results" refers to part of a system that has the function of allowing the server to select the most suitable product based on the analysis results of the input data and display that information on the user's device.

[0952] The "means for capturing user voice input using an in-vehicle voice recognition device" is part of a system that receives user voice commands within an autonomous vehicle and captures them as data.

[0953] "Means of using the in-vehicle infotainment system to visually display search results and allow the user to select products via touchscreen or voice control" refers to a part of a system that displays search results on a display inside an autonomous vehicle and allows the user to select products via touchscreen or voice command.

[0954] "Natural language processing means that generates appropriate search queries even when ambiguous input is given" refers to technology that properly interprets unclear expressions or ambiguous words used by users and generates optimal search queries.

[0955] "Means for receiving voice input and image input, analyzing the input data, and using it as search conditions" refers to part of a system that has the function of receiving voice and image data, analyzing the data, and using it as product search conditions.

[0956] The present invention provides a system that allows a user to comfortably and efficiently perform online shopping while inside an autonomous vehicle. Specific embodiments will be described below.

[0957] The system of the present invention provides a means for users to search for and purchase products using voice and images, and includes the following main components:

[0958] 1. Capture audio and video inputs

[0959] A voice recognition device is used to capture voice inputs from users inside the autonomous vehicle. The device detects the user's voice through a microphone installed in the vehicle and converts it into digital data. Also, if the user uploads an image from their smartphone, it will be captured by the vehicle's infotainment system.

[0960] 2. Data submission and analysis

[0961] The voice and image data captured from the user is sent via the infotainment system to a server, which analyzes the data using the following techniques:

[0962] Speech Recognition: Converts speech data into text using Python and TensorFlow.

[0963] Image analysis: Using OpenCV, features of uploaded images are extracted and search criteria are generated based on them.

[0964] 3. Product Search and Query Generation

[0965] The server uses natural language processing technology to generate a search query based on the received text data and image analysis data, and uses this query to search for matching products in a product database.

[0966] 4. Displaying search results

[0967] Search results are displayed on the infotainment system display, which can be used to select products using a touchscreen or voice control. Search results include detailed information such as product images, prices, and user reviews.

[0968] 5. Purchase Procedure

[0969] A checkout interface is provided to purchase the selected items immediately, with shipping address and payment information securely processed within the system.

[0970] Specific examples

[0971] For example, if a user were to say, "I'm looking for black shoes for the office, especially leather ones," the voice recognition device would capture this speech, convert it into text data, and send it to the server. The server would then use natural language processing technology to generate a search query and search for matching products in a product database. The search results would then be displayed on the infotainment system's display.

[0972] A specific example of a prompt sentence is a voice input such as "I'm looking for black shoes that can be used in the office. I'd especially like leather shoes."

[0973] The system of the present invention allows users to shop online efficiently and comfortably even while inside an autonomous vehicle.

[0974] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0975] Step 1:

[0976] The user inputs voice or uploads images from a smartphone while in an autonomous vehicle. The voice recognition device captures the user's voice through a microphone in the car and converts it into digital voice data. Images are captured through the infotainment system. Input: User's voice or image. Output: Digital voice data or image data.

[0977] Step 2:

[0978] The device sends the captured digital audio or image data to the server. The data is transferred to the server via the infotainment system and over the network. Input: Digital audio or image data. Output: Data sent to the server.

[0979] Step 3:

[0980] The server analyzes the digital voice data it receives and converts it into text data using voice recognition technology. Speech recognition is performed using Python and TensorFlow. In the case of image data, features are extracted using OpenCV. Input: Digital voice data or image data. Output: Text data or image feature data.

[0981] Step 4:

[0982] The server generates search queries using natural language processing technology based on text data and image feature data. A generative AI model is used to generate appropriate search queries even with ambiguous input. Input: Text data or image feature data. Output: Search query.

[0983] Step 5:

[0984] The server uses the generated search query to search for matching products in a product database that contains up-to-date product information from multiple vendors. Input: Search query. Output: Search result list.

[0985] Step 6:

[0986] The server ranks the search results and sends the top results to the terminal. The ranking is done in the optimal order based on the user's past purchase history, ratings, and current search conditions. Input: Search result list. Output: Ranked product list.

[0987] Step 7:

[0988] The terminal displays the ranked product list sent from the server on the display of the infotainment system in the car. The user can select products using the touch screen or voice control. Input: Ranked product list. Output: Product list displayed on the display.

[0989] Step 8:

[0990] The user selects the desired product on the display and begins the purchase process. The terminal prompts the user to enter the shipping address and payment information, which are then sent to the server. Input: User-selected product and purchase information. Output: Purchase request sent to the server.

[0991] Step 9:

[0992] The server receives the purchase request, updates the relevant database, and completes the transaction by contacting an external service responsible for payment processing. The item is then shipped to the specified delivery address. Input: Purchase request. Output: Completed transaction and shipping instructions.

[0993] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0994] The present invention provides a system for providing more personalized search results by combining product search and user emotion recognition in online shopping. This system improves the user experience by recognizing the user's emotions and making appropriate product suggestions based on those emotions. Specific embodiments are described below.

[0995] User Interface Design

[0996] Users access the system via a dedicated application or web browser on their smartphone or PC. The user interface includes a search bar, a voice input button, an image upload button, and a camera icon for emotion recognition.

[0997] Processing User Input

[0998] Users can enter text into the search bar, or use voice input by pressing the microphone button. They can also upload an image by pressing the camera button or click the emotion recognition camera icon. For example, they can turn on emotion recognition by typing, "I want black shoes that go with the office."

[0999] Sending and parsing input

[1000] The device captures input data from the user and transmits it over the network to a server, which can be text, voice, images, or emotion recognition data.

[1001] The server analyzes the received data. For text data, it uses natural language processing technology to generate search queries. For voice data, it uses voice recognition technology to convert it into text and analyze it in the same way. For image data, it uses image analysis technology to extract important features and generate relevant search terms. Furthermore, for emotion recognition data, an emotion engine analyzes facial expressions and voice tone to identify the user's emotional state.

[1002] Linking and searching with product databases

[1003] The server uses the generated search query to search a product database, for example by issuing an SQL query, to retrieve a list of matching products. This database contains up-to-date product information from multiple vendors.

[1004] Product selection and ranking

[1005] The server passes the product list obtained as a search result to the AI ​​model, which then ranks the products. The AI ​​model generates rankings by taking into account the user's past purchase history, ratings, current search criteria, and even their emotional state. For example, if the AI ​​model detects that the user is "tired," it will prioritize products with a relaxing effect.

[1006] Displaying search results

[1007] The device displays a ranked list of products sent from the server to the user, and the search results include detailed information such as product images, prices, user reviews, availability, and even recommendations based on emotional state.

[1008] Feedback and Interaction

[1009] Users can provide feedback on the search results displayed, and the device will send the feedback information to the server, which will then further optimize the search results. In this process, the system will update the search results in real time, even if the user indicates a change in sentiment.

[1010] Start of purchase flow

[1011] When the user finds the product they want, the device displays a checkout screen where they can enter their shipping address and payment information.

[1012] The server receives the purchase request, updates the relevant database, and completes the transaction by connecting with an external service responsible for payment processing. Based on emotion recognition, the user's purchasing experience is personalized.

[1013] This embodiment allows users to obtain search results that reflect their emotional state, allowing them to enjoy more satisfying online shopping. Furthermore, by combining this with voice and image input, the system can accommodate a wider variety of search conditions.

[1014] The processing flow will be explained below.

[1015] Step 1:

[1016] Users access the system through a dedicated application or web browser on their device by entering text into the search bar, pressing the microphone button for voice input, pressing the camera button to upload an image, or clicking the camera icon for emotion recognition.

[1017] Step 2:

[1018] The device captures the user's input data. In the case of text input, it is saved as text data. In the case of voice input, it is converted into text using voice recognition technology. In the case of image input, the image data is saved. It also captures the user's facial expressions and tone of voice using a camera and microphone dedicated to emotion recognition.

[1019] Step 3:

[1020] The device sends the captured input data to the server using a network communication method such as an HTTP POST request, and the data is sent as properly formatted JSON or XML.

[1021] Step 4:

[1022] The server analyzes the received data. In the case of text data, natural language processing (NLP) technology is used to extract the user's intent and convert it into an appropriate search query. In the case of voice data, voice recognition technology is used to convert it into text, which is then analyzed as text data. In the case of image data, image analysis technology is used to extract important features to use as search criteria. Furthermore, in the case of emotion recognition data, an emotion engine analyzes facial expressions and voice tone to identify the user's emotional state.

[1023] Step 5:

[1024] The server uses the generated search query to search the product database using an SQL query or API request to retrieve a list of matching products based on the product information in the database.

[1025] Step 6:

[1026] The server passes the returned product list to the AI ​​model for ranking. This ranking is based on the user's past purchase history, ratings, and current search criteria. In addition, the model takes into account the user's emotional state and prioritizes the most appropriate products based on that emotional state.

[1027] Step 7:

[1028] The server sends the ranked search results to the device, including detailed information such as product images, prices, user reviews, availability, and recommendations (including explanations of the reasons for recommendations if they are based on emotional states).

[1029] Step 8:

[1030] The device will then display the search results received from the server to the user, who can then review the displayed results and use filter options and sorting functions to further refine the search criteria.

[1031] Step 9:

[1032] Users can provide feedback on the search results they see, for example, by setting more specific criteria (price, brand, etc.) based on a particular emotional state (e.g., wanting to relax).

[1033] Step 10:

[1034] The server receives feedback from users, updates the search criteria in real time, and searches again, re-ranking the search results and generating the best product list based on the latest information.

[1035] Step 11:

[1036] When the user finds the product they want, the device displays a checkout screen where they can enter their shipping address and payment information to complete the purchase.

[1037] Step 12:

[1038] The server receives the purchase request, updates the relevant database, and completes the transaction by contacting an external payment processing service. Once the purchase is complete, it sends a confirmation to the user.

[1039] This step allows users to efficiently and personalizedly search and purchase products.

[1040] Example 2

[1041] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1042] Current online shopping systems generate search results based solely on input data provided by the user, making it difficult to provide product suggestions that reflect the user's emotional state or current mood. This can result in a lack of optimal suggestions for the products the user is looking for, which can lead to low satisfaction. Furthermore, they are unable to effectively utilize various input formats, such as voice input or image input, preventing flexible searches. Furthermore, they lack a mechanism for recognizing user emotions in real time and reflecting them in search results, making it difficult to provide more personalized search results for each individual user.

[1043] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1044] In this invention, the server includes a means for receiving input from a user and transmitting the input to the server, a means for analyzing the input received by the server and searching for corresponding products from a product database, a means for selecting optimal products based on the analysis results and providing the user with appropriate search results, a means for recognizing the user's emotions and making product suggestions based on the emotions, and a means for analyzing text, voice, and images provided by the user to generate search criteria. This enables personalized product suggestions that reflect the user's emotional state, improving the user experience. Furthermore, flexible search criteria generation is realized by effectively utilizing various input formats, including voice input and image input.

[1045] "User" refers to an individual or organization that uses the System to search for and purchase products.

[1046] "Input" means data provided by a user to the system, and may include any one or more of text, voice, image, and / or emotion recognition data.

[1047] "Server" refers to a central processing unit that analyzes input data received from users and searches the product database and makes product suggestions based on the results.

[1048] "Product Database" refers to a data storage system that stores up-to-date product information provided by multiple vendors.

[1049] "Analysis" refers to the process by which the server processes input data received from the user and extracts meaningful information.

[1050] "Product suggestion" refers to the server presenting suitable products to the user based on the analysis results.

[1051] "Emotion recognition" refers to the process of analyzing a user's facial expressions and tone of voice to determine their current emotional state.

[1052] "Natural language processing" refers to technology that enables computers to understand and analyze the language that humans use on a daily basis.

[1053] "Speech recognition" refers to the technology of converting voice data into text data.

[1054] "Image analysis" refers to the technology of extracting important features and information from image data.

[1055] "Ranking" refers to the process of prioritizing products based on a user's purchasing history, ratings, current search criteria, and emotional state.

[1056] "Feedback" refers to ratings and comments provided by users that are used to improve the performance of the system and the quality of suggestions.

[1057] "Checkout" refers to the process by which a User enters the necessary information to purchase a selected Product and completes the transaction.

[1058] The present invention provides a system for providing more personalized search results by combining product search and user emotion recognition in online shopping. This system improves the user experience by recognizing the user's emotions and suggesting appropriate products based on those emotions. Specific embodiments are described below.

[1059] User Interface Design

[1060] Users access the system through a dedicated application or a web browser on their smartphone or PC. The user interface includes a search bar, a voice input button, an image upload button, and a camera icon for emotion recognition, allowing users to input search requests in a variety of ways.

[1061] Processing User Input

[1062] Users can enter specific search criteria using text input, voice input, image upload, and emotion recognition. For example, a user may enter text such as "I'm looking for black shoes for the office" and enable emotion recognition. In this case, the user can also use voice input and image upload.

[1063] Sending and parsing input

[1064] The device captures the user's input data and transmits it to a server over a network, including text, voice, images, and emotion recognition data.

[1065] The server analyzes the received data, specifically:

[1066] The text data is analyzed using natural language processing techniques to generate search queries.

[1067] The voice data is converted into text using voice recognition technology, and the text is analyzed.

[1068] Image data is subjected to image analysis techniques to extract features.

[1069] The emotion recognition data is analyzed by an emotion engine to identify the user's emotional state.

[1070] Linking and searching with product databases

[1071] The server uses the generated search query to search a product database, which contains up-to-date product information and data provided by multiple vendors. For example, it issues an SQL query to retrieve a list of matching products.

[1072] Product Selection and Ranking

[1073] The server passes the retrieved product list to an AI model for ranking. The AI ​​model generates rankings by taking into account the user's past purchase history and ratings, current search criteria, and emotional state. For example, if the user's emotion is recognized as "tired," products with a relaxing effect may be displayed preferentially.

[1074] Displaying search results

[1075] The terminal displays a ranked list of products sent from the server to the user, and the search results include detailed information such as product images, prices, user reviews, availability, and reasons for recommendation.

[1076] Feedback and Interaction

[1077] The user can provide feedback on the displayed search results, for example, by clicking a rating button or entering a comment. The device sends the feedback information to the server, and the server further optimizes the search results based on the feedback.

[1078] Start of purchase flow

[1079] Once the user selects the desired product, the terminal displays the checkout screen, where the user enters their shipping address and payment information. The server processes the purchase request, updates the relevant database, and completes the transaction by contacting an external service that handles payment processing.

[1080] Prompt Sentence Examples

[1081] "How can I use natural language processing techniques to turn text into a search query?"

[1082] This system allows users to obtain search results that reflect their emotional state, enabling them to enjoy a more satisfying online shopping experience. In addition, by combining it with voice and image input, the system can accommodate a wider variety of search conditions.

[1083] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1084] Step 1:

[1085] Users access the system by opening a dedicated application or web browser on their smartphone or PC. The user interface displays a search bar, a voice input button, an image upload button, and a camera icon for emotion recognition, allowing users to select which form of input they want.

[1086] Step 2:

[1087] Input is provided in the form of your choice. For example, you can type "I want black shoes for the office" into the search bar, or press the voice input button to search by voice. You can also click the camera icon to turn on emotion recognition. Input data can include text, voice, image data, and emotion data.

[1088] Step 3:

[1089] The device captures the user's input data and sends it to the server via the network. The data sent can be text, audio, images, emotion recognition data, etc. The most common method of transmission is via HTTP request.

[1090] Step 4:

[1091] The server analyzes the received data. In the case of text data, it is analyzed using natural language processing (NLP) technology and converted into a search query. Voice data is converted into text using voice recognition technology, and image data is used to extract important features using image analysis technology. Emotion recognition data is analyzed by an emotion engine to determine the user's emotional state, such as whether they are tired or happy.

[1092] Input: Text, audio, image, and emotion data from users

[1093] Data processing: natural language processing, speech recognition, image analysis, emotion recognition

[1094] Output: Parsed search queries and sentiment data

[1095] Step 5:

[1096] The server uses the generated search query to search a product database, e.g., using an SQL query, to retrieve a list of relevant products. This database contains up-to-date product information from multiple vendors.

[1097] Input: Search query

[1098] Data Calculation: SQL Query Issue

[1099] Output: List of related products

[1100] Step 6:

[1101] The server passes the retrieved product list to an AI model for ranking. The AI ​​model generates rankings based on the user's past purchase history and ratings, current search criteria, and emotional state. For example, if a user is perceived as "tired," products with a relaxing effect will be displayed preferentially.

[1102] Input: product list, user emotional state

[1103] Data Calculation: Ranking by AI Model

[1104] Output: Ranked product list

[1105] Step 7:

[1106] The device displays a ranked list of products sent from the server to the user, and the search results include product images, prices, user reviews, availability, and recommendations based on emotional state.

[1107] Input: Ranked product list

[1108] Data processing: Display processing on HTML or application

[1109] Output: Search results displayed on the user's screen

[1110] Step 8:

[1111] Users can provide feedback on the search results they see by clicking the rating button or entering comments. The device sends this feedback information to the server, which then uses it to optimize search results and rankings.

[1112] Input: User feedback data

[1113] Data processing: Analysis of feedback information and learning

[1114] Output: Optimized ranking and search results

[1115] Step 9:

[1116] If the user decides to purchase the desired product, the device displays a checkout screen, where the user enters the required shipping address and payment information. The server processes the purchase request, updates the product database, and completes the transaction by contacting an external service that handles payment processing.

[1117] Input: User purchase information, payment information

[1118] Data calculation: Processing purchase information and linking with external services

[1119] Output: Confirmation of successful purchase and notification to the user

[1120] (Application example 2)

[1121] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1122] In conventional online shopping systems, when users select from a large number of products, they are unable to receive suggestions for products that match their current emotions or mood, resulting in a uniform, unpersonalized purchasing experience. Furthermore, the payment process does not take into account the user's emotional state, making it difficult to improve user satisfaction. This raises concerns that users' purchasing motivation may decrease and the conversion rate of e-commerce sites may fall.

[1123] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1124] In this invention, the server includes means for receiving input from a user and transmitting the input to the server, means for the server to analyze the input received and search for corresponding products from a product database, means for selecting optimal products based on the analysis results and providing appropriate search results to the user, and means for recognizing the emotional state of the user and adjusting the search results provided based on the emotional state, thereby enabling the provision of personalized search results that take into account the emotional state of the user and the optimization of the payment interface according to the user's emotions.

[1125] "User" means an individual or legal entity that uses the System to search for and purchase Products.

[1126] "Input" is information a user provides to a system, which may be text, audio, images, or a combination of these.

[1127] A "server" is a computer system that receives and analyzes input submitted by a user.

[1128] The "product database" is a database that stores information about products to be searched.

[1129] "Analysis" refers to the means of processing and understanding received input data, and includes natural language processing, speech recognition, image analysis, emotion recognition, etc.

[1130] "Search Results" are lists of relevant products provided by the server based on the user's input and emotional state.

[1131] "Emotional state" refers to the user's psychological and emotional state, as analyzed from the user's facial expressions, tone of voice, etc.

[1132] An "emotion recognizer" is a combination of software and hardware for analyzing a user's emotional state.

[1133] The "payment interface" is a user interface that allows a user to confirm the purchase of a product and carry out payment procedures.

[1134] "Optimization" is the process of improving the user experience by adjusting payment interfaces and search results based on the user's emotional state.

[1135] MODE FOR CARRYING OUT THE INVENTION

[1136] The present invention is an electronic payment system that uses emotion recognition to provide personalized search results and payment interfaces. The system basically consists of a user terminal, a server, an emotion recognition engine, a natural language processing engine, a voice recognition engine, an image analysis engine, and a product database.

[1137] User Interface Design

[1138] Users access the system using a smartphone or personal computer through a dedicated application or a web browser. The user interface includes a search bar, a voice input button, an image upload button, and a camera icon for emotion recognition, allowing users to provide various forms of input to the system.

[1139] Processing User Input

[1140] Users can enter text into the search bar or use voice input by pressing the voice input button. They can also upload an image or click the emotion recognition camera icon to capture natural facial expressions. For example, they can turn on emotion recognition while typing, "I want black shoes that go with the office."

[1141] Data transmission to the server and analysis

[1142] The device captures input data from the user and sends it over the network to a server. The data sent can be text, voice, images, or emotion recognition data. The server analyzes the received data and generates a search query using appropriate technologies, such as a natural language processing engine, voice recognition engine, image analysis engine, or emotion recognition engine.

[1143] Linking and searching with product databases

[1144] The server uses the generated search query to search the product database, which stores the latest product information, and retrieves a list of matching products. The results are retrieved in the form of a database query, such as an SQL query.

[1145] Product selection and ranking

[1146] The server passes the product list obtained as a search result to an artificial intelligence (AI) model for ranking. The AI ​​model generates rankings by taking into account the user's past purchase history, ratings, current search criteria, and emotional state. For example, if the user's emotion is recognized as "tired," products with a relaxing effect will be displayed preferentially.

[1147] Displaying search results

[1148] The device displays a ranked list of products sent from the server to the user, and the search results include detailed information such as product images, prices, user reviews, availability, and even recommendations based on emotional state.

[1149] Optimizing payment interfaces based on emotion recognition

[1150] Once the user finds the product they want, the terminal will prompt them for payment. The payment interface is optimized based on the user's emotional state. For example, if the user is feeling stressed, the terminal will offer the option to complete the payment with one click.

[1151] Feedback and Interaction

[1152] Users can provide feedback on the search results displayed, and the device will send the feedback information to the server and further optimize the search results based on it. If the user indicates a change in their sentiment, the system will update the search results in real time.

[1153] Examples of concrete examples and prompts

[1154] For example, if a user's camera image indicates that they look tired, a simple interface will be provided to complete payment with one click. Additionally, promotions for products that have a relaxing effect will be displayed based on the user's past purchase history and current emotional state.

[1155] Prompt Sentence Examples

[1156] If the user feels tired, generate a prompt to introduce relaxation items, such as massagers, aroma candles, etc., and provide an interface for one-click payment.

[1157] This allows users to easily find the perfect product based on their emotional state and expedite the checkout process.

[1158] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1159] Program processing steps

[1160] Step 1:

[1161] Users can use their device to enter text into the search bar or use voice input by pressing the voice input button, upload an image, or capture facial expressions by clicking the emotion recognition camera icon.

[1162] Input: text, audio, images

[1163] Output: Input data is captured to the terminal

[1164] Step 2:

[1165] The device transmits the captured input data to a server over a network, including text data, voice data, image data, and emotion recognition data.

[1166] Input: Captured data

[1167] Output: The input data is sent to the server

[1168] Step 3:

[1169] The server analyzes the data it receives: for text data, a natural language processing engine generates search queries; for audio data, a speech recognition engine converts it into text; for image data, an image analysis engine extracts key features; for emotion data, an emotion recognition engine analyzes facial expressions and voice tone to identify emotional states.

[1170] Input: Data sent to the server

[1171] Output: search query, emotional state

[1172] Step 4:

[1173] The server uses the generated search query to search a product database, which contains up-to-date product information, to retrieve a list of suitable products.

[1174] Input: search query

[1175] Output: Product list

[1176] Step 5:

[1177] The server passes the search results to an AI model that ranks the products, taking into account the user's past purchase history, ratings, current search criteria, and emotional state.

[1178] Input: Product list, past purchase history, emotional state

[1179] Output: Ranked product list

[1180] Step 6:

[1181] The server sends the ranked product list to the device, which is also based on the user's emotional state, resulting in personalized results.

[1182] Input: Ranked product list

[1183] Output: Ranked product list sent to device

[1184] Step 7:

[1185] The device displays a ranked list of products sent from the server to the user, and the displayed search results include product images, prices, user reviews, availability, and recommendations based on emotional state.

[1186] Input: Ranked product list

[1187] Output: Product information displayed to the user

[1188] Step 8:

[1189] Once the user finds the product they want, the terminal will prompt them for payment. The payment interface is optimized based on the user's emotional state. For example, if the user is feeling stressed, the terminal will offer the option to complete the payment with one click.

[1190] Input: Emotional state, product selection data

[1191] Output: Payment interface

[1192] Step 9:

[1193] Users can provide feedback on the displayed search results, and the terminal will send the feedback information to the server, which will then further optimize the search results.

[1194] Input: User feedback

[1195] Output: Optimized search results

[1196] Specific example of operation

[1197] For example, the following shows how to generate a prompt sentence that determines that the user is "tired" and displays a promotion for a product that has a relaxing effect.

[1198] Example prompts

[1199] If the user feels tired, generate a prompt to introduce relaxation items, such as massagers, aroma candles, etc., and provide an interface for one-click payment.

[1200] This allows users to easily find the perfect product based on their emotional state and expedite the checkout process.

[1201] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1202] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1203] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1204] [Fourth embodiment]

[1205] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1206] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1207] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1208] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1209] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1210] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1211] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1212] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1213] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1214] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1215] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1216] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1217] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1218] The present invention provides a system that utilizes AI technology to improve the product search experience in order to make product searches more efficient in online shopping. Specific embodiments are described below.

[1219] User Interface Design

[1220] Users access the system through a dedicated application or web browser on their device (smartphone or PC). The user interface is simple and easy to use, with a search bar, voice input button, and image upload button.

[1221] Processing User Input

[1222] Users can enter text into the search bar, press the microphone button to speak, or press the camera button to upload an image, for example, "I'm looking for black shoes for the office" or upload a photo of their feet.

[1223] Sending and parsing input

[1224] The device captures the user's input data and transmits it over the network to a server, either as text, audio, or images.

[1225] The server analyzes the received data. For text data, it uses natural language processing technology to generate search queries. For voice data, it uses voice recognition technology to convert it into text and analyze it. For image data, it uses image analysis technology to extract features of the uploaded image and generate search criteria based on them.

[1226] Linking and searching with product databases

[1227] The server uses the generated search query to search for relevant products in a product database, which contains regularly updated product information from multiple vendors, and returns a list of products that match the search query.

[1228] Product selection and ranking

[1229] The server uses an AI model to rank the product list obtained as a search result, in an optimal order based on the user's past purchase history, ratings, and current search criteria.

[1230] Displaying search results

[1231] The terminal displays the ranked product list sent from the server to the user, and the search results include detailed information such as product images, prices, user reviews, and availability.

[1232] Feedback and interactive search tuning

[1233] Users are also provided with an interface to provide feedback on the search results and further refine their search criteria, for example by lowering the price range or selecting a specific brand.

[1234] The server receives feedback from the user, updates the search criteria in real time, and searches again, resulting in more accurate search results.

[1235] Start of purchase flow

[1236] When a user finds a product they want, they are taken to the product detail page and begin the checkout process, which includes entering their shipping address and payment information.

[1237] The server receives the purchase request, updates the relevant database, and works with an external service that handles payment processing to complete the transaction.

[1238] As described above, this invention uses AI technology to provide appropriate search results even when the user provides vague input, and also supports voice and image input, thereby significantly improving the online shopping experience.

[1239] The processing flow will be explained below.

[1240] Step 1:

[1241] Using a device, a user accesses a search screen. The user can choose to enter text, speak, or upload an image. For example, the user enters "I'm looking for black shoes for the office" into the search bar.

[1242] Step 2:

[1243] The device captures input from the user. In the case of text, it is captured as text data, in the case of voice, the voice data is converted to text, and in the case of images, the image data is captured as is.

[1244] Step 3:

[1245] The device sends the captured input data to the server using a network communication method such as an HTTP POST request, with the data being sent in a properly formatted format such as JSON or XML.

[1246] Step 4:

[1247] The server analyzes the received data. In the case of text data, natural language processing (NLP) technology is used to extract search intent and generate appropriate search queries. In the case of audio data, speech recognition technology is used to convert it into text and analyze it in the same way. In the case of image data, image analysis technology is used to extract important features and generate appropriate search terms.

[1248] Step 5:

[1249] The server uses the generated search query to search a product database, for example by issuing an SQL query, to retrieve a list of matching products. This database contains up-to-date product information from multiple vendors.

[1250] Step 6:

[1251] The server then passes the search results to an AI model for ranking, which uses the user's past purchase history, ratings, and other relevant data to rank the most relevant products.

[1252] Step 7:

[1253] The server then sends the ranked search results to the device, which includes detailed information such as product images, prices, user reviews, and availability.

[1254] Step 8:

[1255] The device receives search results from the server and displays them to the user, along with filtering and sorting options to further refine the search criteria.

[1256] Step 9:

[1257] Users can review the search results and then perform new searches by further narrowing the search criteria, such as selecting a price range or a specific brand.

[1258] Step 10:

[1259] The server receives new feedback from the user and performs the product search again, updating the search criteria in real time to generate optimal results.

[1260] Step 11:

[1261] Once the user selects the desired product, the terminal displays a checkout screen where the user enters their shipping address and payment information.

[1262] Step 12:

[1263] The server receives the purchase request, updates the relevant database, coordinates with external services that handle payment processing, and sends a purchase confirmation to the user once the transaction is complete.

[1264] This step allows users to search for products and complete the purchasing process efficiently and conveniently.

[1265] Example 1

[1266] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1267] Conventional online shopping product search systems have difficulty providing accurate search results when user input is ambiguous or when they need to accept a variety of input formats (text, voice, images). Furthermore, there are still challenges in providing a function that reflects user feedback in real time to update search results, and in providing personalized search results based on past purchase history and ratings.

[1268] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1269] In this invention, the server includes means for receiving input from a user and transmitting the input to the server, means for analyzing the input received by the server and searching for corresponding products from a product database, means for selecting optimal products based on the analysis results using a ranking means and providing appropriate search results to the user, means for receiving feedback from the user and updating search conditions in real time, and means for taking into account the user's past purchase history and ratings in subsequent searches. This makes it possible to provide appropriate search results even when the user has entered ambiguous input, support a variety of input formats, and provide personalized search results.

[1270] "User" refers to an individual or corporation that uses the online shopping system.

[1271] "Input" refers to information provided by a user as a search query, and may be in the form of text, audio, images, or the like.

[1272] "Server" refers to a computing resource on a network that receives and analyzes input data from users and executes product searches.

[1273] "Parsing" refers to the process by which the server interprets the input data it receives, understands its meaning, and generates appropriate search queries.

[1274] "Product database" refers to a data storage device that stores product information provided by multiple vendors.

[1275] "Ranking method" refers to an algorithm that prioritizes search results based on a user's past purchasing history, ratings, and current search criteria.

[1276] "Search Results" refers to a list of products selected by the server based on user input.

[1277] "Feedback" refers to additional instructions or ratings given by a user in response to a search result.

[1278] "Real-time update" means that feedback from users is reflected immediately, and product searches are performed again to provide the latest search results.

[1279] "Purchase history" refers to a record of products a user has purchased in the past.

[1280] "Personalized" refers to providing search results that are customized based on the individual user's preferences and behavioral history.

[1281] The present invention is directed to a system for improving the efficiency of product searches in online shopping. Specific embodiments will be described below.

[1282] Users access the system through a dedicated application or a web browser using devices such as smartphones or PCs. The user interface is designed to be easy to use, with a search bar, voice input button, and image upload button located on the screen.

[1283] Users can enter keywords into the search bar, use voice input by pressing the microphone button, or upload an image by pressing the camera icon. For example, they can enter text such as "I'm looking for black shoes that go well with the office," say "I'm looking for casual sneakers," or upload a photo of the product they're looking for.

[1284] The device encodes the captured input data and securely transmits it to the server. The server then analyzes the received data. Specifically, for text data, natural language processing techniques (e.g., BERT or GPT-3) are used to generate search queries. For audio data, speech recognition techniques (e.g., DeepSpeech or Google Speech-to-Text) are used to convert the data into text and analyze that text. For image data, image analysis techniques (e.g., YOLO or ResNet) are used to extract image features and generate search queries based on them.

[1285] The server uses the analyzed search query to search for relevant products in a product database. The product database contains the latest product information provided by numerous vendors. The server then ranks the retrieved product list using an AI model (e.g., random forest or deep learning model). This ranking is based on the user's past purchase history, ratings, and current search criteria.

[1286] The terminal displays the ranked product list sent from the server to the user. The search results include detailed information such as product images, prices, user reviews, and availability. The user is provided with an interface to provide feedback on the displayed search results and further refine the search criteria. For example, the user can change the price range or select a specific brand.

[1287] The server receives feedback from the user, updates the search criteria in real time, and searches again, resulting in more accurate search results. When the user finds the desired product, they are redirected to the product detail page and begin the purchase process, which includes entering shipping address and payment information. The server receives the purchase request, updates the relevant database, and completes the transaction by interacting with external payment processing services (e.g., Stripe or PayPal).

[1288] Specific examples

[1289] A user types "I want black shoes suitable for the office" into the search bar. The device sends this text to the server, which uses natural language processing technology to generate a search query and search a product database. Results such as "black leather shoes suitable for the office" are returned and displayed on the device.

[1290] The user presses the microphone button and says, "I want casual sneakers." The device captures the voice data and sends it to the server. The server uses speech recognition technology to convert the speech into text and generate a search query. Search results for casual sneakers are returned and displayed on the device.

[1291] Example prompt: "If a user is looking for a simple bag, how can you use AI technology to provide the best search results?"

[1292] As described above, the present invention is a system that uses AI technology to provide appropriate search results even when the user enters ambiguous information, and also supports voice and image input, in order to significantly improve the online shopping experience.

[1293] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1294] Step 1: Launch the user interface

[1295] The user launches a dedicated application or web browser on a device such as a smartphone or PC, and the user interface displays a search bar, a voice input button, and an image upload button.

[1296] Input: A user action (launching an application or web browser)

[1297] Output: Display of the user interface

[1298] Step 2: Receiving user input

[1299] Users can enter text keywords into the search bar, press the microphone button to input using their voice, or press the image upload button to upload an image, for example, by typing "black shoes," saying "looking for casual t-shirts," or uploading a photo of a product.

[1300] Input: Text input, voice input, or image upload

[1301] Output: Captured input data (text, audio, images)

[1302] Step 3: Capturing and sending input

[1303] The terminal encodes the input data captured from the user and transmits it over the network to the server.

[1304] Input: Captured input data (text, audio, images)

[1305] Output: Sends the encoded input data to the server

[1306] Step 4: Data analysis by the server

[1307] The server analyzes the received data using appropriate techniques: text data is analyzed using natural language processing techniques (e.g., GPT-3) to generate search queries; voice data is converted to text using speech recognition techniques (e.g., Google Speech-to-Text) and analyzed; and image data is analyzed using image analysis techniques (e.g., YOLO) to extract features and generate search queries.

[1308] Input: Transmitted encoded data (text, audio, image)

[1309] Output: Parsed search query

[1310] Step 5: Product database search

[1311] The server uses the generated search query to search for the relevant product in a product database, which stores up-to-date product information provided by multiple vendors.

[1312] Input: Parsed search query

[1313] Output: Product list as search results

[1314] Step 6: Product Listing Selection and Ranking

[1315] The server then uses an AI model (e.g., a deep learning model) to rank the retrieved product list, based on the user's past purchase history, ratings, and current search criteria.

[1316] Input: Product list as search results

[1317] Output: Ranked product list

[1318] Step 7: Viewing the ranking results

[1319] The terminal receives the ranked product list sent from the server and displays it on the user interface. The search results include information such as product images, prices, user reviews, and stock status.

[1320] Input: Ranked product list

[1321] Output: Display of search results with detailed information

[1322] Step 8: Receive feedback and refine your search criteria

[1323] Users can provide feedback on the displayed search results and enter new criteria, such as a specific price range or brand.

[1324] Input: User feedback (new conditions)

[1325] Output: Updated feedback data

[1326] Step 9: Re-search based on feedback

[1327] The server receives feedback from the user, updates the search criteria in real time, and performs the search again, resulting in more accurate search results.

[1328] Input: Updated feedback data

[1329] Output: Updated product list searched again

[1330] Step 10: Start Checkout

[1331] When the user selects the desired product, they are taken to the product details page and begin the purchase process. They enter their shipping address and payment information to confirm the purchase.

[1332] Input: User's selected product, shipping address, payment information

[1333] Output: Purchase request completed

[1334] Step 11: Complete the purchase process

[1335] The server receives the purchase request, updates the relevant database, and coordinates with an external service (e.g., payment processing software) that handles the payment process and completes the transaction.

[1336] Input: Purchase request and payment information

[1337] Output: Transaction completed and database updated

[1338] (Application example 1)

[1339] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1340] Conventional online shopping systems make it difficult for users to efficiently find the products they are looking for, and the interfaces are cumbersome. Furthermore, especially for use in autonomous vehicles, voice-only operation and image input are important, but there is a lack of technology to properly process these. Therefore, there is a demand for a system that allows users to enjoy a comfortable and efficient online shopping experience inside the car.

[1341] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1342] In this invention, the server includes a means for receiving input from a user and transmitting the input to the server, a means for the server to analyze the received input and search for corresponding products from a product database, and a means for selecting optimal products based on the analysis results and providing appropriate search results to the user. This makes it possible to provide a system that combines a means for capturing a user's voice input using an in-vehicle voice recognition device and a means for visually displaying search results using an in-vehicle infotainment system and allowing the user to select products via a touch screen or voice control. This enables users to efficiently and comfortably shop online even while in an autonomous vehicle.

[1343] The "means for receiving input from a user and transmitting the input to a server" is a part of a system that has the function of capturing input data, such as voice or images, from a user and transmitting the data to a server.

[1344] "Means for the server to analyze the input received and search for the relevant product in the product database" is part of a system that has the processing power to process the text, voice, or image data received by the server from the user to find the relevant product.

[1345] "Means for selecting the most suitable product based on the analysis results and providing the user with appropriate search results" refers to part of a system that has the function of allowing the server to select the most suitable product based on the analysis results of the input data and display that information on the user's device.

[1346] The "means for capturing user voice input using an in-vehicle voice recognition device" is part of a system that receives user voice commands within an autonomous vehicle and captures them as data.

[1347] "Means of using the in-vehicle infotainment system to visually display search results and allow the user to select products via touchscreen or voice control" refers to a part of a system that displays search results on a display inside an autonomous vehicle and allows the user to select products via touchscreen or voice command.

[1348] "Natural language processing means that generates appropriate search queries even when ambiguous input is given" refers to technology that properly interprets unclear expressions or ambiguous words used by users and generates optimal search queries.

[1349] "Means for receiving voice input and image input, analyzing the input data, and using it as search conditions" refers to part of a system that has the function of receiving voice and image data, analyzing the data, and using it as product search conditions.

[1350] The present invention provides a system that allows a user to comfortably and efficiently perform online shopping while inside an autonomous vehicle. Specific embodiments will be described below.

[1351] The system of the present invention provides a means for users to search for and purchase products using voice and images, and includes the following main components:

[1352] 1. Capture audio and video inputs

[1353] A voice recognition device is used to capture voice inputs from users inside the autonomous vehicle. The device detects the user's voice through a microphone installed in the vehicle and converts it into digital data. Also, if the user uploads an image from their smartphone, it will be captured by the vehicle's infotainment system.

[1354] 2. Data submission and analysis

[1355] The voice and image data captured from the user is sent via the infotainment system to a server, which analyzes the data using the following techniques:

[1356] Speech Recognition: Converts speech data into text using Python and TensorFlow.

[1357] Image analysis: Using OpenCV, features of uploaded images are extracted and search criteria are generated based on them.

[1358] 3. Product Search and Query Generation

[1359] The server uses natural language processing technology to generate a search query based on the received text data and image analysis data, and uses this query to search for matching products in a product database.

[1360] 4. Displaying search results

[1361] Search results are displayed on the infotainment system display, which can be used to select products using a touchscreen or voice control. Search results include detailed information such as product images, prices, and user reviews.

[1362] 5. Purchase Procedure

[1363] A checkout interface is provided to purchase the selected items immediately, with shipping address and payment information securely processed within the system.

[1364] Specific examples

[1365] For example, if a user were to say, "I'm looking for black shoes for the office, especially leather ones," the voice recognition device would capture this speech, convert it into text data, and send it to the server. The server would then use natural language processing technology to generate a search query and search for matching products in a product database. The search results would then be displayed on the infotainment system's display.

[1366] A specific example of a prompt sentence is a voice input such as "I'm looking for black shoes that can be used in the office. I'd especially like leather shoes."

[1367] The system of the present invention allows users to shop online efficiently and comfortably even while inside an autonomous vehicle.

[1368] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1369] Step 1:

[1370] The user inputs voice or uploads images from a smartphone while in an autonomous vehicle. The voice recognition device captures the user's voice through a microphone in the car and converts it into digital voice data. Images are captured through the infotainment system. Input: User's voice or image. Output: Digital voice data or image data.

[1371] Step 2:

[1372] The device sends the captured digital audio or image data to the server. The data is transferred to the server via the infotainment system and over the network. Input: Digital audio or image data. Output: Data sent to the server.

[1373] Step 3:

[1374] The server analyzes the digital voice data it receives and converts it into text data using voice recognition technology. Speech recognition is performed using Python and TensorFlow. In the case of image data, features are extracted using OpenCV. Input: Digital voice data or image data. Output: Text data or image feature data.

[1375] Step 4:

[1376] The server generates search queries using natural language processing technology based on text data and image feature data. A generative AI model is used to generate appropriate search queries even with ambiguous input. Input: Text data or image feature data. Output: Search query.

[1377] Step 5:

[1378] The server uses the generated search query to search for matching products in a product database that contains up-to-date product information from multiple vendors. Input: Search query. Output: Search result list.

[1379] Step 6:

[1380] The server ranks the search results and sends the top results to the terminal. The ranking is done in the optimal order based on the user's past purchase history, ratings, and current search conditions. Input: Search result list. Output: Ranked product list.

[1381] Step 7:

[1382] The terminal displays the ranked product list sent from the server on the display of the infotainment system in the car. The user can select products using the touch screen or voice control. Input: Ranked product list. Output: Product list displayed on the display.

[1383] Step 8:

[1384] The user selects the desired product on the display and begins the purchase process. The terminal prompts the user to enter the shipping address and payment information, which are then sent to the server. Input: User-selected product and purchase information. Output: Purchase request sent to the server.

[1385] Step 9:

[1386] The server receives the purchase request, updates the relevant database, and completes the transaction by contacting an external service responsible for payment processing. The item is then shipped to the specified delivery address. Input: Purchase request. Output: Completed transaction and shipping instructions.

[1387] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1388] The present invention provides a system for providing more personalized search results by combining product search and user emotion recognition in online shopping. This system improves the user experience by recognizing the user's emotions and making appropriate product suggestions based on those emotions. Specific embodiments are described below.

[1389] User Interface Design

[1390] Users access the system via a dedicated application or web browser on their smartphone or PC. The user interface includes a search bar, a voice input button, an image upload button, and a camera icon for emotion recognition.

[1391] Processing User Input

[1392] Users can enter text into the search bar, or use voice input by pressing the microphone button. They can also upload an image by pressing the camera button or click the emotion recognition camera icon. For example, they can turn on emotion recognition by typing, "I want black shoes that go with the office."

[1393] Sending and parsing input

[1394] The device captures input data from the user and transmits it over the network to a server, which can be text, voice, images, or emotion recognition data.

[1395] The server analyzes the received data. For text data, it uses natural language processing technology to generate search queries. For voice data, it uses voice recognition technology to convert it into text and analyze it in the same way. For image data, it uses image analysis technology to extract important features and generate relevant search terms. Furthermore, for emotion recognition data, an emotion engine analyzes facial expressions and voice tone to identify the user's emotional state.

[1396] Linking and searching with product databases

[1397] The server uses the generated search query to search a product database, for example by issuing an SQL query, to retrieve a list of matching products. This database contains up-to-date product information from multiple vendors.

[1398] Product selection and ranking

[1399] The server passes the product list obtained as a search result to the AI ​​model, which then ranks the products. The AI ​​model generates rankings by taking into account the user's past purchase history, ratings, current search criteria, and even their emotional state. For example, if the AI ​​model detects that the user is "tired," it will prioritize products with a relaxing effect.

[1400] Displaying search results

[1401] The device displays a ranked list of products sent from the server to the user, and the search results include detailed information such as product images, prices, user reviews, availability, and even recommendations based on emotional state.

[1402] Feedback and Interaction

[1403] Users can provide feedback on the search results displayed, and the device will send the feedback information to the server, which will then further optimize the search results. In this process, the system will update the search results in real time, even if the user indicates a change in sentiment.

[1404] Start of purchase flow

[1405] When the user finds the product they want, the device displays a checkout screen where they can enter their shipping address and payment information.

[1406] The server receives the purchase request, updates the relevant database, and completes the transaction by connecting with an external service responsible for payment processing. Based on emotion recognition, the user's purchasing experience is personalized.

[1407] This embodiment allows users to obtain search results that reflect their emotional state, allowing them to enjoy more satisfying online shopping. Furthermore, by combining this with voice and image input, the system can accommodate a wider variety of search conditions.

[1408] The processing flow will be explained below.

[1409] Step 1:

[1410] Users access the system through a dedicated application or web browser on their device by entering text into the search bar, pressing the microphone button for voice input, pressing the camera button to upload an image, or clicking the camera icon for emotion recognition.

[1411] Step 2:

[1412] The device captures the user's input data. In the case of text input, it is saved as text data. In the case of voice input, it is converted into text using voice recognition technology. In the case of image input, the image data is saved. It also captures the user's facial expressions and tone of voice using a camera and microphone dedicated to emotion recognition.

[1413] Step 3:

[1414] The device sends the captured input data to the server using a network communication method such as an HTTP POST request, and the data is sent as properly formatted JSON or XML.

[1415] Step 4:

[1416] The server analyzes the received data. In the case of text data, natural language processing (NLP) technology is used to extract the user's intent and convert it into an appropriate search query. In the case of voice data, voice recognition technology is used to convert it into text, which is then analyzed as text data. In the case of image data, image analysis technology is used to extract important features to use as search criteria. Furthermore, in the case of emotion recognition data, an emotion engine analyzes facial expressions and voice tone to identify the user's emotional state.

[1417] Step 5:

[1418] The server uses the generated search query to search the product database using an SQL query or API request to retrieve a list of matching products based on the product information in the database.

[1419] Step 6:

[1420] The server passes the returned product list to the AI ​​model for ranking. This ranking is based on the user's past purchase history, ratings, and current search criteria. In addition, the model takes into account the user's emotional state and prioritizes the most appropriate products based on that emotional state.

[1421] Step 7:

[1422] The server sends the ranked search results to the device, including detailed information such as product images, prices, user reviews, availability, and recommendations (including explanations of the reasons for recommendations if they are based on emotional states).

[1423] Step 8:

[1424] The device will then display the search results received from the server to the user, who can then review the displayed results and use filter options and sorting functions to further refine the search criteria.

[1425] Step 9:

[1426] Users can provide feedback on the search results they see, for example, by setting more specific criteria (price, brand, etc.) based on a particular emotional state (e.g., wanting to relax).

[1427] Step 10:

[1428] The server receives feedback from users, updates the search criteria in real time, and searches again, re-ranking the search results and generating the best product list based on the latest information.

[1429] Step 11:

[1430] When the user finds the product they want, the device displays a checkout screen where they can enter their shipping address and payment information to complete the purchase.

[1431] Step 12:

[1432] The server receives the purchase request, updates the relevant database, and completes the transaction by contacting an external payment processing service. Once the purchase is complete, it sends a confirmation to the user.

[1433] This step allows users to efficiently and personalizedly search and purchase products.

[1434] Example 2

[1435] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1436] Current online shopping systems generate search results based solely on input data provided by the user, making it difficult to provide product suggestions that reflect the user's emotional state or current mood. This can result in a lack of optimal suggestions for the products the user is looking for, which can lead to low satisfaction. Furthermore, they are unable to effectively utilize various input formats, such as voice input or image input, preventing flexible searches. Furthermore, they lack a mechanism for recognizing user emotions in real time and reflecting them in search results, making it difficult to provide more personalized search results for each individual user.

[1437] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1438] In this invention, the server includes a means for receiving input from a user and transmitting the input to the server, a means for analyzing the input received by the server and searching for corresponding products from a product database, a means for selecting optimal products based on the analysis results and providing the user with appropriate search results, a means for recognizing the user's emotions and making product suggestions based on the emotions, and a means for analyzing text, voice, and images provided by the user to generate search criteria. This enables personalized product suggestions that reflect the user's emotional state, improving the user experience. Furthermore, flexible search criteria generation is realized by effectively utilizing various input formats, including voice input and image input.

[1439] "User" refers to an individual or organization that uses the System to search for and purchase products.

[1440] "Input" means data provided by a user to the system, and may include any one or more of text, voice, image, and / or emotion recognition data.

[1441] "Server" refers to a central processing unit that analyzes input data received from users and searches the product database and makes product suggestions based on the results.

[1442] "Product Database" refers to a data storage system that stores up-to-date product information provided by multiple vendors.

[1443] "Analysis" refers to the process by which the server processes input data received from the user and extracts meaningful information.

[1444] "Product suggestion" refers to the server presenting suitable products to the user based on the analysis results.

[1445] "Emotion recognition" refers to the process of analyzing a user's facial expressions and tone of voice to determine their current emotional state.

[1446] "Natural language processing" refers to technology that enables computers to understand and analyze the language that humans use on a daily basis.

[1447] "Speech recognition" refers to the technology of converting voice data into text data.

[1448] "Image analysis" refers to the technology of extracting important features and information from image data.

[1449] "Ranking" refers to the process of prioritizing products based on a user's purchasing history, ratings, current search criteria, and emotional state.

[1450] "Feedback" refers to ratings and comments provided by users that are used to improve the performance of the system and the quality of suggestions.

[1451] "Checkout" refers to the process by which a User enters the necessary information to purchase a selected Product and completes the transaction.

[1452] The present invention provides a system for providing more personalized search results by combining product search and user emotion recognition in online shopping. This system improves the user experience by recognizing the user's emotions and suggesting appropriate products based on those emotions. Specific embodiments are described below.

[1453] User Interface Design

[1454] Users access the system through a dedicated application or a web browser on their smartphone or PC. The user interface includes a search bar, a voice input button, an image upload button, and a camera icon for emotion recognition, allowing users to input search requests in a variety of ways.

[1455] Processing User Input

[1456] Users can enter specific search criteria using text input, voice input, image upload, and emotion recognition. For example, a user may enter text such as "I'm looking for black shoes for the office" and enable emotion recognition. In this case, the user can also use voice input and image upload.

[1457] Sending and parsing input

[1458] The device captures the user's input data and transmits it to a server over a network, including text, voice, images, and emotion recognition data.

[1459] The server analyzes the received data, specifically:

[1460] The text data is analyzed using natural language processing techniques to generate search queries.

[1461] The voice data is converted into text using voice recognition technology, and the text is analyzed.

[1462] Image data is subjected to image analysis techniques to extract features.

[1463] The emotion recognition data is analyzed by an emotion engine to identify the user's emotional state.

[1464] Linking and searching with product databases

[1465] The server uses the generated search query to search a product database, which contains up-to-date product information and data provided by multiple vendors. For example, it issues an SQL query to retrieve a list of matching products.

[1466] Product Selection and Ranking

[1467] The server passes the retrieved product list to an AI model for ranking. The AI ​​model generates rankings by taking into account the user's past purchase history and ratings, current search criteria, and emotional state. For example, if the user's emotion is recognized as "tired," products with a relaxing effect may be displayed preferentially.

[1468] Displaying search results

[1469] The terminal displays a ranked list of products sent from the server to the user, and the search results include detailed information such as product images, prices, user reviews, availability, and reasons for recommendation.

[1470] Feedback and Interaction

[1471] The user can provide feedback on the displayed search results, for example, by clicking a rating button or entering a comment. The device sends the feedback information to the server, and the server further optimizes the search results based on the feedback.

[1472] Start of purchase flow

[1473] Once the user selects the desired product, the terminal displays the checkout screen, where the user enters their shipping address and payment information. The server processes the purchase request, updates the relevant database, and completes the transaction by contacting an external service that handles payment processing.

[1474] Prompt Sentence Examples

[1475] "How can I use natural language processing techniques to turn text into a search query?"

[1476] This system allows users to obtain search results that reflect their emotional state, enabling them to enjoy a more satisfying online shopping experience. In addition, by combining it with voice and image input, the system can accommodate a wider variety of search conditions.

[1477] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1478] Step 1:

[1479] Users access the system by opening a dedicated application or web browser on their smartphone or PC. The user interface displays a search bar, a voice input button, an image upload button, and a camera icon for emotion recognition, allowing users to select which form of input they want.

[1480] Step 2:

[1481] Input is provided in the form of your choice. For example, you can type "I want black shoes for the office" into the search bar, or press the voice input button to search by voice. You can also click the camera icon to turn on emotion recognition. Input data can include text, voice, image data, and emotion data.

[1482] Step 3:

[1483] The device captures the user's input data and sends it to the server via the network. The data sent can be text, audio, images, emotion recognition data, etc. The most common method of transmission is via HTTP request.

[1484] Step 4:

[1485] The server analyzes the received data. In the case of text data, it is analyzed using natural language processing (NLP) technology and converted into a search query. Voice data is converted into text using voice recognition technology, and image data is used to extract important features using image analysis technology. Emotion recognition data is analyzed by an emotion engine to determine the user's emotional state, such as whether they are tired or happy.

[1486] Input: Text, audio, image, and emotion data from users

[1487] Data processing: natural language processing, speech recognition, image analysis, emotion recognition

[1488] Output: Parsed search queries and sentiment data

[1489] Step 5:

[1490] The server uses the generated search query to search a product database, e.g., using an SQL query, to retrieve a list of relevant products. This database contains up-to-date product information from multiple vendors.

[1491] Input: Search query

[1492] Data Calculation: SQL Query Issue

[1493] Output: List of related products

[1494] Step 6:

[1495] The server passes the retrieved product list to an AI model for ranking. The AI ​​model generates rankings based on the user's past purchase history and ratings, current search criteria, and emotional state. For example, if a user is perceived as "tired," products with a relaxing effect will be displayed preferentially.

[1496] Input: product list, user emotional state

[1497] Data Calculation: Ranking by AI Model

[1498] Output: Ranked product list

[1499] Step 7:

[1500] The device displays a ranked list of products sent from the server to the user, and the search results include product images, prices, user reviews, availability, and recommendations based on emotional state.

[1501] Input: Ranked product list

[1502] Data processing: Display processing on HTML or application

[1503] Output: Search results displayed on the user's screen

[1504] Step 8:

[1505] Users can provide feedback on the search results they see by clicking the rating button or entering comments. The device sends this feedback information to the server, which then uses it to optimize search results and rankings.

[1506] Input: User feedback data

[1507] Data processing: Analysis of feedback information and learning

[1508] Output: Optimized ranking and search results

[1509] Step 9:

[1510] If the user decides to purchase the desired product, the device displays a checkout screen, where the user enters the required shipping address and payment information. The server processes the purchase request, updates the product database, and completes the transaction by contacting an external service that handles payment processing.

[1511] Input: User purchase information, payment information

[1512] Data calculation: Processing purchase information and linking with external services

[1513] Output: Confirmation of successful purchase and notification to the user

[1514] (Application example 2)

[1515] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1516] In conventional online shopping systems, when users select from a large number of products, they are unable to receive suggestions for products that match their current emotions or mood, resulting in a uniform, unpersonalized purchasing experience. Furthermore, the payment process does not take into account the user's emotional state, making it difficult to improve user satisfaction. This raises concerns that users' purchasing motivation may decrease and the conversion rate of e-commerce sites may fall.

[1517] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1518] In this invention, the server includes means for receiving input from a user and transmitting the input to the server, means for the server to analyze the input received and search for corresponding products from a product database, means for selecting optimal products based on the analysis results and providing appropriate search results to the user, and means for recognizing the emotional state of the user and adjusting the search results provided based on the emotional state, thereby enabling the provision of personalized search results that take into account the emotional state of the user and the optimization of the payment interface according to the user's emotions.

[1519] "User" means an individual or legal entity that uses the System to search for and purchase Products.

[1520] "Input" is information a user provides to a system, which may be text, audio, images, or a combination of these.

[1521] A "server" is a computer system that receives and analyzes input submitted by a user.

[1522] The "product database" is a database that stores information about products to be searched.

[1523] "Analysis" refers to the means of processing and understanding received input data, and includes natural language processing, speech recognition, image analysis, emotion recognition, etc.

[1524] "Search Results" are lists of relevant products provided by the server based on the user's input and emotional state.

[1525] "Emotional state" refers to the user's psychological and emotional state, as analyzed from the user's facial expressions, tone of voice, etc.

[1526] An "emotion recognizer" is a combination of software and hardware for analyzing a user's emotional state.

[1527] The "payment interface" is a user interface that allows a user to confirm the purchase of a product and carry out payment procedures.

[1528] "Optimization" is the process of improving the user experience by adjusting payment interfaces and search results based on the user's emotional state.

[1529] MODE FOR CARRYING OUT THE INVENTION

[1530] The present invention is an electronic payment system that uses emotion recognition to provide personalized search results and payment interfaces. The system basically consists of a user terminal, a server, an emotion recognition engine, a natural language processing engine, a voice recognition engine, an image analysis engine, and a product database.

[1531] User Interface Design

[1532] Users access the system using a smartphone or personal computer through a dedicated application or a web browser. The user interface includes a search bar, a voice input button, an image upload button, and a camera icon for emotion recognition, allowing users to provide various forms of input to the system.

[1533] Processing User Input

[1534] Users can enter text into the search bar or use voice input by pressing the voice input button. They can also upload an image or click the emotion recognition camera icon to capture natural facial expressions. For example, they can turn on emotion recognition while typing, "I want black shoes that go with the office."

[1535] Data transmission to the server and analysis

[1536] The device captures input data from the user and sends it over the network to a server. The data sent can be text, voice, images, or emotion recognition data. The server analyzes the received data and generates a search query using appropriate technologies, such as a natural language processing engine, voice recognition engine, image analysis engine, or emotion recognition engine.

[1537] Linking and searching with product databases

[1538] The server uses the generated search query to search the product database, which stores the latest product information, and retrieves a list of matching products. The results are retrieved in the form of a database query, such as an SQL query.

[1539] Product selection and ranking

[1540] The server passes the product list obtained as a search result to an artificial intelligence (AI) model for ranking. The AI ​​model generates rankings by taking into account the user's past purchase history, ratings, current search criteria, and emotional state. For example, if the user's emotion is recognized as "tired," products with a relaxing effect will be displayed preferentially.

[1541] Displaying search results

[1542] The device displays a ranked list of products sent from the server to the user, and the search results include detailed information such as product images, prices, user reviews, availability, and even recommendations based on emotional state.

[1543] Optimizing payment interfaces based on emotion recognition

[1544] Once the user finds the product they want, the terminal will prompt them for payment. The payment interface is optimized based on the user's emotional state. For example, if the user is feeling stressed, the terminal will offer the option to complete the payment with one click.

[1545] Feedback and Interaction

[1546] Users can provide feedback on the search results displayed, and the device will send the feedback information to the server and further optimize the search results based on it. If the user indicates a change in their sentiment, the system will update the search results in real time.

[1547] Examples of concrete examples and prompts

[1548] For example, if a user's camera image indicates that they look tired, a simple interface will be provided to complete payment with one click. Additionally, promotions for products that have a relaxing effect will be displayed based on the user's past purchase history and current emotional state.

[1549] Prompt Sentence Examples

[1550] If the user feels tired, generate a prompt to introduce relaxation items, such as massagers, aroma candles, etc., and provide an interface for one-click payment.

[1551] This allows users to easily find the perfect product based on their emotional state and expedite the checkout process.

[1552] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1553] Program processing steps

[1554] Step 1:

[1555] Users can use their device to enter text into the search bar or use voice input by pressing the voice input button, upload an image, or capture facial expressions by clicking the emotion recognition camera icon.

[1556] Input: text, audio, images

[1557] Output: Input data is captured to the terminal

[1558] Step 2:

[1559] The device transmits the captured input data to a server over a network, including text data, voice data, image data, and emotion recognition data.

[1560] Input: Captured data

[1561] Output: The input data is sent to the server

[1562] Step 3:

[1563] The server analyzes the data it receives: for text data, a natural language processing engine generates search queries; for audio data, a speech recognition engine converts it into text; for image data, an image analysis engine extracts key features; for emotion data, an emotion recognition engine analyzes facial expressions and voice tone to identify emotional states.

[1564] Input: Data sent to the server

[1565] Output: search query, emotional state

[1566] Step 4:

[1567] The server uses the generated search query to search a product database, which contains up-to-date product information, to retrieve a list of suitable products.

[1568] Input: search query

[1569] Output: Product list

[1570] Step 5:

[1571] The server passes the search results to an AI model that ranks the products, taking into account the user's past purchase history, ratings, current search criteria, and emotional state.

[1572] Input: Product list, past purchase history, emotional state

[1573] Output: Ranked product list

[1574] Step 6:

[1575] The server sends the ranked product list to the device, which is also based on the user's emotional state, resulting in personalized results.

[1576] Input: Ranked product list

[1577] Output: Ranked product list sent to device

[1578] Step 7:

[1579] The device displays a ranked list of products sent from the server to the user, and the displayed search results include product images, prices, user reviews, availability, and recommendations based on emotional state.

[1580] Input: Ranked product list

[1581] Output: Product information displayed to the user

[1582] Step 8:

[1583] Once the user finds the product they want, the terminal will prompt them for payment. The payment interface is optimized based on the user's emotional state. For example, if the user is feeling stressed, the terminal will offer the option to complete the payment with one click.

[1584] Input: Emotional state, product selection data

[1585] Output: Payment interface

[1586] Step 9:

[1587] Users can provide feedback on the displayed search results, and the terminal will send the feedback information to the server, which will then further optimize the search results.

[1588] Input: User feedback

[1589] Output: Optimized search results

[1590] Specific example of operation

[1591] For example, the following shows how to generate a prompt sentence that determines that the user is "tired" and displays a promotion for a product that has a relaxing effect.

[1592] Example prompts

[1593] If the user feels tired, generate a prompt to introduce relaxation items, such as massagers, aroma candles, etc., and provide an interface for one-click payment.

[1594] This allows users to easily find the perfect product based on their emotional state and expedite the checkout process.

[1595] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1596] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1597] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1598] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1599] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1600] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1601] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1602] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1603] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1604] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1605] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1606] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1607] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1608] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1609] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1610] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1611] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1612] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1613] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1614] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1615] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1616] The following is further disclosed regarding the above embodiment.

[1617] (Claim 1)

[1618] means for receiving input from a user and transmitting the input to a server;

[1619] A means for analyzing the input received by the server and searching for a corresponding product from a product database;

[1620] A means to select the most suitable products based on the analysis results and provide appropriate search results to users;

[1621] A system including:

[1622] (Claim 2)

[1623] 2. The system according to claim 1, further comprising natural language processing means for generating an appropriate search query even when an ambiguous input is made.

[1624] (Claim 3)

[1625] 10. The system of claim 1, further comprising means for receiving voice input and image input, analyzing the input data, and using the input data as search criteria.

[1626] "Example 1"

[1627] (Claim 1)

[1628] means for receiving input from a user and transmitting the input to a server;

[1629] A means for analyzing the input received by the server and searching for a corresponding product from a product database;

[1630] A means for selecting the most suitable product based on the analysis results using a ranking means and providing the user with appropriate search results;

[1631] A means to receive user feedback and update search criteria in real time;

[1632] A means to consider the user's past purchase history and ratings in future searches;

[1633] A system including:

[1634] (Claim 2)

[1635] The system according to claim 1, further comprising a natural language processing means for generating an appropriate search query even when an ambiguous input is made, a speech recognition means, and an image recognition means.

[1636] (Claim 3)

[1637] 10. The system of claim 1, further comprising means for receiving voice and image input from a user, analyzing the same and using the same as search criteria.

[1638] "Application Example 1"

[1639] (Claim 1)

[1640] means for receiving input from a user and transmitting the input to a server;

[1641] A means for analyzing the input received by the server and searching for a corresponding product from a product database;

[1642] A means to select the most suitable products based on the analysis results and provide appropriate search results to users;

[1643] means for capturing user voice input using an in-vehicle voice recognition device;

[1644] A means for visually displaying search results using an in-vehicle infotainment system and allowing the user to select products using a touch screen or voice control;

[1645] A system including:

[1646] (Claim 2)

[1647] 2. The system according to claim 1, further comprising natural language processing means for generating an appropriate search query even when an ambiguous input is made.

[1648] (Claim 3)

[1649] 10. The system of claim 1, further comprising means for receiving voice input and image input, analyzing the input data, and using the input data as search criteria.

[1650] "Example 2: Combining Emotion Engines"

[1651] (Claim 1)

[1652] means for receiving input from a user and transmitting the input to a server;

[1653] A means for analyzing the input received by the server and searching for a corresponding product from a product database;

[1654] A means to select the most suitable products based on the analysis results and provide appropriate search results to users;

[1655] A means for recognizing user emotions and making product suggestions based on those emotions;

[1656] A means of generating search criteria by analyzing user-provided text, voice, and images;

[1657] A system including:

[1658] (Claim 2)

[1659] 2. The system according to claim 1, further comprising natural language processing means for generating an appropriate search query even when an ambiguous input is made.

[1660] (Claim 3)

[1661] 10. The system of claim 1, further comprising means for receiving voice input and image input, analyzing the input data, and using the input data as search criteria.

[1662] "Application example 2 when combining emotion engines"

[1663] (Claim 1)

[1664] means for receiving input from a user and transmitting the input to a server;

[1665] A means for analyzing the input received by the server and searching for a corresponding product from a product database;

[1666] A means to select the most suitable products based on the analysis results and provide appropriate search results to users;

[1667] a means for recognizing a user's emotional state and tailoring the search results provided based on the user's emotional state;

[1668] A system including:

[1669] (Claim 2)

[1670] 2. The system according to claim 1, further comprising natural language processing means for generating an appropriate search query even when an ambiguous input is made.

[1671] (Claim 3)

[1672] 10. The system of claim 1, further comprising means for receiving voice input and image input, analyzing the input data, and using the input data as search criteria.

[1673] (Claim 4)

[1674] 10. The system of claim 1, further comprising means for optimizing the payment interface based on the emotional state of the user and providing a corresponding payment procedure. [Explanation of symbols]

[1675] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving input from a user and transmitting the input to a server; A means for analyzing the input received by the server and searching for a corresponding product from a product database; A means to select the most suitable products based on the analysis results and provide appropriate search results to users; A system including:

2. The system according to claim 1 , further comprising natural language processing means for generating an appropriate search query even when an ambiguous input is made.

3. 2. The system of claim 1, further comprising means for receiving voice input and image input, analyzing the input data, and using the input data as search criteria.

Images