System

Abstract

A system is provided.SOLUTION: A system includes a means for receiving input data from a user, a means for analyzing the received input data, a means for proposing a related product or service on the basis of an analysis result, a means for displaying the proposed product or service on a user terminal, and a means for providing a support service after a contract.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional systems have the problem that it takes a lot of time and effort for users to find products and services that meet their needs. They also require users to browse catalogs and read long texts, which hinders efficient purchasing. Furthermore, post-contract support is complicated and the means for quick responses are limited, which can lead to a decline in user satisfaction. [Means for solving the problem]

[0005] The present invention solves the above problems by the following means: a system including means for accepting input data from a user, means for analyzing the accepted input data, means for proposing related products or services based on the analysis results, means for displaying the proposed products or services on a user terminal, and means for providing post-contract support services. This system allows users to easily find products or services that match their preferences and needs, and enables them to receive prompt support even after signing a contract.

[0006] "User" means an individual or entity that uses the System to search for goods or services.

[0007] "Input Data" means information a user provides to the system to determine suggested goods or services, including photographs, images, text, or video.

[0008] The term "means" refers to a physical or logical device, module, program, or combination thereof for realizing a specific function.

[0009] The "receiving means" is a device, module, or program that has the function of taking input data from a user into the system.

[0010] The "analyzing means" is a device, module, or program that has the function of processing received input data and extracting meaningful information.

[0011] The "means for making a suggestion" is a device, module, or program that has the function of selecting related products or services based on the analysis results and presenting them to the user.

[0012] The "displaying means" is a device, module, or program having a function for visually displaying the proposed product or service on the user terminal.

[0013] A "means for providing support services" is a device, module, or program that has the function of providing support to users through chat, messages, voice calls, video calls, etc. in response to questions or problems they may have after signing a contract.

[0014] A "system" is a technical configuration that combines the above means to propose and support comprehensive products or services to users. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The system of the present invention proposes optimal products and services based on data entered by the user, and also provides support after the contract is signed. A specific embodiment of this system will be explained in natural language, with the program processing also being presented, along with specific examples.

[0037] Accepting user input data

[0038] Users access the system using devices such as smartphones, tablets, and PCs. They can input photos, images, text, and videos of products and services they are interested in. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[0039] The terminal receives input data from the user, converts it into an appropriate format, and sends it to the server. For example, photo data in JPEG format can be sent as is.

[0040] Data analysis

[0041] The server runs algorithms to analyze the input data it receives. For photos and videos, image recognition technology is used to extract key features, such as color, shape, and pattern. For text, natural language processing (NLP) technology is used to analyze the user's intent. For example, a keyword like "blue dress" can be used to identify the color and type of item.

[0042] Proposal Generation

[0043] Once the analysis is complete, the server searches the database for relevant products and services and selects those that best meet the user's needs. The results are presented in multiple formats, including photos, text, and video. For example, if the search keyword is "blue dress," product information for multiple blue dresses will be provided along with photos.

[0044] View Suggestions

[0045] The server sends the generated proposal data to the user's device. The device receives the data and displays it visually in an easy-to-understand format. On smartphones, the proposals are displayed on the app screen or via push notifications, while on PCs, they are displayed in a new tab on the browser.

[0046] Post-contract support services

[0047] If a user is interested in a proposed product or service and wishes to sign a contract, they can easily complete the contract procedure through their device. For example, when a user purchases a dress on their smartphone, they simply click the purchase button and enter the necessary payment information to complete the purchase procedure.

[0048] After a purchase or contract is completed, the server synchronizes the contract information in real time and prepares to provide subsequent support services. The server provides support through chat and messaging functions, and also provides real-time support via voice and video calls. For example, if a user has a question about how to use or care for a purchased dress, they can ask through chat and receive an immediate answer.

[0049] Specific examples

[0050] For example, if a user is looking for a particular piece of furniture, the following process applies:

[0051] 1. The user takes a photo of the furniture they want to match with their home's interior using their smartphone and uploads it to the system.

[0052] 2. The device sends the photo data to the server.

[0053] 3. The server runs an image recognition algorithm to analyze the photo and extract features such as furniture type, color, and shape.

[0054] 4. The server searches a database for relevant furniture and generates multiple options, such as a list of sofas and tables with photos.

[0055] 5. The server sends the generated suggestion data to the device, which displays it to the user, who then checks the search results on their smartphone.

[0056] 6. The user selects their favorite furniture from the suggested items and completes the purchase process.

[0057] 7. The server synchronizes the contract information and provides support services as needed, such as video guides on how to assemble furniture.

[0058] In this way, users can easily obtain products and services that suit them and receive comprehensive support even after signing a contract.

[0059] The processing flow will be explained below.

[0060] Step 1:

[0061] Users access the system using devices such as smartphones, tablets, and PCs. They input photos, images, text, and videos of their interests. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[0062] Step 2:

[0063] The terminal receives input data from the user, converts it into an appropriate format, and sends it to the server. For example, photo data in JPEG format can be sent as is.

[0064] Step 3:

[0065] The server runs algorithms to analyze the input data it receives. For photos and videos, image recognition technology is used to extract key features, such as color, shape, and pattern. For text, natural language processing (NLP) technology is used to analyze the user's intent. For example, the color and type of item can be identified from keywords such as "blue dress."

[0066] Step 4:

[0067] Based on the analysis results, the server searches the database for related products and services and selects those that best meet the user's needs. The selected results are generated in multiple formats, including photos, text, and video. For example, if the search keyword is "blue dress," product information for multiple blue dresses will be provided along with photos.

[0068] Step 5:

[0069] The server sends the generated proposal data to the user's device in a format appropriate for the device.

[0070] Step 6:

[0071] The device receives the proposal data and displays it to the user in an easy-to-understand visual format. On smartphones, the proposal content is displayed on the app screen or via push notification, while on PCs, the proposal content is displayed in a new tab on the browser.

[0072] Step 7:

[0073] The user becomes interested in the proposed products or services and makes a selection. The user then checks for more information about the selected product or service, such as detailed product specifications, price, and user reviews.

[0074] Step 8:

[0075] The user finally completes the purchase or contract process by clicking the purchase button and entering the required payment information, such as credit card information and shipping address.

[0076] Step 9:

[0077] The server receives purchase and contract information, stores it in a database, and synchronizes contract information in real time.

[0078] Step 10:

[0079] The server provides channels such as chat, messaging, voice calls, and video calls for post-contract support services. For example, if a user has a question about how to use a product, they can receive support through the chat function.

[0080] Step 11:

[0081] The server provides product usage guides and support materials, such as text guides and video guides, and sends them to the device, which then displays the guides to the user, who can then refer to them.

[0082] This allows users to use the system intuitively throughout the entire process, efficiently obtain products and services that suit them, and even receive prompt support after signing a contract.

[0083] Example 1

[0084] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0085] In conventional systems, it takes a great deal of time and effort for users to search for products and services and find the most suitable ones. Furthermore, there is a problem that support after signing a contract is not provided sufficiently, resulting in a decrease in user satisfaction. The present invention aims to solve these problems by providing a system that allows users to efficiently find products and services that suit them and receive comprehensive support even after signing a contract.

[0086] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0087] In this invention, the server includes means for converting received input data into an appropriate format, means for transmitting the converted data to the server, means for receiving data at the server and checking the format and content of the data, means for analyzing the received data, means for searching for related products or services based on the analysis results, means for generating an optimal proposal based on the search results, means for transmitting the generated proposal to a user terminal, means for displaying the proposed product or service on the user terminal, means for inputting information required for contract procedures, means for transmitting contract information to the server and completing the contract procedures, and means for providing post-contract support services. This enables users to efficiently find products or services, sign contracts, and receive consistent support even after the contract is signed.

[0088] "User" refers to a general user who uses the system to search for and purchase goods and services.

[0089] "Terminal" refers to an electronic device, such as a smartphone, tablet, or PC, that a user uses to access the system, enter information, or receive suggestions.

[0090] "Server" refers to the central processing unit that receives and analyzes data sent by users and generates optimal product and service recommendations.

[0091] "Input data" refers to data such as photographs, images, text, and videos that users input to provide information about products and services to the system.

[0092] "Means for converting into a format" refers to the process by which the terminal receives input data from the user and converts it into an appropriate digital format that can be sent to the server.

[0093] "Means for transmitting data to a server" refers to a function for transmitting converted data to a server via the Internet or other communication network.

[0094] "Means for analyzing data" refers to the function of the server using algorithms to process the data it receives and understand and classify its content.

[0095] "Means for searching for products or services" refers to the function of searching for products or services that meet the user's needs from a database based on the analysis results.

[0096] The "means of generating suggestions" refers to the process of formulating and creating information on optimal products and services to be offered to users based on search results.

[0097] "Means for transmitting suggestions to a user terminal" refers to a communication function for transmitting the generated suggestions to a user terminal and visually displaying them.

[0098] "Means for entering information necessary for contract procedures" refers to the interface through which users enter personal information and payment information required when purchasing or signing a contract for a product or service.

[0099] "Means for completing the contract procedure" refers to the process of sending the entered information to the server and officially concluding a contract for the purchase of goods or services.

[0100] "Means for providing post-contract support services" refers to functions for providing additional information and answering questions to users after the contract is completed.

[0101] The system of the present invention proposes optimal products and services based on data entered by a user, and also provides support after the contract is signed. A specific embodiment of this system will be described in detail below.

[0102] Accepting user input data

[0103] Users access the system using devices such as smartphones, tablets, and PCs. They can input data about products and services that interest them. Data that can be input includes photos, images, text, and videos. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[0104] The terminal receives input data from the user, checks the data format, converts it into an appropriate digital format (e.g., JPEG format, UTF-8 format), and then sends the converted data to the server.

[0105] Data reception and preprocessing

[0106] The server receives the data sent from the device. It checks the format and content of the received data to see if there is any improper data or corrupted files. For images, the server performs preliminary processing such as resizing and noise reduction.

[0107] Data analysis

[0108] The server runs image recognition algorithms to extract key features of the input image (e.g., color code, shape, texture), and uses natural language processing (NLP) algorithms for text data to analyze the user's intent. For example, for "blue dress," the server identifies the color as "blue" and the item as "dress."

[0109] Proposal Generation

[0110] Once the analysis is complete, the server searches the database for relevant products and services. Based on the search results, it generates recommendations that best suit the user's needs. Specifically, it applies filters such as price range, brand, and size, and generates the most suitable product and service information in the form of photos, text, and video.

[0111] View Suggestions

[0112] The server sends the generated proposal data to the device. The device receives the data and displays it in a visually easy-to-understand format for the user. On smartphones, the proposal content is displayed on the app screen or via push notification, and on PCs, the proposal content is displayed in a new tab on the browser.

[0113] Contract Procedures

[0114] The user is interested in the proposed product or service and decides to purchase or enter into a contract. The purchase procedure is completed by entering the necessary information (e.g., name, address, payment information) and sending it to the server via the device. The server synchronizes the contract information in real time, and the contract is officially concluded.

[0115] Post-contract support

[0116] After signing the contract, the server provides the user with additional support services. Specifically, the server provides instant support for user questions through chat and messaging functions. It can also answer user questions in real time using voice and video calls. For example, it can provide usage guides and assembly video tutorials for purchased products.

[0117] Specific examples

[0118] For example, if a user is looking for a particular piece of furniture, the following process applies:

[0119] 1. The user takes a photo of a sofa that suits their living room with their smartphone and uploads it through the app.

[0120] 2. The device converts the photo data into JPEG format and sends it to the server.

[0121] 3. The server receives the photo data and resizes and denoises the image.

[0122] 4. The server uses image recognition algorithms to analyze the sofa's color, shape, and texture.

[0123] 5. The server searches the database for blue living room sofas, filtering by price range and brand.

[0124] 6. The server generates multiple blue sofa photos and details as suggestions.

[0125] 7. The server sends the generated proposal data to the terminal, which displays it to the user.

[0126] 8. The user selects their favorite sofa from the suggestions and proceeds with the purchase.

[0127] 9. The user enters their payment information and clicks the purchase button.

[0128] 10. The device sends the entered information to the server and confirms the purchase.

[0129] 11. The server synchronizes the purchase information and provides the user with assembly instructions via chat.

[0130] 12. If the user asks any follow-up questions, the server will answer them immediately.

[0131] Prompt Sentence Examples

[0132] "I'm looking for a new blue dress. What are my options?"

[0133] "I'm looking for a sofa that will complement my home decor. I've uploaded some photos. Can you show me the options?"

[0134] This allows users to efficiently find products and services that suit them, sign contracts, and receive comprehensive support afterward.

[0135] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0136] Step 1:

[0137] Users access the system using devices such as smartphones, tablets, and PCs. They can input photos, images, text, and videos of products and services that interest them. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system. The input data is accepted by the device.

[0138] Input: User input of photos, images, text, and videos

[0139] Output: Input data stored on the terminal

[0140] Specific behavior: The user opens the app or browser, enters the required information, and clicks the "Submit" button.

[0141] Step 2:

[0142] The terminal checks the data entered by the user and converts it into the appropriate format: if it's a photo, it's converted into JPEG format, if it's text, it's converted into UTF-8 format, and then sends the converted data to the server.

[0143] Input: User-entered photo, image, text, and video data

[0144] Output: Transformed data to send to the server

[0145] Specific operation: The terminal checks the data format, checks for inconsistencies, converts the data format, and sends it to the server via the network.

[0146] Step 3:

[0147] The server receives the data sent from the device. It checks the format and content of the received data to see if there is any improper data or corrupted files. For images, the server performs preliminary processing such as resizing and noise reduction.

[0148] Input: Transformed data sent from the terminal

[0149] Output: Preprocessed data

[0150] Specific operation: The server receives the data, and a resident script checks the data format and content, and performs image resizing and noise removal if necessary.

[0151] Step 4:

[0152] The server runs image recognition algorithms to extract key features of the input image (e.g., color code, shape, texture), and uses natural language processing (NLP) algorithms for text data to analyze the user's intent. For example, for "blue dress," the server identifies the color as "blue" and the item as "dress."

[0153] Input: Preprocessed data (images, text)

[0154] Output: Analysis results (feature extraction information, user intent)

[0155] How it works: An image recognition model on the server analyzes images and extracts color codes and shape information, while an NLP model analyzes text data to identify keywords and intent.

[0156] Step 5:

[0157] The server uses the analyzed data to search for relevant products and services from its database. Based on the search results, it generates recommendations that best suit the user's needs. Specifically, it applies filters such as price range, brand, and size, and generates the most suitable product and service information in the form of photos, text, and video.

[0158] Input: Analysis results

[0159] Output: Product and service proposal data

[0160] Specific operation: The server accesses an internal database, searches for products and services that match the criteria, and generates proposals based on the results.

[0161] Step 6:

[0162] The server sends the generated proposal data to the device. The device receives the data and displays it in a visually easy-to-understand format for the user. On smartphones, the proposal content is displayed on the app screen or via push notification, and on PCs, the proposal content is displayed in a new tab on the browser.

[0163] Input: Generated proposal data

[0164] Output: Proposal displayed on the user's device

[0165] Specific operation: The server sends the proposed data over the network, the device receives it and displays it on the screen, and a push notification is displayed on the smartphone.

[0166] Step 7:

[0167] The user is interested in the proposed product or service and decides to purchase or sign a contract. They enter the necessary information (e.g., name, address, payment information) and send it to the server via their device.

[0168] Input: Name, address, and payment information entered by the user

[0169] Output: Contract information sent to the server

[0170] Specific operation: When the user enters the required information into the form and clicks the "Purchase" or "Contract" button, the information is sent to the server.

[0171] Step 8:

[0172] The server synchronizes the contract information in real time, and the contract is officially established. After the contract is signed, additional support services are provided to the user. Specifically, immediate support for user questions is provided through chat and messaging functions. Users' questions can also be answered in real time using voice and video calls.

[0173] Input: Contract information

[0174] Output: Providing support services

[0175] What it does: The server updates contract information and, if necessary, connects with the support team to provide the necessary support to the user, responding in real time via chat, voice, or video calls.

[0176] (Application example 1)

[0177] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0178] Conventional systems have had difficulty efficiently and accurately recommending products and services that interest users and providing prompt post-purchase support. This problem stems from the difficulty of accurately understanding the data entered by users and making recommendations that meet their needs. Therefore, it is necessary to build a system that can accurately analyze the data entered by users, recommend optimal products and services, and provide prompt post-purchase support.

[0179] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0180] In this invention, the server includes means for accepting input data from a user, means for analyzing the accepted input data, means for proposing related products or services based on the analysis results, means for displaying the proposed products or services on the user terminal, means for providing post-contract support services, means for generating prompt sentences to optimize interpretation of the input data, means for providing real-time support via chat and video calls based on the analysis results, and means for analyzing the accepted input data using image recognition technology and natural language processing technology. This makes it possible to consistently provide product proposals and post-contract support that quickly and accurately meet the user's needs.

[0181] "Means for accepting input data from the user" refers to an interface that allows the user to provide the system with information about products and services in which the user is interested.

[0182] "Means for analyzing received input data" refers to the process of analyzing data provided by users using image recognition technology and natural language processing technology to understand their needs and intentions.

[0183] "Means for suggesting related products or services based on the analysis results" refers to a function for selecting optimal products or services from a database based on the analyzed data and recommending them to the user.

[0184] "Means for displaying proposed products or services on a user's terminal" refers to a function for visually displaying information about proposed products on a terminal such as a user's smartphone.

[0185] "Means for providing post-contract support services" refers to communication means such as chat and video calls for providing after-sales support for products and services purchased by users.

[0186] "Means for generating prompts to optimize interpretation of input data" refers to means for generating precise instructions or questions based on information obtained from the user and for effectively analyzing that data.

[0187] "Means for providing real-time chat and video call support based on the analysis results" refers to a function that uses the analyzed information to respond via chat or video call so that the user can receive immediate support.

[0188] "Means of analyzing input data received using image recognition technology and natural language processing technology" refers to technology that uses advanced algorithms to analyze images and text data provided by users and understand their content.

[0189] The system of the present invention proposes optimal products and services based on user input data and provides support after the contract is signed. To implement this system, the following programs and hardware are used.

[0190] 1. Accepting user input data

[0191] Users access the system using a smart device (smartphone, tablet, or PC) and enter photos, images, text, and videos of products and services they are interested in. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[0192] 2. Data Analysis

[0193] The server analyzes the received input data using image recognition technology (OpenCV and TensorFlow) and natural language processing technology (NLTK). For photos and videos, it extracts image features (color, shape, pattern, etc.), and for text, it analyzes keywords to identify the user's intent.

[0194] 3. Proposal Generation

[0195] Based on the analyzed data, the server searches the database for relevant products and services and generates proposals that best meet the user's needs. These proposals can be in the form of photos, text, videos, etc.

[0196] 4. Display of suggestions

[0197] The server sends the generated proposal data to the user's device, which then displays the received data in an easy-to-understand visual format. On smartphones, the proposals are displayed on the app screen or via push notification, while on PCs, the proposals are displayed in a new tab on the browser.

[0198] 5. Post-contract support services

[0199] If a user is interested in a proposed product or service and signs a contract, they can easily complete the contract process through their device. For example, when a user purchases a dress on their smartphone, they simply click the purchase button and enter their payment information to complete the process. The server synchronizes the contract information in real time and prepares to provide subsequent support services. The server provides support through chat, messaging, voice calls, and video calls.

[0200] Specific examples

[0201] If a user is looking for a particular piece of furniture, the following process applies:

[0202] 1. The user takes a photo of the furniture they want to match with their home's interior using their smartphone and uploads it to the system.

[0203] 2. The device sends the photo data to the server.

[0204] 3. The server runs an image recognition algorithm (OpenCV, TensorFlow) to analyze the photo and extract features such as furniture type, color, and shape.

[0205] 4. The server searches a database for relevant furniture and generates multiple options, such as a list of sofas and tables with photos.

[0206] 5. The server sends the generated suggestion data to the device, which displays it to the user, who then checks the search results on their smartphone.

[0207] 6. The user selects their favorite furniture from the suggested items and completes the purchase process.

[0208] 7. The server synchronizes the contract information and provides support services as needed, such as video guides on how to assemble furniture.

[0209] Prompt Sentence Examples

[0210] "Please analyze this photo and suggest suitable products. The photo shows a blue sofa. Also, please search your database for similar sofas and display them."

[0211] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0212] Step 1:

[0213] Users use their smart devices to input photos, images, text, and videos related to products and services they are interested in and upload them to the system. The input data is sent from the user's device to the server as an HTTP request. At this point, the input is the photo taken or text entered by the user, and the output is unanalyzed data sent to the server.

[0214] Step 2:

[0215] The server analyzes the received data using image recognition technology (OpenCV and TensorFlow). Specifically, it extracts features (color, shape, pattern) from image data and analyzes keywords from text data using natural language processing (NLTK). In this process, the input is unanalyzed data from the user, and the output is analyzed feature information and keywords.

[0216] Step 3:

[0217] The server generates prompts based on the analysis results to optimize the interpretation of the input data. It uses a generative AI model to create prompts to more clearly identify the user's needs and intentions. In this step, the input is the analysis results, and the output is the prompts.

[0218] Step 4:

[0219] The server uses the generated prompt to search a database for relevant products and services. It executes a database query to extract the product information that best suits the user's request. In this operation, the input is the prompt and its parsed results, and the output is a list of suggested products and services.

[0220] Step 5:

[0221] The server sends the information of the proposed products and services to the user's device. The information is encoded in JSON format and returned as an HTTP response. The input is a list of proposed products and services, and the output is the data sent to the user's device.

[0222] Step 6:

[0223] The terminal displays the received proposal data in an easy-to-understand visual format. Specifically, it displays a list of product information on the smart device screen and offers the user options. In this step, the input is the proposal data sent from the server, and the output is the visual information presented to the user.

[0224] Step 7:

[0225] If a user is interested in the proposed product or service and wants to purchase it, he or she inputs the necessary information (e.g., payment information) through the terminal and sends it to the server. In this process, the input is the product information and payment information selected by the user, and the output is the purchase completion status.

[0226] Step 8:

[0227] The server synchronizes contract information in real time and prepares support services after purchase. The contract information is saved in the database and support services are initiated as needed. In this process, the input is the user's purchase information and the output is the support service preparation status.

[0228] Step 9:

[0229] The server provides real-time chat and video support. When a user has a question or problem, support staff are available to respond immediately. The input is the user's support request, and the output is real-time support information.

[0230] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0231] The system of the present invention analyzes input data from users, proposes relevant products or services, recognizes user emotions to optimize proposals, and provides post-contract support. A specific embodiment of this system will be explained below, with program processing explained in natural language and examples provided.

[0232] Accepting user input data

[0233] Users access the system using devices such as smartphones, tablets, and PCs. They input photos, images, text, and videos of their interests. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[0234] The terminal receives input data from the user, converts it into an appropriate format, and sends it to the server. For example, photo data in JPEG format can be sent as is.

[0235] Data analysis

[0236] The server runs algorithms to analyze the input data it receives. For photos and videos, image recognition technology is used to extract key features, such as color, shape, and pattern. For text, natural language processing (NLP) technology is used to analyze the user's intent. For example, a keyword like "blue dress" can be used to identify the color and type of item.

[0237] Furthermore, the server uses an emotion engine to recognize emotions from the user's input data, such as emotion analysis from text data, emotion analysis from audio data, and facial expression analysis from video data, to identify the user's current emotional state (e.g., joy, sadness, surprise).

[0238] Proposal Generation

[0239] Once the analysis is complete, the server searches the database for relevant products and services and selects those that best meet the user's needs. It then adjusts the recommendations based on the user's emotions, as determined by the emotion engine. For example, if the user is in a state of "joy" or "excitement," it will prioritize items with many positive product reviews.

[0240] The selected results are generated in multiple formats, including photos, text, video, etc. For example, if the search keyword is "blue dress," product information for multiple blue dresses will be provided along with photos.

[0241] View Suggestions

[0242] The server transmits the generated proposal data to the user's device in a format appropriate for the device.

[0243] The device then displays the received proposal data in a visually easy-to-understand format to the user. On smartphones, the proposal content is displayed on the app screen or via push notification, while on PCs, the proposal content is displayed in a new tab on the browser.

[0244] Post-contract support services

[0245] If a user is interested in a proposed product or service and wishes to sign a contract, they can easily complete the contract procedure through their device. For example, when a user purchases a dress on their smartphone, they simply click the purchase button and enter the necessary payment information to complete the purchase procedure.

[0246] After a purchase or contract is completed, the server synchronizes the contract information in real time and prepares to provide subsequent support services. The server provides support through chat and messaging functions, and also provides real-time support via voice and video calls. For example, if a user has a question about how to use or care for a purchased dress, they can ask through chat and receive an immediate answer.

[0247] Specific examples

[0248] For example, if a user is looking for a particular piece of furniture, the following process applies:

[0249] 1. The user takes a photo of the furniture they want to match with their home's interior using their smartphone and uploads it to the system.

[0250] 2. The device sends the photo data to the server.

[0251] 3. The server runs an image recognition algorithm to analyze the photo and extract features such as furniture type, color, and shape.

[0252] 4. The server uses an emotion engine to recognize the user's emotions from input data such as photos and text. If the user is recognized as being in a "lively" emotional state, casual, brightly colored furniture will be suggested.

[0253] 5. The server searches a database for relevant furniture and generates multiple options, such as a list of sofas and tables with photos.

[0254] 6. The server sends the generated suggestion data to the device, which displays it to the user, who then checks the search results on their smartphone.

[0255] 7. The user selects their favorite furniture from the suggested items and completes the purchase process.

[0256] 8. The server synchronizes the contract information and provides support services as needed, such as video guides on how to assemble furniture.

[0257] In this way, users can easily obtain products and services that suit them and receive comprehensive support even after signing a contract. In addition, the introduction of an emotion engine will make the user experience more personalized.

[0258] The processing flow will be explained below.

[0259] Step 1:

[0260] Users access the system using devices such as smartphones, tablets, and PCs. They input photos, images, text, and videos related to products and services they are interested in. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[0261] Step 2:

[0262] The terminal receives input data from the user, converts it into an appropriate format, and sends it to the server. For example, it sends photo data in JPEG format as is.

[0263] Step 3:

[0264] The server runs algorithms to analyze the input data it receives. For photos and videos, image recognition technology is used to extract key features, such as color, shape, and pattern. For text, natural language processing (NLP) technology is used to analyze the user's intent. For example, the color and type of item can be identified from keywords such as "blue dress."

[0265] Step 4:

[0266] Furthermore, the server uses an emotion engine to recognize emotions from the user's input data, such as emotion analysis from text data, emotion analysis from audio data, and facial expression analysis from video data, to identify the user's current emotional state (e.g., joy, sadness, surprise).

[0267] Step 5:

[0268] Based on the analysis results, the server searches its database for related products and services and selects those that best meet the user's needs. Furthermore, the server adjusts the recommendations based on the user's emotions, as recognized by the emotion engine. For example, if the user is in a state of "joy" or "excitement," it prioritizes items with many positive product reviews.

[0269] Step 6:

[0270] The server sends the generated proposal data to the user's device in a format appropriate for the device.

[0271] Step 7:

[0272] The device receives the proposal data and displays it to the user in an easy-to-understand visual format. On smartphones, the proposal content is displayed on the app screen or via push notification, while on PCs, the proposal content is displayed in a new tab on the browser.

[0273] Step 8:

[0274] The user becomes interested in the proposed products or services and makes a selection. The user then checks for more information about the selected product or service, such as detailed product specifications, price, and user reviews.

[0275] Step 9:

[0276] The user finally completes the purchase or contract process by clicking the purchase button and entering the required payment information, such as credit card information and shipping address.

[0277] Step 10:

[0278] The server receives purchase and contract information, stores it in a database, and synchronizes contract information in real time.

[0279] Step 11:

[0280] The server provides channels such as chat, messaging, voice calls, and video calls for post-contract support services. For example, if a user has a question about how to use a product, they can receive support through the chat function.

[0281] Step 12:

[0282] The server provides product usage guides and support materials, such as text guides and video guides, and sends them to the device, which then displays the guides to the user, who can then refer to them.

[0283] Example 2

[0284] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0285] Conventional recommendation systems may not be able to provide appropriate products or services because they do not accurately meet user needs or do not take into account the user's emotional state, which calls for an improved user experience. Furthermore, insufficient support services may be provided after the contract is signed, potentially reducing user satisfaction. Therefore, a system is needed that can accurately grasp the user's needs, make optimal recommendations based on the user's emotional state, and provide comprehensive support services after the contract is signed.

[0286] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0287] In this invention, the server includes a means for accepting input data from a user, a means for analyzing the accepted input data, and a means for proposing related products or services based on the analysis results and the user's emotions. This allows the server to accurately grasp the user's needs and make optimal proposals that take into account the user's emotional state. Furthermore, the server can improve user satisfaction by displaying the proposed products or services on the user's terminal and providing post-contract support services.

[0288] "User" refers to a person who uses the system to search for, propose, and contract for goods and services.

[0289] "Input data" refers to information provided by a user to the system, and includes various forms such as photographs, text, and video.

[0290] A "terminal" is a hardware device that a user uses to access the system, and includes a mobile terminal, a computer, and the like.

[0291] "Server" refers to the computer system that acts as the central part of the system, analyzing input data, generating proposals, communicating with the database, and transmitting proposals.

[0292] "Image recognition technology" refers to the technology of extracting specific features (color, shape, pattern, etc.) from photographs and videos.

[0293] "Natural language processing technology" refers to technology that analyzes user intent and meaning from text data.

[0294] "Emotions" are internal states that users express through input data, and include joy, sadness, surprise, etc.

[0295] "Suggestions" refers to information about products and services selected based on the analysis results and user sentiment.

[0296] "Support services" refers to the support a user receives after signing a contract for a proposed product or service, and includes support via chat, voice calls, and video calls.

[0297] The system of the present invention analyzes input data from a user, proposes related products or services, recognizes the user's emotions to optimize the proposals, and also provides post-contract support services. A specific embodiment of this system will be described.

[0298] Accepting user input data

[0299] Users access the system using devices such as smartphones, tablets, and PCs. They input photos, text, and videos of their interests. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[0300] Transfer of input data

[0301] The device receives the data entered by the user, converts it into an appropriate format, and sends it to the server. For example, it sends JPEG photo data as is. In the case of text, it sends the entered keywords as is.

[0302] Data analysis

[0303] The server runs an algorithm to analyze the received input data, specifically using the following techniques:

[0304] Image recognition technology: Algorithms such as YOLO (You Only Look Once) are used to extract key features (color, shape, pattern, etc.) from photos and videos. For example, identifying the features of a "blue dress."

[0305] Natural Language Processing (NLP) techniques: Use NLP models such as BERT (Bidirectional Encoder Representations from Transformers) to analyze user intent from text data. For example, analyze the keyword "blue dress" to identify the color and type of item.

[0306] Emotion Engine: Uses Python's Emotion API to recognize user emotions from text, audio, and video, thereby identifying the user's emotional state (e.g., happy, sad, surprised).

[0307] Proposal Generation

[0308] The server searches the database for relevant products and services and generates optimal suggestions based on the analysis results and the recognized emotions. The server works as follows:

[0309] Product information collection: Pulling relevant product or service information (such as name, price, reviews, and photos) from a database (e.g., a MySQL database).

[0310] Formation of proposal content: Based on the collected information, proposal data is generated in formats such as CSV, JSON, HTML, etc. For example, product information for multiple "blue dresses" is provided along with photos.

[0311] View Suggestions

[0312] The server sends the generated proposal data to the user's device. The device then displays the proposal data in a visually easy-to-understand format. For example, on a smartphone, the proposals are displayed on the app screen or via push notification, while on a PC, they are displayed in a new tab on the browser. React Native is used to generate the smartphone app UI and display the proposals.

[0313] Post-contract support services

[0314] If a user is interested in the proposed products or services and wishes to sign a contract, they can complete the contract procedure through their device. After the purchase procedure is completed, the server synchronizes the contract information in real time and prepares to provide subsequent support services. For example, the Twilio API can be used to implement support functions such as chat, voice calls, and video calls.

[0315] Specific examples

[0316] For example, if a user is looking for a particular piece of furniture, the following process applies:

[0317] The user takes a photo of the furniture they want to match with their home interior using their smartphone and uploads it to the system. The device sends the photo data to the server. The server runs an image recognition algorithm (e.g., YOLO) to analyze the photo and extract features such as the type, color, and shape of the furniture. The server uses an emotion engine (e.g., Emotion API) to recognize the user's emotions. If the user is recognized as being in a "lively" emotional state, casual, brightly colored furniture will be prioritized and suggested. The server searches for related furniture from its database and generates multiple options. The server sends the generated suggestion data to the device, which displays it to the user. The user checks the search results on their smartphone, selects the furniture they like, and completes the purchase process. The server synchronizes contract information and provides support services as needed (e.g., sending video guides on how to assemble the furniture).

[0318] Prompt Sentence Examples

[0319] "Explain how a system can provide product recommendations based on a user-uploaded photo of a blue dress."

[0320] "Please outline a system that analyzes user sentiment and optimizes product recommendations based on that sentiment."

[0321] In this way, users can easily obtain products and services that suit them and receive comprehensive support even after signing a contract. In addition, the introduction of an emotion engine will make the user experience more personalized.

[0322] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0323] Step 1:

[0324] Users access the system using devices such as smartphones, tablets, and PCs. They input photos, text, and videos of their interest. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system. Inputs include photo data (e.g., in JPEG format) and text data (e.g., "blue dress").

[0325] Step 2:

[0326] The terminal receives data entered by the user, converts the format appropriately, and sends it to the server. Specifically, it sends JPEG photo data and text data directly to the server. The input is the photo data and text data received from the user, and the output is the data sent to the server.

[0327] Step 3:

[0328] The server runs an algorithm to analyze the received input data, using the following techniques:

[0329] Image recognition technology: For example, algorithms such as YOLO (You Only Look Once) are used to extract the main features of a photo (color, shape, pattern, etc.). The input is JPEG photo data, and the output is the extracted feature information (e.g., "blue color," "dress shape").

[0330] Natural Language Processing (NLP) technology: For example, an NLP model such as BERT (Bidirectional Encoder Representations from Transformers) is used to analyze user intent from text data. The input is the text data, and the output is the analysis result (e.g., "blue", "dress").

[0331] Emotion engine: For example, Python's Emotion API is used to recognize the user's emotional state from text, audio, and video. The input is text data, and the output is emotional state information (e.g., "joy" or "excitement").

[0332] Step 4:

[0333] The server then suggests relevant products and services based on the analysis results and the recognized emotions. Specifically, it:

[0334] Product information collection: Extract relevant product and service information (such as name, price, reviews, and photos) from a database (e.g., MySQL database). The input is the analysis results and sentiment information, and the output is product information.

[0335] Proposal content generation: Based on the collected product information, proposal data is generated in formats such as CSV, JSON, and HTML. For example, product information for multiple "blue dresses" is provided along with photos. The input is product information, and the output is proposal data.

[0336] Step 5:

[0337] The server sends the generated proposal data to the terminal. The input is the generated proposal data, and the output is the data sent to the terminal.

[0338] Step 6:

[0339] The device receives the proposal data and displays it visually to the user. Specifically, on a smartphone, the proposal is displayed on the app screen or via push notification, and on a PC, the proposal is displayed by opening a new tab in the browser. The input is the proposal data sent from the server, and the output is the information displayed to the user.

[0340] Step 7:

[0341] If a user is interested in a proposed product or service, they can complete the contract process through their device. For example, they can complete the purchase process by simply clicking the "Purchase" button on their smartphone and entering the necessary payment information. The input is the proposal data and contract information, and the output is the completed contract.

[0342] Step 8:

[0343] The server synchronizes the contract information in real time and prepares to provide subsequent support services. Specifically, it uses the Twilio API to implement chat, voice call, and video call support functions. The input is the contract information, and the output is the support service provided.

[0344] (Application example 2)

[0345] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0346] In modern e-commerce, users have difficulty finding the best products and services for them from the vast number of options available. Furthermore, product suggestions that do not match the user's emotional state make it difficult to stimulate their desire to purchase. Furthermore, post-purchase support is often inadequate, resulting in a poor user experience. A system that can solve these problems and improve user satisfaction is needed.

[0347] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0348] In this invention, the server includes means for accepting input data from a user, means for analyzing the accepted input data, means for proposing related products or services based on the analysis results and the user's emotional state, means for displaying the proposed products or services on the user's terminal, and means for providing post-contract support services. This makes it possible to analyze a variety of input data from the user and propose optimal products and services that match the user's emotional state. Furthermore, by providing enhanced post-contract support, user satisfaction can be improved.

[0349] The "means for accepting input data from a user" is an interface that allows a user to provide a photograph, image, text, audio, or video to the system.

[0350] "Means for analyzing received input data" refers to algorithms and techniques for analyzing user-provided input data and extracting data features and emotional states.

[0351] "Means for suggesting relevant products or services based on the analysis results and the user's emotional state" refers to a mechanism by which the system selects and suggests the most appropriate products or services, taking into account the analyzed data and the user's emotional state.

[0352] "Means for displaying suggested products or services on a user terminal" refers to a method by which the system displays the suggested content on the user's smartphone, tablet, personal computer, or smart glasses.

[0353] The "means for providing post-contract support services" is a function for providing support and after-sales services after the user purchases or contracts for the proposed product or service.

[0354] "Photos, images, text, audio, or video" are any form of visual or audio data that a user provides to the system.

[0355] "Means for extracting features and emotional states" refers to techniques for identifying key features (e.g., color, shape) and emotional states (e.g., happy, sad) from user-provided data.

[0356] "User terminal is a smartphone, tablet, personal computer or smart glasses" refers to the electronic device at the user's side that is used to display the system's proposals.

[0357] The system for implementing this invention accepts and analyzes various input data from users and proposes optimal products and services based on the results. It also analyzes the user's emotional state and personalizes the proposals. It also provides post-contract support services to improve user satisfaction.

[0358] Hardware and Software

[0359] Hardware

[0360] Smartphones, tablets, personal computers, smart glasses: electronic devices used by users to provide input data and receive suggestions.

[0361] software

[0362] Image analysis: Uses the Google Cloud Vision API to analyze user-uploaded photos and images and extract features.

[0363] Natural Language Processing (NLP): Uses the Google Cloud Natural Language API to analyze user-entered text and recognize intent and sentiment.

[0364] Emotion Analysis: Uses the Microsoft Azure Emotion API to identify the user's emotional state from audio and video data.

[0365] Database: Use Firebase Realtime Database to retrieve relevant product and service data based on analytics results.

[0366] Front-end: Build the user interface using React Native.

[0367] Example

[0368] An example is given below where the user is looking for a new dress.

[0369] 1. User uploads a photo

[0370] Users use their smartphones to take photos of the dresses and upload them to the application.

[0371] 2. Data Analysis

[0372] The server analyzes uploaded photos using the Google Cloud Vision API to extract features such as the color, shape, and design of the dress, and also analyzes the user's input text and voice using the Google Cloud Natural Language API and Microsoft Azure Emotion API to recognize the user's emotional state.

[0373] 3. Related product suggestions

[0374] The server retrieves relevant product data from the Firebase Realtime Database and selects the most suitable products based on the emotional state. For example, if the emotional state is "joy," it will prioritize dresses with high reviews.

[0375] 4. Display of proposed data

[0376] The suggested products are visually displayed on smartphones and tablets using React Native, and users can choose from the suggested products.

[0377] 5. Post-contract support services

[0378] After the user purchases the selected product, the server stores the purchase information in the Firebase Realtime Database and provides subsequent support services, such as real-time chat support and instructional videos, for any questions about the product.

[0379] Prompt Sentence Examples

[0380] Below is an example of a prompt sentence that can be input into the "generative AI model."

[0381] Analyze photos, extract their features, and generate relevant product suggestions based on the user's emotional state.

[0382] Input: Photo of a blue dress

[0383] Emotion: Joy

[0384] Output: Related blue dress product list and reviews

[0385] In this way, the system of the present invention can propose products and services that accurately reflect the user's needs and emotional state, improving the user experience. Furthermore, the system also provides comprehensive support services after the contract is signed, thereby increasing user satisfaction.

[0386] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0387] Step 1:

[0388] User uploads a photo

[0389] A user takes a photo of a product they want to purchase using their smartphone and uploads it to the application. The input is a JPEG photo, and the output is a notification that the photo has been uploaded. The device receives the photo data from the user and transfers it to the server.

[0390] Step 2:

[0391] Sending data to the server

[0392] The terminal sends JPEG format photo data to the server. The input is a JPEG format photo, and the output is the photo data sent to the server. The terminal performs this process automatically, and the user is not required to perform any special operations.

[0393] Step 3:

[0394] Analyzing photo data

[0395] The server analyzes the received photo data using the Google Cloud Vision API. The input is JPEG photo data, and the output is feature information such as the color, shape, and design of the product. The server runs an image recognition algorithm to extract important features.

[0396] Step 4:

[0397] Analyzing the user's emotional state

[0398] The server uses the Google Cloud Natural Language API and Microsoft Azure Emotion API to analyze the user's input text and voice data. The input is text and voice data, and the output is the user's emotional state information (e.g., joy, surprise, etc.). The server runs an emotion analysis algorithm to identify the user's emotional state.

[0399] Step 5:

[0400] Search and suggest related products

[0401] The server uses Firebase Realtime Database to search for relevant product data based on the analyzed features and the user's emotional state. The input is feature information and emotional state information, and the output is a list of relevant products. The server generates recommendations by prioritizing products according to the emotional state.

[0402] Step 6:

[0403] Send the proposal data to the user device

[0404] The server sends the generated proposal data to the user's terminal. The input is a list of related products, and the output is a notification that the proposal data has been sent to the user's terminal. The server sends the data to the terminal in an appropriate format.

[0405] Step 7:

[0406] View suggested data

[0407] The terminal visually displays the received suggestion data to the user. The input is the suggestion data, and the output is the display screen viewed by the user. The terminal uses React Native to provide a visually easy-to-understand interface.

[0408] Step 8:

[0409] Providing post-contract support

[0410] After the user purchases the recommended product, the server synchronizes the contract information and provides support services. The input is the purchase information, and the output is the support start notification and support content. The server supports the user through support functions such as real-time chat and instructional videos.

[0411] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0412] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0413] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0414] [Second embodiment]

[0415] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0416] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0417] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0418] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0419] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0420] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0421] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0422] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0423] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0424] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0425] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0426] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0427] The system of the present invention proposes optimal products and services based on data entered by the user, and also provides support after the contract is signed. A specific embodiment of this system will be explained in natural language, with the program processing also being presented, along with specific examples.

[0428] Accepting user input data

[0429] Users access the system using devices such as smartphones, tablets, and PCs. They can input photos, images, text, and videos of products and services they are interested in. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[0430] The terminal receives input data from the user, converts it into an appropriate format, and sends it to the server. For example, photo data in JPEG format can be sent as is.

[0431] Data analysis

[0432] The server runs algorithms to analyze the input data it receives. For photos and videos, image recognition technology is used to extract key features, such as color, shape, and pattern. For text, natural language processing (NLP) technology is used to analyze the user's intent. For example, a keyword like "blue dress" can be used to identify the color and type of item.

[0433] Proposal Generation

[0434] Once the analysis is complete, the server searches the database for relevant products and services and selects those that best meet the user's needs. The results are presented in multiple formats, including photos, text, and video. For example, if the search keyword is "blue dress," product information for multiple blue dresses will be provided along with photos.

[0435] View Suggestions

[0436] The server sends the generated proposal data to the user's device. The device receives the data and displays it visually in an easy-to-understand format. On smartphones, the proposals are displayed on the app screen or via push notifications, while on PCs, they are displayed in a new tab on the browser.

[0437] Post-contract support services

[0438] If a user is interested in a proposed product or service and wishes to sign a contract, they can easily complete the contract procedure through their device. For example, when a user purchases a dress on their smartphone, they simply click the purchase button and enter the necessary payment information to complete the purchase procedure.

[0439] After a purchase or contract is completed, the server synchronizes the contract information in real time and prepares to provide subsequent support services. The server provides support through chat and messaging functions, and also provides real-time support via voice and video calls. For example, if a user has a question about how to use or care for a purchased dress, they can ask through chat and receive an immediate answer.

[0440] Specific examples

[0441] For example, if a user is looking for a particular piece of furniture, the following process applies:

[0442] 1. The user takes a photo of the furniture they want to match with their home's interior using their smartphone and uploads it to the system.

[0443] 2. The device sends the photo data to the server.

[0444] 3. The server runs an image recognition algorithm to analyze the photo and extract features such as furniture type, color, and shape.

[0445] 4. The server searches a database for relevant furniture and generates multiple options, such as a list of sofas and tables with photos.

[0446] 5. The server sends the generated suggestion data to the device, which displays it to the user, who then checks the search results on their smartphone.

[0447] 6. The user selects their favorite furniture from the suggested items and completes the purchase process.

[0448] 7. The server synchronizes the contract information and provides support services as needed, such as video guides on how to assemble furniture.

[0449] In this way, users can easily obtain products and services that suit them and receive comprehensive support even after signing a contract.

[0450] The processing flow will be explained below.

[0451] Step 1:

[0452] Users access the system using devices such as smartphones, tablets, and PCs. They input photos, images, text, and videos of their interests. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[0453] Step 2:

[0454] The terminal receives input data from the user, converts it into an appropriate format, and sends it to the server. For example, photo data in JPEG format can be sent as is.

[0455] Step 3:

[0456] The server runs algorithms to analyze the input data it receives. For photos and videos, image recognition technology is used to extract key features, such as color, shape, and pattern. For text, natural language processing (NLP) technology is used to analyze the user's intent. For example, the color and type of item can be identified from keywords such as "blue dress."

[0457] Step 4:

[0458] Based on the analysis results, the server searches the database for related products and services and selects those that best meet the user's needs. The selected results are generated in multiple formats, including photos, text, and video. For example, if the search keyword is "blue dress," product information for multiple blue dresses will be provided along with photos.

[0459] Step 5:

[0460] The server sends the generated proposal data to the user's device in a format appropriate for the device.

[0461] Step 6:

[0462] The device receives the proposal data and displays it to the user in an easy-to-understand visual format. On smartphones, the proposal content is displayed on the app screen or via push notification, while on PCs, the proposal content is displayed in a new tab on the browser.

[0463] Step 7:

[0464] The user becomes interested in the proposed products or services and makes a selection. The user then checks for more information about the selected product or service, such as detailed product specifications, price, and user reviews.

[0465] Step 8:

[0466] The user finally completes the purchase or contract process by clicking the purchase button and entering the required payment information, such as credit card information and shipping address.

[0467] Step 9:

[0468] The server receives purchase and contract information, stores it in a database, and synchronizes contract information in real time.

[0469] Step 10:

[0470] The server provides channels such as chat, messaging, voice calls, and video calls for post-contract support services. For example, if a user has a question about how to use a product, they can receive support through the chat function.

[0471] Step 11:

[0472] The server provides product usage guides and support materials, such as text guides and video guides, and sends them to the device, which then displays the guides to the user, who can then refer to them.

[0473] This allows users to use the system intuitively throughout the entire process, efficiently obtain products and services that suit them, and even receive prompt support after signing a contract.

[0474] Example 1

[0475] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0476] In conventional systems, it takes a great deal of time and effort for users to search for products and services and find the most suitable ones. Furthermore, there is a problem that support after signing a contract is not provided sufficiently, resulting in a decrease in user satisfaction. The present invention aims to solve these problems by providing a system that allows users to efficiently find products and services that suit them and receive comprehensive support even after signing a contract.

[0477] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0478] In this invention, the server includes means for converting received input data into an appropriate format, means for transmitting the converted data to the server, means for receiving data at the server and checking the format and content of the data, means for analyzing the received data, means for searching for related products or services based on the analysis results, means for generating an optimal proposal based on the search results, means for transmitting the generated proposal to a user terminal, means for displaying the proposed product or service on the user terminal, means for inputting information required for contract procedures, means for transmitting contract information to the server and completing the contract procedures, and means for providing post-contract support services. This enables users to efficiently find products or services, sign contracts, and receive consistent support even after the contract is signed.

[0479] "User" refers to a general user who uses the system to search for and purchase goods and services.

[0480] "Terminal" refers to an electronic device, such as a smartphone, tablet, or PC, that a user uses to access the system, enter information, or receive suggestions.

[0481] "Server" refers to the central processing unit that receives and analyzes data sent by users and generates optimal product and service recommendations.

[0482] "Input data" refers to data such as photographs, images, text, and videos that users input to provide information about products and services to the system.

[0483] "Means for converting into a format" refers to the process by which the terminal receives input data from the user and converts it into an appropriate digital format that can be sent to the server.

[0484] "Means for transmitting data to a server" refers to a function for transmitting converted data to a server via the Internet or other communication network.

[0485] "Means for analyzing data" refers to the function of the server using algorithms to process the data it receives and understand and classify its content.

[0486] "Means for searching for products or services" refers to the function of searching for products or services that meet the user's needs from a database based on the analysis results.

[0487] The "means of generating suggestions" refers to the process of formulating and creating information on optimal products and services to be offered to users based on search results.

[0488] "Means for transmitting suggestions to a user terminal" refers to a communication function for transmitting the generated suggestions to a user terminal and visually displaying them.

[0489] "Means for entering information necessary for contract procedures" refers to the interface through which users enter personal information and payment information required when purchasing or signing a contract for a product or service.

[0490] "Means for completing the contract procedure" refers to the process of sending the entered information to the server and officially concluding a contract for the purchase of goods or services.

[0491] "Means for providing post-contract support services" refers to functions for providing additional information and answering questions to users after the contract is completed.

[0492] The system of the present invention proposes optimal products and services based on data entered by a user, and also provides support after the contract is signed. A specific embodiment of this system will be described in detail below.

[0493] Accepting user input data

[0494] Users access the system using devices such as smartphones, tablets, and PCs. They can input data about products and services that interest them. Data that can be input includes photos, images, text, and videos. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[0495] The terminal receives input data from the user, checks the data format, converts it into an appropriate digital format (e.g., JPEG format, UTF-8 format), and then sends the converted data to the server.

[0496] Data reception and preprocessing

[0497] The server receives the data sent from the device. It checks the format and content of the received data to see if there is any improper data or corrupted files. For images, the server performs preliminary processing such as resizing and noise reduction.

[0498] Data analysis

[0499] The server runs image recognition algorithms to extract key features of the input image (e.g., color code, shape, texture), and uses natural language processing (NLP) algorithms for text data to analyze the user's intent. For example, for "blue dress," the server identifies the color as "blue" and the item as "dress."

[0500] Proposal Generation

[0501] Once the analysis is complete, the server searches the database for relevant products and services. Based on the search results, it generates recommendations that best suit the user's needs. Specifically, it applies filters such as price range, brand, and size, and generates the most suitable product and service information in the form of photos, text, and video.

[0502] View Suggestions

[0503] The server sends the generated proposal data to the device. The device receives the data and displays it in a visually easy-to-understand format for the user. On smartphones, the proposal content is displayed on the app screen or via push notification, and on PCs, the proposal content is displayed in a new tab on the browser.

[0504] Contract Procedures

[0505] The user is interested in the proposed product or service and decides to purchase or enter into a contract. The purchase procedure is completed by entering the necessary information (e.g., name, address, payment information) and sending it to the server via the device. The server synchronizes the contract information in real time, and the contract is officially concluded.

[0506] Post-contract support

[0507] After signing the contract, the server provides the user with additional support services. Specifically, the server provides instant support for user questions through chat and messaging functions. It can also answer user questions in real time using voice and video calls. For example, it can provide usage guides and assembly video tutorials for purchased products.

[0508] Specific examples

[0509] For example, if a user is looking for a particular piece of furniture, the following process applies:

[0510] 1. The user takes a photo of a sofa that suits their living room with their smartphone and uploads it through the app.

[0511] 2. The device converts the photo data into JPEG format and sends it to the server.

[0512] 3. The server receives the photo data and resizes and denoises the image.

[0513] 4. The server uses image recognition algorithms to analyze the sofa's color, shape, and texture.

[0514] 5. The server searches the database for blue living room sofas, filtering by price range and brand.

[0515] 6. The server generates multiple blue sofa photos and details as suggestions.

[0516] 7. The server sends the generated proposal data to the terminal, which displays it to the user.

[0517] 8. The user selects their favorite sofa from the suggestions and proceeds with the purchase.

[0518] 9. The user enters their payment information and clicks the purchase button.

[0519] 10. The device sends the entered information to the server and confirms the purchase.

[0520] 11. The server synchronizes the purchase information and provides the user with assembly instructions via chat.

[0521] 12. If the user asks any follow-up questions, the server will answer them immediately.

[0522] Prompt Sentence Examples

[0523] "I'm looking for a new blue dress. What are my options?"

[0524] "I'm looking for a sofa that will complement my home decor. I've uploaded some photos. Can you show me the options?"

[0525] This allows users to efficiently find products and services that suit them, sign contracts, and receive comprehensive support afterward.

[0526] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0527] Step 1:

[0528] Users access the system using devices such as smartphones, tablets, and PCs. They can input photos, images, text, and videos of products and services that interest them. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system. The input data is accepted by the device.

[0529] Input: User input of photos, images, text, and videos

[0530] Output: Input data stored on the terminal

[0531] Specific behavior: The user opens the app or browser, enters the required information, and clicks the "Submit" button.

[0532] Step 2:

[0533] The terminal checks the data entered by the user and converts it into the appropriate format: if it's a photo, it's converted into JPEG format, if it's text, it's converted into UTF-8 format, and then sends the converted data to the server.

[0534] Input: User-entered photo, image, text, and video data

[0535] Output: Transformed data to send to the server

[0536] Specific operation: The terminal checks the data format, checks for inconsistencies, converts the data format, and sends it to the server via the network.

[0537] Step 3:

[0538] The server receives the data sent from the device. It checks the format and content of the received data to see if there is any improper data or corrupted files. For images, the server performs preliminary processing such as resizing and noise reduction.

[0539] Input: Transformed data sent from the terminal

[0540] Output: Preprocessed data

[0541] Specific operation: The server receives the data, and a resident script checks the data format and content, and performs image resizing and noise removal if necessary.

[0542] Step 4:

[0543] The server runs image recognition algorithms to extract key features of the input image (e.g., color code, shape, texture), and uses natural language processing (NLP) algorithms for text data to analyze the user's intent. For example, for "blue dress," the server identifies the color as "blue" and the item as "dress."

[0544] Input: Preprocessed data (images, text)

[0545] Output: Analysis results (feature extraction information, user intent)

[0546] How it works: An image recognition model on the server analyzes images and extracts color codes and shape information, while an NLP model analyzes text data to identify keywords and intent.

[0547] Step 5:

[0548] The server uses the analyzed data to search for relevant products and services from its database. Based on the search results, it generates recommendations that best suit the user's needs. Specifically, it applies filters such as price range, brand, and size, and generates the most suitable product and service information in the form of photos, text, and video.

[0549] Input: Analysis results

[0550] Output: Product and service proposal data

[0551] Specific operation: The server accesses an internal database, searches for products and services that match the criteria, and generates proposals based on the results.

[0552] Step 6:

[0553] The server sends the generated proposal data to the device. The device receives the data and displays it in a visually easy-to-understand format for the user. On smartphones, the proposal content is displayed on the app screen or via push notification, and on PCs, the proposal content is displayed in a new tab on the browser.

[0554] Input: Generated proposal data

[0555] Output: Proposal displayed on the user's device

[0556] Specific operation: The server sends the proposed data over the network, the device receives it and displays it on the screen, and a push notification is displayed on the smartphone.

[0557] Step 7:

[0558] The user is interested in the proposed product or service and decides to purchase or sign a contract. They enter the necessary information (e.g., name, address, payment information) and send it to the server via their device.

[0559] Input: Name, address, and payment information entered by the user

[0560] Output: Contract information sent to the server

[0561] Specific operation: When the user enters the required information into the form and clicks the "Purchase" or "Contract" button, the information is sent to the server.

[0562] Step 8:

[0563] The server synchronizes the contract information in real time, and the contract is officially established. After the contract is signed, additional support services are provided to the user. Specifically, immediate support for user questions is provided through chat and messaging functions. Users' questions can also be answered in real time using voice and video calls.

[0564] Input: Contract information

[0565] Output: Providing support services

[0566] What it does: The server updates contract information and, if necessary, connects with the support team to provide the necessary support to the user, responding in real time via chat, voice, or video calls.

[0567] (Application example 1)

[0568] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0569] Conventional systems have had difficulty efficiently and accurately recommending products and services that interest users and providing prompt post-purchase support. This problem stems from the difficulty of accurately understanding the data entered by users and making recommendations that meet their needs. Therefore, it is necessary to build a system that can accurately analyze the data entered by users, recommend optimal products and services, and provide prompt post-purchase support.

[0570] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0571] In this invention, the server includes means for accepting input data from a user, means for analyzing the accepted input data, means for proposing related products or services based on the analysis results, means for displaying the proposed products or services on the user terminal, means for providing post-contract support services, means for generating prompt sentences to optimize interpretation of the input data, means for providing real-time support via chat and video calls based on the analysis results, and means for analyzing the accepted input data using image recognition technology and natural language processing technology. This makes it possible to consistently provide product proposals and post-contract support that quickly and accurately meet the user's needs.

[0572] "Means for accepting input data from the user" refers to an interface that allows the user to provide the system with information about products and services in which the user is interested.

[0573] "Means for analyzing received input data" refers to the process of analyzing data provided by users using image recognition technology and natural language processing technology to understand their needs and intentions.

[0574] "Means for suggesting related products or services based on the analysis results" refers to a function for selecting optimal products or services from a database based on the analyzed data and recommending them to the user.

[0575] "Means for displaying proposed products or services on a user's terminal" refers to a function for visually displaying information about proposed products on a terminal such as a user's smartphone.

[0576] "Means for providing post-contract support services" refers to communication means such as chat and video calls for providing after-sales support for products and services purchased by users.

[0577] "Means for generating prompts to optimize interpretation of input data" refers to means for generating precise instructions or questions based on information obtained from the user and for effectively analyzing that data.

[0578] "Means for providing real-time chat and video call support based on the analysis results" refers to a function that uses the analyzed information to respond via chat or video call so that the user can receive immediate support.

[0579] "Means of analyzing input data received using image recognition technology and natural language processing technology" refers to technology that uses advanced algorithms to analyze images and text data provided by users and understand their content.

[0580] The system of the present invention proposes optimal products and services based on user input data and provides support after the contract is signed. To implement this system, the following programs and hardware are used.

[0581] 1. Accepting user input data

[0582] Users access the system using a smart device (smartphone, tablet, or PC) and enter photos, images, text, and videos of products and services they are interested in. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[0583] 2. Data Analysis

[0584] The server analyzes the received input data using image recognition technology (OpenCV and TensorFlow) and natural language processing technology (NLTK). For photos and videos, it extracts image features (color, shape, pattern, etc.), and for text, it analyzes keywords to identify the user's intent.

[0585] 3. Proposal Generation

[0586] Based on the analyzed data, the server searches the database for relevant products and services and generates proposals that best meet the user's needs. These proposals can be in the form of photos, text, videos, etc.

[0587] 4. Display of suggestions

[0588] The server sends the generated proposal data to the user's device, which then displays the received data in an easy-to-understand visual format. On smartphones, the proposals are displayed on the app screen or via push notification, while on PCs, the proposals are displayed in a new tab on the browser.

[0589] 5. Post-contract support services

[0590] If a user is interested in a proposed product or service and signs a contract, they can easily complete the contract process through their device. For example, when a user purchases a dress on their smartphone, they simply click the purchase button and enter their payment information to complete the process. The server synchronizes the contract information in real time and prepares to provide subsequent support services. The server provides support through chat, messaging, voice calls, and video calls.

[0591] Specific examples

[0592] If a user is looking for a particular piece of furniture, the following process applies:

[0593] 1. The user takes a photo of the furniture they want to match with their home's interior using their smartphone and uploads it to the system.

[0594] 2. The device sends the photo data to the server.

[0595] 3. The server runs an image recognition algorithm (OpenCV, TensorFlow) to analyze the photo and extract features such as furniture type, color, and shape.

[0596] 4. The server searches a database for relevant furniture and generates multiple options, such as a list of sofas and tables with photos.

[0597] 5. The server sends the generated suggestion data to the device, which displays it to the user, who then checks the search results on their smartphone.

[0598] 6. The user selects their favorite furniture from the suggested items and completes the purchase process.

[0599] 7. The server synchronizes the contract information and provides support services as needed, such as video guides on how to assemble furniture.

[0600] Prompt Sentence Examples

[0601] "Please analyze this photo and suggest suitable products. The photo shows a blue sofa. Also, please search your database for similar sofas and display them."

[0602] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0603] Step 1:

[0604] Users use their smart devices to input photos, images, text, and videos related to products and services they are interested in and upload them to the system. The input data is sent from the user's device to the server as an HTTP request. At this point, the input is the photo taken or text entered by the user, and the output is unanalyzed data sent to the server.

[0605] Step 2:

[0606] The server analyzes the received data using image recognition technology (OpenCV and TensorFlow). Specifically, it extracts features (color, shape, pattern) from image data and analyzes keywords from text data using natural language processing (NLTK). In this process, the input is unanalyzed data from the user, and the output is analyzed feature information and keywords.

[0607] Step 3:

[0608] The server generates prompts based on the analysis results to optimize the interpretation of the input data. It uses a generative AI model to create prompts to more clearly identify the user's needs and intentions. In this step, the input is the analysis results, and the output is the prompts.

[0609] Step 4:

[0610] The server uses the generated prompt to search a database for relevant products and services. It executes a database query to extract the product information that best suits the user's request. In this operation, the input is the prompt and its parsed results, and the output is a list of suggested products and services.

[0611] Step 5:

[0612] The server sends the information of the proposed products and services to the user's device. The information is encoded in JSON format and returned as an HTTP response. The input is a list of proposed products and services, and the output is the data sent to the user's device.

[0613] Step 6:

[0614] The terminal displays the received proposal data in an easy-to-understand visual format. Specifically, it displays a list of product information on the smart device screen and offers the user options. In this step, the input is the proposal data sent from the server, and the output is the visual information presented to the user.

[0615] Step 7:

[0616] If a user is interested in the proposed product or service and wants to purchase it, he or she inputs the necessary information (e.g., payment information) through the terminal and sends it to the server. In this process, the input is the product information and payment information selected by the user, and the output is the purchase completion status.

[0617] Step 8:

[0618] The server synchronizes contract information in real time and prepares support services after purchase. The contract information is saved in the database and support services are initiated as needed. In this process, the input is the user's purchase information and the output is the support service preparation status.

[0619] Step 9:

[0620] The server provides real-time chat and video support. When a user has a question or problem, support staff are available to respond immediately. The input is the user's support request, and the output is real-time support information.

[0621] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0622] The system of the present invention analyzes input data from users, proposes relevant products or services, recognizes user emotions to optimize proposals, and provides post-contract support. A specific embodiment of this system will be explained below, with program processing explained in natural language and examples provided.

[0623] Accepting user input data

[0624] Users access the system using devices such as smartphones, tablets, and PCs. They input photos, images, text, and videos of their interests. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[0625] The terminal receives input data from the user, converts it into an appropriate format, and sends it to the server. For example, photo data in JPEG format can be sent as is.

[0626] Data analysis

[0627] The server runs algorithms to analyze the input data it receives. For photos and videos, image recognition technology is used to extract key features, such as color, shape, and pattern. For text, natural language processing (NLP) technology is used to analyze the user's intent. For example, a keyword like "blue dress" can be used to identify the color and type of item.

[0628] Furthermore, the server uses an emotion engine to recognize emotions from the user's input data, such as emotion analysis from text data, emotion analysis from audio data, and facial expression analysis from video data, to identify the user's current emotional state (e.g., joy, sadness, surprise).

[0629] Proposal Generation

[0630] Once the analysis is complete, the server searches the database for relevant products and services and selects those that best meet the user's needs. It then adjusts the recommendations based on the user's emotions, as determined by the emotion engine. For example, if the user is in a state of "joy" or "excitement," it will prioritize items with many positive product reviews.

[0631] The selected results are generated in multiple formats, including photos, text, video, etc. For example, if the search keyword is "blue dress," product information for multiple blue dresses will be provided along with photos.

[0632] View Suggestions

[0633] The server transmits the generated proposal data to the user's device in a format appropriate for the device.

[0634] The device then displays the received proposal data in a visually easy-to-understand format to the user. On smartphones, the proposal content is displayed on the app screen or via push notification, while on PCs, the proposal content is displayed in a new tab on the browser.

[0635] Post-contract support services

[0636] If a user is interested in a proposed product or service and wishes to sign a contract, they can easily complete the contract procedure through their device. For example, when a user purchases a dress on their smartphone, they simply click the purchase button and enter the necessary payment information to complete the purchase procedure.

[0637] After a purchase or contract is completed, the server synchronizes the contract information in real time and prepares to provide subsequent support services. The server provides support through chat and messaging functions, and also provides real-time support via voice and video calls. For example, if a user has a question about how to use or care for a purchased dress, they can ask through chat and receive an immediate answer.

[0638] Specific examples

[0639] For example, if a user is looking for a particular piece of furniture, the following process applies:

[0640] 1. The user takes a photo of the furniture they want to match with their home's interior using their smartphone and uploads it to the system.

[0641] 2. The device sends the photo data to the server.

[0642] 3. The server runs an image recognition algorithm to analyze the photo and extract features such as furniture type, color, and shape.

[0643] 4. The server uses an emotion engine to recognize the user's emotions from input data such as photos and text. If the user is recognized as being in a "lively" emotional state, casual, brightly colored furniture will be suggested.

[0644] 5. The server searches a database for relevant furniture and generates multiple options, such as a list of sofas and tables with photos.

[0645] 6. The server sends the generated suggestion data to the device, which displays it to the user, who then checks the search results on their smartphone.

[0646] 7. The user selects their favorite furniture from the suggested items and completes the purchase process.

[0647] 8. The server synchronizes the contract information and provides support services as needed, such as video guides on how to assemble furniture.

[0648] In this way, users can easily obtain products and services that suit them and receive comprehensive support even after signing a contract. In addition, the introduction of an emotion engine will make the user experience more personalized.

[0649] The processing flow will be explained below.

[0650] Step 1:

[0651] Users access the system using devices such as smartphones, tablets, and PCs. They input photos, images, text, and videos related to products and services they are interested in. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[0652] Step 2:

[0653] The terminal receives input data from the user, converts it into an appropriate format, and sends it to the server. For example, it sends photo data in JPEG format as is.

[0654] Step 3:

[0655] The server runs algorithms to analyze the input data it receives. For photos and videos, image recognition technology is used to extract key features, such as color, shape, and pattern. For text, natural language processing (NLP) technology is used to analyze the user's intent. For example, the color and type of item can be identified from keywords such as "blue dress."

[0656] Step 4:

[0657] Furthermore, the server uses an emotion engine to recognize emotions from the user's input data, such as emotion analysis from text data, emotion analysis from audio data, and facial expression analysis from video data, to identify the user's current emotional state (e.g., joy, sadness, surprise).

[0658] Step 5:

[0659] Based on the analysis results, the server searches its database for related products and services and selects those that best meet the user's needs. Furthermore, the server adjusts the recommendations based on the user's emotions, as recognized by the emotion engine. For example, if the user is in a state of "joy" or "excitement," it prioritizes items with many positive product reviews.

[0660] Step 6:

[0661] The server sends the generated proposal data to the user's device in a format appropriate for the device.

[0662] Step 7:

[0663] The device receives the proposal data and displays it to the user in an easy-to-understand visual format. On smartphones, the proposal content is displayed on the app screen or via push notification, while on PCs, the proposal content is displayed in a new tab on the browser.

[0664] Step 8:

[0665] The user becomes interested in the proposed products or services and makes a selection. The user then checks for more information about the selected product or service, such as detailed product specifications, price, and user reviews.

[0666] Step 9:

[0667] The user finally completes the purchase or contract process by clicking the purchase button and entering the required payment information, such as credit card information and shipping address.

[0668] Step 10:

[0669] The server receives purchase and contract information, stores it in a database, and synchronizes contract information in real time.

[0670] Step 11:

[0671] The server provides channels such as chat, messaging, voice calls, and video calls for post-contract support services. For example, if a user has a question about how to use a product, they can receive support through the chat function.

[0672] Step 12:

[0673] The server provides product usage guides and support materials, such as text guides and video guides, and sends them to the device, which then displays the guides to the user, who can then refer to them.

[0674] Example 2

[0675] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0676] Conventional recommendation systems may not be able to provide appropriate products or services because they do not accurately meet user needs or do not take into account the user's emotional state, which calls for an improved user experience. Furthermore, insufficient support services may be provided after the contract is signed, potentially reducing user satisfaction. Therefore, a system is needed that can accurately grasp the user's needs, make optimal recommendations based on the user's emotional state, and provide comprehensive support services after the contract is signed.

[0677] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0678] In this invention, the server includes a means for accepting input data from a user, a means for analyzing the accepted input data, and a means for proposing related products or services based on the analysis results and the user's emotions. This allows the server to accurately grasp the user's needs and make optimal proposals that take into account the user's emotional state. Furthermore, the server can improve user satisfaction by displaying the proposed products or services on the user's terminal and providing post-contract support services.

[0679] "User" refers to a person who uses the system to search for, propose, and contract for goods and services.

[0680] "Input data" refers to information provided by a user to the system, and includes various forms such as photographs, text, and video.

[0681] A "terminal" is a hardware device that a user uses to access the system, and includes a mobile terminal, a computer, and the like.

[0682] "Server" refers to the computer system that acts as the central part of the system, analyzing input data, generating proposals, communicating with the database, and transmitting proposals.

[0683] "Image recognition technology" refers to the technology of extracting specific features (color, shape, pattern, etc.) from photographs and videos.

[0684] "Natural language processing technology" refers to technology that analyzes user intent and meaning from text data.

[0685] "Emotions" are internal states that users express through input data, and include joy, sadness, surprise, etc.

[0686] "Suggestions" refers to information about products and services selected based on the analysis results and user sentiment.

[0687] "Support services" refers to the support a user receives after signing a contract for a proposed product or service, and includes support via chat, voice calls, and video calls.

[0688] The system of the present invention analyzes input data from a user, proposes related products or services, recognizes the user's emotions to optimize the proposals, and also provides post-contract support services. A specific embodiment of this system will be described.

[0689] Accepting user input data

[0690] Users access the system using devices such as smartphones, tablets, and PCs. They input photos, text, and videos of their interests. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[0691] Transfer of input data

[0692] The device receives the data entered by the user, converts it into an appropriate format, and sends it to the server. For example, it sends JPEG photo data as is. In the case of text, it sends the entered keywords as is.

[0693] Data analysis

[0694] The server runs an algorithm to analyze the received input data, specifically using the following techniques:

[0695] Image recognition technology: Algorithms such as YOLO (You Only Look Once) are used to extract key features (color, shape, pattern, etc.) from photos and videos. For example, identifying the features of a "blue dress."

[0696] Natural Language Processing (NLP) techniques: Use NLP models such as BERT (Bidirectional Encoder Representations from Transformers) to analyze user intent from text data. For example, analyze the keyword "blue dress" to identify the color and type of item.

[0697] Emotion Engine: Uses Python's Emotion API to recognize user emotions from text, audio, and video, thereby identifying the user's emotional state (e.g., happy, sad, surprised).

[0698] Proposal Generation

[0699] The server searches the database for relevant products and services and generates optimal suggestions based on the analysis results and the recognized emotions. The server works as follows:

[0700] Product information collection: Pulling relevant product or service information (such as name, price, reviews, and photos) from a database (e.g., a MySQL database).

[0701] Formation of proposal content: Based on the collected information, proposal data is generated in formats such as CSV, JSON, HTML, etc. For example, product information for multiple "blue dresses" is provided along with photos.

[0702] View Suggestions

[0703] The server sends the generated proposal data to the user's device. The device then displays the proposal data in a visually easy-to-understand format. For example, on a smartphone, the proposals are displayed on the app screen or via push notification, while on a PC, they are displayed in a new tab on the browser. React Native is used to generate the smartphone app UI and display the proposals.

[0704] Post-contract support services

[0705] If a user is interested in the proposed products or services and wishes to sign a contract, they can complete the contract procedure through their device. After the purchase procedure is completed, the server synchronizes the contract information in real time and prepares to provide subsequent support services. For example, the Twilio API can be used to implement support functions such as chat, voice calls, and video calls.

[0706] Specific examples

[0707] For example, if a user is looking for a particular piece of furniture, the following process applies:

[0708] The user takes a photo of the furniture they want to match with their home interior using their smartphone and uploads it to the system. The device sends the photo data to the server. The server runs an image recognition algorithm (e.g., YOLO) to analyze the photo and extract features such as the type, color, and shape of the furniture. The server uses an emotion engine (e.g., Emotion API) to recognize the user's emotions. If the user is recognized as being in a "lively" emotional state, casual, brightly colored furniture will be prioritized and suggested. The server searches for related furniture from its database and generates multiple options. The server sends the generated suggestion data to the device, which displays it to the user. The user checks the search results on their smartphone, selects the furniture they like, and completes the purchase process. The server synchronizes contract information and provides support services as needed (e.g., sending video guides on how to assemble the furniture).

[0709] Prompt Sentence Examples

[0710] "Explain how a system can provide product recommendations based on a user-uploaded photo of a blue dress."

[0711] "Please outline a system that analyzes user sentiment and optimizes product recommendations based on that sentiment."

[0712] In this way, users can easily obtain products and services that suit them and receive comprehensive support even after signing a contract. In addition, the introduction of an emotion engine will make the user experience more personalized.

[0713] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0714] Step 1:

[0715] Users access the system using devices such as smartphones, tablets, and PCs. They input photos, text, and videos of their interest. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system. Inputs include photo data (e.g., in JPEG format) and text data (e.g., "blue dress").

[0716] Step 2:

[0717] The terminal receives data entered by the user, converts the format appropriately, and sends it to the server. Specifically, it sends JPEG photo data and text data directly to the server. The input is the photo data and text data received from the user, and the output is the data sent to the server.

[0718] Step 3:

[0719] The server runs an algorithm to analyze the received input data, using the following techniques:

[0720] Image recognition technology: For example, algorithms such as YOLO (You Only Look Once) are used to extract the main features of a photo (color, shape, pattern, etc.). The input is JPEG photo data, and the output is the extracted feature information (e.g., "blue color," "dress shape").

[0721] Natural Language Processing (NLP) technology: For example, an NLP model such as BERT (Bidirectional Encoder Representations from Transformers) is used to analyze user intent from text data. The input is the text data, and the output is the analysis result (e.g., "blue", "dress").

[0722] Emotion engine: For example, Python's Emotion API is used to recognize the user's emotional state from text, audio, and video. The input is text data, and the output is emotional state information (e.g., "joy" or "excitement").

[0723] Step 4:

[0724] The server then suggests relevant products and services based on the analysis results and the recognized emotions. Specifically, it:

[0725] Product information collection: Extract relevant product and service information (such as name, price, reviews, and photos) from a database (e.g., MySQL database). The input is the analysis results and sentiment information, and the output is product information.

[0726] Proposal content generation: Based on the collected product information, proposal data is generated in formats such as CSV, JSON, and HTML. For example, product information for multiple "blue dresses" is provided along with photos. The input is product information, and the output is proposal data.

[0727] Step 5:

[0728] The server sends the generated proposal data to the terminal. The input is the generated proposal data, and the output is the data sent to the terminal.

[0729] Step 6:

[0730] The device receives the proposal data and displays it visually to the user. Specifically, on a smartphone, the proposal is displayed on the app screen or via push notification, and on a PC, the proposal is displayed by opening a new tab in the browser. The input is the proposal data sent from the server, and the output is the information displayed to the user.

[0731] Step 7:

[0732] If a user is interested in a proposed product or service, they can complete the contract process through their device. For example, they can complete the purchase process by simply clicking the "Purchase" button on their smartphone and entering the necessary payment information. The input is the proposal data and contract information, and the output is the completed contract.

[0733] Step 8:

[0734] The server synchronizes the contract information in real time and prepares to provide subsequent support services. Specifically, it uses the Twilio API to implement chat, voice call, and video call support functions. The input is the contract information, and the output is the support service provided.

[0735] (Application example 2)

[0736] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0737] In modern e-commerce, users have difficulty finding the best products and services for them from the vast number of options available. Furthermore, product suggestions that do not match the user's emotional state make it difficult to stimulate their desire to purchase. Furthermore, post-purchase support is often inadequate, resulting in a poor user experience. A system that can solve these problems and improve user satisfaction is needed.

[0738] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0739] In this invention, the server includes means for accepting input data from a user, means for analyzing the accepted input data, means for proposing related products or services based on the analysis results and the user's emotional state, means for displaying the proposed products or services on the user's terminal, and means for providing post-contract support services. This makes it possible to analyze a variety of input data from the user and propose optimal products and services that match the user's emotional state. Furthermore, by providing enhanced post-contract support, user satisfaction can be improved.

[0740] The "means for accepting input data from a user" is an interface that allows a user to provide a photograph, image, text, audio, or video to the system.

[0741] "Means for analyzing received input data" refers to algorithms and techniques for analyzing user-provided input data and extracting data features and emotional states.

[0742] "Means for suggesting relevant products or services based on the analysis results and the user's emotional state" refers to a mechanism by which the system selects and suggests the most appropriate products or services, taking into account the analyzed data and the user's emotional state.

[0743] "Means for displaying suggested products or services on a user terminal" refers to a method by which the system displays the suggested content on the user's smartphone, tablet, personal computer, or smart glasses.

[0744] The "means for providing post-contract support services" is a function for providing support and after-sales services after the user purchases or contracts for the proposed product or service.

[0745] "Photos, images, text, audio, or video" are any form of visual or audio data that a user provides to the system.

[0746] "Means for extracting features and emotional states" refers to techniques for identifying key features (e.g., color, shape) and emotional states (e.g., happy, sad) from user-provided data.

[0747] "User terminal is a smartphone, tablet, personal computer or smart glasses" refers to the electronic device at the user's side that is used to display the system's proposals.

[0748] The system for implementing this invention accepts and analyzes various input data from users and proposes optimal products and services based on the results. It also analyzes the user's emotional state and personalizes the proposals. It also provides post-contract support services to improve user satisfaction.

[0749] Hardware and Software

[0750] Hardware

[0751] Smartphones, tablets, personal computers, smart glasses: electronic devices used by users to provide input data and receive suggestions.

[0752] software

[0753] Image analysis: Uses the Google Cloud Vision API to analyze user-uploaded photos and images and extract features.

[0754] Natural Language Processing (NLP): Uses the Google Cloud Natural Language API to analyze user-entered text and recognize intent and sentiment.

[0755] Emotion Analysis: Uses the Microsoft Azure Emotion API to identify the user's emotional state from audio and video data.

[0756] Database: Use Firebase Realtime Database to retrieve relevant product and service data based on analytics results.

[0757] Front-end: Build the user interface using React Native.

[0758] Example

[0759] An example is given below where the user is looking for a new dress.

[0760] 1. User uploads a photo

[0761] Users use their smartphones to take photos of the dresses and upload them to the application.

[0762] 2. Data Analysis

[0763] The server analyzes uploaded photos using the Google Cloud Vision API to extract features such as the color, shape, and design of the dress, and also analyzes the user's input text and voice using the Google Cloud Natural Language API and Microsoft Azure Emotion API to recognize the user's emotional state.

[0764] 3. Related product suggestions

[0765] The server retrieves relevant product data from the Firebase Realtime Database and selects the most suitable products based on the emotional state. For example, if the emotional state is "joy," it will prioritize dresses with high reviews.

[0766] 4. Display of proposed data

[0767] The suggested products are visually displayed on smartphones and tablets using React Native, and users can choose from the suggested products.

[0768] 5. Post-contract support services

[0769] After the user purchases the selected product, the server stores the purchase information in the Firebase Realtime Database and provides subsequent support services, such as real-time chat support and instructional videos, for any questions about the product.

[0770] Prompt Sentence Examples

[0771] Below is an example of a prompt sentence that can be input into the "generative AI model."

[0772] Analyze photos, extract their features, and generate relevant product suggestions based on the user's emotional state.

[0773] Input: Photo of a blue dress

[0774] Emotion: Joy

[0775] Output: Related blue dress product list and reviews

[0776] In this way, the system of the present invention can propose products and services that accurately reflect the user's needs and emotional state, improving the user experience. Furthermore, the system also provides comprehensive support services after the contract is signed, thereby increasing user satisfaction.

[0777] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0778] Step 1:

[0779] User uploads a photo

[0780] A user takes a photo of a product they want to purchase using their smartphone and uploads it to the application. The input is a JPEG photo, and the output is a notification that the photo has been uploaded. The device receives the photo data from the user and transfers it to the server.

[0781] Step 2:

[0782] Sending data to the server

[0783] The terminal sends JPEG format photo data to the server. The input is a JPEG format photo, and the output is the photo data sent to the server. The terminal performs this process automatically, and the user is not required to perform any special operations.

[0784] Step 3:

[0785] Analyzing photo data

[0786] The server analyzes the received photo data using the Google Cloud Vision API. The input is JPEG photo data, and the output is feature information such as the color, shape, and design of the product. The server runs an image recognition algorithm to extract important features.

[0787] Step 4:

[0788] Analyzing the user's emotional state

[0789] The server uses the Google Cloud Natural Language API and Microsoft Azure Emotion API to analyze the user's input text and voice data. The input is text and voice data, and the output is the user's emotional state information (e.g., joy, surprise, etc.). The server runs an emotion analysis algorithm to identify the user's emotional state.

[0790] Step 5:

[0791] Search and suggest related products

[0792] The server uses Firebase Realtime Database to search for relevant product data based on the analyzed features and the user's emotional state. The input is feature information and emotional state information, and the output is a list of relevant products. The server generates recommendations by prioritizing products according to the emotional state.

[0793] Step 6:

[0794] Send the proposal data to the user device

[0795] The server sends the generated proposal data to the user's terminal. The input is a list of related products, and the output is a notification that the proposal data has been sent to the user's terminal. The server sends the data to the terminal in an appropriate format.

[0796] Step 7:

[0797] View suggested data

[0798] The terminal visually displays the received suggestion data to the user. The input is the suggestion data, and the output is the display screen viewed by the user. The terminal uses React Native to provide a visually easy-to-understand interface.

[0799] Step 8:

[0800] Providing post-contract support

[0801] After the user purchases the recommended product, the server synchronizes the contract information and provides support services. The input is the purchase information, and the output is the support start notification and support content. The server supports the user through support functions such as real-time chat and instructional videos.

[0802] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0803] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0804] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0805] [Third embodiment]

[0806] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0807] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0808] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0809] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0810] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0811] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0812] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0813] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0814] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0815] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0816] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0817] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0818] The system of the present invention proposes optimal products and services based on data entered by the user, and also provides support after the contract is signed. A specific embodiment of this system will be explained in natural language, with the program processing also being presented, along with specific examples.

[0819] Accepting user input data

[0820] Users access the system using devices such as smartphones, tablets, and PCs. They can input photos, images, text, and videos of products and services they are interested in. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[0821] The terminal receives input data from the user, converts it into an appropriate format, and sends it to the server. For example, photo data in JPEG format can be sent as is.

[0822] Data analysis

[0823] The server runs algorithms to analyze the input data it receives. For photos and videos, image recognition technology is used to extract key features, such as color, shape, and pattern. For text, natural language processing (NLP) technology is used to analyze the user's intent. For example, a keyword like "blue dress" can be used to identify the color and type of item.

[0824] Proposal Generation

[0825] Once the analysis is complete, the server searches the database for relevant products and services and selects those that best meet the user's needs. The results are presented in multiple formats, including photos, text, and video. For example, if the search keyword is "blue dress," product information for multiple blue dresses will be provided along with photos.

[0826] View Suggestions

[0827] The server sends the generated proposal data to the user's device. The device receives the data and displays it visually in an easy-to-understand format. On smartphones, the proposals are displayed on the app screen or via push notifications, while on PCs, they are displayed in a new tab on the browser.

[0828] Post-contract support services

[0829] If a user is interested in a proposed product or service and wishes to sign a contract, they can easily complete the contract procedure through their device. For example, when a user purchases a dress on their smartphone, they simply click the purchase button and enter the necessary payment information to complete the purchase procedure.

[0830] After a purchase or contract is completed, the server synchronizes the contract information in real time and prepares to provide subsequent support services. The server provides support through chat and messaging functions, and also provides real-time support via voice and video calls. For example, if a user has a question about how to use or care for a purchased dress, they can ask through chat and receive an immediate answer.

[0831] Specific examples

[0832] For example, if a user is looking for a particular piece of furniture, the following process applies:

[0833] 1. The user takes a photo of the furniture they want to match with their home's interior using their smartphone and uploads it to the system.

[0834] 2. The device sends the photo data to the server.

[0835] 3. The server runs an image recognition algorithm to analyze the photo and extract features such as furniture type, color, and shape.

[0836] 4. The server searches a database for relevant furniture and generates multiple options, such as a list of sofas and tables with photos.

[0837] 5. The server sends the generated suggestion data to the device, which displays it to the user, who then checks the search results on their smartphone.

[0838] 6. The user selects their favorite furniture from the suggested items and completes the purchase process.

[0839] 7. The server synchronizes the contract information and provides support services as needed, such as video guides on how to assemble furniture.

[0840] In this way, users can easily obtain products and services that suit them and receive comprehensive support even after signing a contract.

[0841] The processing flow will be explained below.

[0842] Step 1:

[0843] Users access the system using devices such as smartphones, tablets, and PCs. They input photos, images, text, and videos of their interests. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[0844] Step 2:

[0845] The terminal receives input data from the user, converts it into an appropriate format, and sends it to the server. For example, photo data in JPEG format can be sent as is.

[0846] Step 3:

[0847] The server runs algorithms to analyze the input data it receives. For photos and videos, image recognition technology is used to extract key features, such as color, shape, and pattern. For text, natural language processing (NLP) technology is used to analyze the user's intent. For example, the color and type of item can be identified from keywords such as "blue dress."

[0848] Step 4:

[0849] Based on the analysis results, the server searches the database for related products and services and selects those that best meet the user's needs. The selected results are generated in multiple formats, including photos, text, and video. For example, if the search keyword is "blue dress," product information for multiple blue dresses will be provided along with photos.

[0850] Step 5:

[0851] The server sends the generated proposal data to the user's device in a format appropriate for the device.

[0852] Step 6:

[0853] The device receives the proposal data and displays it to the user in an easy-to-understand visual format. On smartphones, the proposal content is displayed on the app screen or via push notification, while on PCs, the proposal content is displayed in a new tab on the browser.

[0854] Step 7:

[0855] The user becomes interested in the proposed products or services and makes a selection. The user then checks for more information about the selected product or service, such as detailed product specifications, price, and user reviews.

[0856] Step 8:

[0857] The user finally completes the purchase or contract process by clicking the purchase button and entering the required payment information, such as credit card information and shipping address.

[0858] Step 9:

[0859] The server receives purchase and contract information, stores it in a database, and synchronizes contract information in real time.

[0860] Step 10:

[0861] The server provides channels such as chat, messaging, voice calls, and video calls for post-contract support services. For example, if a user has a question about how to use a product, they can receive support through the chat function.

[0862] Step 11:

[0863] The server provides product usage guides and support materials, such as text guides and video guides, and sends them to the device, which then displays the guides to the user, who can then refer to them.

[0864] This allows users to use the system intuitively throughout the entire process, efficiently obtain products and services that suit them, and even receive prompt support after signing a contract.

[0865] Example 1

[0866] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0867] In conventional systems, it takes a great deal of time and effort for users to search for products and services and find the most suitable ones. Furthermore, there is a problem that support after signing a contract is not provided sufficiently, resulting in a decrease in user satisfaction. The present invention aims to solve these problems by providing a system that allows users to efficiently find products and services that suit them and receive comprehensive support even after signing a contract.

[0868] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0869] In this invention, the server includes means for converting received input data into an appropriate format, means for transmitting the converted data to the server, means for receiving data at the server and checking the format and content of the data, means for analyzing the received data, means for searching for related products or services based on the analysis results, means for generating an optimal proposal based on the search results, means for transmitting the generated proposal to a user terminal, means for displaying the proposed product or service on the user terminal, means for inputting information required for contract procedures, means for transmitting contract information to the server and completing the contract procedures, and means for providing post-contract support services. This enables users to efficiently find products or services, sign contracts, and receive consistent support even after the contract is signed.

[0870] "User" refers to a general user who uses the system to search for and purchase goods and services.

[0871] "Terminal" refers to an electronic device, such as a smartphone, tablet, or PC, that a user uses to access the system, enter information, or receive suggestions.

[0872] "Server" refers to the central processing unit that receives and analyzes data sent by users and generates optimal product and service recommendations.

[0873] "Input data" refers to data such as photographs, images, text, and videos that users input to provide information about products and services to the system.

[0874] "Means for converting into a format" refers to the process by which the terminal receives input data from the user and converts it into an appropriate digital format that can be sent to the server.

[0875] "Means for transmitting data to a server" refers to a function for transmitting converted data to a server via the Internet or other communication network.

[0876] "Means for analyzing data" refers to the function of the server using algorithms to process the data it receives and understand and classify its content.

[0877] "Means for searching for products or services" refers to the function of searching for products or services that meet the user's needs from a database based on the analysis results.

[0878] The "means of generating suggestions" refers to the process of formulating and creating information on optimal products and services to be offered to users based on search results.

[0879] "Means for transmitting suggestions to a user terminal" refers to a communication function for transmitting the generated suggestions to a user terminal and visually displaying them.

[0880] "Means for entering information necessary for contract procedures" refers to the interface through which users enter personal information and payment information required when purchasing or signing a contract for a product or service.

[0881] "Means for completing the contract procedure" refers to the process of sending the entered information to the server and officially concluding a contract for the purchase of goods or services.

[0882] "Means for providing post-contract support services" refers to functions for providing additional information and answering questions to users after the contract is completed.

[0883] The system of the present invention proposes optimal products and services based on data entered by a user, and also provides support after the contract is signed. A specific embodiment of this system will be described in detail below.

[0884] Accepting user input data

[0885] Users access the system using devices such as smartphones, tablets, and PCs. They can input data about products and services that interest them. Data that can be input includes photos, images, text, and videos. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[0886] The terminal receives input data from the user, checks the data format, converts it into an appropriate digital format (e.g., JPEG format, UTF-8 format), and then sends the converted data to the server.

[0887] Data reception and preprocessing

[0888] The server receives the data sent from the device. It checks the format and content of the received data to see if there is any improper data or corrupted files. For images, the server performs preliminary processing such as resizing and noise reduction.

[0889] Data analysis

[0890] The server runs image recognition algorithms to extract key features of the input image (e.g., color code, shape, texture), and uses natural language processing (NLP) algorithms for text data to analyze the user's intent. For example, for "blue dress," the server identifies the color as "blue" and the item as "dress."

[0891] Proposal Generation

[0892] Once the analysis is complete, the server searches the database for relevant products and services. Based on the search results, it generates recommendations that best suit the user's needs. Specifically, it applies filters such as price range, brand, and size, and generates the most suitable product and service information in the form of photos, text, and video.

[0893] View Suggestions

[0894] The server sends the generated proposal data to the device. The device receives the data and displays it in a visually easy-to-understand format for the user. On smartphones, the proposal content is displayed on the app screen or via push notification, and on PCs, the proposal content is displayed in a new tab on the browser.

[0895] Contract Procedures

[0896] The user is interested in the proposed product or service and decides to purchase or enter into a contract. The purchase procedure is completed by entering the necessary information (e.g., name, address, payment information) and sending it to the server via the device. The server synchronizes the contract information in real time, and the contract is officially concluded.

[0897] Post-contract support

[0898] After signing the contract, the server provides the user with additional support services. Specifically, the server provides instant support for user questions through chat and messaging functions. It can also answer user questions in real time using voice and video calls. For example, it can provide usage guides and assembly video tutorials for purchased products.

[0899] Specific examples

[0900] For example, if a user is looking for a particular piece of furniture, the following process applies:

[0901] 1. The user takes a photo of a sofa that suits their living room with their smartphone and uploads it through the app.

[0902] 2. The device converts the photo data into JPEG format and sends it to the server.

[0903] 3. The server receives the photo data and resizes and denoises the image.

[0904] 4. The server uses image recognition algorithms to analyze the sofa's color, shape, and texture.

[0905] 5. The server searches the database for blue living room sofas, filtering by price range and brand.

[0906] 6. The server generates multiple blue sofa photos and details as suggestions.

[0907] 7. The server sends the generated proposal data to the terminal, which displays it to the user.

[0908] 8. The user selects their favorite sofa from the suggestions and proceeds with the purchase.

[0909] 9. The user enters their payment information and clicks the purchase button.

[0910] 10. The device sends the entered information to the server and confirms the purchase.

[0911] 11. The server synchronizes the purchase information and provides the user with assembly instructions via chat.

[0912] 12. If the user asks any follow-up questions, the server will answer them immediately.

[0913] Prompt Sentence Examples

[0914] "I'm looking for a new blue dress. What are my options?"

[0915] "I'm looking for a sofa that will complement my home decor. I've uploaded some photos. Can you show me the options?"

[0916] This allows users to efficiently find products and services that suit them, sign contracts, and receive comprehensive support afterward.

[0917] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0918] Step 1:

[0919] Users access the system using devices such as smartphones, tablets, and PCs. They can input photos, images, text, and videos of products and services that interest them. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system. The input data is accepted by the device.

[0920] Input: User input of photos, images, text, and videos

[0921] Output: Input data stored on the terminal

[0922] Specific behavior: The user opens the app or browser, enters the required information, and clicks the "Submit" button.

[0923] Step 2:

[0924] The terminal checks the data entered by the user and converts it into the appropriate format: if it's a photo, it's converted into JPEG format, if it's text, it's converted into UTF-8 format, and then sends the converted data to the server.

[0925] Input: User-entered photo, image, text, and video data

[0926] Output: Transformed data to send to the server

[0927] Specific operation: The terminal checks the data format, checks for inconsistencies, converts the data format, and sends it to the server via the network.

[0928] Step 3:

[0929] The server receives the data sent from the device. It checks the format and content of the received data to see if there is any improper data or corrupted files. For images, the server performs preliminary processing such as resizing and noise reduction.

[0930] Input: Transformed data sent from the terminal

[0931] Output: Preprocessed data

[0932] Specific operation: The server receives the data, and a resident script checks the data format and content, and performs image resizing and noise removal if necessary.

[0933] Step 4:

[0934] The server runs image recognition algorithms to extract key features of the input image (e.g., color code, shape, texture), and uses natural language processing (NLP) algorithms for text data to analyze the user's intent. For example, for "blue dress," the server identifies the color as "blue" and the item as "dress."

[0935] Input: Preprocessed data (images, text)

[0936] Output: Analysis results (feature extraction information, user intent)

[0937] How it works: An image recognition model on the server analyzes images and extracts color codes and shape information, while an NLP model analyzes text data to identify keywords and intent.

[0938] Step 5:

[0939] The server uses the analyzed data to search for relevant products and services from its database. Based on the search results, it generates recommendations that best suit the user's needs. Specifically, it applies filters such as price range, brand, and size, and generates the most suitable product and service information in the form of photos, text, and video.

[0940] Input: Analysis results

[0941] Output: Product and service proposal data

[0942] Specific operation: The server accesses an internal database, searches for products and services that match the criteria, and generates proposals based on the results.

[0943] Step 6:

[0944] The server sends the generated proposal data to the device. The device receives the data and displays it in a visually easy-to-understand format for the user. On smartphones, the proposal content is displayed on the app screen or via push notification, and on PCs, the proposal content is displayed in a new tab on the browser.

[0945] Input: Generated proposal data

[0946] Output: Proposal displayed on the user's device

[0947] Specific operation: The server sends the proposed data over the network, the device receives it and displays it on the screen, and a push notification is displayed on the smartphone.

[0948] Step 7:

[0949] The user is interested in the proposed product or service and decides to purchase or sign a contract. They enter the necessary information (e.g., name, address, payment information) and send it to the server via their device.

[0950] Input: Name, address, and payment information entered by the user

[0951] Output: Contract information sent to the server

[0952] Specific operation: When the user enters the required information into the form and clicks the "Purchase" or "Contract" button, the information is sent to the server.

[0953] Step 8:

[0954] The server synchronizes the contract information in real time, and the contract is officially established. After the contract is signed, additional support services are provided to the user. Specifically, immediate support for user questions is provided through chat and messaging functions. Users' questions can also be answered in real time using voice and video calls.

[0955] Input: Contract information

[0956] Output: Providing support services

[0957] What it does: The server updates contract information and, if necessary, connects with the support team to provide the necessary support to the user, responding in real time via chat, voice, or video calls.

[0958] (Application example 1)

[0959] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0960] Conventional systems have had difficulty efficiently and accurately recommending products and services that interest users and providing prompt post-purchase support. This problem stems from the difficulty of accurately understanding the data entered by users and making recommendations that meet their needs. Therefore, it is necessary to build a system that can accurately analyze the data entered by users, recommend optimal products and services, and provide prompt post-purchase support.

[0961] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0962] In this invention, the server includes means for accepting input data from a user, means for analyzing the accepted input data, means for proposing related products or services based on the analysis results, means for displaying the proposed products or services on the user terminal, means for providing post-contract support services, means for generating prompt sentences to optimize interpretation of the input data, means for providing real-time support via chat and video calls based on the analysis results, and means for analyzing the accepted input data using image recognition technology and natural language processing technology. This makes it possible to consistently provide product proposals and post-contract support that quickly and accurately meet the user's needs.

[0963] "Means for accepting input data from the user" refers to an interface that allows the user to provide the system with information about products and services in which the user is interested.

[0964] "Means for analyzing received input data" refers to the process of analyzing data provided by users using image recognition technology and natural language processing technology to understand their needs and intentions.

[0965] "Means for suggesting related products or services based on the analysis results" refers to a function for selecting optimal products or services from a database based on the analyzed data and recommending them to the user.

[0966] "Means for displaying proposed products or services on a user's terminal" refers to a function for visually displaying information about proposed products on a terminal such as a user's smartphone.

[0967] "Means for providing post-contract support services" refers to communication means such as chat and video calls for providing after-sales support for products and services purchased by users.

[0968] "Means for generating prompts to optimize interpretation of input data" refers to means for generating precise instructions or questions based on information obtained from the user and for effectively analyzing that data.

[0969] "Means for providing real-time chat and video call support based on the analysis results" refers to a function that uses the analyzed information to respond via chat or video call so that the user can receive immediate support.

[0970] "Means of analyzing input data received using image recognition technology and natural language processing technology" refers to technology that uses advanced algorithms to analyze images and text data provided by users and understand their content.

[0971] The system of the present invention proposes optimal products and services based on user input data and provides support after the contract is signed. To implement this system, the following programs and hardware are used.

[0972] 1. Accepting user input data

[0973] Users access the system using a smart device (smartphone, tablet, or PC) and enter photos, images, text, and videos of products and services they are interested in. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[0974] 2. Data Analysis

[0975] The server analyzes the received input data using image recognition technology (OpenCV and TensorFlow) and natural language processing technology (NLTK). For photos and videos, it extracts image features (color, shape, pattern, etc.), and for text, it analyzes keywords to identify the user's intent.

[0976] 3. Proposal Generation

[0977] Based on the analyzed data, the server searches the database for relevant products and services and generates proposals that best meet the user's needs. These proposals can be in the form of photos, text, videos, etc.

[0978] 4. Display of suggestions

[0979] The server sends the generated proposal data to the user's device, which then displays the received data in an easy-to-understand visual format. On smartphones, the proposals are displayed on the app screen or via push notification, while on PCs, the proposals are displayed in a new tab on the browser.

[0980] 5. Post-contract support services

[0981] If a user is interested in a proposed product or service and signs a contract, they can easily complete the contract process through their device. For example, when a user purchases a dress on their smartphone, they simply click the purchase button and enter their payment information to complete the process. The server synchronizes the contract information in real time and prepares to provide subsequent support services. The server provides support through chat, messaging, voice calls, and video calls.

[0982] Specific examples

[0983] If a user is looking for a particular piece of furniture, the following process applies:

[0984] 1. The user takes a photo of the furniture they want to match with their home's interior using their smartphone and uploads it to the system.

[0985] 2. The device sends the photo data to the server.

[0986] 3. The server runs an image recognition algorithm (OpenCV, TensorFlow) to analyze the photo and extract features such as furniture type, color, and shape.

[0987] 4. The server searches a database for relevant furniture and generates multiple options, such as a list of sofas and tables with photos.

[0988] 5. The server sends the generated suggestion data to the device, which displays it to the user, who then checks the search results on their smartphone.

[0989] 6. The user selects their favorite furniture from the suggested items and completes the purchase process.

[0990] 7. The server synchronizes the contract information and provides support services as needed, such as video guides on how to assemble furniture.

[0991] Prompt Sentence Examples

[0992] "Please analyze this photo and suggest suitable products. The photo shows a blue sofa. Also, please search your database for similar sofas and display them."

[0993] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0994] Step 1:

[0995] Users use their smart devices to input photos, images, text, and videos related to products and services they are interested in and upload them to the system. The input data is sent from the user's device to the server as an HTTP request. At this point, the input is the photo taken or text entered by the user, and the output is unanalyzed data sent to the server.

[0996] Step 2:

[0997] The server analyzes the received data using image recognition technology (OpenCV and TensorFlow). Specifically, it extracts features (color, shape, pattern) from image data and analyzes keywords from text data using natural language processing (NLTK). In this process, the input is unanalyzed data from the user, and the output is analyzed feature information and keywords.

[0998] Step 3:

[0999] The server generates prompts based on the analysis results to optimize the interpretation of the input data. It uses a generative AI model to create prompts to more clearly identify the user's needs and intentions. In this step, the input is the analysis results, and the output is the prompts.

[1000] Step 4:

[1001] The server uses the generated prompt to search a database for relevant products and services. It executes a database query to extract the product information that best suits the user's request. In this operation, the input is the prompt and its parsed results, and the output is a list of suggested products and services.

[1002] Step 5:

[1003] The server sends the information of the proposed products and services to the user's device. The information is encoded in JSON format and returned as an HTTP response. The input is a list of proposed products and services, and the output is the data sent to the user's device.

[1004] Step 6:

[1005] The terminal displays the received proposal data in an easy-to-understand visual format. Specifically, it displays a list of product information on the smart device screen and offers the user options. In this step, the input is the proposal data sent from the server, and the output is the visual information presented to the user.

[1006] Step 7:

[1007] If a user is interested in the proposed product or service and wants to purchase it, he or she inputs the necessary information (e.g., payment information) through the terminal and sends it to the server. In this process, the input is the product information and payment information selected by the user, and the output is the purchase completion status.

[1008] Step 8:

[1009] The server synchronizes contract information in real time and prepares support services after purchase. The contract information is saved in the database and support services are initiated as needed. In this process, the input is the user's purchase information and the output is the support service preparation status.

[1010] Step 9:

[1011] The server provides real-time chat and video support. When a user has a question or problem, support staff are available to respond immediately. The input is the user's support request, and the output is real-time support information.

[1012] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1013] The system of the present invention analyzes input data from users, proposes relevant products or services, recognizes user emotions to optimize proposals, and provides post-contract support. A specific embodiment of this system will be explained below, with program processing explained in natural language and examples provided.

[1014] Accepting user input data

[1015] Users access the system using devices such as smartphones, tablets, and PCs. They input photos, images, text, and videos of their interests. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[1016] The terminal receives input data from the user, converts it into an appropriate format, and sends it to the server. For example, photo data in JPEG format can be sent as is.

[1017] Data analysis

[1018] The server runs algorithms to analyze the input data it receives. For photos and videos, image recognition technology is used to extract key features, such as color, shape, and pattern. For text, natural language processing (NLP) technology is used to analyze the user's intent. For example, a keyword like "blue dress" can be used to identify the color and type of item.

[1019] Furthermore, the server uses an emotion engine to recognize emotions from the user's input data, such as emotion analysis from text data, emotion analysis from audio data, and facial expression analysis from video data, to identify the user's current emotional state (e.g., joy, sadness, surprise).

[1020] Proposal Generation

[1021] Once the analysis is complete, the server searches the database for relevant products and services and selects those that best meet the user's needs. It then adjusts the recommendations based on the user's emotions, as determined by the emotion engine. For example, if the user is in a state of "joy" or "excitement," it will prioritize items with many positive product reviews.

[1022] The selected results are generated in multiple formats, including photos, text, video, etc. For example, if the search keyword is "blue dress," product information for multiple blue dresses will be provided along with photos.

[1023] View Suggestions

[1024] The server transmits the generated proposal data to the user's device in a format appropriate for the device.

[1025] The device then displays the received proposal data in a visually easy-to-understand format to the user. On smartphones, the proposal content is displayed on the app screen or via push notification, while on PCs, the proposal content is displayed in a new tab on the browser.

[1026] Post-contract support services

[1027] If a user is interested in a proposed product or service and wishes to sign a contract, they can easily complete the contract procedure through their device. For example, when a user purchases a dress on their smartphone, they simply click the purchase button and enter the necessary payment information to complete the purchase procedure.

[1028] After a purchase or contract is completed, the server synchronizes the contract information in real time and prepares to provide subsequent support services. The server provides support through chat and messaging functions, and also provides real-time support via voice and video calls. For example, if a user has a question about how to use or care for a purchased dress, they can ask through chat and receive an immediate answer.

[1029] Specific examples

[1030] For example, if a user is looking for a particular piece of furniture, the following process applies:

[1031] 1. The user takes a photo of the furniture they want to match with their home's interior using their smartphone and uploads it to the system.

[1032] 2. The device sends the photo data to the server.

[1033] 3. The server runs an image recognition algorithm to analyze the photo and extract features such as furniture type, color, and shape.

[1034] 4. The server uses an emotion engine to recognize the user's emotions from input data such as photos and text. If the user is recognized as being in a "lively" emotional state, casual, brightly colored furniture will be suggested.

[1035] 5. The server searches a database for relevant furniture and generates multiple options, such as a list of sofas and tables with photos.

[1036] 6. The server sends the generated suggestion data to the device, which displays it to the user, who then checks the search results on their smartphone.

[1037] 7. The user selects their favorite furniture from the suggested items and completes the purchase process.

[1038] 8. The server synchronizes the contract information and provides support services as needed, such as video guides on how to assemble furniture.

[1039] In this way, users can easily obtain products and services that suit them and receive comprehensive support even after signing a contract. In addition, the introduction of an emotion engine will make the user experience more personalized.

[1040] The processing flow will be explained below.

[1041] Step 1:

[1042] Users access the system using devices such as smartphones, tablets, and PCs. They input photos, images, text, and videos related to products and services they are interested in. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[1043] Step 2:

[1044] The terminal receives input data from the user, converts it into an appropriate format, and sends it to the server. For example, it sends photo data in JPEG format as is.

[1045] Step 3:

[1046] The server runs algorithms to analyze the input data it receives. For photos and videos, image recognition technology is used to extract key features, such as color, shape, and pattern. For text, natural language processing (NLP) technology is used to analyze the user's intent. For example, the color and type of item can be identified from keywords such as "blue dress."

[1047] Step 4:

[1048] Furthermore, the server uses an emotion engine to recognize emotions from the user's input data, such as emotion analysis from text data, emotion analysis from audio data, and facial expression analysis from video data, to identify the user's current emotional state (e.g., joy, sadness, surprise).

[1049] Step 5:

[1050] Based on the analysis results, the server searches its database for related products and services and selects those that best meet the user's needs. Furthermore, the server adjusts the recommendations based on the user's emotions, as recognized by the emotion engine. For example, if the user is in a state of "joy" or "excitement," it prioritizes items with many positive product reviews.

[1051] Step 6:

[1052] The server sends the generated proposal data to the user's device in a format appropriate for the device.

[1053] Step 7:

[1054] The device receives the proposal data and displays it to the user in an easy-to-understand visual format. On smartphones, the proposal content is displayed on the app screen or via push notification, while on PCs, the proposal content is displayed in a new tab on the browser.

[1055] Step 8:

[1056] The user becomes interested in the proposed products or services and makes a selection. The user then checks for more information about the selected product or service, such as detailed product specifications, price, and user reviews.

[1057] Step 9:

[1058] The user finally completes the purchase or contract process by clicking the purchase button and entering the required payment information, such as credit card information and shipping address.

[1059] Step 10:

[1060] The server receives purchase and contract information, stores it in a database, and synchronizes contract information in real time.

[1061] Step 11:

[1062] The server provides channels such as chat, messaging, voice calls, and video calls for post-contract support services. For example, if a user has a question about how to use a product, they can receive support through the chat function.

[1063] Step 12:

[1064] The server provides product usage guides and support materials, such as text guides and video guides, and sends them to the device, which then displays the guides to the user, who can then refer to them.

[1065] Example 2

[1066] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1067] Conventional recommendation systems may not be able to provide appropriate products or services because they do not accurately meet user needs or do not take into account the user's emotional state, which calls for an improved user experience. Furthermore, insufficient support services may be provided after the contract is signed, potentially reducing user satisfaction. Therefore, a system is needed that can accurately grasp the user's needs, make optimal recommendations based on the user's emotional state, and provide comprehensive support services after the contract is signed.

[1068] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1069] In this invention, the server includes a means for accepting input data from a user, a means for analyzing the accepted input data, and a means for proposing related products or services based on the analysis results and the user's emotions. This allows the server to accurately grasp the user's needs and make optimal proposals that take into account the user's emotional state. Furthermore, the server can improve user satisfaction by displaying the proposed products or services on the user's terminal and providing post-contract support services.

[1070] "User" refers to a person who uses the system to search for, propose, and contract for goods and services.

[1071] "Input data" refers to information provided by a user to the system, and includes various forms such as photographs, text, and video.

[1072] A "terminal" is a hardware device that a user uses to access the system, and includes a mobile terminal, a computer, and the like.

[1073] "Server" refers to the computer system that acts as the central part of the system, analyzing input data, generating proposals, communicating with the database, and transmitting proposals.

[1074] "Image recognition technology" refers to the technology of extracting specific features (color, shape, pattern, etc.) from photographs and videos.

[1075] "Natural language processing technology" refers to technology that analyzes user intent and meaning from text data.

[1076] "Emotions" are internal states that users express through input data, and include joy, sadness, surprise, etc.

[1077] "Suggestions" refers to information about products and services selected based on the analysis results and user sentiment.

[1078] "Support services" refers to the support a user receives after signing a contract for a proposed product or service, and includes support via chat, voice calls, and video calls.

[1079] The system of the present invention analyzes input data from a user, proposes related products or services, recognizes the user's emotions to optimize the proposals, and also provides post-contract support services. A specific embodiment of this system will be described.

[1080] Accepting user input data

[1081] Users access the system using devices such as smartphones, tablets, and PCs. They input photos, text, and videos of their interests. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[1082] Transfer of input data

[1083] The device receives the data entered by the user, converts it into an appropriate format, and sends it to the server. For example, it sends JPEG photo data as is. In the case of text, it sends the entered keywords as is.

[1084] Data analysis

[1085] The server runs an algorithm to analyze the received input data, specifically using the following techniques:

[1086] Image recognition technology: Algorithms such as YOLO (You Only Look Once) are used to extract key features (color, shape, pattern, etc.) from photos and videos. For example, identifying the features of a "blue dress."

[1087] Natural Language Processing (NLP) techniques: Use NLP models such as BERT (Bidirectional Encoder Representations from Transformers) to analyze user intent from text data. For example, analyze the keyword "blue dress" to identify the color and type of item.

[1088] Emotion Engine: Uses Python's Emotion API to recognize user emotions from text, audio, and video, thereby identifying the user's emotional state (e.g., happy, sad, surprised).

[1089] Proposal Generation

[1090] The server searches the database for relevant products and services and generates optimal suggestions based on the analysis results and the recognized emotions. The server works as follows:

[1091] Product information collection: Pulling relevant product or service information (such as name, price, reviews, and photos) from a database (e.g., a MySQL database).

[1092] Formation of proposal content: Based on the collected information, proposal data is generated in formats such as CSV, JSON, HTML, etc. For example, product information for multiple "blue dresses" is provided along with photos.

[1093] View Suggestions

[1094] The server sends the generated proposal data to the user's device. The device then displays the proposal data in a visually easy-to-understand format. For example, on a smartphone, the proposals are displayed on the app screen or via push notification, while on a PC, they are displayed in a new tab on the browser. React Native is used to generate the smartphone app UI and display the proposals.

[1095] Post-contract support services

[1096] If a user is interested in the proposed products or services and wishes to sign a contract, they can complete the contract procedure through their device. After the purchase procedure is completed, the server synchronizes the contract information in real time and prepares to provide subsequent support services. For example, the Twilio API can be used to implement support functions such as chat, voice calls, and video calls.

[1097] Specific examples

[1098] For example, if a user is looking for a particular piece of furniture, the following process applies:

[1099] The user takes a photo of the furniture they want to match with their home interior using their smartphone and uploads it to the system. The device sends the photo data to the server. The server runs an image recognition algorithm (e.g., YOLO) to analyze the photo and extract features such as the type, color, and shape of the furniture. The server uses an emotion engine (e.g., Emotion API) to recognize the user's emotions. If the user is recognized as being in a "lively" emotional state, casual, brightly colored furniture will be prioritized and suggested. The server searches for related furniture from its database and generates multiple options. The server sends the generated suggestion data to the device, which displays it to the user. The user checks the search results on their smartphone, selects the furniture they like, and completes the purchase process. The server synchronizes contract information and provides support services as needed (e.g., sending video guides on how to assemble the furniture).

[1100] Prompt Sentence Examples

[1101] "Explain how a system can provide product recommendations based on a user-uploaded photo of a blue dress."

[1102] "Please outline a system that analyzes user sentiment and optimizes product recommendations based on that sentiment."

[1103] In this way, users can easily obtain products and services that suit them and receive comprehensive support even after signing a contract. In addition, the introduction of an emotion engine will make the user experience more personalized.

[1104] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1105] Step 1:

[1106] Users access the system using devices such as smartphones, tablets, and PCs. They input photos, text, and videos of their interest. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system. Inputs include photo data (e.g., in JPEG format) and text data (e.g., "blue dress").

[1107] Step 2:

[1108] The terminal receives data entered by the user, converts the format appropriately, and sends it to the server. Specifically, it sends JPEG photo data and text data directly to the server. The input is the photo data and text data received from the user, and the output is the data sent to the server.

[1109] Step 3:

[1110] The server runs an algorithm to analyze the received input data, using the following techniques:

[1111] Image recognition technology: For example, algorithms such as YOLO (You Only Look Once) are used to extract the main features of a photo (color, shape, pattern, etc.). The input is JPEG photo data, and the output is the extracted feature information (e.g., "blue color," "dress shape").

[1112] Natural Language Processing (NLP) technology: For example, an NLP model such as BERT (Bidirectional Encoder Representations from Transformers) is used to analyze user intent from text data. The input is the text data, and the output is the analysis result (e.g., "blue", "dress").

[1113] Emotion engine: For example, Python's Emotion API is used to recognize the user's emotional state from text, audio, and video. The input is text data, and the output is emotional state information (e.g., "joy" or "excitement").

[1114] Step 4:

[1115] The server then suggests relevant products and services based on the analysis results and the recognized emotions. Specifically, it:

[1116] Product information collection: Extract relevant product and service information (such as name, price, reviews, and photos) from a database (e.g., MySQL database). The input is the analysis results and sentiment information, and the output is product information.

[1117] Proposal content generation: Based on the collected product information, proposal data is generated in formats such as CSV, JSON, and HTML. For example, product information for multiple "blue dresses" is provided along with photos. The input is product information, and the output is proposal data.

[1118] Step 5:

[1119] The server sends the generated proposal data to the terminal. The input is the generated proposal data, and the output is the data sent to the terminal.

[1120] Step 6:

[1121] The device receives the proposal data and displays it visually to the user. Specifically, on a smartphone, the proposal is displayed on the app screen or via push notification, and on a PC, the proposal is displayed by opening a new tab in the browser. The input is the proposal data sent from the server, and the output is the information displayed to the user.

[1122] Step 7:

[1123] If a user is interested in a proposed product or service, they can complete the contract process through their device. For example, they can complete the purchase process by simply clicking the "Purchase" button on their smartphone and entering the necessary payment information. The input is the proposal data and contract information, and the output is the completed contract.

[1124] Step 8:

[1125] The server synchronizes the contract information in real time and prepares to provide subsequent support services. Specifically, it uses the Twilio API to implement chat, voice call, and video call support functions. The input is the contract information, and the output is the support service provided.

[1126] (Application example 2)

[1127] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1128] In modern e-commerce, users have difficulty finding the best products and services for them from the vast number of options available. Furthermore, product suggestions that do not match the user's emotional state make it difficult to stimulate their desire to purchase. Furthermore, post-purchase support is often inadequate, resulting in a poor user experience. A system that can solve these problems and improve user satisfaction is needed.

[1129] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1130] In this invention, the server includes means for accepting input data from a user, means for analyzing the accepted input data, means for proposing related products or services based on the analysis results and the user's emotional state, means for displaying the proposed products or services on the user's terminal, and means for providing post-contract support services. This makes it possible to analyze a variety of input data from the user and propose optimal products and services that match the user's emotional state. Furthermore, by providing enhanced post-contract support, user satisfaction can be improved.

[1131] The "means for accepting input data from a user" is an interface that allows a user to provide a photograph, image, text, audio, or video to the system.

[1132] "Means for analyzing received input data" refers to algorithms and techniques for analyzing user-provided input data and extracting data features and emotional states.

[1133] "Means for suggesting relevant products or services based on the analysis results and the user's emotional state" refers to a mechanism by which the system selects and suggests the most appropriate products or services, taking into account the analyzed data and the user's emotional state.

[1134] "Means for displaying suggested products or services on a user terminal" refers to a method by which the system displays the suggested content on the user's smartphone, tablet, personal computer, or smart glasses.

[1135] The "means for providing post-contract support services" is a function for providing support and after-sales services after the user purchases or contracts for the proposed product or service.

[1136] "Photos, images, text, audio, or video" are any form of visual or audio data that a user provides to the system.

[1137] "Means for extracting features and emotional states" refers to techniques for identifying key features (e.g., color, shape) and emotional states (e.g., happy, sad) from user-provided data.

[1138] "User terminal is a smartphone, tablet, personal computer or smart glasses" refers to the electronic device at the user's side that is used to display the system's proposals.

[1139] The system for implementing this invention accepts and analyzes various input data from users and proposes optimal products and services based on the results. It also analyzes the user's emotional state and personalizes the proposals. It also provides post-contract support services to improve user satisfaction.

[1140] Hardware and Software

[1141] Hardware

[1142] Smartphones, tablets, personal computers, smart glasses: electronic devices used by users to provide input data and receive suggestions.

[1143] software

[1144] Image analysis: Uses the Google Cloud Vision API to analyze user-uploaded photos and images and extract features.

[1145] Natural Language Processing (NLP): Uses the Google Cloud Natural Language API to analyze user-entered text and recognize intent and sentiment.

[1146] Emotion Analysis: Uses the Microsoft Azure Emotion API to identify the user's emotional state from audio and video data.

[1147] Database: Use Firebase Realtime Database to retrieve relevant product and service data based on analytics results.

[1148] Front-end: Build the user interface using React Native.

[1149] Example

[1150] An example is given below where the user is looking for a new dress.

[1151] 1. User uploads a photo

[1152] Users use their smartphones to take photos of the dresses and upload them to the application.

[1153] 2. Data Analysis

[1154] The server analyzes uploaded photos using the Google Cloud Vision API to extract features such as the color, shape, and design of the dress, and also analyzes the user's input text and voice using the Google Cloud Natural Language API and Microsoft Azure Emotion API to recognize the user's emotional state.

[1155] 3. Related product suggestions

[1156] The server retrieves relevant product data from the Firebase Realtime Database and selects the most suitable products based on the emotional state. For example, if the emotional state is "joy," it will prioritize dresses with high reviews.

[1157] 4. Display of proposed data

[1158] The suggested products are visually displayed on smartphones and tablets using React Native, and users can choose from the suggested products.

[1159] 5. Post-contract support services

[1160] After the user purchases the selected product, the server stores the purchase information in the Firebase Realtime Database and provides subsequent support services, such as real-time chat support and instructional videos, for any questions about the product.

[1161] Prompt Sentence Examples

[1162] Below is an example of a prompt sentence that can be input into the "generative AI model."

[1163] Analyze photos, extract their features, and generate relevant product suggestions based on the user's emotional state.

[1164] Input: Photo of a blue dress

[1165] Emotion: Joy

[1166] Output: Related blue dress product list and reviews

[1167] In this way, the system of the present invention can propose products and services that accurately reflect the user's needs and emotional state, improving the user experience. Furthermore, the system also provides comprehensive support services after the contract is signed, thereby increasing user satisfaction.

[1168] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1169] Step 1:

[1170] User uploads a photo

[1171] A user takes a photo of a product they want to purchase using their smartphone and uploads it to the application. The input is a JPEG photo, and the output is a notification that the photo has been uploaded. The device receives the photo data from the user and transfers it to the server.

[1172] Step 2:

[1173] Sending data to the server

[1174] The terminal sends JPEG format photo data to the server. The input is a JPEG format photo, and the output is the photo data sent to the server. The terminal performs this process automatically, and the user is not required to perform any special operations.

[1175] Step 3:

[1176] Analyzing photo data

[1177] The server analyzes the received photo data using the Google Cloud Vision API. The input is JPEG photo data, and the output is feature information such as the color, shape, and design of the product. The server runs an image recognition algorithm to extract important features.

[1178] Step 4:

[1179] Analyzing the user's emotional state

[1180] The server uses the Google Cloud Natural Language API and Microsoft Azure Emotion API to analyze the user's input text and voice data. The input is text and voice data, and the output is the user's emotional state information (e.g., joy, surprise, etc.). The server runs an emotion analysis algorithm to identify the user's emotional state.

[1181] Step 5:

[1182] Search and suggest related products

[1183] The server uses Firebase Realtime Database to search for relevant product data based on the analyzed features and the user's emotional state. The input is feature information and emotional state information, and the output is a list of relevant products. The server generates recommendations by prioritizing products according to the emotional state.

[1184] Step 6:

[1185] Send the proposal data to the user device

[1186] The server sends the generated proposal data to the user's terminal. The input is a list of related products, and the output is a notification that the proposal data has been sent to the user's terminal. The server sends the data to the terminal in an appropriate format.

[1187] Step 7:

[1188] View suggested data

[1189] The terminal visually displays the received suggestion data to the user. The input is the suggestion data, and the output is the display screen viewed by the user. The terminal uses React Native to provide a visually easy-to-understand interface.

[1190] Step 8:

[1191] Providing post-contract support

[1192] After the user purchases the recommended product, the server synchronizes the contract information and provides support services. The input is the purchase information, and the output is the support start notification and support content. The server supports the user through support functions such as real-time chat and instructional videos.

[1193] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1194] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1195] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1196] [Fourth embodiment]

[1197] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1198] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1199] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1200] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1201] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1202] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1203] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1204] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1205] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1206] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1207] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1208] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1209] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1210] The system of the present invention proposes optimal products and services based on data entered by the user, and also provides support after the contract is signed. A specific embodiment of this system will be explained in natural language, with the program processing also being presented, along with specific examples.

[1211] Accepting user input data

[1212] Users access the system using devices such as smartphones, tablets, and PCs. They can input photos, images, text, and videos of products and services they are interested in. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[1213] The terminal receives input data from the user, converts it into an appropriate format, and sends it to the server. For example, photo data in JPEG format can be sent as is.

[1214] Data analysis

[1215] The server runs algorithms to analyze the input data it receives. For photos and videos, image recognition technology is used to extract key features, such as color, shape, and pattern. For text, natural language processing (NLP) technology is used to analyze the user's intent. For example, a keyword like "blue dress" can be used to identify the color and type of item.

[1216] Proposal Generation

[1217] Once the analysis is complete, the server searches the database for relevant products and services and selects those that best meet the user's needs. The results are presented in multiple formats, including photos, text, and video. For example, if the search keyword is "blue dress," product information for multiple blue dresses will be provided along with photos.

[1218] View Suggestions

[1219] The server sends the generated proposal data to the user's device. The device receives the data and displays it visually in an easy-to-understand format. On smartphones, the proposals are displayed on the app screen or via push notifications, while on PCs, they are displayed in a new tab on the browser.

[1220] Post-contract support services

[1221] If a user is interested in a proposed product or service and wishes to sign a contract, they can easily complete the contract procedure through their device. For example, when a user purchases a dress on their smartphone, they simply click the purchase button and enter the necessary payment information to complete the purchase procedure.

[1222] After a purchase or contract is completed, the server synchronizes the contract information in real time and prepares to provide subsequent support services. The server provides support through chat and messaging functions, and also provides real-time support via voice and video calls. For example, if a user has a question about how to use or care for a purchased dress, they can ask through chat and receive an immediate answer.

[1223] Specific examples

[1224] For example, if a user is looking for a particular piece of furniture, the following process applies:

[1225] 1. The user takes a photo of the furniture they want to match with their home's interior using their smartphone and uploads it to the system.

[1226] 2. The device sends the photo data to the server.

[1227] 3. The server runs an image recognition algorithm to analyze the photo and extract features such as furniture type, color, and shape.

[1228] 4. The server searches a database for relevant furniture and generates multiple options, such as a list of sofas and tables with photos.

[1229] 5. The server sends the generated suggestion data to the device, which displays it to the user, who then checks the search results on their smartphone.

[1230] 6. The user selects their favorite furniture from the suggested items and completes the purchase process.

[1231] 7. The server synchronizes the contract information and provides support services as needed, such as video guides on how to assemble furniture.

[1232] In this way, users can easily obtain products and services that suit them and receive comprehensive support even after signing a contract.

[1233] The processing flow will be explained below.

[1234] Step 1:

[1235] Users access the system using devices such as smartphones, tablets, and PCs. They input photos, images, text, and videos of their interests. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[1236] Step 2:

[1237] The terminal receives input data from the user, converts it into an appropriate format, and sends it to the server. For example, photo data in JPEG format can be sent as is.

[1238] Step 3:

[1239] The server runs algorithms to analyze the input data it receives. For photos and videos, image recognition technology is used to extract key features, such as color, shape, and pattern. For text, natural language processing (NLP) technology is used to analyze the user's intent. For example, the color and type of item can be identified from keywords such as "blue dress."

[1240] Step 4:

[1241] Based on the analysis results, the server searches the database for related products and services and selects those that best meet the user's needs. The selected results are generated in multiple formats, including photos, text, and video. For example, if the search keyword is "blue dress," product information for multiple blue dresses will be provided along with photos.

[1242] Step 5:

[1243] The server sends the generated proposal data to the user's device in a format appropriate for the device.

[1244] Step 6:

[1245] The device receives the proposal data and displays it to the user in an easy-to-understand visual format. On smartphones, the proposal content is displayed on the app screen or via push notification, while on PCs, the proposal content is displayed in a new tab on the browser.

[1246] Step 7:

[1247] The user becomes interested in the proposed products or services and makes a selection. The user then checks for more information about the selected product or service, such as detailed product specifications, price, and user reviews.

[1248] Step 8:

[1249] The user finally completes the purchase or contract process by clicking the purchase button and entering the required payment information, such as credit card information and shipping address.

[1250] Step 9:

[1251] The server receives purchase and contract information, stores it in a database, and synchronizes contract information in real time.

[1252] Step 10:

[1253] The server provides channels such as chat, messaging, voice calls, and video calls for post-contract support services. For example, if a user has a question about how to use a product, they can receive support through the chat function.

[1254] Step 11:

[1255] The server provides product usage guides and support materials, such as text guides and video guides, and sends them to the device, which then displays the guides to the user, who can then refer to them.

[1256] This allows users to use the system intuitively throughout the entire process, efficiently obtain products and services that suit them, and even receive prompt support after signing a contract.

[1257] Example 1

[1258] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1259] In conventional systems, it takes a great deal of time and effort for users to search for products and services and find the most suitable ones. Furthermore, there is a problem that support after signing a contract is not provided sufficiently, resulting in a decrease in user satisfaction. The present invention aims to solve these problems by providing a system that allows users to efficiently find products and services that suit them and receive comprehensive support even after signing a contract.

[1260] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1261] In this invention, the server includes means for converting received input data into an appropriate format, means for transmitting the converted data to the server, means for receiving data at the server and checking the format and content of the data, means for analyzing the received data, means for searching for related products or services based on the analysis results, means for generating an optimal proposal based on the search results, means for transmitting the generated proposal to a user terminal, means for displaying the proposed product or service on the user terminal, means for inputting information required for contract procedures, means for transmitting contract information to the server and completing the contract procedures, and means for providing post-contract support services. This enables users to efficiently find products or services, sign contracts, and receive consistent support even after the contract is signed.

[1262] "User" refers to a general user who uses the system to search for and purchase goods and services.

[1263] "Terminal" refers to an electronic device, such as a smartphone, tablet, or PC, that a user uses to access the system, enter information, or receive suggestions.

[1264] "Server" refers to the central processing unit that receives and analyzes data sent by users and generates optimal product and service recommendations.

[1265] "Input data" refers to data such as photographs, images, text, and videos that users input to provide information about products and services to the system.

[1266] "Means for converting into a format" refers to the process by which the terminal receives input data from the user and converts it into an appropriate digital format that can be sent to the server.

[1267] "Means for transmitting data to a server" refers to a function for transmitting converted data to a server via the Internet or other communication network.

[1268] "Means for analyzing data" refers to the function of the server using algorithms to process the data it receives and understand and classify its content.

[1269] "Means for searching for products or services" refers to the function of searching for products or services that meet the user's needs from a database based on the analysis results.

[1270] The "means of generating suggestions" refers to the process of formulating and creating information on optimal products and services to be offered to users based on search results.

[1271] "Means for transmitting suggestions to a user terminal" refers to a communication function for transmitting the generated suggestions to a user terminal and visually displaying them.

[1272] "Means for entering information necessary for contract procedures" refers to the interface through which users enter personal information and payment information required when purchasing or signing a contract for a product or service.

[1273] "Means for completing the contract procedure" refers to the process of sending the entered information to the server and officially concluding a contract for the purchase of goods or services.

[1274] "Means for providing post-contract support services" refers to functions for providing additional information and answering questions to users after the contract is completed.

[1275] The system of the present invention proposes optimal products and services based on data entered by a user, and also provides support after the contract is signed. A specific embodiment of this system will be described in detail below.

[1276] Accepting user input data

[1277] Users access the system using devices such as smartphones, tablets, and PCs. They can input data about products and services that interest them. Data that can be input includes photos, images, text, and videos. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[1278] The terminal receives input data from the user, checks the data format, converts it into an appropriate digital format (e.g., JPEG format, UTF-8 format), and then sends the converted data to the server.

[1279] Data reception and preprocessing

[1280] The server receives the data sent from the device. It checks the format and content of the received data to see if there is any improper data or corrupted files. For images, the server performs preliminary processing such as resizing and noise reduction.

[1281] Data analysis

[1282] The server runs image recognition algorithms to extract key features of the input image (e.g., color code, shape, texture), and uses natural language processing (NLP) algorithms for text data to analyze the user's intent. For example, for "blue dress," the server identifies the color as "blue" and the item as "dress."

[1283] Proposal Generation

[1284] Once the analysis is complete, the server searches the database for relevant products and services. Based on the search results, it generates recommendations that best suit the user's needs. Specifically, it applies filters such as price range, brand, and size, and generates the most suitable product and service information in the form of photos, text, and video.

[1285] View Suggestions

[1286] The server sends the generated proposal data to the device. The device receives the data and displays it in a visually easy-to-understand format for the user. On smartphones, the proposal content is displayed on the app screen or via push notification, and on PCs, the proposal content is displayed in a new tab on the browser.

[1287] Contract Procedures

[1288] The user is interested in the proposed product or service and decides to purchase or enter into a contract. The purchase procedure is completed by entering the necessary information (e.g., name, address, payment information) and sending it to the server via the device. The server synchronizes the contract information in real time, and the contract is officially concluded.

[1289] Post-contract support

[1290] After signing the contract, the server provides the user with additional support services. Specifically, the server provides instant support for user questions through chat and messaging functions. It can also answer user questions in real time using voice and video calls. For example, it can provide usage guides and assembly video tutorials for purchased products.

[1291] Specific examples

[1292] For example, if a user is looking for a particular piece of furniture, the following process applies:

[1293] 1. The user takes a photo of a sofa that suits their living room with their smartphone and uploads it through the app.

[1294] 2. The device converts the photo data into JPEG format and sends it to the server.

[1295] 3. The server receives the photo data and resizes and denoises the image.

[1296] 4. The server uses image recognition algorithms to analyze the sofa's color, shape, and texture.

[1297] 5. The server searches the database for blue living room sofas, filtering by price range and brand.

[1298] 6. The server generates multiple blue sofa photos and details as suggestions.

[1299] 7. The server sends the generated proposal data to the terminal, which displays it to the user.

[1300] 8. The user selects their favorite sofa from the suggestions and proceeds with the purchase.

[1301] 9. The user enters their payment information and clicks the purchase button.

[1302] 10. The device sends the entered information to the server and confirms the purchase.

[1303] 11. The server synchronizes the purchase information and provides the user with assembly instructions via chat.

[1304] 12. If the user asks any follow-up questions, the server will answer them immediately.

[1305] Prompt Sentence Examples

[1306] "I'm looking for a new blue dress. What are my options?"

[1307] "I'm looking for a sofa that will complement my home decor. I've uploaded some photos. Can you show me the options?"

[1308] This allows users to efficiently find products and services that suit them, sign contracts, and receive comprehensive support afterward.

[1309] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1310] Step 1:

[1311] Users access the system using devices such as smartphones, tablets, and PCs. They can input photos, images, text, and videos of products and services that interest them. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system. The input data is accepted by the device.

[1312] Input: User input of photos, images, text, and videos

[1313] Output: Input data stored on the terminal

[1314] Specific behavior: The user opens the app or browser, enters the required information, and clicks the "Submit" button.

[1315] Step 2:

[1316] The terminal checks the data entered by the user and converts it into the appropriate format: if it's a photo, it's converted into JPEG format, if it's text, it's converted into UTF-8 format, and then sends the converted data to the server.

[1317] Input: User-entered photo, image, text, and video data

[1318] Output: Transformed data to send to the server

[1319] Specific operation: The terminal checks the data format, checks for inconsistencies, converts the data format, and sends it to the server via the network.

[1320] Step 3:

[1321] The server receives the data sent from the device. It checks the format and content of the received data to see if there is any improper data or corrupted files. For images, the server performs preliminary processing such as resizing and noise reduction.

[1322] Input: Transformed data sent from the terminal

[1323] Output: Preprocessed data

[1324] Specific operation: The server receives the data, and a resident script checks the data format and content, and performs image resizing and noise removal if necessary.

[1325] Step 4:

[1326] The server runs image recognition algorithms to extract key features of the input image (e.g., color code, shape, texture), and uses natural language processing (NLP) algorithms for text data to analyze the user's intent. For example, for "blue dress," the server identifies the color as "blue" and the item as "dress."

[1327] Input: Preprocessed data (images, text)

[1328] Output: Analysis results (feature extraction information, user intent)

[1329] How it works: An image recognition model on the server analyzes images and extracts color codes and shape information, while an NLP model analyzes text data to identify keywords and intent.

[1330] Step 5:

[1331] The server uses the analyzed data to search for relevant products and services from its database. Based on the search results, it generates recommendations that best suit the user's needs. Specifically, it applies filters such as price range, brand, and size, and generates the most suitable product and service information in the form of photos, text, and video.

[1332] Input: Analysis results

[1333] Output: Product and service proposal data

[1334] Specific operation: The server accesses an internal database, searches for products and services that match the criteria, and generates proposals based on the results.

[1335] Step 6:

[1336] The server sends the generated proposal data to the device. The device receives the data and displays it in a visually easy-to-understand format for the user. On smartphones, the proposal content is displayed on the app screen or via push notification, and on PCs, the proposal content is displayed in a new tab on the browser.

[1337] Input: Generated proposal data

[1338] Output: Proposal displayed on the user's device

[1339] Specific operation: The server sends the proposed data over the network, and the device receives it and displays it on the screen. A push notification is displayed on the smartphone.

[1340] Step 7:

[1341] The user is interested in the proposed product or service and decides to purchase or sign a contract. They enter the necessary information (e.g., name, address, payment information) and send it to the server via their device.

[1342] Input: Name, address, and payment information entered by the user

[1343] Output: Contract information sent to the server

[1344] Specific operation: When the user enters the required information into the form and clicks the "Purchase" or "Contract" button, the information is sent to the server.

[1345] Step 8:

[1346] The server synchronizes the contract information in real time, and the contract is officially established. After the contract is signed, additional support services are provided to the user. Specifically, immediate support for user questions is provided through chat and messaging functions. Users' questions can also be answered in real time using voice and video calls.

[1347] Input: Contract information

[1348] Output: Providing support services

[1349] What it does: The server updates the contract information and, if necessary, connects with the support team to provide the necessary support to the user, in real time via chat, voice, or video calls.

[1350] (Application example 1)

[1351] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1352] Conventional systems have had difficulty efficiently and accurately recommending products and services that interest users and providing prompt post-purchase support. This problem stems from the difficulty of accurately understanding the data entered by users and making recommendations that meet their needs. Therefore, it is necessary to build a system that can accurately analyze the data entered by users, recommend optimal products and services, and provide prompt post-purchase support.

[1353] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1354] In this invention, the server includes means for accepting input data from a user, means for analyzing the accepted input data, means for proposing related products or services based on the analysis results, means for displaying the proposed products or services on the user terminal, means for providing post-contract support services, means for generating prompt sentences to optimize interpretation of the input data, means for providing real-time support via chat and video calls based on the analysis results, and means for analyzing the accepted input data using image recognition technology and natural language processing technology. This makes it possible to consistently provide product proposals and post-contract support that quickly and accurately meet the user's needs.

[1355] "Means for accepting input data from the user" refers to an interface that allows the user to provide the system with information about products and services in which the user is interested.

[1356] "Means for analyzing received input data" refers to the process of analyzing data provided by users using image recognition technology and natural language processing technology to understand their needs and intentions.

[1357] "Means for suggesting related products or services based on the analysis results" refers to a function for selecting optimal products or services from a database based on the analyzed data and recommending them to the user.

[1358] "Means for displaying proposed products or services on a user's terminal" refers to a function for visually displaying information about the proposed products on a terminal such as a user's smartphone.

[1359] "Means for providing post-contract support services" refers to communication means such as chat and video calls for providing after-sales support for products and services purchased by users.

[1360] "Means for generating prompts to optimize interpretation of input data" refers to means for generating precise instructions or questions based on information obtained from the user and for effectively analyzing that data.

[1361] "Means for providing real-time chat and video call support based on the analysis results" refers to a function that uses the analyzed information to respond via chat or video call so that the user can receive immediate support.

[1362] "Means of analyzing input data received using image recognition technology and natural language processing technology" refers to technology that uses advanced algorithms to analyze images and text data provided by users and understand their content.

[1363] The system of the present invention proposes optimal products and services based on user input data and provides support after the contract is signed. To implement this system, the following programs and hardware are used.

[1364] 1. Accepting user input data

[1365] Users access the system using a smart device (smartphone, tablet, or PC) and enter photos, images, text, and videos of products and services they are interested in. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[1366] 2. Data Analysis

[1367] The server analyzes the received input data using image recognition technology (OpenCV and TensorFlow) and natural language processing technology (NLTK). For photos and videos, it extracts image features (color, shape, pattern, etc.), and for text, it analyzes keywords to identify the user's intent.

[1368] 3. Proposal Generation

[1369] The server then searches the database for relevant products and services based on the analyzed data and generates suggestions that best meet the user's needs. These suggestions can be in the form of photos, text, videos, etc.

[1370] 4. Display of suggestions

[1371] The server sends the generated proposal data to the user's device, which then displays the received data in an easy-to-understand visual format. On smartphones, the proposals are displayed on the app screen or via push notification, while on PCs, they are displayed in a new tab on the browser.

[1372] 5. Post-contract support services

[1373] If a user is interested in a proposed product or service and signs a contract, they can easily complete the contract process through their device. For example, when a user purchases a dress on their smartphone, they simply click the purchase button and enter their payment information to complete the process. The server synchronizes the contract information in real time and prepares to provide subsequent support services. The server provides support through chat, messaging, voice calls, and video calls.

[1374] Specific examples

[1375] If a user is looking for a particular piece of furniture, the following process applies:

[1376] 1. The user takes a photo of the furniture they want to match with their home's interior using their smartphone and uploads it to the system.

[1377] 2. The device sends the photo data to the server.

[1378] 3. The server runs an image recognition algorithm (OpenCV, TensorFlow) to analyze the photo and extract features such as furniture type, color, and shape.

[1379] 4. The server searches a database for relevant furniture and generates multiple options, such as a list of sofas and tables with photos.

[1380] 5. The server sends the generated suggestion data to the device, which displays it to the user, who then checks the search results on their smartphone.

[1381] 6. The user selects their favorite furniture from the suggested items and completes the purchase process.

[1382] 7. The server synchronizes the contract information and provides support services as needed, such as video guides on how to assemble furniture.

[1383] Prompt Sentence Examples

[1384] "Please analyze this photo and suggest suitable products. The photo shows a blue sofa. Also, please search your database for similar sofas and display them."

[1385] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1386] Step 1:

[1387] Users use their smart devices to input photos, images, text, and videos related to products and services they are interested in and upload them to the system. The input data is sent from the user's device to the server as an HTTP request. At this point, the input is the photo taken or text entered by the user, and the output is unanalyzed data sent to the server.

[1388] Step 2:

[1389] The server analyzes the received data using image recognition technology (OpenCV and TensorFlow). Specifically, it extracts features (color, shape, pattern) from image data and analyzes keywords from text data using natural language processing (NLTK). In this process, the input is unanalyzed data from the user, and the output is analyzed feature information and keywords.

[1390] Step 3:

[1391] The server generates prompts based on the analysis results to optimize the interpretation of the input data. It uses a generative AI model to create prompts to more clearly identify the user's needs and intentions. In this step, the input is the analysis results, and the output is the prompts.

[1392] Step 4:

[1393] The server uses the generated prompt to search a database for relevant products and services. It executes a database query to extract the product information that best suits the user's request. In this operation, the input is the prompt and its parsed results, and the output is a list of suggested products and services.

[1394] Step 5:

[1395] The server sends the information of the proposed products and services to the user's device. The information is encoded in JSON format and returned as an HTTP response. The input is a list of proposed products and services, and the output is the data sent to the user's device.

[1396] Step 6:

[1397] The terminal displays the received proposal data in an easy-to-understand visual format. Specifically, it displays a list of product information on the smart device screen and offers the user options. In this step, the input is the proposal data sent from the server, and the output is the visual information presented to the user.

[1398] Step 7:

[1399] If a user is interested in the proposed product or service and wants to purchase it, he or she inputs the necessary information (e.g., payment information) through the terminal and sends it to the server. In this process, the input is the product information and payment information selected by the user, and the output is the purchase completion status.

[1400] Step 8:

[1401] The server synchronizes contract information in real time and prepares support services after purchase. The contract information is saved in the database and support services are initiated as needed. In this process, the input is the user's purchase information and the output is the support service preparation status.

[1402] Step 9:

[1403] The server provides real-time chat and video support. When a user has a question or problem, support staff are available to respond immediately. The input is the user's support request, and the output is real-time support information.

[1404] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1405] The system of the present invention analyzes input data from users, proposes relevant products or services, recognizes user emotions to optimize proposals, and provides post-contract support. A specific embodiment of this system will be explained below, with program processing explained in natural language and examples provided.

[1406] Accepting user input data

[1407] Users access the system using devices such as smartphones, tablets, and PCs. They input photos, images, text, and videos of their interests. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[1408] The terminal receives input data from the user, converts it into an appropriate format, and sends it to the server. For example, photo data in JPEG format can be sent as is.

[1409] Data analysis

[1410] The server runs algorithms to analyze the input data it receives. For photos and videos, image recognition technology is used to extract key features, such as color, shape, and pattern. For text, natural language processing (NLP) technology is used to analyze the user's intent. For example, a keyword like "blue dress" can be used to identify the color and type of item.

[1411] Furthermore, the server uses an emotion engine to recognize emotions from the user's input data, such as emotion analysis from text data, emotion analysis from audio data, and facial expression analysis from video data, to identify the user's current emotional state (e.g., joy, sadness, surprise).

[1412] Proposal Generation

[1413] Once the analysis is complete, the server searches the database for relevant products and services and selects those that best meet the user's needs. It then adjusts the recommendations based on the user's emotions, as determined by the emotion engine. For example, if the user is in a state of "joy" or "excitement," it will prioritize items with many positive product reviews.

[1414] The selected results are generated in multiple formats, including photos, text, video, etc. For example, if the search keyword is "blue dress," product information for multiple blue dresses will be provided along with photos.

[1415] View Suggestions

[1416] The server transmits the generated proposal data to the user's device in a format appropriate for the device.

[1417] The device then displays the received proposal data in a visually easy-to-understand format to the user. On smartphones, the proposal content is displayed on the app screen or via push notification, while on PCs, the proposal content is displayed in a new tab on the browser.

[1418] Post-contract support services

[1419] If a user is interested in a proposed product or service and wishes to sign a contract, they can easily complete the contract procedure through their device. For example, when a user purchases a dress on their smartphone, they simply click the purchase button and enter the necessary payment information to complete the purchase procedure.

[1420] After a purchase or contract is completed, the server synchronizes the contract information in real time and prepares to provide subsequent support services. The server provides support through chat and messaging functions, and also provides real-time support via voice and video calls. For example, if a user has a question about how to use or care for a purchased dress, they can ask through chat and receive an immediate answer.

[1421] Specific examples

[1422] For example, if a user is looking for a particular piece of furniture, the following process applies:

[1423] 1. The user takes a photo of the furniture they want to match with their home's interior using their smartphone and uploads it to the system.

[1424] 2. The device sends the photo data to the server.

[1425] 3. The server runs an image recognition algorithm to analyze the photo and extract features such as furniture type, color, and shape.

[1426] 4. The server uses an emotion engine to recognize the user's emotions from input data such as photos and text. If the user is recognized as being in a "lively" emotional state, casual, brightly colored furniture will be suggested.

[1427] 5. The server searches a database for relevant furniture and generates multiple options, such as a list of sofas and tables with photos.

[1428] 6. The server sends the generated suggestion data to the device, which displays it to the user, who then checks the search results on their smartphone.

[1429] 7. The user selects their favorite furniture from the suggested items and completes the purchase process.

[1430] 8. The server synchronizes the contract information and provides support services as needed, such as video guides on how to assemble furniture.

[1431] In this way, users can easily obtain products and services that suit them and receive comprehensive support even after signing a contract. In addition, the introduction of an emotion engine will make the user experience more personalized.

[1432] The processing flow will be explained below.

[1433] Step 1:

[1434] Users access the system using devices such as smartphones, tablets, and PCs. They input photos, images, text, and videos related to products and services they are interested in. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[1435] Step 2:

[1436] The terminal receives input data from the user, converts it into an appropriate format, and sends it to the server. For example, it sends photo data in JPEG format as is.

[1437] Step 3:

[1438] The server runs algorithms to analyze the input data it receives. For photos and videos, image recognition technology is used to extract key features, such as color, shape, and pattern. For text, natural language processing (NLP) technology is used to analyze the user's intent. For example, the color and type of item can be identified from keywords such as "blue dress."

[1439] Step 4:

[1440] Furthermore, the server uses an emotion engine to recognize emotions from the user's input data, such as emotion analysis from text data, emotion analysis from audio data, and facial expression analysis from video data, to identify the user's current emotional state (e.g., joy, sadness, surprise).

[1441] Step 5:

[1442] Based on the analysis results, the server searches its database for related products and services and selects those that best meet the user's needs. Furthermore, the server adjusts the recommendations based on the user's emotions, as recognized by the emotion engine. For example, if the user is in a state of "joy" or "excitement," it prioritizes items with many positive product reviews.

[1443] Step 6:

[1444] The server sends the generated proposal data to the user's device in a format appropriate for the device.

[1445] Step 7:

[1446] The device receives the proposal data and displays it to the user in an easy-to-understand visual format. On smartphones, the proposal content is displayed on the app screen or via push notification, while on PCs, the proposal content is displayed in a new tab on the browser.

[1447] Step 8:

[1448] The user becomes interested in the proposed products or services and makes a selection. The user then checks for more information about the selected product or service, such as detailed product specifications, price, and user reviews.

[1449] Step 9:

[1450] The user finally completes the purchase or contract process by clicking the purchase button and entering the required payment information, such as credit card information and shipping address.

[1451] Step 10:

[1452] The server receives purchase and contract information, stores it in a database, and synchronizes contract information in real time.

[1453] Step 11:

[1454] The server provides channels such as chat, messaging, voice calls, and video calls for post-contract support services. For example, if a user has a question about how to use a product, they can receive support through the chat function.

[1455] Step 12:

[1456] The server provides product usage guides and support materials, such as text guides and video guides, and sends them to the device, which then displays the guides to the user, who can then refer to them.

[1457] Example 2

[1458] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1459] Conventional recommendation systems may not be able to provide appropriate products or services because they do not accurately meet user needs or do not take into account the user's emotional state, which calls for an improved user experience. Furthermore, insufficient support services may be provided after the contract is signed, potentially reducing user satisfaction. Therefore, a system is needed that can accurately grasp the user's needs, make optimal recommendations based on the user's emotional state, and provide comprehensive support services after the contract is signed.

[1460] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1461] In this invention, the server includes a means for accepting input data from a user, a means for analyzing the accepted input data, and a means for proposing related products or services based on the analysis results and the user's emotions. This allows the server to accurately grasp the user's needs and make optimal proposals that take into account the user's emotional state. Furthermore, the server can improve user satisfaction by displaying the proposed products or services on the user's terminal and providing post-contract support services.

[1462] "User" refers to a person who uses the system to search for, propose, and contract for goods and services.

[1463] "Input data" refers to information provided by a user to the system, and includes various forms such as photographs, text, and video.

[1464] A "terminal" is a hardware device that a user uses to access the system, and includes a mobile terminal, a computer, and the like.

[1465] "Server" refers to the computer system that acts as the central part of the system, analyzing input data, generating proposals, communicating with the database, and transmitting proposals.

[1466] "Image recognition technology" refers to the technology of extracting specific features (color, shape, pattern, etc.) from photographs and videos.

[1467] "Natural language processing technology" refers to technology that analyzes user intent and meaning from text data.

[1468] "Emotions" are internal states that users express through input data, and include joy, sadness, surprise, etc.

[1469] "Suggestions" refers to information about products and services selected based on the analysis results and user sentiment.

[1470] "Support services" refers to the support a user receives after signing a contract for a proposed product or service, and includes support via chat, voice calls, and video calls.

[1471] The system of the present invention analyzes input data from a user, proposes related products or services, recognizes the user's emotions to optimize the proposals, and also provides post-contract support services. A specific embodiment of this system will be described.

[1472] Accepting user input data

[1473] Users access the system using devices such as smartphones, tablets, and PCs. They input photos, text, and videos of their interests. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system.

[1474] Transfer of input data

[1475] The device receives the data entered by the user, converts it into an appropriate format, and sends it to the server. For example, it sends JPEG photo data as is. In the case of text, it sends the entered keywords as is.

[1476] Data analysis

[1477] The server runs an algorithm to analyze the received input data, specifically using the following techniques:

[1478] Image recognition technology: Algorithms such as YOLO (You Only Look Once) are used to extract key features (color, shape, pattern, etc.) from photos and videos. For example, identifying the features of a "blue dress."

[1479] Natural Language Processing (NLP) techniques: Use NLP models such as BERT (Bidirectional Encoder Representations from Transformers) to analyze user intent from text data. For example, analyze the keyword "blue dress" to identify the color and type of item.

[1480] Emotion Engine: Uses Python's Emotion API to recognize user emotions from text, audio, and video, thereby identifying the user's emotional state (e.g., happy, sad, surprised).

[1481] Proposal Generation

[1482] The server searches the database for relevant products and services and generates optimal suggestions based on the analysis results and the recognized emotions. The server works as follows:

[1483] Product information collection: Pulling relevant product or service information (such as name, price, reviews, and photos) from a database (e.g., a MySQL database).

[1484] Formation of proposal content: Based on the collected information, proposal data is generated in formats such as CSV, JSON, HTML, etc. For example, product information for multiple "blue dresses" is provided along with photos.

[1485] View Suggestions

[1486] The server sends the generated proposal data to the user's device. The device then displays the proposal data in a visually easy-to-understand format. For example, on a smartphone, the proposals are displayed on the app screen or via push notification, while on a PC, they are displayed in a new tab on the browser. React Native is used to generate the smartphone app UI and display the proposals.

[1487] Post-contract support services

[1488] If a user is interested in the proposed products or services and wishes to sign a contract, they can complete the contract procedure through their device. After the purchase procedure is completed, the server synchronizes the contract information in real time and prepares to provide subsequent support services. For example, the Twilio API can be used to implement support functions such as chat, voice calls, and video calls.

[1489] Specific examples

[1490] For example, if a user is looking for a particular piece of furniture, the following process applies:

[1491] The user takes a photo of the furniture they want to match with their home interior using their smartphone and uploads it to the system. The device sends the photo data to the server. The server runs an image recognition algorithm (e.g., YOLO) to analyze the photo and extract features such as the type, color, and shape of the furniture. The server uses an emotion engine (e.g., Emotion API) to recognize the user's emotions. If the user is recognized as being in a "lively" emotional state, casual, brightly colored furniture will be prioritized and suggested. The server searches for related furniture from its database and generates multiple options. The server sends the generated suggestion data to the device, which displays it to the user. The user checks the search results on their smartphone, selects the furniture they like, and completes the purchase process. The server synchronizes contract information and provides support services as needed (e.g., sending video guides on how to assemble the furniture).

[1492] Prompt Sentence Examples

[1493] "Explain how a system can provide product recommendations based on a user-uploaded photo of a blue dress."

[1494] "Please outline a system that analyzes user sentiment and optimizes product recommendations based on that sentiment."

[1495] In this way, users can easily obtain products and services that suit them and receive comprehensive support even after signing a contract. In addition, the introduction of an emotion engine will make the user experience more personalized.

[1496] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1497] Step 1:

[1498] Users access the system using devices such as smartphones, tablets, and PCs. They input photos, text, and videos of their interest. For example, if a user is looking for new clothing, they can take a photo of an outfit they like with their smartphone and upload it to the system. Inputs include photo data (e.g., in JPEG format) and text data (e.g., "blue dress").

[1499] Step 2:

[1500] The terminal receives data entered by the user, converts the format appropriately, and sends it to the server. Specifically, it sends JPEG photo data and text data directly to the server. The input is the photo data and text data received from the user, and the output is the data sent to the server.

[1501] Step 3:

[1502] The server runs an algorithm to analyze the received input data, using the following techniques:

[1503] Image recognition technology: For example, algorithms such as YOLO (You Only Look Once) are used to extract the main features of a photo (color, shape, pattern, etc.). The input is JPEG photo data, and the output is the extracted feature information (e.g., "blue color," "dress shape").

[1504] Natural Language Processing (NLP) technology: For example, an NLP model such as BERT (Bidirectional Encoder Representations from Transformers) is used to analyze user intent from text data. The input is the text data, and the output is the analysis result (e.g., "blue", "dress").

[1505] Emotion engine: For example, Python's Emotion API is used to recognize the user's emotional state from text, audio, and video. The input is text data, and the output is emotional state information (e.g., "joy" or "excitement").

[1506] Step 4:

[1507] The server then suggests relevant products and services based on the analysis results and the recognized emotions. Specifically, it:

[1508] Product information collection: Extract relevant product and service information (such as name, price, reviews, and photos) from a database (e.g., MySQL database). The input is the analysis results and sentiment information, and the output is product information.

[1509] Proposal content generation: Based on the collected product information, proposal data is generated in formats such as CSV, JSON, and HTML. For example, product information for multiple "blue dresses" is provided along with photos. The input is product information, and the output is proposal data.

[1510] Step 5:

[1511] The server sends the generated proposal data to the terminal. The input is the generated proposal data, and the output is the data sent to the terminal.

[1512] Step 6:

[1513] The device receives the proposal data and displays it visually to the user. Specifically, on a smartphone, the proposal is displayed on the app screen or via push notification, and on a PC, the proposal is displayed by opening a new tab in the browser. The input is the proposal data sent from the server, and the output is the information displayed to the user.

[1514] Step 7:

[1515] If a user is interested in a proposed product or service, they can complete the contract process through their device. For example, they can complete the purchase process by simply clicking the "Purchase" button on their smartphone and entering the necessary payment information. The input is the proposal data and contract information, and the output is the completed contract.

[1516] Step 8:

[1517] The server synchronizes the contract information in real time and prepares to provide subsequent support services. Specifically, it uses the Twilio API to implement chat, voice call, and video call support functions. The input is the contract information, and the output is the support service provided.

[1518] (Application example 2)

[1519] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1520] In modern e-commerce, users have difficulty finding the best products and services for them from the vast number of options available. Furthermore, product suggestions that do not match the user's emotional state make it difficult to stimulate their desire to purchase. Furthermore, post-purchase support is often inadequate, resulting in a poor user experience. A system that can solve these problems and improve user satisfaction is needed.

[1521] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1522] In this invention, the server includes means for accepting input data from a user, means for analyzing the accepted input data, means for proposing related products or services based on the analysis results and the user's emotional state, means for displaying the proposed products or services on the user's terminal, and means for providing post-contract support services. This makes it possible to analyze a variety of input data from the user and propose optimal products and services that match the user's emotional state. Furthermore, by providing enhanced post-contract support, user satisfaction can be improved.

[1523] The "means for accepting input data from a user" is an interface that allows a user to provide a photograph, image, text, audio, or video to the system.

[1524] "Means for analyzing received input data" refers to algorithms and techniques for analyzing user-provided input data and extracting data features and emotional states.

[1525] "Means for suggesting relevant products or services based on the analysis results and the user's emotional state" refers to a mechanism by which the system selects and suggests the most appropriate products or services, taking into account the analyzed data and the user's emotional state.

[1526] "Means for displaying suggested products or services on a user terminal" refers to a method by which the system displays the suggested content on the user's smartphone, tablet, personal computer, or smart glasses.

[1527] The "means for providing post-contract support services" is a function for providing support and after-sales services after the user purchases or contracts for the proposed product or service.

[1528] "Photos, images, text, audio, or video" are any form of visual or audio data that a user provides to the system.

[1529] "Means for extracting features and emotional states" refers to techniques for identifying key features (e.g., color, shape) and emotional states (e.g., happy, sad) from user-provided data.

[1530] "User terminal is a smartphone, tablet, personal computer or smart glasses" refers to the electronic device at the user's side that is used to display the system's proposals.

[1531] The system for implementing this invention accepts and analyzes various input data from users and proposes optimal products and services based on the results. It also analyzes the user's emotional state and personalizes the proposals. It also provides post-contract support services to improve user satisfaction.

[1532] Hardware and Software

[1533] Hardware

[1534] Smartphones, tablets, personal computers, smart glasses: electronic devices used by users to provide input data and receive suggestions.

[1535] software

[1536] Image analysis: Uses the Google Cloud Vision API to analyze user-uploaded photos and images and extract features.

[1537] Natural Language Processing (NLP): Uses the Google Cloud Natural Language API to analyze user-entered text and recognize intent and sentiment.

[1538] Emotion Analysis: Uses the Microsoft Azure Emotion API to identify the user's emotional state from audio and video data.

[1539] Database: Use Firebase Realtime Database to retrieve relevant product and service data based on analytics results.

[1540] Front-end: Build the user interface using React Native.

[1541] Example

[1542] An example is given below where the user is looking for a new dress.

[1543] 1. User uploads a photo

[1544] Users use their smartphones to take photos of the dresses and upload them to the application.

[1545] 2. Data Analysis

[1546] The server analyzes uploaded photos using the Google Cloud Vision API to extract features such as the color, shape, and design of the dress, and also analyzes the user's input text and voice using the Google Cloud Natural Language API and Microsoft Azure Emotion API to recognize the user's emotional state.

[1547] 3. Related product suggestions

[1548] The server retrieves relevant product data from the Firebase Realtime Database and selects the most suitable products based on the emotional state. For example, if the emotional state is "joy," it will prioritize dresses with high reviews.

[1549] 4. Display of proposed data

[1550] The suggested products are visually displayed on smartphones and tablets using React Native, and users can choose from the suggested products.

[1551] 5. Post-contract support services

[1552] After the user purchases the selected product, the server stores the purchase information in the Firebase Realtime Database and provides subsequent support services, such as real-time chat support and instructional videos, for any questions about the product.

[1553] Prompt Sentence Examples

[1554] Below is an example of a prompt sentence that can be input into the "generative AI model."

[1555] Analyze photos, extract their features, and generate relevant product suggestions based on the user's emotional state.

[1556] Input: Photo of a blue dress

[1557] Emotion: Joy

[1558] Output: Related blue dress product list and reviews

[1559] In this way, the system of the present invention can propose products and services that accurately reflect the user's needs and emotional state, improving the user experience. Furthermore, the system also provides comprehensive support services after the contract is signed, thereby increasing user satisfaction.

[1560] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1561] Step 1:

[1562] User uploads a photo

[1563] A user takes a photo of a product they want to purchase using their smartphone and uploads it to the application. The input is a JPEG photo, and the output is a notification that the photo has been uploaded. The device receives the photo data from the user and transfers it to the server.

[1564] Step 2:

[1565] Sending data to the server

[1566] The terminal sends JPEG format photo data to the server. The input is a JPEG format photo, and the output is the photo data sent to the server. The terminal performs this process automatically, and the user is not required to perform any special operations.

[1567] Step 3:

[1568] Analyzing photo data

[1569] The server analyzes the received photo data using the Google Cloud Vision API. The input is JPEG photo data, and the output is feature information such as the color, shape, and design of the product. The server runs an image recognition algorithm to extract important features.

[1570] Step 4:

[1571] Analyzing the user's emotional state

[1572] The server uses the Google Cloud Natural Language API and Microsoft Azure Emotion API to analyze the user's input text and voice data. The input is text and voice data, and the output is the user's emotional state information (e.g., joy, surprise, etc.). The server runs an emotion analysis algorithm to identify the user's emotional state.

[1573] Step 5:

[1574] Search and suggest related products

[1575] The server uses Firebase Realtime Database to search for relevant product data based on the analyzed features and the user's emotional state. The input is feature information and emotional state information, and the output is a list of relevant products. The server generates recommendations by prioritizing products according to the emotional state.

[1576] Step 6:

[1577] Send the proposal data to the user device

[1578] The server sends the generated proposal data to the user's terminal. The input is a list of related products, and the output is a notification that the proposal data has been sent to the user's terminal. The server sends the data to the terminal in an appropriate format.

[1579] Step 7:

[1580] View suggested data

[1581] The terminal visually displays the received suggestion data to the user. The input is the suggestion data, and the output is the display screen viewed by the user. The terminal uses React Native to provide a visually easy-to-understand interface.

[1582] Step 8:

[1583] Providing post-contract support

[1584] After the user purchases the recommended product, the server synchronizes the contract information and provides support services. The input is the purchase information, and the output is the support start notification and support content. The server supports the user through support functions such as real-time chat and instructional videos.

[1585] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1586] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1587] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1588] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1589] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1590] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1591] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1592] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1593] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1594] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1595] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1596] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1597] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1598] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1599] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1600] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1601] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1602] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1603] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1604] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1605] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1606] The following is further disclosed regarding the above embodiment.

[1607] (Claim 1)

[1608] means for accepting input data from a user;

[1609] means for analyzing the received input data;

[1610] A means for suggesting related products or services based on the analysis results;

[1611] means for displaying the proposed product or service on a user terminal;

[1612] a means of providing post-contract support services;

[1613] A system including:

[1614] (Claim 2)

[1615] 10. The system of claim 1, further comprising means for analyzing input data from a user when the input data is a photograph, an image, text, or video.

[1616] (Claim 3)

[1617] 2. The system of claim 1, further comprising means for displaying the suggestions on the user terminal when the terminal is a smartphone, a tablet, or a personal computer.

[1618] "Example 1"

[1619] (Claim 1)

[1620] means for accepting input data from a user;

[1621] a means for converting accepted input data into an appropriate format;

[1622] means for transmitting the converted data to a server;

[1623] a means for receiving the data at the server and verifying its format and content;

[1624] means for analyzing received data;

[1625] a means for searching for related products or services based on the analysis results;

[1626] A means for generating optimal suggestions based on search results;

[1627] means for transmitting the generated proposal to a user terminal;

[1628] means for displaying the suggested goods or services on a user terminal;

[1629] A means for inputting information required for contract procedures;

[1630] means for transmitting contract information to a server and completing contract procedures;

[1631] a means of providing post-contract support services;

[1632] A system including:

[1633] (Claim 2)

[1634] 10. The system of claim 1, further comprising means for analyzing input data from a user when the input data is a photograph, an image, text, or video.

[1635] (Claim 3)

[1636] 2. The system of claim 1, further comprising means for displaying the suggestions on the user terminal when the terminal is a smartphone, a tablet, or a personal computer.

[1637] "Application Example 1"

[1638] (Claim 1)

[1639] means for accepting input data from a user;

[1640] means for analyzing the received input data;

[1641] A means for suggesting related products or services based on the analysis results;

[1642] means for displaying the proposed product or service on a user terminal;

[1643] a means of providing post-contract support services;

[1644] a means for generating prompt sentences to optimize interpretation of input data;

[1645] A means to provide real-time chat and video support based on the analysis results;

[1646] means for analyzing the received input data using image recognition technology and natural language processing technology;

[1647] A system including:

[1648] (Claim 2)

[1649] 10. The system of claim 1, further comprising means for analyzing input data from a user when the input data is a photograph, an image, text, or video.

[1650] (Claim 3)

[1651] 2. The system of claim 1, further comprising means for displaying the suggestions on the user terminal when the terminal is a smart device or a computer.

[1652] "Example 2: Combining Emotion Engines"

[1653] (Claim 1)

[1654] means for accepting input data from a user;

[1655] means for analyzing the received input data;

[1656] means for suggesting relevant products or services based on the analysis results and user sentiment;

[1657] means for displaying the proposed product or service on a user terminal;

[1658] a means of providing post-contract support services;

[1659] A system including:

[1660] (Claim 2)

[1661] 2. The system according to claim 1, further comprising means for analyzing input data from a user using image recognition technology and natural language processing technology when the input data is a photograph, text, or video.

[1662] (Claim 3)

[1663] 2. The system of claim 1, further comprising means for displaying the suggestions on the user terminal when the terminal is a mobile terminal or a computer.

[1664] "Application example 2 when combining emotion engines"

[1665] (Claim 1)

[1666] means for accepting input data from a user;

[1667] means for analyzing the received input data;

[1668] means for suggesting relevant products or services based on the analysis results and the user's emotional state;

[1669] means for displaying the proposed product or service on a user terminal;

[1670] a means of providing post-contract support services;

[1671] A system including:

[1672] (Claim 2)

[1673] 10. The system of claim 1, further comprising means for analyzing input data from a user and extracting features and emotional states when the input data is a photograph, an image, text, audio, or video.

[1674] (Claim 3)

[1675] 10. The system of claim 1, further comprising means for displaying the suggestions on a user terminal when the terminal is a...

Claims

1. means for accepting input data from a user; means for analyzing the received input data; A means for suggesting related products or services based on the analysis results; means for displaying the proposed product or service on a user terminal; a means of providing post-contract support services; A system including:

2. 2. The system of claim 1, further comprising means for analyzing input data from a user when the input data is a photograph, an image, text, or video.

3. The system of claim 1 further comprising means for displaying the suggestions on the user terminal when the user terminal is a smartphone, a tablet, or a personal computer.

Images