system

Abstract

Provide a system. [Solution] means for receiving inquiries from users regarding tour plans; A means for analyzing the content of a user's question and extracting keywords; A means to query databases or external APIs based on the extracted keywords to retrieve relevant data; means for aggregating the retrieved data and generating a response to return to the user; means for transmitting the generated response to the user's terminal; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, when planning a trip, users need to gather a lot of information and select appropriate tourist spots and restaurants. However, with the vast amount of information available on the Internet, it is not easy to find the best one for you. Furthermore, integrating information from different platforms and databases increases the time and effort required for users. Therefore, there is a need for a system that allows users to create tour plans easily and quickly. [Means for solving the problem]

[0005] In order to solve the above problems, the present invention provides a system including: means for receiving a question about a tour plan from a user; means for analyzing the content of the user's question and extracting keywords; means for sending a query to a database or an external API based on the extracted keywords to obtain relevant data; means for integrating the obtained data and generating a response to return to the user; and means for sending the generated response to the user's terminal. This system enables users to easily obtain the information necessary for travel planning and quickly create the optimal tour plan for themselves.

[0006] "User" means an individual or organization that uses the system and requests information about tour plans.

[0007] A "means" is a method, device, or process used to accomplish a particular purpose.

[0008] A "question" is a statement or phrase that a user enters through the system requesting specific information.

[0009] "Analysis" is the process of breaking down input information and understanding its structure and meaning.

[0010] "Keywords" are words or phrases extracted from the content of a question that play an important role in search and information retrieval.

[0011] A "database" is a collection of information for efficiently storing, retrieving, and managing structured data.

[0012] An "external API" is an interface for exchanging information with other systems or services, and is managed by an external provider.

[0013] A "query" is a search request made to a database or external API, a command statement to retrieve specific data.

[0014] A "response" is information that responds to a user's question and is generated based on the results of analysis and data acquisition.

[0015] A "terminal" is an electronic device such as a computer, smartphone, or tablet that is directly operated by a user.

[0016] "Natural language processing algorithms" are a set of techniques and methods that allow machines to understand and analyze human language. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The system of the present invention is designed to enable users to create tour plans easily and quickly. This system allows users to input questions about the tour plan into a terminal, and provides information on appropriate tourist spots and restaurants based on the questions. Specific embodiments of this system are described below.

[0039] This system consists of three main components: the user, the terminal, and the server. The user accesses the system through the terminal and inputs questions related to the tour plan they want. For example, the user might input, "Please tell me the recommended tourist spots and restaurants in Tokyo."

[0040] When the device receives user input, it sends the input to the server. The client application on the device captures the user's question in text format, converts it to JSON format, and sends it as an HTTP request to the server. The server receives this request and begins analyzing it.

[0041] The server first parses the received JSON data and uses a natural language processing algorithm (NLP) to analyze the user's question. At this stage, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted from the question. These keywords are used as queries to retrieve related data.

[0042] Next, the server sends queries to a database or external API based on the extracted keywords. For example, the server queries an external API for "popular tourist spots in Tokyo" and receives a list of tourist spots in response. Similarly, it retrieves data on "popular restaurants in Tokyo."

[0043] Based on the acquired data, the server generates a response to return to the user. This response may include tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza." In this way, the server creates a list of information that best suits the user's question and formats it in a format that is easy for the user to understand.

[0044] The generated response is converted back to JSON format and sent to the device as an HTTP response. When the device receives this response, the client application parses the data and formats it into a layout to display to the user. Finally, the user can see a list of tourist attractions and restaurants on the device screen.

[0045] For example, a user receives the following tour plan information:

[0046] Tourist attractions: Tokyo Tower, Sensoji Temple, Skytree

[0047] Restaurants: Tsukiji Market, Odaiba restaurants, Ginza sushi restaurants

[0048] In this way, users can easily obtain the information they need to plan their trip and quickly create the tour plan that best suits them, reducing the burden on users and supporting comfortable trip planning.

[0049] The processing flow will be explained below.

[0050] Step 1:

[0051] The user types into the terminal, "Please tell me some recommended tourist spots and restaurants in Tokyo."

[0052] Step 2:

[0053] The terminal captures the user's input in text format and converts it to JSON format.

[0054] Step 3:

[0055] The terminal sends the converted JSON data to the server as an HTTP request.

[0056] Step 4:

[0057] The server receives the JSON data, parses it, and extracts the user's question.

[0058] Step 5:

[0059] The server uses a natural language processing algorithm (NLP) to analyze the question and extract important keywords (e.g., "Tokyo," "tourist spots," "restaurants").

[0060] Step 6:

[0061] The server sends a query to a database or external API based on the extracted keywords. For example, it sends a query such as "popular tourist spots in Tokyo."

[0062] Step 7:

[0063] The server receives the response data to the query and obtains information about tourist spots and restaurants.

[0064] Step 8:

[0065] The server integrates the acquired data and generates a response to return to the user. For example, it creates a list of tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as a list of restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza."

[0066] Step 9:

[0067] The response generated by the server is converted to JSON format and sent to the terminal as an HTTP response.

[0068] Step 10:

[0069] The device parses the JSON data received from the server and formats it into a layout to display to the user.

[0070] Step 11:

[0071] Users can view a list of tourist attractions and restaurants on their device screen.

[0072] Example 1

[0073] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0074] Conventional travel planning systems have had the problem of requiring a great deal of time and effort when users research trip details. In particular, the task of collecting and integrating necessary information from multiple sources is extremely time-consuming. It is also difficult to provide accurate information that meets the user's needs. This tends to make users' travel planning complicated and inefficient.

[0075] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0076] In this invention, the server includes means for receiving a plan-related question from a user, means for analyzing the content of the user's question and extracting keywords, means for sending a query to information sources based on the extracted keywords to obtain related data, means for integrating the obtained data and generating a response to return to the user, and means for transmitting the generated response to the user's terminal, thereby enabling the user to quickly and accurately obtain information necessary for planning and easily create a travel plan.

[0077] "Plan" is a plan that allows a user to organize and execute details of trips and activities.

[0078] A "question" is text or a query that a user enters to request information about a plan.

[0079] "Analysis" is the process of understanding the user's question and identifying important elements and information.

[0080] "Keywords" are important words or phrases extracted from a user's question that serve as the basis for conducting a search or query.

[0081] An "information source" is a data provider, such as a database or external API, that is used to obtain the required information.

[0082] A "query" is a request sent to a source to search for or retrieve specific information.

[0083] "Data" refers to the information and responses related to a user's question.

[0084] A "response" is information or a reply provided to a user's question, and includes content that is useful to the user.

[0085] A "terminal" is a device that a user uses to access and operate the system. Examples include a smartphone or computer.

[0086] A "natural language processing algorithm" is a technology for analyzing text data and understanding its meaning and structure, and is used to accurately analyze the content of user questions.

[0087] The "JSON format" is a text format for structuring and expressing data, and is a format that is often used for sending and receiving data.

[0088] An "HTTP request" is a protocol used when a client requests information from a server.

[0089] An "HTTP response" is a protocol used when a server returns information to a client.

[0090] The system of the present invention is designed to enable users to create plans quickly and accurately. This system allows users to input questions about the plan into a terminal, and provides appropriate information based on the questions. Specific embodiments of this system are described below.

[0091] Hardware and software used

[0092] This system consists of three main components: the user, the terminal, and the server. The terminal can be a smartphone or a computer. The server is a server computer operated in a cloud environment or on-premise. The software includes client applications, server-side programs, natural language processing algorithms (NLP tools), and various APIs.

[0093] Explanation of program processing

[0094] 1. User Input

[0095] The user uses the client application on the terminal to input a question about the plan. For example, the user inputs, "Please tell me recommended tourist spots and restaurants in Tokyo."

[0096] 2. Data transmission by the terminal

[0097] The terminal captures the user's question in text format and converts it into JSON format. The converted JSON data is sent to the server as an HTTP request. The software used for this process is a request library such as fetch or axios.

[0098] 3. Data analysis by the server

[0099] The server parses the received JSON data and analyzes the question using natural language processing (NLP) algorithms, for example by extracting keywords using libraries such as spaCy or NLTK.

[0100] For example, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted. These keywords are used as queries to search for related information.

[0101] 4. Acquiring relevant data

[0102] Based on the extracted keywords, the server sends queries to databases or external APIs to retrieve relevant data. For example, it uses the requests library to access an API and retrieve information about tourist attractions and restaurants.

[0103] 5. Generating and Sending the Response

[0104] The server generates a response based on the acquired data. This response includes information on tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza." The generated response is converted to JSON format and sent to the device as an HTTP response.

[0105] 6. Displaying data on a terminal

[0106] The client application on the device parses the received response and displays it in a user-friendly format, using HTML and CSS to format the layout.

[0107] Specific examples

[0108] For example, a user enters the following question:

[0109] > "What are some recommended tourist spots and restaurants in Tokyo?"

[0110] In response to this question, the system can return information such as:

[0111] Tourist attractions:

[0112] Tokyo Tower

[0113] Sensoji Temple

[0114] Skytree

[0115] Restaurant:

[0116] Tsukiji Market

[0117] Restaurants in Odaiba

[0118] Sushi restaurant in Ginza

[0119] In this way, the user can quickly and easily obtain the necessary information and create an optimal plan.

[0120] The system of the present invention reduces the burden on the user and makes it possible to support comfortable planning.

[0121] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0122] Step 1:

[0123] The user inputs a question into the terminal. The user opens the client application on the terminal and inputs a question about the plan. For example, the user inputs, "Please tell me recommended tourist spots and restaurants in Tokyo." This input text becomes input data for the next processing step.

[0124] Step 2:

[0125] The device sends a question to the server. The device captures the question text from the user in text format and converts it to JSON format. This converted data becomes the body of an HTTP request and is sent to the server. Specifically, the device uses a request library such as fetch or axios to send the data. The output of this step is JSON format data that the server can process.

[0126] Step 3:

[0127] The server analyzes the question. The server parses the received JSON data and uses a natural language processing algorithm (NLP) to analyze the question. This analysis process uses libraries such as spaCy and NLTK to tokenize the text and extract keywords. The input for this process is the JSON data, and the output is a list of extracted keywords. For example, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted.

[0128] Step 4:

[0129] The server retrieves relevant data from a database or external API. Based on the extracted keywords, the server creates a database query or external API query to retrieve the required data. Specifically, it uses the requests library to access the API and retrieve information about tourist attractions and restaurants. The input for this step is the keywords, and the output is a list of the retrieved tourist attractions and restaurants.

[0130] Step 5:

[0131] The server generates a response. Based on the acquired data, the server generates a response to return to the user. This response includes a list of tourist spots and restaurants. For example, tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree" and restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi Restaurants in Ginza" are listed. The input of this step is the acquired data, and the output is the formatted response data.

[0132] Step 6:

[0133] The server sends the response to the terminal. The generated response data is converted back to JSON format and sent to the terminal as an HTTP response. Specifically, the server generates the HTTP response using a web framework (e.g., Flask). The input of this step is the formatted response data, and the output is JSON-formatted response data.

[0134] Step 7:

[0135] The device displays the response. The device's client application parses the received response and displays it in a format that is easy for the user to understand. Specifically, it formats the data using HTML and CSS and provides it to the user. The input to this step is the response data in JSON format, and the output is a displayed list of tourist attractions and restaurants.

[0136] By performing the above steps, the system can quickly and accurately provide the necessary information based on the user's question.

[0137] (Application example 1)

[0138] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0139] Conventional tour planning systems only provide users with information in a static format, making it difficult to provide intuitive understanding or a realistic experience. In particular, there is a lack of technology for providing tour planning information visually and interactively, which has led to issues such as reduced convenience and satisfaction for users.

[0140] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0141] In this invention, the server includes means for receiving a question about a tour plan from a user, means for analyzing the content of the user's question and extracting keywords, means for sending a query to a database or an external API based on the extracted keywords to obtain related data, means for integrating the obtained data and generating a response to return to the user, means for sending the generated response to the user's terminal, and means for reproducing the obtained data in a virtual environment using a head-mounted display. This allows the user to visually experience tourist spots and restaurants in the virtual environment, enabling them to create a more intuitive and realistic tour plan.

[0142] "User" refers to a person who uses the system to ask questions or obtain information about tour plans.

[0143] A "tour plan question" refers to an inquiry from a user to ask about tourist attractions and restaurant information for a particular travel destination.

[0144] "Keywords" refer to important words or phrases extracted from the content of a user's question and are used as search queries.

[0145] A "database" refers to a collection of information about tourist spots and restaurants.

[0146] An "external API" refers to an interface for obtaining data from other systems or services.

[0147] A "query" is a search request sent to a database or external API requesting information.

[0148] A "response" refers to a set of information that a system returns in response to a question from a user.

[0149] "Terminal" refers to a device through which a user accesses the system, including smartphones, tablets, and PCs.

[0150] "Head-mounted display" refers to a display device worn by a user to visually experience a virtual environment.

[0151] A "virtual environment" refers to a virtual space constructed using computer technology that users can experience visually and interactively.

[0152] The embodiment of this invention is a system that allows users to easily and quickly create a tour plan and visually experience it in a virtual environment. The system mainly consists of a user, a terminal, a server, and a head-mounted display.

[0153] First, a user accesses the system through a terminal and inputs a question about a tour plan. For example, a user might input a question like, "Please tell me the recommended tourist spots and restaurants in Tokyo."

[0154] The device captures the user's question in text format, converts it to JSON format, and sends it to the server as an HTTP request. The server receives this request and begins analyzing it. The natural language processing algorithm used here is spaCy, for example.

[0155] The server parses the received JSON data and analyzes the user's question. Through analysis, it extracts keywords such as "Tokyo," "tourist spots," and "restaurants." These keywords are used as queries to retrieve related data. The server then sends queries to databases or external APIs based on the extracted keywords to retrieve data on tourist spots and restaurants.

[0156] The server generates a response based on the acquired data. This response includes tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza." The generated response is again converted to JSON format and sent to the device as an HTTP response.

[0157] When the device receives this response, it parses the data and formats it into a layout for display to the user. However, a distinctive feature of the system is that it uses a head-mounted display (HMD) to recreate the acquired data in a virtual environment. This virtual environment is built using a game engine such as Unity.

[0158] By wearing the HMD, users can visually experience the tourist spots and restaurants provided in the virtual environment, allowing them to check and plan their tour plans as if they were actually visiting the places.

[0159] As a concrete example, if a user asks the device, "What are some recommended tourist spots and restaurants in Tokyo?", the system will recreate tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza" in the virtual environment. In this process, the following prompt sentence can be input to the generative AI model: "What are some recommended tourist spots and restaurants in Tokyo?"

[0160] In this way, users can visually receive tour plan suggestions and plan their trip through a virtual experience on the spot. The hardware used includes head-mounted displays such as Oculus Rift and HTC Vive. The software used includes Unity, Python, spaCy, and external APIs.

[0161] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0162] Step 1:

[0163] The user inputs a question about the tour plan into the terminal. For example, the user inputs the text "Please tell me the recommended tourist spots and restaurants in Tokyo."

[0164] Step 2:

[0165] The terminal receives the user's question in text format and converts it to JSON format for subsequent server processing. The input value is the user's question text, and the output is a JSON object.

[0166] Step 3:

[0167] The terminal sends the converted JSON data to the server as an HTTP request. This request includes the user's question and is sent to the server's URL. The input value is a JSON object, and the output is an HTTP request.

[0168] Step 4:

[0169] The server receives the HTTP request and parses the JSON data. During this process, the parsed data is converted into a structured format and the user's question is analyzed. The input value is the JSON data of the HTTP request, and the output is structured data.

[0170] Step 5:

[0171] The server uses a natural language processing (NLP) algorithm to extract keywords from the data analyzed. In this process, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted from the user's question. The input is structured data, and the output is a set of extracted keywords.

[0172] Step 6:

[0173] The server uses the extracted keywords to send queries to databases or external APIs to obtain the required information. Specifically, queries to obtain data on tourist attractions and restaurants are created and sent to the API or database. The input is a set of keywords, and the output is data on tourist attractions and restaurants.

[0174] Step 7:

[0175] The server generates a response to return to the user based on the data it has acquired. The response includes information about tourist spots and restaurants, and is formatted in a way that is easy for the user to understand. The input is the acquired data, and the output is the formatted response.

[0176] Step 8:

[0177] The server converts the generated response back into JSON format and sends it as an HTTP response to the terminal. The input is a formatted response, and the output is a JSON-formatted HTTP response.

[0178] Step 9:

[0179] The device receives the JSON data returned from the server, parses the data, and displays it to the user. This display is not only displayed on the device screen, but also reproduced in a virtual environment via a head-mounted display (HMD). The input is the JSON data of the HTTP response, and the output is the visual information displayed to the user.

[0180] Step 10:

[0181] By wearing an HMD, users can visually experience tourist spots and restaurants in a virtual environment, giving them the feeling that they are actually visiting those places. The input is the visual information in the virtual environment, and the output is the user's experience.

[0182] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0183] The system of the present invention is designed to enable users to create tour plans easily and quickly. The system allows users to input questions about the tour plan into a terminal, and provides information on appropriate tourist spots and restaurants based on the questions. Furthermore, the system incorporates an emotion engine that recognizes the user's emotions, allowing it to provide appropriate responses based on the user's emotions.

[0184] This system consists of four main components: the user, the terminal, the server, and the emotion engine. The user accesses the system through the terminal and inputs questions about the tour plan they want. For example, consider the case where a user inputs, "Please tell me recommended tourist spots and restaurants in Tokyo."

[0185] When the device receives user input, it sends the input to the server. The client application on the device captures the user's question in text format, converts it to JSON format, and sends it as an HTTP request to the server. The server receives this request and begins analyzing it.

[0186] The server first parses the received JSON data and uses a natural language processing algorithm (NLP) to analyze the user's question. At this stage, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted from the question. Next, an emotion engine is used to analyze the user's emotions from the text input. This emotion analysis provides information such as whether the user is happy or anxious.

[0187] Next, the server sends a query to a database or external API based on the extracted keywords and the results of the sentiment analysis. For example, the server queries an external API for "popular tourist spots in Tokyo" and receives a list of tourist spots in response. Similarly, it retrieves data on "popular restaurants in Tokyo." In addition, it adjusts the response content to match the user's emotions based on the results of the sentiment analysis. For example, if the user is feeling anxious, it adds reassuring language to the response.

[0188] Based on the retrieved data, the server generates a response to return to the user. This response might include tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza." This information is then formatted and summarized in a form that is easy for the user to understand.

[0189] The generated response is converted back to JSON format and sent to the device as an HTTP response. When the device receives this response, the client application parses the data and formats it into a layout to display to the user. Finally, the user can see a list of tourist attractions and restaurants on the device screen.

[0190] For example, a user receives the following tour plan information:

[0191] Tourist attractions: Tokyo Tower, Sensoji Temple, Skytree

[0192] Restaurants: Tsukiji Market, Odaiba restaurants, Ginza sushi restaurants

[0193] In addition, responses are adjusted based on sentiment analysis, so if the user is feeling anxious, the phrase "Enjoy with peace of mind" can be added, increasing the user's sense of trust.

[0194] In this way, users can easily obtain the information they need for travel planning and quickly create the tour plan that best suits them. Sentiment analysis provides responses that are more tailored to the user's needs, improving user satisfaction.

[0195] The processing flow will be explained below.

[0196] Step 1:

[0197] The user types into the terminal, "Please tell me some recommended tourist spots and restaurants in Tokyo."

[0198] Step 2:

[0199] The terminal captures the user's input in text format and converts it to JSON format.

[0200] Step 3:

[0201] The terminal sends the converted JSON data to the server as an HTTP request.

[0202] Step 4:

[0203] The server receives the JSON data, parses it, and extracts the user's question.

[0204] Step 5:

[0205] The server uses a natural language processing algorithm (NLP) to analyze the question and extract important keywords (e.g., "Tokyo," "tourist spots," "restaurants").

[0206] Step 6:

[0207] The server uses an emotion engine to analyze emotions from the user's text input, for example, to determine whether the user is happy or anxious.

[0208] Step 7:

[0209] The server sends a query to a database or external API based on the extracted keywords and analyzed sentiment. For example, it sends a query such as "popular tourist spots in Tokyo."

[0210] Step 8:

[0211] The server receives the response data to the query and obtains information about tourist spots and restaurants.

[0212] Step 9:

[0213] Based on the data acquired by the server, it generates a response to be sent to the user. For example, it creates a list of tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as a list of restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza."

[0214] Step 10:

[0215] The server adjusts the response based on the results of the sentiment analysis. For example, if the user is feeling anxious, it adds reassuring language to the response.

[0216] Step 11:

[0217] The response generated by the server is converted to JSON format and sent to the terminal as an HTTP response.

[0218] Step 12:

[0219] The device parses the JSON data received from the server and formats it into a layout to display to the user.

[0220] Step 13:

[0221] Users can view a list of tourist attractions and restaurants on their device screen.

[0222] Example 2

[0223] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0224] Conventional tour planning systems only provide information on tourist spots and restaurants that users desire, without considering the user's emotional state, which can lead to low user satisfaction. Furthermore, many systems have limited application of natural language processing, which can result in inaccurate responses to user input.

[0225] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0226] In this invention, the server includes means for receiving a question about a tour plan from a user, means for analyzing the content of the user's question and extracting keywords, means for sending a query to a database or an external API based on the extracted keywords to obtain related data, means for integrating the obtained data and generating a response to be returned to the user, means for analyzing the user's emotions, means for adjusting the response content based on the result of the emotion analysis, and means for transmitting the generated response to the user's terminal. This makes it possible to propose a tour plan that is adapted to the user's emotional state and improve user satisfaction.

[0227] A "user" is someone who uses the system to input questions about tour plans and obtain information.

[0228] A "terminal" is a device that allows a user to input questions and view received information, and includes computers such as smartphones and personal computers.

[0229] A "server" is a computer system that has the function of analyzing a question received from a user, acquiring related data, generating a response, and sending it to a terminal.

[0230] The term "means" refers to a method or device for realizing a specific function, and is a technical element used in an embodiment of the present invention.

[0231] A "tour plan" is a plan that includes travel-related information such as tourist spots and restaurants.

[0232] The "question content" is a textual question entered by the user regarding the tour plan.

[0233] "Keywords" are the main words and phrases that make up the tour plan, extracted by analyzing the content of the question.

[0234] A "database" is a collection of information that stores information on related tourist spots and restaurants, etc., and can be searched as needed.

[0235] An "external API" is an interface for using functions provided by services or programs outside the system.

[0236] A "query" is a request sent to a database or external API to obtain required information.

[0237] "Emotion analysis" is a technology that analyzes a user's emotional state (e.g., joy, anxiety) from the content of their question.

[0238] A "response" is an answer generated by the server in response to a user's question, and includes information about tourist spots and restaurants.

[0239] A "natural language processing algorithm" is a technology for analyzing text entered by a user into a form that is easy for a machine to understand.

[0240] A "generative AI model" is a machine learning model used to automatically generate responses to user questions.

[0241] A "prompt sentence" is an instruction sentence that can be input into a generative AI model to obtain a desired response.

[0242] MODE FOR CARRYING OUT THE INVENTION

[0243] The system of the present invention is designed to enable users to easily and quickly create tour plans. The system allows users to input questions about the tour plan into a terminal, and provides information on suitable tourist spots and restaurants based on the questions. Furthermore, the system incorporates an emotion engine for analyzing the user's emotions, and also has the function of adjusting the response content according to the user's emotions. This improves user satisfaction.

[0244] The system is mainly composed of four main components: the user, the terminal, the server, and the emotion engine. Each component of the system is explained in detail below.

[0245] 1. Users

[0246] Users access the system using a device such as a smartphone or PC, enter questions about the tour plan in the input field on the device, and send the questions to the system.

[0247] 2. Terminal

[0248] The device is responsible for receiving user input and sending it to the server. The client application on the device captures the questions entered by the user in text format and converts them into JSON format. It then sends this JSON data to the server as an HTTP request. Specific examples of this include web browsers and mobile apps installed on smartphones and PCs.

[0249] 3. Server

[0250] The server receives and processes the HTTP request sent from the terminal. The server first analyzes the received JSON data and uses a natural language processing (NLP) algorithm to analyze the user's question. For example, a general-purpose natural language processing engine is used as the NLP algorithm.

[0251] Next, the server uses an emotion engine to analyze the user's emotions. Emotion analysis determines whether the user is feeling happy, anxious, etc. This allows the system to reflect the user's emotional state.

[0252] The server then sends queries to external databases or APIs based on the extracted keywords and the results of sentiment analysis. For example, an external API used to obtain tourist spot information is an API that provides geographic information. For example, a general geographic information API is used to obtain tourist spot information. Similarly, a query to obtain restaurant information is also sent.

[0253] Based on the retrieved data, the server generates a response to return to the user, which may include a list of tourist attractions or restaurants, and further adjusts the response based on the user's emotional state.

[0254] The server converts the generated response into JSON format and sends it to the terminal as an HTTP response.

[0255] 4. Emotion Engine

[0256] An emotion engine is an engine for analyzing emotions from text entered by a user. For example, it uses an emotion analysis API to analyze the user's emotional state and adjusts the response content based on the results.

[0257] Specific examples

[0258] The following concrete examples will help you understand how the system works:

[0259] The user enters a question such as:

[0260] What are some recommended tourist spots and restaurants in Tokyo?

[0261] The device sends this input to the server, which parses it and generates a response like this:

[0262] Tourist attractions: Tokyo Tower, Sensoji Temple, Skytree

[0263] Restaurants: Tsukiji Market, Odaiba restaurants, Ginza sushi restaurants

[0264] Additionally, if the user is feeling anxious, the response will include the following:

[0265] There are tourist spots you can enjoy without worry: Tokyo Tower, Sensoji Temple, and Skytree.

[0266] In this way, users can easily obtain the information they need to plan their trip. Sentiment analysis improves user satisfaction by providing appropriate responses based on the user's needs.

[0267] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0268] Step 1:

[0269] The user inputs a question into the terminal.

[0270] Input: The user enters text into the terminal, such as "Please tell me some recommended tourist spots and restaurants in Tokyo."

[0271] Data processing: The user enters text using a keyboard or touchscreen and presses the send button.

[0272] Output: The entered question is saved on the device.

[0273] Specific operation: The user enters a question into the input field on the terminal and clicks the "Submit" button.

[0274] Step 2:

[0275] The terminal sends user input to the server.

[0276] Input: The text data of the question entered by the user.

[0277] Data processing: The client application on the device converts the text into JSON format and sends it to the server as an HTTP request.

[0278] Output: The user's question is converted into JSON format and sent to the server as an HTTP request.

[0279] Specific operation: The client application makes an HTTP POST request using the fetch function of JavaScript (registered trademark).

[0280] Step 3:

[0281] The server parses the JSON data and performs natural language processing.

[0282] Input: The user's question in JSON format sent from the device.

[0283] Data processing: The server parses the received JSON data and extracts keywords using natural language processing algorithms.

[0284] Output: Extracted keywords (e.g., "Tokyo", "tourist spots", "restaurants").

[0285] Specific operation: The server parses the JSON data using a Python library and extracts keywords using an NLP engine (e.g., a general natural language processing engine).

[0286] Step 4:

[0287] The server performs emotion analysis using an emotion engine.

[0288] Input: Keywords extracted by natural language processing and the user's question.

[0289] Data processing: Using sentiment analysis algorithms to determine emotional states from text.

[0290] Output: User's emotional state (e.g., happy, anxious).

[0291] Specific operation: The server calls the emotion analysis API and analyzes the user's emotional state from the content of the question.

[0292] Step 5:

[0293] The server queries the external API to retrieve the relevant data.

[0294] Input: Extracted keywords (e.g., "Tokyo", "tourist spots", "restaurants") and emotional states.

[0295] Data processing: The server sends a query to an external API to obtain information about tourist attractions and restaurants.

[0296] Output: Data returned from the external API (list of tourist attractions and restaurants).

[0297] Specific operation: The server sends queries to multiple external APIs using HTTP requests and analyzes the responses.

[0298] Step 6:

[0299] The server generates a response based on the data it retrieves.

[0300] Input: Data obtained from external API and sentiment analysis results.

[0301] Data processing: Integrate the acquired information on tourist spots and restaurants, and add text based on the emotional state as needed.

[0302] Output: Generated response data (e.g., "Tokyo Tower," "Sensoji Temple," "Skytree," "Tsukiji Market," "Restaurants in Odaiba," "Sushi restaurants in Ginza").

[0303] Specific operation: The server uses a scripting language such as Python to integrate the data, add text, and convert it into JSON format.

[0304] Step 7:

[0305] The server generates a response and sends it to the terminal.

[0306] Input: The generated response data.

[0307] Data processing: Convert the response data into JSON format and send it to the terminal as an HTTP response.

[0308] Output: JSON formatted response data received by the device.

[0309] Specific operation: The server sends the HTTP response to the client application via a framework (e.g., Flask).

[0310] Step 8:

[0311] The terminal displays the received response to the user.

[0312] Input: JSON formatted response data received from the server.

[0313] Data processing: The client application parses the JSON data and converts it into a format that is easy for the user to read.

[0314] Output: A list of attractions and restaurants displayed on the user's device.

[0315] Specific operation: The client application uses a JavaScript framework (e.g., React.js) to format the response data into HTML and display it as a web page.

[0316] (Application example 2)

[0317] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0318] Conventional tour planning systems have difficulty adjusting responses flexibly based on user emotions, which can result in a poor user experience. Furthermore, there is a lack of methods to optimize the in-store shopping experience, leaving users with few efficient ways to obtain in-store product recommendations and sales information. Therefore, a method is needed to quickly and appropriately provide users with the information they desire.

[0319] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0320] In this invention, the server includes means including an emotion engine that recognizes a user's emotions, means for adjusting response content based on the results of emotion analysis, means for allowing a user to input a question using a smart device in a physical store in order to provide in-store product recommendations and sales information, and means for using a generative AI model, thereby generating an appropriate response based on the user's emotions and improving the shopping experience in the physical store.

[0321] A "user" is an individual who accesses the system through a terminal and inputs a question regarding a tour plan.

[0322] A "terminal" is a device used by a user, such as a smartphone or smart glasses.

[0323] A "server" is a central system that analyzes user questions and generates responses by sending queries to databases or external APIs.

[0324] An "emotion engine" is a technology that analyzes emotions from a user's text input and adjusts responses based on those emotions.

[0325] A "natural language processing algorithm (NLP)" is an algorithm that analyzes the content of a user's question and extracts keywords.

[0326] An "external API" is an interface for obtaining information from external systems such as databases.

[0327] A "generative AI model" is a machine learning model for performing tasks such as natural language processing and sentiment analysis.

[0328] A "physical store" is a sales point that exists in a physical location and where users can visit and purchase products.

[0329] A "query" is a request statement that requests information from a database or external API.

[0330] A "smart device" is a device that can connect to the Internet, such as a smartphone or smart glasses.

[0331] "Adjusting the answer content" means optimizing the information provided based on the user's emotions and the content of the question.

[0332] "Shopping experience" refers to the overall experience a user has when searching for and purchasing products in a physical store.

[0333] "Sale information" is information about discounts and special offers being offered in stores.

[0334] The system of this invention is designed to enable users to create tour plans easily and quickly. The system allows users to input questions about the tour plan into a terminal, and provides information on appropriate tourist spots and restaurants based on the questions. Furthermore, the system incorporates an emotion engine that recognizes the user's emotions, and can adjust the response content based on the user's emotions.

[0335] This system consists of four main components: the user, the terminal, the server, and the emotion engine. The user accesses the system through the terminal and inputs questions about the tour plan they want. For example, consider the case where a user inputs, "Please tell me the recommended tourist spots and restaurants in Tokyo."

[0336] When the device receives user input, it sends the input to the server. The client application on the device captures the user's question in text format, converts it to JSON format, and sends it as an HTTP request to the server. The server receives this request and begins analyzing it.

[0337] The server first parses the received JSON data and uses a natural language processing algorithm (NLP) to analyze the user's question. At this stage, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted from the question. Next, an emotion engine is used to analyze the user's emotions from the text input. This emotion analysis provides information such as whether the user is happy or anxious.

[0338] Next, the server sends a query to a database or external API based on the extracted keywords and the results of the sentiment analysis. For example, the server queries an external API for "popular tourist spots in Tokyo" and receives a list of tourist spots in response. Similarly, it retrieves data on "popular restaurants in Tokyo." In addition, it adjusts the response content to match the user's emotions based on the results of the sentiment analysis. For example, if the user is feeling anxious, it adds reassuring language to the response.

[0339] Based on the retrieved data, the server generates a response to return to the user. This response might include tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza." This information is then formatted and summarized in a form that is easy for the user to understand.

[0340] The generated response is converted back to JSON format and sent to the device as an HTTP response. When the device receives this response, the client application parses the data and formats it into a layout to display to the user. Finally, the user can see a list of tourist attractions and restaurants on the device screen. For example, the user receives the following tour plan information:

[0341] Tourist attractions: Tokyo Tower, Sensoji Temple, Skytree

[0342] Restaurants: Tsukiji Market, Odaiba restaurants, Ginza sushi restaurants

[0343] In addition, responses are adjusted based on sentiment analysis, so if the user is feeling anxious, the phrase "Enjoy with peace of mind" can be added, increasing the user's sense of trust.

[0344] To illustrate, here is an example prompt using a generative AI model (e.g., GPT-4®):

[0345] User Input:

[0346] "What products do you recommend at this store?"

[0347] Example prompts for generative AI models:

[0348] A user asks, "What products do you recommend at this store?" Sentiment analysis shows they are feeling anxious. What kind of response should be returned as a recommendation?

[0349] In this way, users can easily obtain the information they need for travel planning and quickly create the tour plan that best suits them. Sentiment analysis provides responses that are more tailored to the user's needs, improving user satisfaction.

[0350] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0351] Step 1:

[0352] The user inputs a question about the tour plan into the terminal. The input format is text or voice, and the terminal converts this into text data and then converts it into JSON format. The input data is the "question content," and the output data is the "text-formatted question content."

[0353] Step 2:

[0354] The device sends JSON data containing the "text question" to the server as an HTTP request. At this time, the input sent from the device is "text data," and the output is "a request to the server."

[0355] Step 3:

[0356] The server parses the received JSON data and applies natural language processing (NLP) algorithms to analyze the user's question, where the input data is the "JSON-formatted question" and the output data is the "analyzed keywords."

[0357] Step 4:

[0358] The server uses an emotion engine to analyze the user's emotions from the question content. The input data is the "text question content" and the output data is the "user's emotional state." Specifically, the server identifies emotions from the text data and determines the emotional state, such as joy, anxiety, or excitement.

[0359] Step 5:

[0360] The server sends queries to databases and external APIs based on the extracted keywords and the results of sentiment analysis to retrieve relevant data. The input data is the "analyzed keywords" and "user emotional state," and the output data is the "retrieved tourist spot and restaurant information." Specifically, queries such as "popular tourist spots in Tokyo" and "popular restaurants in Tokyo" are sent to the external API.

[0361] Step 6:

[0362] The server integrates the acquired data and generates a response to return to the user. The input data is the acquired tourist spot and restaurant information and the user's emotional state, and the output data is the formatted response. Specifically, if the user is feeling anxious, the server adds the phrase "Enjoy yourself with peace of mind" to the response.

[0363] Step 7:

[0364] The server converts the generated response content back into JSON format and sends it to the terminal as an HTTP response. The input data is the "formatted response content" and the output data is the "response to the terminal."

[0365] Step 8:

[0366] The terminal receives the response from the server, and the client application parses the data and formats it into a layout to be displayed to the user. The input data is the "response from the server," and the output data is the "information to be displayed to the user." Specifically, information such as "Tourist spots: Tokyo Tower, Sensoji Temple, Skytree" is displayed.

[0367] The above is the specific flow of the program's processing. This system generates appropriate responses based on the user's emotions, enabling efficient tour planning. In addition, by using a generative AI model, it is possible to provide sophisticated responses tailored to the user's needs. An example of a prompt sentence is shown below:

[0368] User Input:

[0369] "What products do you recommend at this store?"

[0370] Example prompts for generative AI models:

[0371] A user asks, "What products do you recommend at this store?" Sentiment analysis shows they are feeling anxious. What kind of response should be returned as a recommendation?

[0372] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0373] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0374] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0375] [Second embodiment]

[0376] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0377] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0378] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0379] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0380] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0381] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0382] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0383] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0384] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0385] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0386] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0387] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0388] The system of the present invention is designed to enable users to create tour plans easily and quickly. This system allows users to input questions about the tour plan into a terminal, and provides information on appropriate tourist spots and restaurants based on the questions. Specific embodiments of this system are described below.

[0389] This system consists of three main components: the user, the terminal, and the server. The user accesses the system through the terminal and inputs questions related to the tour plan they want. For example, the user might input, "Please tell me the recommended tourist spots and restaurants in Tokyo."

[0390] When the device receives user input, it sends the input to the server. The client application on the device captures the user's question in text format, converts it to JSON format, and sends it as an HTTP request to the server. The server receives this request and begins analyzing it.

[0391] The server first parses the received JSON data and uses a natural language processing algorithm (NLP) to analyze the user's question. At this stage, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted from the question. These keywords are used as queries to retrieve related data.

[0392] Next, the server sends queries to a database or external API based on the extracted keywords. For example, the server queries an external API for "popular tourist spots in Tokyo" and receives a list of tourist spots in response. Similarly, it retrieves data on "popular restaurants in Tokyo."

[0393] Based on the acquired data, the server generates a response to return to the user. This response may include tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza." In this way, the server creates a list of information that best suits the user's question and formats it in a format that is easy for the user to understand.

[0394] The generated response is converted back to JSON format and sent to the device as an HTTP response. When the device receives this response, the client application parses the data and formats it into a layout to display to the user. Finally, the user can see a list of tourist attractions and restaurants on the device screen.

[0395] For example, a user receives the following tour plan information:

[0396] Tourist attractions: Tokyo Tower, Sensoji Temple, Skytree

[0397] Restaurants: Tsukiji Market, Odaiba restaurants, Ginza sushi restaurants

[0398] In this way, users can easily obtain the information they need to plan their trip and quickly create the tour plan that best suits them, reducing the burden on users and supporting comfortable trip planning.

[0399] The processing flow will be explained below.

[0400] Step 1:

[0401] The user types into the terminal, "Please tell me some recommended tourist spots and restaurants in Tokyo."

[0402] Step 2:

[0403] The terminal captures the user's input in text format and converts it to JSON format.

[0404] Step 3:

[0405] The terminal sends the converted JSON data to the server as an HTTP request.

[0406] Step 4:

[0407] The server receives the JSON data, parses it, and extracts the user's question.

[0408] Step 5:

[0409] The server uses a natural language processing algorithm (NLP) to analyze the question and extract important keywords (e.g., "Tokyo," "tourist spots," "restaurants").

[0410] Step 6:

[0411] The server sends a query to a database or external API based on the extracted keywords. For example, it sends a query such as "popular tourist spots in Tokyo."

[0412] Step 7:

[0413] The server receives the response data to the query and obtains information about tourist spots and restaurants.

[0414] Step 8:

[0415] The server integrates the acquired data and generates a response to return to the user. For example, it creates a list of tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as a list of restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza."

[0416] Step 9:

[0417] The response generated by the server is converted to JSON format and sent to the terminal as an HTTP response.

[0418] Step 10:

[0419] The device parses the JSON data received from the server and formats it into a layout to display to the user.

[0420] Step 11:

[0421] Users can view a list of tourist attractions and restaurants on their device screen.

[0422] Example 1

[0423] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0424] Conventional travel planning systems have had the problem of requiring a great deal of time and effort when users research trip details. In particular, the task of collecting and integrating necessary information from multiple sources is extremely time-consuming. It is also difficult to provide accurate information that meets the user's needs. This tends to make users' travel planning complicated and inefficient.

[0425] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0426] In this invention, the server includes means for receiving a plan-related question from a user, means for analyzing the content of the user's question and extracting keywords, means for sending a query to information sources based on the extracted keywords to obtain related data, means for integrating the obtained data and generating a response to return to the user, and means for transmitting the generated response to the user's terminal, thereby enabling the user to quickly and accurately obtain information necessary for planning and easily create a travel plan.

[0427] "Plan" is a plan that allows a user to organize and execute details of trips and activities.

[0428] A "question" is text or a query that a user enters to request information about a plan.

[0429] "Analysis" is the process of understanding the user's question and identifying important elements and information.

[0430] "Keywords" are important words or phrases extracted from a user's question that serve as the basis for conducting a search or query.

[0431] An "information source" is a data provider, such as a database or external API, that is used to obtain the required information.

[0432] A "query" is a request sent to a source to search for or retrieve specific information.

[0433] "Data" refers to the information and responses related to a user's question.

[0434] A "response" is information or a reply provided to a user's question, and includes content that is useful to the user.

[0435] A "terminal" is a device that a user uses to access and operate the system. Examples include a smartphone or computer.

[0436] A "natural language processing algorithm" is a technology for analyzing text data and understanding its meaning and structure, and is used to accurately analyze the content of user questions.

[0437] The "JSON format" is a text format for structuring and expressing data, and is a format that is often used for sending and receiving data.

[0438] An "HTTP request" is a protocol used when a client requests information from a server.

[0439] An "HTTP response" is a protocol used when a server returns information to a client.

[0440] The system of the present invention is designed to enable users to create plans quickly and accurately. This system allows users to input questions about the plan into a terminal, and provides appropriate information based on the questions. Specific embodiments of this system are described below.

[0441] Hardware and software used

[0442] This system consists of three main components: the user, the terminal, and the server. The terminal can be a smartphone or a computer. The server is a server computer operated in a cloud environment or on-premise. The software includes client applications, server-side programs, natural language processing algorithms (NLP tools), and various APIs.

[0443] Explanation of program processing

[0444] 1. User Input

[0445] The user uses the client application on the terminal to input a question about the plan. For example, the user inputs, "Please tell me recommended tourist spots and restaurants in Tokyo."

[0446] 2. Data transmission by the terminal

[0447] The terminal captures the user's question in text format and converts it into JSON format. The converted JSON data is sent to the server as an HTTP request. The software used for this process is a request library such as fetch or axios.

[0448] 3. Data analysis by the server

[0449] The server parses the received JSON data and analyzes the question using natural language processing (NLP) algorithms, for example by extracting keywords using libraries such as spaCy or NLTK.

[0450] For example, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted. These keywords are used as queries to search for related information.

[0451] 4. Acquiring relevant data

[0452] Based on the extracted keywords, the server sends queries to databases or external APIs to retrieve relevant data. For example, it uses the requests library to access an API and retrieve information about tourist attractions and restaurants.

[0453] 5. Generating and Sending the Response

[0454] The server generates a response based on the acquired data. This response includes information on tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza." The generated response is converted to JSON format and sent to the device as an HTTP response.

[0455] 6. Displaying data on a terminal

[0456] The client application on the device parses the received response and displays it in a user-friendly format, using HTML and CSS to format the layout.

[0457] Specific examples

[0458] For example, a user enters the following question:

[0459] > "What are some recommended tourist spots and restaurants in Tokyo?"

[0460] In response to this question, the system can return information such as:

[0461] Tourist attractions:

[0462] Tokyo Tower

[0463] Sensoji Temple

[0464] Skytree

[0465] Restaurant:

[0466] Tsukiji Market

[0467] Restaurants in Odaiba

[0468] Sushi restaurant in Ginza

[0469] In this way, the user can quickly and easily obtain the necessary information and create an optimal plan.

[0470] The system of the present invention reduces the burden on the user and makes it possible to support comfortable planning.

[0471] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0472] Step 1:

[0473] The user inputs a question into the terminal. The user opens the client application on the terminal and inputs a question about the plan. For example, the user inputs, "Please tell me recommended tourist spots and restaurants in Tokyo." This input text becomes input data for the next processing step.

[0474] Step 2:

[0475] The device sends a question to the server. The device captures the question text from the user in text format and converts it to JSON format. This converted data becomes the body of an HTTP request and is sent to the server. Specifically, the device uses a request library such as fetch or axios to send the data. The output of this step is JSON format data that the server can process.

[0476] Step 3:

[0477] The server analyzes the question. The server parses the received JSON data and uses a natural language processing algorithm (NLP) to analyze the question. This analysis process uses libraries such as spaCy and NLTK to tokenize the text and extract keywords. The input for this process is the JSON data, and the output is a list of extracted keywords. For example, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted.

[0478] Step 4:

[0479] The server retrieves relevant data from a database or external API. Based on the extracted keywords, the server creates a database query or external API query to retrieve the required data. Specifically, it uses the requests library to access the API and retrieve information about tourist attractions and restaurants. The input for this step is the keywords, and the output is a list of the retrieved tourist attractions and restaurants.

[0480] Step 5:

[0481] The server generates a response. Based on the acquired data, the server generates a response to return to the user. This response includes a list of tourist spots and restaurants. For example, tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree" and restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi Restaurants in Ginza" are listed. The input of this step is the acquired data, and the output is the formatted response data.

[0482] Step 6:

[0483] The server sends the response to the terminal. The generated response data is converted back to JSON format and sent to the terminal as an HTTP response. Specifically, the server generates the HTTP response using a web framework (e.g., Flask). The input of this step is the formatted response data, and the output is JSON-formatted response data.

[0484] Step 7:

[0485] The device displays the response. The device's client application parses the received response and displays it in a format that is easy for the user to understand. Specifically, it formats the data using HTML and CSS and provides it to the user. The input to this step is the response data in JSON format, and the output is a displayed list of tourist attractions and restaurants.

[0486] By performing the above steps, the system can quickly and accurately provide the necessary information based on the user's question.

[0487] (Application example 1)

[0488] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0489] Conventional tour planning systems only provide users with information in a static format, making it difficult to provide intuitive understanding or a realistic experience. In particular, there is a lack of technology for providing tour planning information visually and interactively, which has led to issues such as reduced convenience and satisfaction for users.

[0490] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0491] In this invention, the server includes means for receiving a question about a tour plan from a user, means for analyzing the content of the user's question and extracting keywords, means for sending a query to a database or an external API based on the extracted keywords to obtain related data, means for integrating the obtained data and generating a response to return to the user, means for sending the generated response to the user's terminal, and means for reproducing the obtained data in a virtual environment using a head-mounted display. This allows the user to visually experience tourist spots and restaurants in the virtual environment, enabling them to create a more intuitive and realistic tour plan.

[0492] "User" refers to a person who uses the system to ask questions or obtain information about tour plans.

[0493] A "tour plan question" refers to an inquiry from a user to ask about tourist attractions and restaurant information for a particular travel destination.

[0494] "Keywords" refer to important words or phrases extracted from the content of a user's question and are used as search queries.

[0495] A "database" refers to a collection of information about tourist spots and restaurants.

[0496] An "external API" refers to an interface for obtaining data from other systems or services.

[0497] A "query" is a search request sent to a database or external API requesting information.

[0498] A "response" refers to a set of information that a system returns in response to a question from a user.

[0499] "Terminal" refers to a device through which a user accesses the system, including smartphones, tablets, and PCs.

[0500] "Head-mounted display" refers to a display device worn by a user to visually experience a virtual environment.

[0501] A "virtual environment" refers to a virtual space constructed using computer technology that users can experience visually and interactively.

[0502] The embodiment of this invention is a system that allows users to easily and quickly create a tour plan and visually experience it in a virtual environment. The system mainly consists of a user, a terminal, a server, and a head-mounted display.

[0503] First, a user accesses the system through a terminal and inputs a question about a tour plan. For example, a user might input a question like, "Please tell me the recommended tourist spots and restaurants in Tokyo."

[0504] The device captures the user's question in text format, converts it to JSON format, and sends it to the server as an HTTP request. The server receives this request and begins analyzing it. The natural language processing algorithm used here is spaCy, for example.

[0505] The server parses the received JSON data and analyzes the user's question. Through analysis, it extracts keywords such as "Tokyo," "tourist spots," and "restaurants." These keywords are used as queries to retrieve related data. The server then sends queries to databases or external APIs based on the extracted keywords to retrieve data on tourist spots and restaurants.

[0506] The server generates a response based on the acquired data. This response includes tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza." The generated response is again converted to JSON format and sent to the device as an HTTP response.

[0507] When the device receives this response, it parses the data and formats it into a layout for display to the user. However, a distinctive feature of the system is that it uses a head-mounted display (HMD) to recreate the acquired data in a virtual environment. This virtual environment is built using a game engine such as Unity.

[0508] By wearing the HMD, users can visually experience the tourist spots and restaurants provided in the virtual environment, allowing them to check and plan their tour plans as if they were actually visiting the places.

[0509] As a concrete example, if a user asks the device, "What are some recommended tourist spots and restaurants in Tokyo?", the system will recreate tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza" in the virtual environment. In this process, the following prompt sentence can be input to the generative AI model: "What are some recommended tourist spots and restaurants in Tokyo?"

[0510] In this way, users can visually receive tour plan suggestions and plan their trip through a virtual experience on the spot. The hardware used includes head-mounted displays such as Oculus Rift and HTC Vive. The software used includes Unity, Python, spaCy, and external APIs.

[0511] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0512] Step 1:

[0513] The user inputs a question about the tour plan into the terminal. For example, the user inputs the text "Please tell me the recommended tourist spots and restaurants in Tokyo."

[0514] Step 2:

[0515] The terminal receives the user's question in text format and converts it to JSON format for subsequent server processing. The input value is the user's question text, and the output is a JSON object.

[0516] Step 3:

[0517] The terminal sends the converted JSON data to the server as an HTTP request. This request includes the user's question and is sent to the server's URL. The input value is a JSON object, and the output is an HTTP request.

[0518] Step 4:

[0519] The server receives the HTTP request and parses the JSON data. During this process, the parsed data is converted into a structured format and the user's question is analyzed. The input value is the JSON data of the HTTP request, and the output is structured data.

[0520] Step 5:

[0521] The server uses a natural language processing (NLP) algorithm to extract keywords from the data analyzed. In this process, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted from the user's question. The input is structured data, and the output is a set of extracted keywords.

[0522] Step 6:

[0523] The server uses the extracted keywords to send queries to databases or external APIs to obtain the required information. Specifically, queries to obtain data on tourist attractions and restaurants are created and sent to the API or database. The input is a set of keywords, and the output is data on tourist attractions and restaurants.

[0524] Step 7:

[0525] The server generates a response to return to the user based on the data it has acquired. The response includes information about tourist spots and restaurants, and is formatted in a way that is easy for the user to understand. The input is the acquired data, and the output is the formatted response.

[0526] Step 8:

[0527] The server converts the generated response back into JSON format and sends it as an HTTP response to the terminal. The input is a formatted response, and the output is a JSON-formatted HTTP response.

[0528] Step 9:

[0529] The device receives the JSON data returned from the server, parses the data, and displays it to the user. This display is not only displayed on the device screen, but also reproduced in a virtual environment via a head-mounted display (HMD). The input is the JSON data of the HTTP response, and the output is the visual information displayed to the user.

[0530] Step 10:

[0531] By wearing an HMD, users can visually experience tourist spots and restaurants in a virtual environment, giving them the feeling that they are actually visiting those places. The input is the visual information in the virtual environment, and the output is the user's experience.

[0532] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0533] The system of the present invention is designed to enable users to create tour plans easily and quickly. The system allows users to input questions about the tour plan into a terminal, and provides information on appropriate tourist spots and restaurants based on the questions. Furthermore, the system incorporates an emotion engine that recognizes the user's emotions, allowing it to provide appropriate responses based on the user's emotions.

[0534] This system consists of four main components: the user, the terminal, the server, and the emotion engine. The user accesses the system through the terminal and inputs questions about the tour plan they want. For example, consider the case where a user inputs, "Please tell me recommended tourist spots and restaurants in Tokyo."

[0535] When the device receives user input, it sends the input to the server. The client application on the device captures the user's question in text format, converts it to JSON format, and sends it as an HTTP request to the server. The server receives this request and begins analyzing it.

[0536] The server first parses the received JSON data and uses a natural language processing algorithm (NLP) to analyze the user's question. At this stage, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted from the question. Next, an emotion engine is used to analyze the user's emotions from the text input. This emotion analysis provides information such as whether the user is happy or anxious.

[0537] Next, the server sends a query to a database or external API based on the extracted keywords and the results of the sentiment analysis. For example, the server queries an external API for "popular tourist spots in Tokyo" and receives a list of tourist spots in response. Similarly, it retrieves data on "popular restaurants in Tokyo." In addition, it adjusts the response content to match the user's emotions based on the results of the sentiment analysis. For example, if the user is feeling anxious, it adds reassuring language to the response.

[0538] Based on the retrieved data, the server generates a response to return to the user. This response might include tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza." This information is then formatted and summarized in a form that is easy for the user to understand.

[0539] The generated response is converted back to JSON format and sent to the device as an HTTP response. When the device receives this response, the client application parses the data and formats it into a layout to display to the user. Finally, the user can see a list of tourist attractions and restaurants on the device screen.

[0540] For example, a user receives the following tour plan information:

[0541] Tourist attractions: Tokyo Tower, Sensoji Temple, Skytree

[0542] Restaurants: Tsukiji Market, Odaiba restaurants, Ginza sushi restaurants

[0543] In addition, responses are adjusted based on sentiment analysis, so if the user is feeling anxious, the phrase "Enjoy with peace of mind" can be added, increasing the user's sense of trust.

[0544] In this way, users can easily obtain the information they need for travel planning and quickly create the tour plan that best suits them. Sentiment analysis provides responses that are more tailored to the user's needs, improving user satisfaction.

[0545] The processing flow will be explained below.

[0546] Step 1:

[0547] The user types into the terminal, "Please tell me some recommended tourist spots and restaurants in Tokyo."

[0548] Step 2:

[0549] The terminal captures the user's input in text format and converts it to JSON format.

[0550] Step 3:

[0551] The terminal sends the converted JSON data to the server as an HTTP request.

[0552] Step 4:

[0553] The server receives the JSON data, parses it, and extracts the user's question.

[0554] Step 5:

[0555] The server uses a natural language processing algorithm (NLP) to analyze the question and extract important keywords (e.g., "Tokyo," "tourist spots," "restaurants").

[0556] Step 6:

[0557] The server uses an emotion engine to analyze emotions from the user's text input, for example, to determine whether the user is happy or anxious.

[0558] Step 7:

[0559] The server sends a query to a database or external API based on the extracted keywords and analyzed sentiment. For example, it sends a query such as "popular tourist spots in Tokyo."

[0560] Step 8:

[0561] The server receives the response data to the query and obtains information about tourist spots and restaurants.

[0562] Step 9:

[0563] Based on the data acquired by the server, it generates a response to be sent to the user. For example, it creates a list of tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as a list of restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza."

[0564] Step 10:

[0565] The server adjusts the response based on the results of the sentiment analysis. For example, if the user is feeling anxious, it adds reassuring language to the response.

[0566] Step 11:

[0567] The response generated by the server is converted to JSON format and sent to the terminal as an HTTP response.

[0568] Step 12:

[0569] The device parses the JSON data received from the server and formats it into a layout to display to the user.

[0570] Step 13:

[0571] Users can view a list of tourist attractions and restaurants on their device screen.

[0572] Example 2

[0573] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0574] Conventional tour planning systems only provide information on tourist spots and restaurants that users desire, without considering the user's emotional state, which can lead to low user satisfaction. Furthermore, many systems have limited application of natural language processing, which can result in inaccurate responses to user input.

[0575] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0576] In this invention, the server includes means for receiving a question about a tour plan from a user, means for analyzing the content of the user's question and extracting keywords, means for sending a query to a database or an external API based on the extracted keywords to obtain related data, means for integrating the obtained data and generating a response to be returned to the user, means for analyzing the user's emotions, means for adjusting the response content based on the result of the emotion analysis, and means for transmitting the generated response to the user's terminal. This makes it possible to propose a tour plan that is adapted to the user's emotional state and improve user satisfaction.

[0577] A "user" is someone who uses the system to input questions about tour plans and obtain information.

[0578] A "terminal" is a device that allows a user to input questions and view received information, and includes computers such as smartphones and personal computers.

[0579] A "server" is a computer system that has the function of analyzing a question received from a user, acquiring related data, generating a response, and sending it to a terminal.

[0580] The term "means" refers to a method or device for realizing a specific function, and is a technical element used in an embodiment of the present invention.

[0581] A "tour plan" is a plan that includes travel-related information such as tourist spots and restaurants.

[0582] The "question content" is a textual question entered by the user regarding the tour plan.

[0583] "Keywords" are the main words and phrases that make up the tour plan, extracted by analyzing the content of the question.

[0584] A "database" is a collection of information that stores information on related tourist spots and restaurants, etc., and can be searched as needed.

[0585] An "external API" is an interface for using functions provided by services or programs outside the system.

[0586] A "query" is a request sent to a database or external API to obtain required information.

[0587] "Emotion analysis" is a technology that analyzes a user's emotional state (e.g., joy, anxiety) from the content of their question.

[0588] A "response" is an answer generated by the server in response to a user's question, and includes information about tourist spots and restaurants.

[0589] A "natural language processing algorithm" is a technology for analyzing text entered by a user into a form that is easy for a machine to understand.

[0590] A "generative AI model" is a machine learning model used to automatically generate responses to user questions.

[0591] A "prompt sentence" is an instruction sentence that can be input into a generative AI model to obtain a desired response.

[0592] MODE FOR CARRYING OUT THE INVENTION

[0593] The system of the present invention is designed to enable users to easily and quickly create tour plans. The system allows users to input questions about the tour plan into a terminal, and provides information on suitable tourist spots and restaurants based on the questions. Furthermore, the system incorporates an emotion engine for analyzing the user's emotions, and also has the function of adjusting the response content according to the user's emotions. This improves user satisfaction.

[0594] The system is mainly composed of four main components: the user, the terminal, the server, and the emotion engine. Each component of the system is explained in detail below.

[0595] 1. Users

[0596] Users access the system using a device such as a smartphone or PC, enter questions about the tour plan in the input field on the device, and send the questions to the system.

[0597] 2. Terminal

[0598] The device is responsible for receiving user input and sending it to the server. The client application on the device captures the questions entered by the user in text format and converts them into JSON format. It then sends this JSON data to the server as an HTTP request. Specific examples of this include web browsers and mobile apps installed on smartphones and PCs.

[0599] 3. Server

[0600] The server receives and processes the HTTP request sent from the terminal. The server first analyzes the received JSON data and uses a natural language processing (NLP) algorithm to analyze the user's question. For example, a general-purpose natural language processing engine is used as the NLP algorithm.

[0601] Next, the server uses an emotion engine to analyze the user's emotions. Emotion analysis determines whether the user is feeling happy, anxious, etc. This allows the system to reflect the user's emotional state.

[0602] The server then sends queries to external databases or APIs based on the extracted keywords and the results of sentiment analysis. For example, an external API used to obtain tourist spot information is an API that provides geographic information. For example, a general geographic information API is used to obtain tourist spot information. Similarly, a query to obtain restaurant information is also sent.

[0603] Based on the retrieved data, the server generates a response to return to the user, which may include a list of tourist attractions or restaurants, and further adjusts the response based on the user's emotional state.

[0604] The server converts the generated response into JSON format and sends it to the terminal as an HTTP response.

[0605] 4. Emotion Engine

[0606] An emotion engine is an engine for analyzing emotions from text entered by a user. For example, it uses an emotion analysis API to analyze the user's emotional state and adjusts the response content based on the results.

[0607] Specific examples

[0608] The following concrete examples will help you understand how the system works:

[0609] The user enters a question such as:

[0610] What are some recommended tourist spots and restaurants in Tokyo?

[0611] The device sends this input to the server, which parses it and generates a response like this:

[0612] Tourist attractions: Tokyo Tower, Sensoji Temple, Skytree

[0613] Restaurants: Tsukiji Market, Odaiba restaurants, Ginza sushi restaurants

[0614] Additionally, if the user is feeling anxious, the response will include the following:

[0615] There are tourist spots you can enjoy without worry: Tokyo Tower, Sensoji Temple, and Skytree.

[0616] In this way, users can easily obtain the information they need to plan their trip. Sentiment analysis improves user satisfaction by providing appropriate responses based on the user's needs.

[0617] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0618] Step 1:

[0619] The user inputs a question into the terminal.

[0620] Input: The user enters text into the terminal, such as "Please tell me some recommended tourist spots and restaurants in Tokyo."

[0621] Data processing: The user enters text using a keyboard or touchscreen and presses the send button.

[0622] Output: The entered question is saved on the device.

[0623] Specific operation: The user enters a question into the input field on the terminal and clicks the "Submit" button.

[0624] Step 2:

[0625] The terminal sends user input to the server.

[0626] Input: The text data of the question entered by the user.

[0627] Data processing: The client application on the device converts the text into JSON format and sends it to the server as an HTTP request.

[0628] Output: The user's question is converted into JSON format and sent to the server as an HTTP request.

[0629] Specific behavior: The client application makes an HTTP POST request using JavaScript's fetch function.

[0630] Step 3:

[0631] The server parses the JSON data and performs natural language processing.

[0632] Input: The user's question in JSON format sent from the device.

[0633] Data processing: The server parses the received JSON data and extracts keywords using natural language processing algorithms.

[0634] Output: Extracted keywords (e.g., "Tokyo", "tourist spots", "restaurants").

[0635] Specific operation: The server parses the JSON data using a Python library and extracts keywords using an NLP engine (e.g., a general natural language processing engine).

[0636] Step 4:

[0637] The server performs emotion analysis using an emotion engine.

[0638] Input: Keywords extracted by natural language processing and the user's question.

[0639] Data processing: Using sentiment analysis algorithms to determine emotional states from text.

[0640] Output: User's emotional state (e.g., happy, anxious).

[0641] Specific operation: The server calls the emotion analysis API and analyzes the user's emotional state from the content of the question.

[0642] Step 5:

[0643] The server queries the external API to retrieve the relevant data.

[0644] Input: Extracted keywords (e.g., "Tokyo", "tourist spots", "restaurants") and emotional states.

[0645] Data processing: The server sends a query to an external API to obtain information about tourist attractions and restaurants.

[0646] Output: Data returned from the external API (list of tourist attractions and restaurants).

[0647] Specific operation: The server sends queries to multiple external APIs using HTTP requests and analyzes the responses.

[0648] Step 6:

[0649] The server generates a response based on the data it retrieves.

[0650] Input: Data obtained from external API and sentiment analysis results.

[0651] Data processing: Integrate the acquired information on tourist spots and restaurants, and add text based on the emotional state as needed.

[0652] Output: Generated response data (e.g., "Tokyo Tower," "Sensoji Temple," "Skytree," "Tsukiji Market," "Restaurants in Odaiba," "Sushi restaurants in Ginza").

[0653] Specific operation: The server uses a scripting language such as Python to integrate the data, add text, and convert it into JSON format.

[0654] Step 7:

[0655] The server generates a response and sends it to the terminal.

[0656] Input: The generated response data.

[0657] Data processing: Convert the response data into JSON format and send it to the terminal as an HTTP response.

[0658] Output: JSON formatted response data received by the device.

[0659] Specific operation: The server sends the HTTP response to the client application via a framework (e.g., Flask).

[0660] Step 8:

[0661] The terminal displays the received response to the user.

[0662] Input: JSON formatted response data received from the server.

[0663] Data processing: The client application parses the JSON data and converts it into a format that is easy for the user to read.

[0664] Output: A list of attractions and restaurants displayed on the user's device.

[0665] Specific operation: The client application uses a JavaScript framework (e.g., React.js) to format the response data into HTML and display it as a web page.

[0666] (Application example 2)

[0667] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0668] Conventional tour planning systems have difficulty adjusting responses flexibly based on user emotions, which can result in a poor user experience. Furthermore, there is a lack of methods to optimize the in-store shopping experience, leaving users with few efficient ways to obtain in-store product recommendations and sales information. Therefore, a method is needed to quickly and appropriately provide users with the information they desire.

[0669] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0670] In this invention, the server includes means including an emotion engine that recognizes a user's emotions, means for adjusting response content based on the results of emotion analysis, means for allowing a user to input a question using a smart device in a physical store in order to provide in-store product recommendations and sales information, and means for using a generative AI model, thereby generating an appropriate response based on the user's emotions and improving the shopping experience in the physical store.

[0671] A "user" is an individual who accesses the system through a terminal and inputs a question regarding a tour plan.

[0672] A "terminal" is a device used by a user, such as a smartphone or smart glasses.

[0673] A "server" is a central system that analyzes user questions and generates responses by sending queries to databases or external APIs.

[0674] An "emotion engine" is a technology that analyzes emotions from a user's text input and adjusts responses based on those emotions.

[0675] A "natural language processing algorithm (NLP)" is an algorithm that analyzes the content of a user's question and extracts keywords.

[0676] An "external API" is an interface for obtaining information from external systems such as databases.

[0677] A "generative AI model" is a machine learning model for performing tasks such as natural language processing and sentiment analysis.

[0678] A "physical store" is a sales point that exists in a physical location and where users can visit and purchase products.

[0679] A "query" is a request statement that requests information from a database or external API.

[0680] A "smart device" is a device that can connect to the Internet, such as a smartphone or smart glasses.

[0681] "Adjusting the answer content" means optimizing the information provided based on the user's emotions and the content of the question.

[0682] "Shopping experience" refers to the overall experience a user has when searching for and purchasing products in a physical store.

[0683] "Sale information" is information about discounts and special offers being offered in stores.

[0684] The system of this invention is designed to enable users to create tour plans easily and quickly. The system allows users to input questions about the tour plan into a terminal, and provides information on appropriate tourist spots and restaurants based on the questions. Furthermore, the system incorporates an emotion engine that recognizes the user's emotions, and can adjust the response content based on the user's emotions.

[0685] This system consists of four main components: the user, the terminal, the server, and the emotion engine. The user accesses the system through the terminal and inputs questions about the tour plan they want. For example, consider the case where a user inputs, "Please tell me the recommended tourist spots and restaurants in Tokyo."

[0686] When the device receives user input, it sends the input to the server. The client application on the device captures the user's question in text format, converts it to JSON format, and sends it as an HTTP request to the server. The server receives this request and begins analyzing it.

[0687] The server first parses the received JSON data and uses a natural language processing algorithm (NLP) to analyze the user's question. At this stage, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted from the question. Next, an emotion engine is used to analyze the user's emotions from the text input. This emotion analysis provides information such as whether the user is happy or anxious.

[0688] Next, the server sends a query to a database or external API based on the extracted keywords and the results of the sentiment analysis. For example, the server queries an external API for "popular tourist spots in Tokyo" and receives a list of tourist spots in response. Similarly, it retrieves data on "popular restaurants in Tokyo." In addition, it adjusts the response content to match the user's emotions based on the results of the sentiment analysis. For example, if the user is feeling anxious, it adds reassuring language to the response.

[0689] Based on the retrieved data, the server generates a response to return to the user. This response might include tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza." This information is then formatted and summarized in a form that is easy for the user to understand.

[0690] The generated response is converted back to JSON format and sent to the device as an HTTP response. When the device receives this response, the client application parses the data and formats it into a layout to display to the user. Finally, the user can see a list of tourist attractions and restaurants on the device screen. For example, the user receives the following tour plan information:

[0691] Tourist attractions: Tokyo Tower, Sensoji Temple, Skytree

[0692] Restaurants: Tsukiji Market, Odaiba restaurants, Ginza sushi restaurants

[0693] In addition, responses are adjusted based on sentiment analysis, so if the user is feeling anxious, the phrase "Enjoy with peace of mind" can be added, increasing the user's sense of trust.

[0694] To illustrate, here are some example prompts using a generative AI model (e.g., GPT-4):

[0695] User Input:

[0696] "What products do you recommend at this store?"

[0697] Example prompts for generative AI models:

[0698] A user asks, "What products do you recommend at this store?" Sentiment analysis shows they are feeling anxious. What kind of response should be returned as a recommendation?

[0699] In this way, users can easily obtain the information they need for travel planning and quickly create the tour plan that best suits them. Sentiment analysis provides responses that are more tailored to the user's needs, improving user satisfaction.

[0700] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0701] Step 1:

[0702] The user inputs a question about the tour plan into the terminal. The input format is text or voice, and the terminal converts this into text data and then converts it into JSON format. The input data is the "question content," and the output data is the "text-formatted question content."

[0703] Step 2:

[0704] The device sends JSON data containing the "text question" to the server as an HTTP request. At this time, the input sent from the device is "text data," and the output is "a request to the server."

[0705] Step 3:

[0706] The server parses the received JSON data and applies natural language processing (NLP) algorithms to analyze the user's question, where the input data is the "JSON-formatted question" and the output data is the "analyzed keywords."

[0707] Step 4:

[0708] The server uses an emotion engine to analyze the user's emotions from the question content. The input data is the "text question content" and the output data is the "user's emotional state." Specifically, the server identifies emotions from the text data and determines the emotional state, such as joy, anxiety, or excitement.

[0709] Step 5:

[0710] The server sends queries to databases and external APIs based on the extracted keywords and the results of sentiment analysis to retrieve relevant data. The input data is the "analyzed keywords" and "user emotional state," and the output data is the "retrieved tourist spot and restaurant information." Specifically, queries such as "popular tourist spots in Tokyo" and "popular restaurants in Tokyo" are sent to the external API.

[0711] Step 6:

[0712] The server integrates the acquired data and generates a response to return to the user. The input data is the acquired tourist spot and restaurant information and the user's emotional state, and the output data is the formatted response. Specifically, if the user is feeling anxious, the server adds the phrase "Enjoy yourself with peace of mind" to the response.

[0713] Step 7:

[0714] The server converts the generated response content back into JSON format and sends it to the terminal as an HTTP response. The input data is the "formatted response content" and the output data is the "response to the terminal."

[0715] Step 8:

[0716] The terminal receives the response from the server, and the client application parses the data and formats it into a layout to be displayed to the user. The input data is the "response from the server," and the output data is the "information to be displayed to the user." Specifically, information such as "Tourist spots: Tokyo Tower, Sensoji Temple, Skytree" is displayed.

[0717] The above is the specific flow of the program's processing. This system generates appropriate responses based on the user's emotions, enabling efficient tour planning. In addition, by using a generative AI model, it is possible to provide sophisticated responses tailored to the user's needs. An example of a prompt sentence is shown below:

[0718] User Input:

[0719] "What products do you recommend at this store?"

[0720] Example prompts for generative AI models:

[0721] A user asks, "What products do you recommend at this store?" Sentiment analysis shows they are feeling anxious. What kind of response should be returned as a recommendation?

[0722] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0723] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0724] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0725] [Third embodiment]

[0726] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0727] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0728] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0729] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0730] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0731] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0732] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0733] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0734] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0735] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0736] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0737] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0738] The system of the present invention is designed to enable users to create tour plans easily and quickly. This system allows users to input questions about the tour plan into a terminal, and provides information on appropriate tourist spots and restaurants based on the questions. Specific embodiments of this system are described below.

[0739] This system consists of three main components: the user, the terminal, and the server. The user accesses the system through the terminal and inputs questions related to the tour plan they want. For example, the user might input, "Please tell me the recommended tourist spots and restaurants in Tokyo."

[0740] When the device receives user input, it sends the input to the server. The client application on the device captures the user's question in text format, converts it to JSON format, and sends it as an HTTP request to the server. The server receives this request and begins analyzing it.

[0741] The server first parses the received JSON data and uses a natural language processing algorithm (NLP) to analyze the user's question. At this stage, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted from the question. These keywords are used as queries to retrieve related data.

[0742] Next, the server sends queries to a database or external API based on the extracted keywords. For example, the server queries an external API for "popular tourist spots in Tokyo" and receives a list of tourist spots in response. Similarly, it retrieves data on "popular restaurants in Tokyo."

[0743] Based on the acquired data, the server generates a response to return to the user. This response may include tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza." In this way, the server creates a list of information that best suits the user's question and formats it in a format that is easy for the user to understand.

[0744] The generated response is converted back to JSON format and sent to the device as an HTTP response. When the device receives this response, the client application parses the data and formats it into a layout to display to the user. Finally, the user can see a list of tourist attractions and restaurants on the device screen.

[0745] For example, a user receives the following tour plan information:

[0746] Tourist attractions: Tokyo Tower, Sensoji Temple, Skytree

[0747] Restaurants: Tsukiji Market, Odaiba restaurants, Ginza sushi restaurants

[0748] In this way, users can easily obtain the information they need to plan their trip and quickly create the tour plan that best suits them, reducing the burden on users and supporting comfortable trip planning.

[0749] The processing flow will be explained below.

[0750] Step 1:

[0751] The user types into the terminal, "Please tell me some recommended tourist spots and restaurants in Tokyo."

[0752] Step 2:

[0753] The terminal captures the user's input in text format and converts it to JSON format.

[0754] Step 3:

[0755] The terminal sends the converted JSON data to the server as an HTTP request.

[0756] Step 4:

[0757] The server receives the JSON data, parses it, and extracts the user's question.

[0758] Step 5:

[0759] The server uses a natural language processing algorithm (NLP) to analyze the question and extract important keywords (e.g., "Tokyo," "tourist spots," "restaurants").

[0760] Step 6:

[0761] The server sends a query to a database or external API based on the extracted keywords. For example, it sends a query such as "popular tourist spots in Tokyo."

[0762] Step 7:

[0763] The server receives the response data to the query and obtains information about tourist spots and restaurants.

[0764] Step 8:

[0765] The server integrates the acquired data and generates a response to return to the user. For example, it creates a list of tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as a list of restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza."

[0766] Step 9:

[0767] The response generated by the server is converted to JSON format and sent to the terminal as an HTTP response.

[0768] Step 10:

[0769] The device parses the JSON data received from the server and formats it into a layout to display to the user.

[0770] Step 11:

[0771] Users can view a list of tourist attractions and restaurants on their device screen.

[0772] Example 1

[0773] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0774] Conventional travel planning systems have had the problem of requiring a great deal of time and effort when users research trip details. In particular, the task of collecting and integrating necessary information from multiple sources is extremely time-consuming. It is also difficult to provide accurate information that meets the user's needs. This tends to make users' travel planning complicated and inefficient.

[0775] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0776] In this invention, the server includes means for receiving a plan-related question from a user, means for analyzing the content of the user's question and extracting keywords, means for sending a query to information sources based on the extracted keywords to obtain related data, means for integrating the obtained data and generating a response to return to the user, and means for transmitting the generated response to the user's terminal, thereby enabling the user to quickly and accurately obtain information necessary for planning and easily create a travel plan.

[0777] "Plan" is a plan that allows a user to organize and execute details of trips and activities.

[0778] A "question" is text or a query that a user enters to request information about a plan.

[0779] "Analysis" is the process of understanding the user's question and identifying important elements and information.

[0780] "Keywords" are important words or phrases extracted from a user's question that serve as the basis for conducting a search or query.

[0781] An "information source" is a data provider, such as a database or external API, that is used to obtain the required information.

[0782] A "query" is a request sent to a source to search for or retrieve specific information.

[0783] "Data" refers to the information and responses related to a user's question.

[0784] A "response" is information or a reply provided to a user's question, and includes content that is useful to the user.

[0785] A "terminal" is a device that a user uses to access and operate the system. Examples include a smartphone or computer.

[0786] A "natural language processing algorithm" is a technology for analyzing text data and understanding its meaning and structure, and is used to accurately analyze the content of user questions.

[0787] The "JSON format" is a text format for structuring and expressing data, and is a format that is often used for sending and receiving data.

[0788] An "HTTP request" is a protocol used when a client requests information from a server.

[0789] An "HTTP response" is a protocol used when a server returns information to a client.

[0790] The system of the present invention is designed to enable users to create plans quickly and accurately. This system allows users to input questions about the plan into a terminal, and provides appropriate information based on the questions. Specific embodiments of this system are described below.

[0791] Hardware and software used

[0792] This system consists of three main components: the user, the terminal, and the server. The terminal can be a smartphone or a computer. The server is a server computer operated in a cloud environment or on-premise. The software includes client applications, server-side programs, natural language processing algorithms (NLP tools), and various APIs.

[0793] Explanation of program processing

[0794] 1. User Input

[0795] The user uses the client application on the terminal to input a question about the plan. For example, the user inputs, "Please tell me recommended tourist spots and restaurants in Tokyo."

[0796] 2. Data transmission by the terminal

[0797] The terminal captures the user's question in text format and converts it into JSON format. The converted JSON data is sent to the server as an HTTP request. The software used for this process is a request library such as fetch or axios.

[0798] 3. Data analysis by the server

[0799] The server parses the received JSON data and analyzes the question using natural language processing (NLP) algorithms, for example by extracting keywords using libraries such as spaCy or NLTK.

[0800] For example, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted. These keywords are used as queries to search for related information.

[0801] 4. Acquiring relevant data

[0802] Based on the extracted keywords, the server sends queries to databases or external APIs to retrieve relevant data. For example, it uses the requests library to access an API and retrieve information about tourist attractions and restaurants.

[0803] 5. Generating and Sending the Response

[0804] The server generates a response based on the acquired data. This response includes information on tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza." The generated response is converted to JSON format and sent to the device as an HTTP response.

[0805] 6. Displaying data on a terminal

[0806] The client application on the device parses the received response and displays it in a user-friendly format, using HTML and CSS to format the layout.

[0807] Specific examples

[0808] For example, a user enters the following question:

[0809] > "What are some recommended tourist spots and restaurants in Tokyo?"

[0810] In response to this question, the system can return information such as:

[0811] Tourist attractions:

[0812] Tokyo Tower

[0813] Sensoji Temple

[0814] Skytree

[0815] Restaurant:

[0816] Tsukiji Market

[0817] Restaurants in Odaiba

[0818] Sushi restaurant in Ginza

[0819] In this way, the user can quickly and easily obtain the necessary information and create an optimal plan.

[0820] The system of the present invention reduces the burden on the user and makes it possible to support comfortable planning.

[0821] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0822] Step 1:

[0823] The user inputs a question into the terminal. The user opens the client application on the terminal and inputs a question about the plan. For example, the user inputs, "Please tell me recommended tourist spots and restaurants in Tokyo." This input text becomes input data for the next processing step.

[0824] Step 2:

[0825] The device sends a question to the server. The device captures the question text from the user in text format and converts it to JSON format. This converted data becomes the body of an HTTP request and is sent to the server. Specifically, the device uses a request library such as fetch or axios to send the data. The output of this step is JSON format data that the server can process.

[0826] Step 3:

[0827] The server analyzes the question. The server parses the received JSON data and uses a natural language processing algorithm (NLP) to analyze the question. This analysis process uses libraries such as spaCy and NLTK to tokenize the text and extract keywords. The input for this process is the JSON data, and the output is a list of extracted keywords. For example, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted.

[0828] Step 4:

[0829] The server retrieves relevant data from a database or external API. Based on the extracted keywords, the server creates a database query or external API query to retrieve the required data. Specifically, it uses the requests library to access the API and retrieve information about tourist attractions and restaurants. The input for this step is the keywords, and the output is a list of the retrieved tourist attractions and restaurants.

[0830] Step 5:

[0831] The server generates a response. Based on the acquired data, the server generates a response to return to the user. This response includes a list of tourist spots and restaurants. For example, tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree" and restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi Restaurants in Ginza" are listed. The input of this step is the acquired data, and the output is the formatted response data.

[0832] Step 6:

[0833] The server sends the response to the terminal. The generated response data is converted back to JSON format and sent to the terminal as an HTTP response. Specifically, the server generates the HTTP response using a web framework (e.g., Flask). The input of this step is the formatted response data, and the output is JSON-formatted response data.

[0834] Step 7:

[0835] The device displays the response. The device's client application parses the received response and displays it in a format that is easy for the user to understand. Specifically, it formats the data using HTML and CSS and provides it to the user. The input to this step is the response data in JSON format, and the output is a displayed list of tourist attractions and restaurants.

[0836] By performing the above steps, the system can quickly and accurately provide the necessary information based on the user's question.

[0837] (Application example 1)

[0838] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0839] Conventional tour planning systems only provide users with information in a static format, making it difficult to provide intuitive understanding or a realistic experience. In particular, there is a lack of technology for providing tour planning information visually and interactively, which has led to issues such as reduced convenience and satisfaction for users.

[0840] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0841] In this invention, the server includes means for receiving a question about a tour plan from a user, means for analyzing the content of the user's question and extracting keywords, means for sending a query to a database or an external API based on the extracted keywords to obtain related data, means for integrating the obtained data and generating a response to return to the user, means for sending the generated response to the user's terminal, and means for reproducing the obtained data in a virtual environment using a head-mounted display. This allows the user to visually experience tourist spots and restaurants in the virtual environment, enabling them to create a more intuitive and realistic tour plan.

[0842] "User" refers to a person who uses the system to ask questions or obtain information about tour plans.

[0843] A "tour plan question" refers to an inquiry from a user to ask about tourist attractions and restaurant information for a particular travel destination.

[0844] "Keywords" refer to important words or phrases extracted from the content of a user's question and are used as search queries.

[0845] A "database" refers to a collection of information about tourist spots and restaurants.

[0846] An "external API" refers to an interface for obtaining data from other systems or services.

[0847] A "query" is a search request sent to a database or external API requesting information.

[0848] A "response" refers to a set of information that a system returns in response to a question from a user.

[0849] "Terminal" refers to a device through which a user accesses the system, including smartphones, tablets, and PCs.

[0850] "Head-mounted display" refers to a display device worn by a user to visually experience a virtual environment.

[0851] A "virtual environment" refers to a virtual space constructed using computer technology that users can experience visually and interactively.

[0852] The embodiment of this invention is a system that allows users to easily and quickly create a tour plan and visually experience it in a virtual environment. The system mainly consists of a user, a terminal, a server, and a head-mounted display.

[0853] First, a user accesses the system through a terminal and inputs a question about a tour plan. For example, a user might input a question like, "Please tell me the recommended tourist spots and restaurants in Tokyo."

[0854] The device captures the user's question in text format, converts it to JSON format, and sends it to the server as an HTTP request. The server receives this request and begins analyzing it. The natural language processing algorithm used here is spaCy, for example.

[0855] The server parses the received JSON data and analyzes the user's question. Through analysis, it extracts keywords such as "Tokyo," "tourist spots," and "restaurants." These keywords are used as queries to retrieve related data. The server then sends queries to databases or external APIs based on the extracted keywords to retrieve data on tourist spots and restaurants.

[0856] The server generates a response based on the acquired data. This response includes tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza." The generated response is again converted to JSON format and sent to the device as an HTTP response.

[0857] When the device receives this response, it parses the data and formats it into a layout for display to the user. However, a distinctive feature of the system is that it uses a head-mounted display (HMD) to recreate the acquired data in a virtual environment. This virtual environment is built using a game engine such as Unity.

[0858] By wearing the HMD, users can visually experience the tourist spots and restaurants provided in the virtual environment, allowing them to check and plan their tour plans as if they were actually visiting the places.

[0859] As a concrete example, if a user asks the device, "What are some recommended tourist spots and restaurants in Tokyo?", the system will recreate tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza" in the virtual environment. In this process, the following prompt sentence can be input to the generative AI model: "What are some recommended tourist spots and restaurants in Tokyo?"

[0860] In this way, users can visually receive tour plan suggestions and plan their trip through a virtual experience on the spot. The hardware used includes head-mounted displays such as Oculus Rift and HTC Vive. The software used includes Unity, Python, spaCy, and external APIs.

[0861] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0862] Step 1:

[0863] The user inputs a question about the tour plan into the terminal. For example, the user inputs the text "Please tell me the recommended tourist spots and restaurants in Tokyo."

[0864] Step 2:

[0865] The terminal receives the user's question in text format and converts it to JSON format for subsequent server processing. The input value is the user's question text, and the output is a JSON object.

[0866] Step 3:

[0867] The terminal sends the converted JSON data to the server as an HTTP request. This request includes the user's question and is sent to the server's URL. The input value is a JSON object, and the output is an HTTP request.

[0868] Step 4:

[0869] The server receives the HTTP request and parses the JSON data. During this process, the parsed data is converted into a structured format and the user's question is analyzed. The input value is the JSON data of the HTTP request, and the output is structured data.

[0870] Step 5:

[0871] The server uses a natural language processing (NLP) algorithm to extract keywords from the data analyzed. In this process, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted from the user's question. The input is structured data, and the output is a set of extracted keywords.

[0872] Step 6:

[0873] The server uses the extracted keywords to send queries to databases or external APIs to obtain the required information. Specifically, queries to obtain data on tourist attractions and restaurants are created and sent to the API or database. The input is a set of keywords, and the output is data on tourist attractions and restaurants.

[0874] Step 7:

[0875] The server generates a response to return to the user based on the data it has acquired. The response includes information about tourist spots and restaurants, and is formatted in a way that is easy for the user to understand. The input is the acquired data, and the output is the formatted response.

[0876] Step 8:

[0877] The server converts the generated response back into JSON format and sends it as an HTTP response to the terminal. The input is a formatted response, and the output is a JSON-formatted HTTP response.

[0878] Step 9:

[0879] The device receives the JSON data returned from the server, parses the data, and displays it to the user. This display is not only displayed on the device screen, but also reproduced in a virtual environment via a head-mounted display (HMD). The input is the JSON data of the HTTP response, and the output is the visual information displayed to the user.

[0880] Step 10:

[0881] By wearing an HMD, users can visually experience tourist spots and restaurants in a virtual environment, giving them the feeling that they are actually visiting those places. The input is the visual information in the virtual environment, and the output is the user's experience.

[0882] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0883] The system of the present invention is designed to enable users to create tour plans easily and quickly. The system allows users to input questions about the tour plan into a terminal, and provides information on appropriate tourist spots and restaurants based on the questions. Furthermore, the system incorporates an emotion engine that recognizes the user's emotions, allowing it to provide appropriate responses based on the user's emotions.

[0884] This system consists of four main components: the user, the terminal, the server, and the emotion engine. The user accesses the system through the terminal and inputs questions about the tour plan they want. For example, consider the case where a user inputs, "Please tell me recommended tourist spots and restaurants in Tokyo."

[0885] When the device receives user input, it sends the input to the server. The client application on the device captures the user's question in text format, converts it to JSON format, and sends it as an HTTP request to the server. The server receives this request and begins analyzing it.

[0886] The server first parses the received JSON data and uses a natural language processing algorithm (NLP) to analyze the user's question. At this stage, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted from the question. Next, an emotion engine is used to analyze the user's emotions from the text input. This emotion analysis provides information such as whether the user is happy or anxious.

[0887] Next, the server sends a query to a database or external API based on the extracted keywords and the results of the sentiment analysis. For example, the server queries an external API for "popular tourist spots in Tokyo" and receives a list of tourist spots in response. Similarly, it retrieves data on "popular restaurants in Tokyo." In addition, it adjusts the response content to match the user's emotions based on the results of the sentiment analysis. For example, if the user is feeling anxious, it adds reassuring language to the response.

[0888] Based on the retrieved data, the server generates a response to return to the user. This response might include tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza." This information is then formatted and summarized in a form that is easy for the user to understand.

[0889] The generated response is converted back to JSON format and sent to the device as an HTTP response. When the device receives this response, the client application parses the data and formats it into a layout to display to the user. Finally, the user can see a list of tourist attractions and restaurants on the device screen.

[0890] For example, a user receives the following tour plan information:

[0891] Tourist attractions: Tokyo Tower, Sensoji Temple, Skytree

[0892] Restaurants: Tsukiji Market, Odaiba restaurants, Ginza sushi restaurants

[0893] In addition, responses are adjusted based on sentiment analysis, so if the user is feeling anxious, the phrase "Enjoy with peace of mind" can be added, increasing the user's sense of trust.

[0894] In this way, users can easily obtain the information they need for travel planning and quickly create the tour plan that best suits them. Sentiment analysis provides responses that are more tailored to the user's needs, improving user satisfaction.

[0895] The processing flow will be explained below.

[0896] Step 1:

[0897] The user types into the terminal, "Please tell me some recommended tourist spots and restaurants in Tokyo."

[0898] Step 2:

[0899] The terminal captures the user's input in text format and converts it to JSON format.

[0900] Step 3:

[0901] The terminal sends the converted JSON data to the server as an HTTP request.

[0902] Step 4:

[0903] The server receives the JSON data, parses it, and extracts the user's question.

[0904] Step 5:

[0905] The server uses a natural language processing algorithm (NLP) to analyze the question and extract important keywords (e.g., "Tokyo," "tourist spots," "restaurants").

[0906] Step 6:

[0907] The server uses an emotion engine to analyze emotions from the user's text input, for example, to determine whether the user is happy or anxious.

[0908] Step 7:

[0909] The server sends a query to a database or external API based on the extracted keywords and analyzed sentiment. For example, it sends a query such as "popular tourist spots in Tokyo."

[0910] Step 8:

[0911] The server receives the response data to the query and obtains information about tourist spots and restaurants.

[0912] Step 9:

[0913] Based on the data acquired by the server, it generates a response to be sent to the user. For example, it creates a list of tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as a list of restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza."

[0914] Step 10:

[0915] The server adjusts the response based on the results of the sentiment analysis. For example, if the user is feeling anxious, it adds reassuring language to the response.

[0916] Step 11:

[0917] The response generated by the server is converted to JSON format and sent to the terminal as an HTTP response.

[0918] Step 12:

[0919] The device parses the JSON data received from the server and formats it into a layout to display to the user.

[0920] Step 13:

[0921] Users can view a list of tourist attractions and restaurants on their device screen.

[0922] Example 2

[0923] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0924] Conventional tour planning systems only provide information on tourist spots and restaurants that users desire, without considering the user's emotional state, which can lead to low user satisfaction. Furthermore, many systems have limited application of natural language processing, which can result in inaccurate responses to user input.

[0925] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0926] In this invention, the server includes means for receiving a question about a tour plan from a user, means for analyzing the content of the user's question and extracting keywords, means for sending a query to a database or an external API based on the extracted keywords to obtain related data, means for integrating the obtained data and generating a response to be returned to the user, means for analyzing the user's emotions, means for adjusting the response content based on the result of the emotion analysis, and means for transmitting the generated response to the user's terminal. This makes it possible to propose a tour plan that is adapted to the user's emotional state and improve user satisfaction.

[0927] A "user" is someone who uses the system to input questions about tour plans and obtain information.

[0928] A "terminal" is a device that allows a user to input questions and view received information, and includes computers such as smartphones and personal computers.

[0929] A "server" is a computer system that has the function of analyzing a question received from a user, acquiring related data, generating a response, and sending it to a terminal.

[0930] The term "means" refers to a method or device for realizing a specific function, and is a technical element used in an embodiment of the present invention.

[0931] A "tour plan" is a plan that includes travel-related information such as tourist spots and restaurants.

[0932] The "question content" is a textual question entered by the user regarding the tour plan.

[0933] "Keywords" are the main words and phrases that make up the tour plan, extracted by analyzing the content of the question.

[0934] A "database" is a collection of information that stores information on related tourist spots and restaurants, etc., and can be searched as needed.

[0935] An "external API" is an interface for using functions provided by services or programs outside the system.

[0936] A "query" is a request sent to a database or external API to obtain required information.

[0937] "Emotion analysis" is a technology that analyzes a user's emotional state (e.g., joy, anxiety) from the content of their question.

[0938] A "response" is an answer generated by the server in response to a user's question, and includes information about tourist spots and restaurants.

[0939] A "natural language processing algorithm" is a technology for analyzing text entered by a user into a form that is easy for a machine to understand.

[0940] A "generative AI model" is a machine learning model used to automatically generate responses to user questions.

[0941] A "prompt sentence" is an instruction sentence that can be input into a generative AI model to obtain a desired response.

[0942] MODE FOR CARRYING OUT THE INVENTION

[0943] The system of the present invention is designed to enable users to easily and quickly create tour plans. The system allows users to input questions about the tour plan into a terminal, and provides information on suitable tourist spots and restaurants based on the questions. Furthermore, the system incorporates an emotion engine for analyzing the user's emotions, and also has the function of adjusting the response content according to the user's emotions. This improves user satisfaction.

[0944] The system is mainly composed of four main components: the user, the terminal, the server, and the emotion engine. Each component of the system is explained in detail below.

[0945] 1. Users

[0946] Users access the system using a device such as a smartphone or PC, enter questions about the tour plan in the input field on the device, and send the questions to the system.

[0947] 2. Terminal

[0948] The device is responsible for receiving user input and sending it to the server. The client application on the device captures the questions entered by the user in text format and converts them into JSON format. It then sends this JSON data to the server as an HTTP request. Specific examples of this include web browsers and mobile apps installed on smartphones and PCs.

[0949] 3. Server

[0950] The server receives and processes the HTTP request sent from the terminal. The server first analyzes the received JSON data and uses a natural language processing (NLP) algorithm to analyze the user's question. For example, a general-purpose natural language processing engine is used as the NLP algorithm.

[0951] Next, the server uses an emotion engine to analyze the user's emotions. Emotion analysis determines whether the user is feeling happy, anxious, etc. This allows the system to reflect the user's emotional state.

[0952] The server then sends queries to external databases or APIs based on the extracted keywords and the results of sentiment analysis. For example, an external API used to obtain tourist spot information is an API that provides geographic information. For example, a general geographic information API is used to obtain tourist spot information. Similarly, a query to obtain restaurant information is also sent.

[0953] Based on the retrieved data, the server generates a response to return to the user, which may include a list of tourist attractions or restaurants, and further adjusts the response based on the user's emotional state.

[0954] The server converts the generated response into JSON format and sends it to the terminal as an HTTP response.

[0955] 4. Emotion Engine

[0956] An emotion engine is an engine for analyzing emotions from text entered by a user. For example, it uses an emotion analysis API to analyze the user's emotional state and adjusts the response content based on the results.

[0957] Specific examples

[0958] The following concrete examples will help you understand how the system works:

[0959] The user enters a question such as:

[0960] What are some recommended tourist spots and restaurants in Tokyo?

[0961] The device sends this input to the server, which parses it and generates a response like this:

[0962] Tourist attractions: Tokyo Tower, Sensoji Temple, Skytree

[0963] Restaurants: Tsukiji Market, Odaiba restaurants, Ginza sushi restaurants

[0964] Additionally, if the user is feeling anxious, the response will include the following:

[0965] There are tourist spots you can enjoy without worry: Tokyo Tower, Sensoji Temple, and Skytree.

[0966] In this way, users can easily obtain the information they need to plan their trip. Sentiment analysis improves user satisfaction by providing appropriate responses based on the user's needs.

[0967] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0968] Step 1:

[0969] The user inputs a question into the terminal.

[0970] Input: The user enters text into the terminal, such as "Please tell me some recommended tourist spots and restaurants in Tokyo."

[0971] Data processing: The user enters text using a keyboard or touchscreen and presses the send button.

[0972] Output: The entered question is saved on the device.

[0973] Specific operation: The user enters a question into the input field on the terminal and clicks the "Submit" button.

[0974] Step 2:

[0975] The terminal sends user input to the server.

[0976] Input: The text data of the question entered by the user.

[0977] Data processing: The client application on the device converts the text into JSON format and sends it to the server as an HTTP request.

[0978] Output: The user's question is converted into JSON format and sent to the server as an HTTP request.

[0979] Specific behavior: The client application makes an HTTP POST request using JavaScript's fetch function.

[0980] Step 3:

[0981] The server parses the JSON data and performs natural language processing.

[0982] Input: The user's question in JSON format sent from the device.

[0983] Data processing: The server parses the received JSON data and extracts keywords using natural language processing algorithms.

[0984] Output: Extracted keywords (e.g., "Tokyo", "tourist spots", "restaurants").

[0985] Specific operation: The server parses the JSON data using a Python library and extracts keywords using an NLP engine (e.g., a general natural language processing engine).

[0986] Step 4:

[0987] The server performs emotion analysis using an emotion engine.

[0988] Input: Keywords extracted by natural language processing and the user's question.

[0989] Data processing: Using sentiment analysis algorithms to determine emotional states from text.

[0990] Output: User's emotional state (e.g., happy, anxious).

[0991] Specific operation: The server calls the emotion analysis API and analyzes the user's emotional state from the content of the question.

[0992] Step 5:

[0993] The server queries the external API to retrieve the relevant data.

[0994] Input: Extracted keywords (e.g., "Tokyo", "tourist spots", "restaurants") and emotional states.

[0995] Data processing: The server sends a query to an external API to obtain information about tourist attractions and restaurants.

[0996] Output: Data returned from the external API (list of tourist attractions and restaurants).

[0997] Specific operation: The server sends queries to multiple external APIs using HTTP requests and analyzes the responses.

[0998] Step 6:

[0999] The server generates a response based on the data it retrieves.

[1000] Input: Data obtained from external API and sentiment analysis results.

[1001] Data processing: Integrate the acquired information on tourist spots and restaurants, and add text based on the emotional state as needed.

[1002] Output: Generated response data (e.g., "Tokyo Tower," "Sensoji Temple," "Skytree," "Tsukiji Market," "Restaurants in Odaiba," "Sushi restaurants in Ginza").

[1003] Specific operation: The server uses a scripting language such as Python to integrate the data, add text, and convert it into JSON format.

[1004] Step 7:

[1005] The server generates a response and sends it to the terminal.

[1006] Input: The generated response data.

[1007] Data processing: Convert the response data into JSON format and send it to the terminal as an HTTP response.

[1008] Output: JSON formatted response data received by the device.

[1009] Specific operation: The server sends the HTTP response to the client application via a framework (e.g., Flask).

[1010] Step 8:

[1011] The terminal displays the received response to the user.

[1012] Input: JSON formatted response data received from the server.

[1013] Data processing: The client application parses the JSON data and converts it into a format that is easy for the user to read.

[1014] Output: A list of attractions and restaurants displayed on the user's device.

[1015] Specific operation: The client application uses a JavaScript framework (e.g., React.js) to format the response data into HTML and display it as a web page.

[1016] (Application example 2)

[1017] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1018] Conventional tour planning systems have difficulty adjusting responses flexibly based on user emotions, which can result in a poor user experience. Furthermore, there is a lack of methods to optimize the in-store shopping experience, leaving users with few efficient ways to obtain in-store product recommendations and sales information. Therefore, a method is needed to quickly and appropriately provide users with the information they desire.

[1019] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1020] In this invention, the server includes means including an emotion engine that recognizes a user's emotions, means for adjusting response content based on the results of emotion analysis, means for allowing a user to input a question using a smart device in a physical store in order to provide in-store product recommendations and sales information, and means for using a generative AI model, thereby generating an appropriate response based on the user's emotions and improving the shopping experience in the physical store.

[1021] A "user" is an individual who accesses the system through a terminal and inputs a question regarding a tour plan.

[1022] A "terminal" is a device used by a user, such as a smartphone or smart glasses.

[1023] A "server" is a central system that analyzes user questions and generates responses by sending queries to databases or external APIs.

[1024] An "emotion engine" is a technology that analyzes emotions from a user's text input and adjusts responses based on those emotions.

[1025] A "natural language processing algorithm (NLP)" is an algorithm that analyzes the content of a user's question and extracts keywords.

[1026] An "external API" is an interface for obtaining information from external systems such as databases.

[1027] A "generative AI model" is a machine learning model for performing tasks such as natural language processing and sentiment analysis.

[1028] A "physical store" is a sales point that exists in a physical location and where users can visit and purchase products.

[1029] A "query" is a request statement that requests information from a database or external API.

[1030] A "smart device" is a device that can connect to the Internet, such as a smartphone or smart glasses.

[1031] "Adjusting the answer content" means optimizing the information provided based on the user's emotions and the content of the question.

[1032] "Shopping experience" refers to the overall experience a user has when searching for and purchasing products in a physical store.

[1033] "Sale information" is information about discounts and special offers being offered in stores.

[1034] The system of this invention is designed to enable users to create tour plans easily and quickly. The system allows users to input questions about the tour plan into a terminal, and provides information on appropriate tourist spots and restaurants based on the questions. Furthermore, the system incorporates an emotion engine that recognizes the user's emotions, and can adjust the response content based on the user's emotions.

[1035] This system consists of four main components: the user, the terminal, the server, and the emotion engine. The user accesses the system through the terminal and inputs questions about the tour plan they want. For example, consider the case where a user inputs, "Please tell me the recommended tourist spots and restaurants in Tokyo."

[1036] When the device receives user input, it sends the input to the server. The client application on the device captures the user's question in text format, converts it to JSON format, and sends it as an HTTP request to the server. The server receives this request and begins analyzing it.

[1037] The server first parses the received JSON data and uses a natural language processing algorithm (NLP) to analyze the user's question. At this stage, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted from the question. Next, an emotion engine is used to analyze the user's emotions from the text input. This emotion analysis provides information such as whether the user is happy or anxious.

[1038] Next, the server sends a query to a database or external API based on the extracted keywords and the results of the sentiment analysis. For example, the server queries an external API for "popular tourist spots in Tokyo" and receives a list of tourist spots in response. Similarly, it retrieves data on "popular restaurants in Tokyo." In addition, it adjusts the response content to match the user's emotions based on the results of the sentiment analysis. For example, if the user is feeling anxious, it adds reassuring language to the response.

[1039] Based on the retrieved data, the server generates a response to return to the user. This response might include tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza." This information is then formatted and summarized in a form that is easy for the user to understand.

[1040] The generated response is converted back to JSON format and sent to the device as an HTTP response. When the device receives this response, the client application parses the data and formats it into a layout to display to the user. Finally, the user can see a list of tourist attractions and restaurants on the device screen. For example, the user receives the following tour plan information:

[1041] Tourist attractions: Tokyo Tower, Sensoji Temple, Skytree

[1042] Restaurants: Tsukiji Market, Odaiba restaurants, Ginza sushi restaurants

[1043] In addition, responses are adjusted based on sentiment analysis, so if the user is feeling anxious, the phrase "Enjoy with peace of mind" can be added, increasing the user's sense of trust.

[1044] To illustrate, here are some example prompts using a generative AI model (e.g., GPT-4):

[1045] User Input:

[1046] "What products do you recommend at this store?"

[1047] Example prompts for generative AI models:

[1048] A user asks, "What products do you recommend at this store?" Sentiment analysis shows they are feeling anxious. What kind of response should be returned as a recommendation?

[1049] In this way, users can easily obtain the information they need for travel planning and quickly create the tour plan that best suits them. Sentiment analysis provides responses that are more tailored to the user's needs, improving user satisfaction.

[1050] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1051] Step 1:

[1052] The user inputs a question about the tour plan into the terminal. The input format is text or voice, and the terminal converts this into text data and then converts it into JSON format. The input data is the "question content," and the output data is the "text-formatted question content."

[1053] Step 2:

[1054] The device sends JSON data containing the "text question" to the server as an HTTP request. At this time, the input sent from the device is "text data," and the output is "a request to the server."

[1055] Step 3:

[1056] The server parses the received JSON data and applies natural language processing (NLP) algorithms to analyze the user's question, where the input data is the "JSON-formatted question" and the output data is the "analyzed keywords."

[1057] Step 4:

[1058] The server uses an emotion engine to analyze the user's emotions from the question content. The input data is the "text question content" and the output data is the "user's emotional state." Specifically, the server identifies emotions from the text data and determines the emotional state, such as joy, anxiety, or excitement.

[1059] Step 5:

[1060] The server sends queries to databases and external APIs based on the extracted keywords and the results of sentiment analysis to retrieve relevant data. The input data is the "analyzed keywords" and "user emotional state," and the output data is the "retrieved tourist spot and restaurant information." Specifically, queries such as "popular tourist spots in Tokyo" and "popular restaurants in Tokyo" are sent to the external API.

[1061] Step 6:

[1062] The server integrates the acquired data and generates a response to return to the user. The input data is the acquired tourist spot and restaurant information and the user's emotional state, and the output data is the formatted response. Specifically, if the user is feeling anxious, the server adds the phrase "Enjoy yourself with peace of mind" to the response.

[1063] Step 7:

[1064] The server converts the generated response content back into JSON format and sends it to the terminal as an HTTP response. The input data is the "formatted response content" and the output data is the "response to the terminal."

[1065] Step 8:

[1066] The terminal receives the response from the server, and the client application parses the data and formats it into a layout to be displayed to the user. The input data is the "response from the server," and the output data is the "information to be displayed to the user." Specifically, information such as "Tourist spots: Tokyo Tower, Sensoji Temple, Skytree" is displayed.

[1067] The above is the specific flow of the program's processing. This system generates appropriate responses based on the user's emotions, enabling efficient tour planning. In addition, by using a generative AI model, it is possible to provide sophisticated responses tailored to the user's needs. An example of a prompt sentence is shown below:

[1068] User Input:

[1069] "What products do you recommend at this store?"

[1070] Example prompts for generative AI models:

[1071] A user asks, "What products do you recommend at this store?" Sentiment analysis shows they are feeling anxious. What kind of response should be returned as a recommendation?

[1072] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1073] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1074] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1075] [Fourth embodiment]

[1076] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1077] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1078] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1079] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1080] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1081] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1082] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1083] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1084] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1085] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1086] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1087] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1088] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1089] The system of the present invention is designed to enable users to create tour plans easily and quickly. This system allows users to input questions about the tour plan into a terminal, and provides information on appropriate tourist spots and restaurants based on the questions. Specific embodiments of this system are described below.

[1090] This system consists of three main components: the user, the terminal, and the server. The user accesses the system through the terminal and inputs questions related to the tour plan they want. For example, the user might input, "Please tell me the recommended tourist spots and restaurants in Tokyo."

[1091] When the device receives user input, it sends the input to the server. The client application on the device captures the user's question in text format, converts it to JSON format, and sends it as an HTTP request to the server. The server receives this request and begins analyzing it.

[1092] The server first parses the received JSON data and uses a natural language processing algorithm (NLP) to analyze the user's question. At this stage, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted from the question. These keywords are used as queries to retrieve related data.

[1093] Next, the server sends queries to a database or external API based on the extracted keywords. For example, the server queries an external API for "popular tourist spots in Tokyo" and receives a list of tourist spots in response. Similarly, it retrieves data on "popular restaurants in Tokyo."

[1094] Based on the acquired data, the server generates a response to return to the user. This response may include tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza." In this way, the server creates a list of information that best suits the user's question and formats it in a format that is easy for the user to understand.

[1095] The generated response is converted back to JSON format and sent to the device as an HTTP response. When the device receives this response, the client application parses the data and formats it into a layout to display to the user. Finally, the user can see a list of tourist attractions and restaurants on the device screen.

[1096] For example, a user receives the following tour plan information:

[1097] Tourist attractions: Tokyo Tower, Sensoji Temple, Skytree

[1098] Restaurants: Tsukiji Market, Odaiba restaurants, Ginza sushi restaurants

[1099] In this way, users can easily obtain the information they need to plan their trip and quickly create the tour plan that best suits them, reducing the burden on users and supporting comfortable trip planning.

[1100] The processing flow will be explained below.

[1101] Step 1:

[1102] The user types into the terminal, "Please tell me some recommended tourist spots and restaurants in Tokyo."

[1103] Step 2:

[1104] The terminal captures the user's input in text format and converts it to JSON format.

[1105] Step 3:

[1106] The terminal sends the converted JSON data to the server as an HTTP request.

[1107] Step 4:

[1108] The server receives the JSON data, parses it, and extracts the user's question.

[1109] Step 5:

[1110] The server uses a natural language processing algorithm (NLP) to analyze the question and extract important keywords (e.g., "Tokyo," "tourist spots," "restaurants").

[1111] Step 6:

[1112] The server sends a query to a database or external API based on the extracted keywords. For example, it sends a query such as "popular tourist spots in Tokyo."

[1113] Step 7:

[1114] The server receives the response data to the query and obtains information about tourist spots and restaurants.

[1115] Step 8:

[1116] The server integrates the acquired data and generates a response to return to the user. For example, it creates a list of tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as a list of restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza."

[1117] Step 9:

[1118] The response generated by the server is converted to JSON format and sent to the terminal as an HTTP response.

[1119] Step 10:

[1120] The device parses the JSON data received from the server and formats it into a layout to display to the user.

[1121] Step 11:

[1122] Users can view a list of tourist attractions and restaurants on their device screen.

[1123] Example 1

[1124] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1125] Conventional travel planning systems have had the problem of requiring a great deal of time and effort when users research trip details. In particular, the task of collecting and integrating necessary information from multiple sources is extremely time-consuming. It is also difficult to provide accurate information that meets the user's needs. This tends to make users' travel planning complicated and inefficient.

[1126] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1127] In this invention, the server includes means for receiving a plan-related question from a user, means for analyzing the content of the user's question and extracting keywords, means for sending a query to information sources based on the extracted keywords to obtain related data, means for integrating the obtained data and generating a response to return to the user, and means for transmitting the generated response to the user's terminal, thereby enabling the user to quickly and accurately obtain information necessary for planning and easily create a travel plan.

[1128] "Plan" is a plan that allows a user to organize and execute details of trips and activities.

[1129] A "question" is text or a query that a user enters to request information about a plan.

[1130] "Analysis" is the process of understanding the user's question and identifying important elements and information.

[1131] "Keywords" are important words or phrases extracted from a user's question that serve as the basis for conducting a search or query.

[1132] An "information source" is a data provider, such as a database or external API, that is used to obtain the required information.

[1133] A "query" is a request sent to a source to search for or retrieve specific information.

[1134] "Data" refers to the information and responses related to a user's question.

[1135] A "response" is information or a reply provided to a user's question, and includes content that is useful to the user.

[1136] A "terminal" is a device that a user uses to access and operate the system. Examples include a smartphone or computer.

[1137] A "natural language processing algorithm" is a technology for analyzing text data and understanding its meaning and structure, and is used to accurately analyze the content of user questions.

[1138] The "JSON format" is a text format for structuring and expressing data, and is a format that is often used for sending and receiving data.

[1139] An "HTTP request" is a protocol used when a client requests information from a server.

[1140] An "HTTP response" is a protocol used when a server returns information to a client.

[1141] The system of the present invention is designed to enable users to create plans quickly and accurately. This system allows users to input questions about the plan into a terminal, and provides appropriate information based on the questions. Specific embodiments of this system are described below.

[1142] Hardware and software used

[1143] This system consists of three main components: the user, the terminal, and the server. The terminal can be a smartphone or a computer. The server is a server computer operated in a cloud environment or on-premise. The software includes client applications, server-side programs, natural language processing algorithms (NLP tools), and various APIs.

[1144] Explanation of program processing

[1145] 1. User Input

[1146] The user uses the client application on the terminal to input a question about the plan. For example, the user inputs, "Please tell me recommended tourist spots and restaurants in Tokyo."

[1147] 2. Data transmission by the terminal

[1148] The terminal captures the user's question in text format and converts it into JSON format. The converted JSON data is sent to the server as an HTTP request. The software used for this process is a request library such as fetch or axios.

[1149] 3. Data analysis by the server

[1150] The server parses the received JSON data and analyzes the question using natural language processing (NLP) algorithms, for example by extracting keywords using libraries such as spaCy or NLTK.

[1151] For example, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted. These keywords are used as queries to search for related information.

[1152] 4. Acquiring relevant data

[1153] Based on the extracted keywords, the server sends queries to databases or external APIs to retrieve relevant data. For example, it uses the requests library to access an API and retrieve information about tourist attractions and restaurants.

[1154] 5. Generating and Sending the Response

[1155] The server generates a response based on the acquired data. This response includes information on tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza." The generated response is converted to JSON format and sent to the device as an HTTP response.

[1156] 6. Displaying data on a terminal

[1157] The client application on the device parses the received response and displays it in a user-friendly format, using HTML and CSS to format the layout.

[1158] Specific examples

[1159] For example, a user enters the following question:

[1160] > "What are some recommended tourist spots and restaurants in Tokyo?"

[1161] In response to this question, the system can return information such as:

[1162] Tourist attractions:

[1163] Tokyo Tower

[1164] Sensoji Temple

[1165] Skytree

[1166] Restaurant:

[1167] Tsukiji Market

[1168] Restaurants in Odaiba

[1169] Sushi restaurant in Ginza

[1170] In this way, the user can quickly and easily obtain the necessary information and create an optimal plan.

[1171] The system of the present invention reduces the burden on the user and makes it possible to support comfortable planning.

[1172] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1173] Step 1:

[1174] The user inputs a question into the terminal. The user opens the client application on the terminal and inputs a question about the plan. For example, the user inputs, "Please tell me recommended tourist spots and restaurants in Tokyo." This input text becomes input data for the next processing step.

[1175] Step 2:

[1176] The device sends a question to the server. The device captures the question text from the user in text format and converts it to JSON format. This converted data becomes the body of an HTTP request and is sent to the server. Specifically, the device uses a request library such as fetch or axios to send the data. The output of this step is JSON format data that the server can process.

[1177] Step 3:

[1178] The server analyzes the question. The server parses the received JSON data and uses a natural language processing algorithm (NLP) to analyze the question. This analysis process uses libraries such as spaCy and NLTK to tokenize the text and extract keywords. The input for this process is the JSON data, and the output is a list of extracted keywords. For example, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted.

[1179] Step 4:

[1180] The server retrieves relevant data from a database or external API. Based on the extracted keywords, the server creates a database query or external API query to retrieve the required data. Specifically, it uses the requests library to access the API and retrieve information about tourist attractions and restaurants. The input for this step is the keywords, and the output is a list of the retrieved tourist attractions and restaurants.

[1181] Step 5:

[1182] The server generates a response. Based on the acquired data, the server generates a response to return to the user. This response includes a list of tourist spots and restaurants. For example, tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree" and restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi Restaurants in Ginza" are listed. The input of this step is the acquired data, and the output is the formatted response data.

[1183] Step 6:

[1184] The server sends the response to the terminal. The generated response data is converted back to JSON format and sent to the terminal as an HTTP response. Specifically, the server generates the HTTP response using a web framework (e.g., Flask). The input of this step is the formatted response data, and the output is JSON-formatted response data.

[1185] Step 7:

[1186] The device displays the response. The device's client application parses the received response and displays it in a format that is easy for the user to understand. Specifically, it formats the data using HTML and CSS and provides it to the user. The input to this step is the response data in JSON format, and the output is a displayed list of tourist attractions and restaurants.

[1187] By performing the above steps, the system can quickly and accurately provide the necessary information based on the user's question.

[1188] (Application example 1)

[1189] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1190] Conventional tour planning systems only provide users with information in a static format, making it difficult to provide intuitive understanding or a realistic experience. In particular, there is a lack of technology for providing tour planning information visually and interactively, which has led to issues such as reduced convenience and satisfaction for users.

[1191] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1192] In this invention, the server includes means for receiving a question about a tour plan from a user, means for analyzing the content of the user's question and extracting keywords, means for sending a query to a database or an external API based on the extracted keywords to obtain related data, means for integrating the obtained data and generating a response to return to the user, means for sending the generated response to the user's terminal, and means for reproducing the obtained data in a virtual environment using a head-mounted display. This allows the user to visually experience tourist spots and restaurants in the virtual environment, enabling them to create a more intuitive and realistic tour plan.

[1193] "User" refers to a person who uses the system to ask questions or obtain information about tour plans.

[1194] A "tour plan question" refers to an inquiry from a user to ask about tourist attractions and restaurant information for a particular travel destination.

[1195] "Keywords" refer to important words or phrases extracted from the content of a user's question and are used as search queries.

[1196] A "database" refers to a collection of information about tourist spots and restaurants.

[1197] An "external API" refers to an interface for obtaining data from other systems or services.

[1198] A "query" is a search request sent to a database or external API requesting information.

[1199] A "response" refers to a set of information that a system returns in response to a question from a user.

[1200] "Terminal" refers to a device through which a user accesses the system, including smartphones, tablets, and PCs.

[1201] "Head-mounted display" refers to a display device worn by a user to visually experience a virtual environment.

[1202] A "virtual environment" refers to a virtual space constructed using computer technology that users can experience visually and interactively.

[1203] The embodiment of this invention is a system that allows users to easily and quickly create a tour plan and visually experience it in a virtual environment. The system mainly consists of a user, a terminal, a server, and a head-mounted display.

[1204] First, a user accesses the system through a terminal and inputs a question about a tour plan. For example, a user might input a question like, "Please tell me the recommended tourist spots and restaurants in Tokyo."

[1205] The device captures the user's question in text format, converts it to JSON format, and sends it to the server as an HTTP request. The server receives this request and begins analyzing it. The natural language processing algorithm used here is spaCy, for example.

[1206] The server parses the received JSON data and analyzes the user's question. Through analysis, it extracts keywords such as "Tokyo," "tourist spots," and "restaurants." These keywords are used as queries to retrieve related data. The server then sends queries to databases or external APIs based on the extracted keywords to retrieve data on tourist spots and restaurants.

[1207] The server generates a response based on the acquired data. This response includes tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza." The generated response is again converted to JSON format and sent to the device as an HTTP response.

[1208] When the device receives this response, it parses the data and formats it into a layout for display to the user. However, a distinctive feature of the system is that it uses a head-mounted display (HMD) to recreate the acquired data in a virtual environment. This virtual environment is built using a game engine such as Unity.

[1209] By wearing the HMD, users can visually experience the tourist spots and restaurants provided in the virtual environment, allowing them to check and plan their tour plans as if they were actually visiting the places.

[1210] As a concrete example, if a user asks the device, "What are some recommended tourist spots and restaurants in Tokyo?", the system will recreate tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza" in the virtual environment. In this process, the following prompt sentence can be input to the generative AI model: "What are some recommended tourist spots and restaurants in Tokyo?"

[1211] In this way, users can visually receive tour plan suggestions and plan their trip through a virtual experience on the spot. The hardware used includes head-mounted displays such as Oculus Rift and HTC Vive. The software used includes Unity, Python, spaCy, and external APIs.

[1212] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1213] Step 1:

[1214] The user inputs a question about the tour plan into the terminal. For example, the user inputs the text "Please tell me the recommended tourist spots and restaurants in Tokyo."

[1215] Step 2:

[1216] The terminal receives the user's question in text format and converts it to JSON format for subsequent server processing. The input value is the user's question text, and the output is a JSON object.

[1217] Step 3:

[1218] The terminal sends the converted JSON data to the server as an HTTP request. This request includes the user's question and is sent to the server's URL. The input value is a JSON object, and the output is an HTTP request.

[1219] Step 4:

[1220] The server receives the HTTP request and parses the JSON data. During this process, the parsed data is converted into a structured format and the user's question is analyzed. The input value is the JSON data of the HTTP request, and the output is structured data.

[1221] Step 5:

[1222] The server uses a natural language processing (NLP) algorithm to extract keywords from the data analyzed. In this process, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted from the user's question. The input is structured data, and the output is a set of extracted keywords.

[1223] Step 6:

[1224] The server uses the extracted keywords to send queries to databases or external APIs to obtain the required information. Specifically, queries to obtain data on tourist attractions and restaurants are created and sent to the API or database. The input is a set of keywords, and the output is data on tourist attractions and restaurants.

[1225] Step 7:

[1226] The server generates a response to return to the user based on the data it has acquired. The response includes information about tourist spots and restaurants, and is formatted in a way that is easy for the user to understand. The input is the acquired data, and the output is the formatted response.

[1227] Step 8:

[1228] The server converts the generated response back into JSON format and sends it as an HTTP response to the terminal. The input is a formatted response, and the output is a JSON-formatted HTTP response.

[1229] Step 9:

[1230] The device receives the JSON data returned from the server, parses the data, and displays it to the user. This display is not only displayed on the device screen, but also reproduced in a virtual environment via a head-mounted display (HMD). The input is the JSON data of the HTTP response, and the output is the visual information displayed to the user.

[1231] Step 10:

[1232] By wearing an HMD, users can visually experience tourist spots and restaurants in a virtual environment, giving them the feeling that they are actually visiting those places. The input is the visual information in the virtual environment, and the output is the user's experience.

[1233] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1234] The system of the present invention is designed to enable users to create tour plans easily and quickly. The system allows users to input questions about the tour plan into a terminal, and provides information on appropriate tourist spots and restaurants based on the questions. Furthermore, the system incorporates an emotion engine that recognizes the user's emotions, allowing it to provide appropriate responses based on the user's emotions.

[1235] This system consists of four main components: the user, the terminal, the server, and the emotion engine. The user accesses the system through the terminal and inputs questions about the tour plan they want. For example, consider the case where a user inputs, "Please tell me recommended tourist spots and restaurants in Tokyo."

[1236] When the device receives user input, it sends the input to the server. The client application on the device captures the user's question in text format, converts it to JSON format, and sends it as an HTTP request to the server. The server receives this request and begins analyzing it.

[1237] The server first parses the received JSON data and uses a natural language processing algorithm (NLP) to analyze the user's question. At this stage, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted from the question. Next, an emotion engine is used to analyze the user's emotions from the text input. This emotion analysis provides information such as whether the user is happy or anxious.

[1238] Next, the server sends a query to a database or external API based on the extracted keywords and the results of the sentiment analysis. For example, the server queries an external API for "popular tourist spots in Tokyo" and receives a list of tourist spots in response. Similarly, it retrieves data on "popular restaurants in Tokyo." In addition, it adjusts the response content to match the user's emotions based on the results of the sentiment analysis. For example, if the user is feeling anxious, it adds reassuring language to the response.

[1239] Based on the retrieved data, the server generates a response to return to the user. This response might include tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza." This information is then formatted and summarized in a form that is easy for the user to understand.

[1240] The generated response is converted back to JSON format and sent to the device as an HTTP response. When the device receives this response, the client application parses the data and formats it into a layout to display to the user. Finally, the user can see a list of tourist attractions and restaurants on the device screen.

[1241] For example, a user receives the following tour plan information:

[1242] Tourist attractions: Tokyo Tower, Sensoji Temple, Skytree

[1243] Restaurants: Tsukiji Market, Odaiba restaurants, Ginza sushi restaurants

[1244] In addition, responses are adjusted based on sentiment analysis, so if the user is feeling anxious, the phrase "Enjoy with peace of mind" can be added, increasing the user's sense of trust.

[1245] In this way, users can easily obtain the information they need for travel planning and quickly create the tour plan that best suits them. Sentiment analysis provides responses that are more tailored to the user's needs, improving user satisfaction.

[1246] The processing flow will be explained below.

[1247] Step 1:

[1248] The user types into the terminal, "Please tell me some recommended tourist spots and restaurants in Tokyo."

[1249] Step 2:

[1250] The terminal captures the user's input in text format and converts it to JSON format.

[1251] Step 3:

[1252] The terminal sends the converted JSON data to the server as an HTTP request.

[1253] Step 4:

[1254] The server receives the JSON data, parses it, and extracts the user's question.

[1255] Step 5:

[1256] The server uses a natural language processing algorithm (NLP) to analyze the question and extract important keywords (e.g., "Tokyo," "tourist spots," "restaurants").

[1257] Step 6:

[1258] The server uses an emotion engine to analyze emotions from the user's text input, for example, to determine whether the user is happy or anxious.

[1259] Step 7:

[1260] The server sends a query to a database or external API based on the extracted keywords and analyzed sentiment. For example, it sends a query such as "popular tourist spots in Tokyo."

[1261] Step 8:

[1262] The server receives the response data to the query and obtains information about tourist spots and restaurants.

[1263] Step 9:

[1264] Based on the data acquired by the server, it generates a response to be sent to the user. For example, it creates a list of tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as a list of restaurants such as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza."

[1265] Step 10:

[1266] The server adjusts the response based on the results of the sentiment analysis. For example, if the user is feeling anxious, it adds reassuring language to the response.

[1267] Step 11:

[1268] The response generated by the server is converted to JSON format and sent to the terminal as an HTTP response.

[1269] Step 12:

[1270] The device parses the JSON data received from the server and formats it into a layout to display to the user.

[1271] Step 13:

[1272] Users can view a list of tourist attractions and restaurants on their device screen.

[1273] Example 2

[1274] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1275] Conventional tour planning systems only provide information on tourist spots and restaurants that users desire, without considering the user's emotional state, which can lead to low user satisfaction. Furthermore, many systems have limited application of natural language processing, which can result in inaccurate responses to user input.

[1276] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1277] In this invention, the server includes means for receiving a question about a tour plan from a user, means for analyzing the content of the user's question and extracting keywords, means for sending a query to a database or an external API based on the extracted keywords to obtain related data, means for integrating the obtained data and generating a response to be returned to the user, means for analyzing the user's emotions, means for adjusting the response content based on the result of the emotion analysis, and means for transmitting the generated response to the user's terminal. This makes it possible to propose a tour plan that is adapted to the user's emotional state and improve user satisfaction.

[1278] A "user" is someone who uses the system to input questions about tour plans and obtain information.

[1279] A "terminal" is a device that allows a user to input questions and view received information, and includes computers such as smartphones and personal computers.

[1280] A "server" is a computer system that has the function of analyzing a question received from a user, acquiring related data, generating a response, and sending it to a terminal.

[1281] The term "means" refers to a method or device for realizing a specific function, and is a technical element used in an embodiment of the present invention.

[1282] A "tour plan" is a plan that includes travel-related information such as tourist spots and restaurants.

[1283] The "question content" is a textual question entered by the user regarding the tour plan.

[1284] "Keywords" are the main words and phrases that make up the tour plan, extracted by analyzing the content of the question.

[1285] A "database" is a collection of information that stores information on related tourist spots and restaurants, etc., and can be searched as needed.

[1286] An "external API" is an interface for using functions provided by services or programs outside the system.

[1287] A "query" is a request sent to a database or external API to obtain required information.

[1288] "Emotion analysis" is a technology that analyzes a user's emotional state (e.g., joy, anxiety) from the content of their question.

[1289] A "response" is an answer generated by the server in response to a user's question, and includes information about tourist spots and restaurants.

[1290] A "natural language processing algorithm" is a technology for analyzing text entered by a user into a form that is easy for a machine to understand.

[1291] A "generative AI model" is a machine learning model used to automatically generate responses to user questions.

[1292] A "prompt sentence" is an instruction sentence that can be input into a generative AI model to obtain a desired response.

[1293] MODE FOR CARRYING OUT THE INVENTION

[1294] The system of the present invention is designed to enable users to easily and quickly create tour plans. The system allows users to input questions about the tour plan into a terminal, and provides information on suitable tourist spots and restaurants based on the questions. Furthermore, the system incorporates an emotion engine for analyzing the user's emotions, and also has the function of adjusting the response content according to the user's emotions. This improves user satisfaction.

[1295] The system is mainly composed of four main components: the user, the terminal, the server, and the emotion engine. Each component of the system is explained in detail below.

[1296] 1. Users

[1297] Users access the system using a device such as a smartphone or PC, enter questions about the tour plan in the input field on the device, and send the questions to the system.

[1298] 2. Terminal

[1299] The device is responsible for receiving user input and sending it to the server. The client application on the device captures the questions entered by the user in text format and converts them into JSON format. It then sends this JSON data to the server as an HTTP request. Specific examples of this include web browsers and mobile apps installed on smartphones and PCs.

[1300] 3. Server

[1301] The server receives and processes the HTTP request sent from the terminal. The server first analyzes the received JSON data and uses a natural language processing (NLP) algorithm to analyze the user's question. For example, a general-purpose natural language processing engine is used as the NLP algorithm.

[1302] Next, the server uses an emotion engine to analyze the user's emotions. Emotion analysis determines whether the user is feeling happy, anxious, etc. This allows the system to reflect the user's emotional state.

[1303] The server then sends queries to external databases or APIs based on the extracted keywords and the results of sentiment analysis. For example, an external API used to obtain tourist spot information is an API that provides geographic information. For example, a general geographic information API is used to obtain tourist spot information. Similarly, a query to obtain restaurant information is also sent.

[1304] Based on the retrieved data, the server generates a response to return to the user, which may include a list of tourist attractions or restaurants, and further adjusts the response based on the user's emotional state.

[1305] The server converts the generated response into JSON format and sends it to the terminal as an HTTP response.

[1306] 4. Emotion Engine

[1307] An emotion engine is an engine for analyzing emotions from text entered by a user. For example, it uses an emotion analysis API to analyze the user's emotional state and adjusts the response content based on the results.

[1308] Specific examples

[1309] The following concrete examples will help you understand how the system works:

[1310] The user enters a question such as:

[1311] What are some recommended tourist spots and restaurants in Tokyo?

[1312] The device sends this input to the server, which parses it and generates a response like this:

[1313] Tourist attractions: Tokyo Tower, Sensoji Temple, Skytree

[1314] Restaurants: Tsukiji Market, Odaiba restaurants, Ginza sushi restaurants

[1315] Additionally, if the user is feeling anxious, the response will include the following:

[1316] There are tourist spots you can enjoy without worry: Tokyo Tower, Sensoji Temple, and Skytree.

[1317] In this way, users can easily obtain the information they need to plan their trip. Sentiment analysis improves user satisfaction by providing appropriate responses based on the user's needs.

[1318] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1319] Step 1:

[1320] The user inputs a question into the terminal.

[1321] Input: The user enters text into the terminal, such as "Please tell me some recommended tourist spots and restaurants in Tokyo."

[1322] Data processing: The user enters text using a keyboard or touchscreen and presses the send button.

[1323] Output: The entered question is saved on the device.

[1324] Specific operation: The user enters a question into the input field on the terminal and clicks the "Submit" button.

[1325] Step 2:

[1326] The terminal sends user input to the server.

[1327] Input: The text data of the question entered by the user.

[1328] Data processing: The client application on the device converts the text into JSON format and sends it to the server as an HTTP request.

[1329] Output: The user's question is converted into JSON format and sent to the server as an HTTP request.

[1330] Specific behavior: The client application makes an HTTP POST request using JavaScript's fetch function.

[1331] Step 3:

[1332] The server parses the JSON data and performs natural language processing.

[1333] Input: The user's question in JSON format sent from the device.

[1334] Data processing: The server parses the received JSON data and extracts keywords using natural language processing algorithms.

[1335] Output: Extracted keywords (e.g., "Tokyo", "tourist spots", "restaurants").

[1336] Specific operation: The server parses the JSON data using a Python library and extracts keywords using an NLP engine (e.g., a general natural language processing engine).

[1337] Step 4:

[1338] The server performs emotion analysis using an emotion engine.

[1339] Input: Keywords extracted by natural language processing and the user's question.

[1340] Data processing: Using sentiment analysis algorithms to determine emotional states from text.

[1341] Output: User's emotional state (e.g., happy, anxious).

[1342] Specific operation: The server calls the emotion analysis API and analyzes the user's emotional state from the content of the question.

[1343] Step 5:

[1344] The server queries the external API to retrieve the relevant data.

[1345] Input: Extracted keywords (e.g., "Tokyo", "tourist spots", "restaurants") and emotional states.

[1346] Data processing: The server sends a query to an external API to obtain information about tourist attractions and restaurants.

[1347] Output: Data returned from the external API (list of tourist attractions and restaurants).

[1348] Specific operation: The server sends queries to multiple external APIs using HTTP requests and analyzes the responses.

[1349] Step 6:

[1350] The server generates a response based on the data it retrieves.

[1351] Input: Data obtained from external API and sentiment analysis results.

[1352] Data processing: Integrate the acquired information on tourist spots and restaurants, and add text based on the emotional state as needed.

[1353] Output: Generated response data (e.g., "Tokyo Tower," "Sensoji Temple," "Skytree," "Tsukiji Market," "Restaurants in Odaiba," "Sushi restaurants in Ginza").

[1354] Specific operation: The server uses a scripting language such as Python to integrate the data, add text, and convert it into JSON format.

[1355] Step 7:

[1356] The server generates a response and sends it to the terminal.

[1357] Input: The generated response data.

[1358] Data processing: Convert the response data into JSON format and send it to the terminal as an HTTP response.

[1359] Output: JSON formatted response data received by the device.

[1360] Specific operation: The server sends the HTTP response to the client application via a framework (e.g., Flask).

[1361] Step 8:

[1362] The terminal displays the received response to the user.

[1363] Input: JSON formatted response data received from the server.

[1364] Data processing: The client application parses the JSON data and converts it into a format that is easy for the user to read.

[1365] Output: A list of attractions and restaurants displayed on the user's device.

[1366] Specific operation: The client application uses a JavaScript framework (e.g., React.js) to format the response data into HTML and display it as a web page.

[1367] (Application example 2)

[1368] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1369] Conventional tour planning systems have difficulty adjusting responses flexibly based on user emotions, which can result in a poor user experience. Furthermore, there is a lack of methods to optimize the in-store shopping experience, leaving users with few efficient ways to obtain in-store product recommendations and sales information. Therefore, a method is needed to quickly and appropriately provide users with the information they desire.

[1370] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1371] In this invention, the server includes means including an emotion engine that recognizes a user's emotions, means for adjusting response content based on the results of emotion analysis, means for allowing a user to input a question using a smart device in a physical store in order to provide in-store product recommendations and sales information, and means for using a generative AI model, thereby generating an appropriate response based on the user's emotions and improving the shopping experience in the physical store.

[1372] A "user" is an individual who accesses the system through a terminal and inputs a question regarding a tour plan.

[1373] A "terminal" is a device used by a user, such as a smartphone or smart glasses.

[1374] A "server" is a central system that analyzes user questions and generates responses by sending queries to databases or external APIs.

[1375] An "emotion engine" is a technology that analyzes emotions from a user's text input and adjusts responses based on those emotions.

[1376] A "natural language processing algorithm (NLP)" is an algorithm that analyzes the content of a user's question and extracts keywords.

[1377] An "external API" is an interface for obtaining information from external systems such as databases.

[1378] A "generative AI model" is a machine learning model for performing tasks such as natural language processing and sentiment analysis.

[1379] A "physical store" is a sales point that exists in a physical location and where users can visit and purchase products.

[1380] A "query" is a request statement that requests information from a database or external API.

[1381] A "smart device" is a device that can connect to the Internet, such as a smartphone or smart glasses.

[1382] "Adjusting the answer content" means optimizing the information provided based on the user's emotions and the content of the question.

[1383] "Shopping experience" refers to the overall experience a user has when searching for and purchasing products in a physical store.

[1384] "Sale information" is information about discounts and special offers being offered in stores.

[1385] The system of this invention is designed to enable users to create tour plans easily and quickly. The system allows users to input questions about the tour plan into a terminal, and provides information on appropriate tourist spots and restaurants based on the questions. Furthermore, the system incorporates an emotion engine that recognizes the user's emotions, and can adjust the response content based on the user's emotions.

[1386] This system consists of four main components: the user, the terminal, the server, and the emotion engine. The user accesses the system through the terminal and inputs questions about the tour plan they want. For example, consider the case where a user inputs, "Please tell me the recommended tourist spots and restaurants in Tokyo."

[1387] When the device receives user input, it sends the input to the server. The client application on the device captures the user's question in text format, converts it to JSON format, and sends it as an HTTP request to the server. The server receives this request and begins analyzing it.

[1388] The server first parses the received JSON data and uses a natural language processing algorithm (NLP) to analyze the user's question. At this stage, keywords such as "Tokyo," "tourist spots," and "restaurants" are extracted from the question. Next, an emotion engine is used to analyze the user's emotions from the text input. This emotion analysis provides information such as whether the user is happy or anxious.

[1389] Next, the server sends a query to a database or external API based on the extracted keywords and the results of the sentiment analysis. For example, the server queries an external API for "popular tourist spots in Tokyo" and receives a list of tourist spots in response. Similarly, it retrieves data on "popular restaurants in Tokyo." In addition, it adjusts the response content to match the user's emotions based on the results of the sentiment analysis. For example, if the user is feeling anxious, it adds reassuring language to the response.

[1390] Based on the retrieved data, the server generates a response to return to the user. This response might include tourist spots such as "Tokyo Tower," "Sensoji Temple," and "Skytree," as well as "Tsukiji Market," "Restaurants in Odaiba," and "Sushi restaurants in Ginza." This information is then formatted and summarized in a form that is easy for the user to understand.

[1391] The generated response is converted back to JSON format and sent to the device as an HTTP response. When the device receives this response, the client application parses the data and formats it into a layout to display to the user. Finally, the user can see a list of tourist attractions and restaurants on the device screen. For example, the user receives the following tour plan information:

[1392] Tourist attractions: Tokyo Tower, Sensoji Temple, Skytree

[1393] Restaurants: Tsukiji Market, Odaiba restaurants, Ginza sushi restaurants

[1394] In addition, responses are adjusted based on sentiment analysis, so if the user is feeling anxious, the phrase "Enjoy with peace of mind" can be added, increasing the user's sense of trust.

[1395] To illustrate, here are some example prompts using a generative AI model (e.g., GPT-4):

[1396] User Input:

[1397] "What products do you recommend at this store?"

[1398] Example prompts for generative AI models:

[1399] A user asks, "What products do you recommend at this store?" Sentiment analysis shows they are feeling anxious. What kind of response should be returned as a recommendation?

[1400] In this way, users can easily obtain the information they need for travel planning and quickly create the tour plan that best suits them. Sentiment analysis provides responses that are more tailored to the user's needs, improving user satisfaction.

[1401] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1402] Step 1:

[1403] The user inputs a question about the tour plan into the terminal. The input format is text or voice, and the terminal converts this into text data and then converts it into JSON format. The input data is the "question content," and the output data is the "text-formatted question content."

[1404] Step 2:

[1405] The device sends JSON data containing the "text question" to the server as an HTTP request. At this time, the input sent from the device is "text data," and the output is "a request to the server."

[1406] Step 3:

[1407] The server parses the received JSON data and applies natural language processing (NLP) algorithms to analyze the user's question, where the input data is the "JSON-formatted question" and the output data is the "analyzed keywords."

[1408] Step 4:

[1409] The server uses an emotion engine to analyze the user's emotions from the question content. The input data is the "text question content" and the output data is the "user's emotional state." Specifically, the server identifies emotions from the text data and determines the emotional state, such as joy, anxiety, or excitement.

[1410] Step 5:

[1411] The server sends queries to databases and external APIs based on the extracted keywords and the results of sentiment analysis to retrieve relevant data. The input data is the "analyzed keywords" and "user emotional state," and the output data is the "retrieved tourist spot and restaurant information." Specifically, queries such as "popular tourist spots in Tokyo" and "popular restaurants in Tokyo" are sent to the external API.

[1412] Step 6:

[1413] The server integrates the acquired data and generates a response to return to the user. The input data is the acquired tourist spot and restaurant information and the user's emotional state, and the output data is the formatted response. Specifically, if the user is feeling anxious, the server adds the phrase "Enjoy yourself with peace of mind" to the response.

[1414] Step 7:

[1415] The server converts the generated response content back into JSON format and sends it to the terminal as an HTTP response. The input data is the "formatted response content" and the output data is the "response to the terminal."

[1416] Step 8:

[1417] The terminal receives the response from the server, and the client application parses the data and formats it into a layout to be displayed to the user. The input data is the "response from the server," and the output data is the "information to be displayed to the user." Specifically, information such as "Tourist spots: Tokyo Tower, Sensoji Temple, Skytree" is displayed.

[1418] The above is the specific flow of the program's processing. This system generates appropriate responses based on the user's emotions, enabling efficient tour planning. In addition, by using a generative AI model, it is possible to provide sophisticated responses tailored to the user's needs. An example of a prompt sentence is shown below:

[1419] User Input:

[1420] "What products do you recommend at this store?"

[1421] Example prompts for generative AI models:

[1422] A user asks, "What products do you recommend at this store?" Sentiment analysis shows they are feeling anxious. What kind of response should be returned as a recommendation?

[1423] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1424] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1425] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1426] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1427] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1428] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1429] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1430] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1431] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1432] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1433] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1434] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1435] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1436] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1437] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1438] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1439] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1440] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1441] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1442] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1443] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1444] The following is further disclosed regarding the above embodiment.

[1445] (Claim 1)

[1446] means for receiving inquiries from users regarding tour plans;

[1447] A means for analyzing the content of a user's question and extracting keywords;

[1448] A means to query databases or external APIs based on the extracted keywords to retrieve relevant data;

[1449] means for aggregating the retrieved data and generating a response to return to the user;

[1450] means for transmitting the generated response to the user's terminal;

[1451] A system including:

[1452] (Claim 2)

[1453] 10. The system of claim 1, further comprising: means for displaying the generated response on a user terminal.

[1454] (Claim 3)

[1455] 10. The system of claim 1, further comprising means for analyzing user queries using natural language processing algorithms.

[1456] "Example 1"

[1457] (Claim 1)

[1458] means for receiving planning queries from a user;

[1459] A means for analyzing the content of a user's question and extracting keywords;

[1460] means for sending queries to information sources based on the extracted keywords to retrieve relevant data;

[1461] means for aggregating the retrieved data and generating a response to return to the user;

[1462] means for transmitting the generated response to the user's terminal;

[1463] A system including:

[1464] (Claim 2)

[1465] 10. The system of claim 1, further comprising: means for displaying the generated response on a user terminal.

[1466] (Claim 3)

[1467] 10. The system of claim 1, further comprising means for analyzing user queries using natural language processing algorithms.

[1468] (Claim 4)

[1469] 10. The system of claim 1, further comprising means for capturing received questions in text format, converting the questions into JSON format, and transmitting the JSON format to the server.

[1470] (Claim 5)

[1471] 10. The system of claim 1, further comprising means for the server to convert the response into a JSON format and send it to the terminal as an HTTP response.

[1472] "Application Example 1"

[1473] (Claim 1)

[1474] means for receiving inquiries from users regarding tour plans;

[1475] A means for analyzing the content of a user's question and extracting keywords;

[1476] A means to query databases or external APIs based on the extracted keywords to retrieve relevant data;

[1477] means for aggregating the retrieved data and generating a response to return to the user;

[1478] means for transmitting the generated response to the user's terminal;

[1479] means for reproducing the acquired data in a virtual environment using a head-mounted display;

[1480] A system including:

[1481] (Claim 2)

[1482] 10. The system of claim 1, further comprising means for displaying the generated responses on a user terminal to visually present a tour plan within the virtual environment.

[1483] (Claim 3)

[1484] 10. The system of claim 1, further comprising means for analyzing a user's question using a natural language processing algorithm to generate data for visual representation within the virtual environment.

[1485] "Example 2: Combining Emotion Engines"

[1486] (Claim 1)

[1487] means for receiving inquiries from users regarding tour plans;

[1488] A means for analyzing the content of a user's question and extracting keywords;

[1489] A means to query databases or external APIs based on the extracted keywords to retrieve relevant data;

[1490] means for aggregating the retrieved data and generating a response to return to the user;

[1491] means for analyzing user emotions;

[1492] means for adjusting the response content based on the results of the sentiment analysis;

[1493] means for transmitting the generated response to the user's terminal;

[1494] A system including:

[1495] (Claim 2)

[1496] 10. The system of claim 1, further comprising: means for displaying the generated response on a user terminal.

[1497] (Claim 3)

[1498] 10. The system of claim 1, further comprising means for analyzing user queries using natural language processing algorithms.

[1499] "Application example 2 when combining emotion engines"

[1500] (Claim 1)

[1501] means for receiving inquiries from users regarding tour plans;

[1502] A means for analyzing the content of a user's question and extracting keywords;

[1503] A means to query databases or external APIs based on the extracted keywords to retrieve relevant data;

[1504] means for aggregating the retrieved data and generating a response to return to the user;

[1505] means for transmitting the generated response to the user's terminal;

[1506] means including an emotion engine for recognizing an emotion of a user;

[1507] means for adjusting the response content based on the results of the sentiment analysis;

[1508] A system including:

[1509] (Claim 2)

[1510] 10. The system of claim 1, further comprising: means for displaying the generated response on a user terminal.

[1511] (Claim 3)

[1512] 10. The system of claim 1, further comprising means for analyzing user queries using natural language processing algorithms.

[1513] (Claim 4)

[1514] 10. The system of claim 1, further comprising means for a user to input a question using a smart device in a physical store to provide in-store product recommendations and sales information.

[1515] (Claim 5)

[1516] 10. The system of claim 1, further comprising: means for using the generative AI model to generate appropriate responses to questions from a user. [Explanation of symbols]

[1517] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving inquiries from users regarding tour plans; A means for analyzing the content of a user's question and extracting keywords; A means to query databases or external APIs based on the extracted keywords to retrieve relevant data; means for aggregating the retrieved data and generating a response to return to the user; means for transmitting the generated response to the user's terminal; A system including:

2. The system of claim 1 further comprising means for displaying the generated response on a user terminal.

3. 10. The system of claim 1, further comprising means for analyzing the content of a user's question using a natural language processing algorithm.

Images