system

Abstract

We provide the system. [Solution] For setting a destination in an autonomous vehicle, A means for receiving user input and generating departure point and destination information, A means for calculating the optimal route using the received departure point and destination information, A means of presenting calculated route information to the user and confirming whether or not highways can be used, A system that includes means for optimizing routes based on user responses and generating final route information.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In conventional autonomous vehicles and public transportation, various services provided by crew members such as taxi drivers, ticket sellers, and in-vehicle salespersons require manual labor, and improvements in efficiency and cost have been demanded. Also, new technologies are needed to achieve flexible and prompt responses to user requests. The present invention aims to solve these problems and provide efficient and user-friendly services in autonomous vehicles and public transportation.

Means for Solving the Problems

[0005] The present invention provides a system for realizing various services in autonomous vehicles and public transportation as virtual crew members. Specifically, it is a system including the following means.

[0006] 1. The system includes means for receiving user input and generating departure point and destination information, means for calculating the optimal route using the received departure point and destination information, means for presenting the calculated route information to the user and confirming the availability of expressways, and means for optimizing the route based on the user's response and generating final route information, thereby streamlining destination setting and route calculation.

[0007] 2. To provide in-vehicle sales services, the system includes means for receiving user in-vehicle sales requests and sending them to a server, means for obtaining in-vehicle sales menus and inventory information, means for presenting the obtained menu information to the user, means for receiving user selection information and sending it to a server, means for checking the inventory of selected products and presenting a purchase confirmation message to the user, thereby improving the efficiency of in-vehicle sales and the user experience.

[0008] 3. To change a train ticket, the system will simplify and streamline the ticket change procedure by including means for receiving a user's ticket change request and sending it to the server, means for presenting the user with changeable option information calculated by the server, means for receiving the user's selection information and sending it to the server, and means for generating new ticket information and presenting it to the user.

[0009] As a result, the present invention reduces the effort required for driving and providing services in autonomous vehicles and public transportation, enabling faster and more efficient service delivery to users.

[0010] A "user" refers to a person who makes decisions regarding passengers or drivers using autonomous vehicles or public transportation.

[0011] A "terminal" refers to an electronic device or apparatus that a user uses to input information and receive information for using a service.

[0012] A "server" refers to a computer system that receives information sent from a terminal, processes it, and returns the results to the terminal.

[0013] An "autonomous vehicle" refers to a vehicle in which all driving operations are automatically performed by a system without the need for a human driver to operate.

[0014] A "destination" refers to a point set by the user as the target location.

[0015] A "starting point" refers to the location where an autonomous vehicle begins to move.

[0016] A "route" refers to the path from the starting point to the destination, and specifically refers to the travel route calculated by a navigation system.

[0017] A "highway" refers to a road with a higher speed limit compared to ordinary roads, which is used for efficient long-distance travel.

[0018] "In-vehicle sales" refers to a service for selling goods inside a vehicle, including the provision of food and beverages, groceries, etc.

[0019] A "ticket" refers to a paid certificate for using public transportation.

[0020] A "request" refers to a demand by the user for a specific service or operation.

[0021] An "option" refers to multiple choices or settings that can be selected by the user.

[0022] "Inventory information" refers to data indicating the current remaining quantity and status of goods for sale.

[0023] A "purchase confirmation message" refers to a notification for confirming the intention to purchase the goods selected by the user.

[0024] A "change request" refers to a demand by the user to change the content of a ticket that the user has already obtained.

Brief Description of Drawings

[0025] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0026] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0027] First, let's explain the terminology used in the following explanation.

[0028] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0029] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0030] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0031] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0032] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0033] [First Embodiment]

[0034] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0035] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0036] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0037] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0038] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0039] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0040] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0041] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0042] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0043] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0044] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0045] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0046] This invention is a system that provides various services in autonomous vehicles and public transportation by having virtual crew members. The specific system configuration and operation are described below.

[0047] System Configuration

[0048] This system consists of users, terminals, and servers. Users operate terminals such as tablets and smartphones to use autonomous vehicles and public transportation. The terminal receives user input, processes it, and sends it to the server. The server performs calculations and data processing based on the received information and sends the results to the terminal.

[0049] Setting the destination and route

[0050] First, the user specifies their starting point and destination. For example, the user sets the starting point to "Tokyo Station" and the destination to "Shinagawa Station" on the map application on their device. This information is sent from the device to the server. The server calculates the optimal route based on the received starting point and destination information. The calculated route information (distance, travel time, whether or not highways can be used, etc.) is sent to the device and displayed to the user. The user then selects whether or not to use highways. For example, if the user selects "Use highways," this information is sent from the device to the server. The server optimizes the route based on this selection, generates the final route information, and sends it to the device. The device then displays the final route on its navigation system.

[0051] In-train sales

[0052] Next, let's consider the case where a user requests an in-train purchase. The user operates the in-train sales application on their terminal and requests, "I want to order a bento box." This request is sent from the terminal to the server. The server retrieves the in-train sales menu and inventory information based on the specified request and sends it to the terminal. The terminal displays the retrieved menu information to the user. The user selects the desired product (for example, "sushi bento"). This selection information is sent from the terminal to the server. The server checks the inventory of the selected product and sends a purchase confirmation message to the terminal. The terminal displays this purchase confirmation message to the user. Once the user confirms, the ordering process is complete.

[0053] Ticket change

[0054] Finally, consider the case where a user requests a change to their ticket. The user operates the ticket management application on their terminal and requests a "change ticket." This request is sent from the terminal to the server. The server calculates the available change options based on the request and sends that information to the terminal. The terminal presents the user with the available change options (e.g., change time, change seat). The user selects the desired change option. This selection information is sent from the terminal to the server. The server generates new ticket information based on the selection and sends it to the terminal. The terminal displays the new ticket information to the user.

[0055] As described above, the present invention is a system in which users, terminals, and servers cooperate to provide efficient and user-friendly services.

[0056] The following describes the processing flow.

[0057] Setting the destination and route

[0058] Step 1:

[0059] The user operates the terminal and enters the departure point and destination. For example, the user enters "Tokyo Station" as the departure point and "Shinagawa Station" as the destination.

[0060] Step 2:

[0061] The terminal receives user input information and sends it to the server. Specifically, it sends data on the departure point and destination to the server.

[0062] Step 3:

[0063] The server calculates the optimal route based on the information it receives. This calculation uses external map APIs or the company's own database.

[0064] Step 4:

[0065] The server sends calculated route information to the terminal, including distance, estimated travel time, and whether highways can be used.

[0066] Step 5:

[0067] The terminal displays the route information it has received to the user. The user can then choose whether or not to use the highway.

[0068] Step 6:

[0069] The user selects whether or not to use the highway, and the terminal sends this selection information to the server.

[0070] Step 7:

[0071] The server recalculates the route based on the user's selection information and generates the final route information.

[0072] Step 8:

[0073] The server sends the final route information to the terminal. The terminal displays the final route on its navigation system.

[0074] In-train sales

[0075] Step 1:

[0076] The user operates the terminal and selects a request for in-car sales. For example, they might select the "Order a bento box" option.

[0077] Step 2:

[0078] The terminal sends the user's request to the server. Specifically, this includes the user ID and the request details.

[0079] Step 3:

[0080] The server retrieves the in-vehicle sales menu and inventory information. For example, it reads the latest menu and inventory numbers from the database.

[0081] Step 4:

[0082] The server sends the retrieved menu information to the terminal. This includes the product name, price, and stock quantity.

[0083] Step 5:

[0084] The terminal displays menu information to the user. The user selects the desired product.

[0085] Step 6:

[0086] The user selects an item, and the device sends that selection information to the server. Specifically, this includes the item ID and quantity.

[0087] Step 7:

[0088] The server checks the inventory of the selected product. If it is in stock, it generates a purchase confirmation message and sends it to the terminal.

[0089] Step 8:

[0090] The device displays a purchase confirmation message to the user. The order is finalized once the user confirms it.

[0091] Ticket change

[0092] Step 1:

[0093] The user operates the terminal and selects a request to change their ticket. For example, they might select the "Change boarding time" menu.

[0094] Step 2:

[0095] The terminal sends the user's request to the server. Specifically, this includes current ticket information and conditions for change.

[0096] Step 3:

[0097] The server calculates the options that can be changed. For example, it searches the database for available trains and seats.

[0098] Step 4:

[0099] The server sends the calculation results to the terminal. This includes information about trains and seats that can be changed.

[0100] Step 5:

[0101] The device displays change options to the user. The user selects the desired change option.

[0102] Step 6:

[0103] The user selects a change option, and the device sends that selection information to the server. Specifically, this includes the changed train ID and seat number.

[0104] Step 7:

[0105] The server generates new ticket information, which includes new train and seat information.

[0106] Step 8:

[0107] The server sends the new ticket information to the terminal. The terminal displays the new ticket information to the user.

[0108] (Example 1)

[0109] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0110] To improve user convenience and comfort in autonomous vehicles and public transportation, it is necessary to efficiently provide a wide range of services, such as destination setting, in-vehicle sales, and ticket changes. However, in conventional systems, these services are often fragmented, making it difficult to provide a unified experience. Furthermore, advanced data processing technology is required to process user input information quickly and accurately and provide users with the necessary information.

[0111] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0112] In this invention, the server includes means for receiving user input and generating departure point and destination information; means for calculating an optimal route using the received departure point and destination information; means for presenting the calculated route information to the user and confirming the availability of highways; means for optimizing the route based on the user's response and generating final route information; means for displaying the final route information on a navigation system; means for receiving the user's in-vehicle sales request and transmitting it to the server; means for acquiring in-vehicle sales menus and inventory information; means for presenting the acquired menu information to the user; means for receiving the user's selection information and transmitting it to the server; means for confirming the inventory of selected items and presenting a purchase confirmation message to the user; means for receiving user confirmation and completing the order process; means for receiving the user's ticket change request and transmitting it to the server; means for presenting the user with changeable option information calculated by the server; means for receiving the user's selection information and transmitting it to the server; means for generating new ticket information and presenting it to the user; and means for displaying the new ticket information on a navigation system. This enables the provision of a consistent user experience and the efficient execution of various services.

[0113] "User" refers to a person who operates this system in order to use autonomous vehicles or public transportation.

[0114] A "device" is an electronic device used by a user, and includes tablets, smartphones, and other similar devices.

[0115] A "server" refers to a central system that receives information from users, performs data processing and calculations, and sends the results to terminals.

[0116] The "starting point" refers to the initial location from which the user boards the vehicle, and is one of the location information settings in a map application.

[0117] "Destination information" refers to information about the location that the user sets as their destination.

[0118] "Optimal route" refers to the shortest or most efficient route information calculated by the server based on the origin and destination information received.

[0119] A "navigation system" is a system that displays optimal route information and final route information to guide the user to their destination.

[0120] An "in-car sales request" refers to a request sent from a terminal to a server when a user wishes to purchase goods while in a vehicle.

[0121] The "in-train sales menu" refers to a list of products offered on board the train.

[0122] "Inventory information" refers to information showing the availability of each product included in the in-train sales menu.

[0123] "Selected information" refers to information about products or services that the user has selected on their device.

[0124] A "purchase confirmation message" refers to a confirmation message sent by the server to the user to complete the purchase process for the selected product.

[0125] A "ticket change request" refers to a request sent from a user's device to a server when the user wishes to change the details of a ticket they have already purchased.

[0126] "Changeable option information" refers to the ticket modification options that the server calculates and that the user can select.

[0127] "New ticket information" refers to information about a new ticket generated based on the user's selection.

[0128] This invention relates to a virtual crew system that provides a unified range of services in autonomous vehicles and public transportation. The following describes embodiments of this system.

[0129] System Configuration

[0130] This system consists of users, terminals, and servers. Users operate terminals such as tablets and smartphones to use autonomous vehicles and public transportation. The terminal receives user input, processes it, and sends it to the server. The server performs calculations and data processing based on the received information and sends the results to the terminal.

[0131] Setting the destination and route

[0132] First, the user specifies their pick-up and destination. For example, the user uses their device's map application to set the departure point to "Tokyo Station" and the destination to "Shinagawa Station." This information is sent from the device to the server. The server uses the Google® Maps API to calculate the optimal route based on the received departure and destination information. The calculated route information (distance, travel time, whether highways can be used, etc.) is sent to the device and displayed to the user. The user then selects whether or not to use highways. For example, if they select "Use highways," this information is sent from the device to the server. The server optimizes the route based on this selection, generates the final route information, and sends it to the device. The device then displays the final route on its navigation system.

[0133] Specific example:

[0134] Prompt message:

[0135] "Please set the departure point to Tokyo Station and the destination to Shinagawa Station."

[0136] In-train sales

[0137] Next, let's consider the case where a user requests an in-train purchase. The user uses the in-train sales application on their terminal to request, "I want to order a bento box." This request is sent from the terminal to the server. The server retrieves the in-train sales menu and inventory information based on the specified request and sends it to the terminal. The terminal displays the retrieved menu information to the user. The user selects the desired item (for example, "sushi bento"). This selection information is sent from the terminal to the server. The server checks the inventory of the selected item and sends a purchase confirmation message to the terminal. The terminal displays this purchase confirmation message to the user. Once the user confirms, the ordering process is complete.

[0138] Specific example:

[0139] Prompt message:

[0140] "Please select and order your bento box."

[0141] Ticket change

[0142] Finally, consider the case where a user requests a change to their ticket. The user requests a "change ticket" by operating the ticket management application on their terminal. This request is sent from the terminal to the server. The server calculates the available change options based on the request and sends that information to the terminal. The server uses its own backend system and reservation management system. It presents the user with the available change options (e.g., change time, change seat). The user selects the desired change option and sends the selection information from the terminal to the server. The server generates new ticket information based on the selection and sends it to the terminal. The terminal displays the new ticket information to the user.

[0143] Specific example:

[0144] Prompt message:

[0145] "Please view and select the ticket change options."

[0146] As described above, this system enables efficient and user-friendly services through the collaboration of users, terminals, and servers. By utilizing data processing technologies, including the Google Maps API, this system makes it possible to provide advanced services.

[0147] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0148] Setting the destination and route

[0149] Step 1:

[0150] The user operates their device (tablet or smartphone) and opens a map application. The user sets the starting point to "Tokyo Station" and the destination to "Shinagawa Station," and then presses the send button.

[0151] Input: Departure point and destination information (Tokyo Station, Shinagawa Station)

[0152] Specific operation: The user specifies the starting point and destination by dragging and dropping pins on the map using their device.

[0153] Output: The departure point and destination information are set on the terminal.

[0154] Step 2:

[0155] The device sends its departure point and destination information to the server.

[0156] Input: Departure point and destination information set by the user.

[0157] Specific operation: The app on the device creates an HTTP request and sends it to the server.

[0158] Output: The server receives the departure and destination information.

[0159] Step 3:

[0160] The server calculates the optimal route based on the information it receives. The server uses the Google Maps API to obtain the optimal route (distance, travel time, whether highways can be used, etc.).

[0161] Input: Departure point and destination information

[0162] Specific operation: The server accesses the Google Maps API route search endpoint and sends the starting point and destination parameters.

[0163] Output: Optimal route information is generated on the server.

[0164] Step 4:

[0165] The server sends the calculated optimal route information to the terminal.

[0166] Input: Optimal route information

[0167] Specific operation: The server sends data in JSON format to the terminal.

[0168] Output: The terminal receives optimal route information.

[0169] Step 5:

[0170] The terminal displays the route information it has received to the user. The user can choose whether or not to use highways.

[0171] Input: Optimal route information

[0172] Specific action: The app on the device displays route information on the map and shows a checkbox indicating whether to "use highways."

[0173] Output: The user's highway usage selection information is determined.

[0174] Step 6:

[0175] The user selects the option to use the highway and presses the submit button. The terminal sends the selection information to the server.

[0176] Input: User's highway usage selection information

[0177] Specific operation: The user checks / unchecks a checkbox within the app on their device and presses the submit button. The device sends the selection information to the server via an HTTP request.

[0178] Output: The server receives the selection information.

[0179] Step 7:

[0180] The server recalculates the route based on the selected information and generates the final route information. It then sends the final route information to the terminal.

[0181] Input: User's highway usage selection information

[0182] Specific operation: The server uses the selected information to re-execute the optimal route calculation algorithm and generate the final route information.

[0183] Output: The final route information is generated on the server and sent to the terminal.

[0184] Step 8:

[0185] The device displays the final route information on the navigation system.

[0186] Input: Final route information

[0187] Specific operation: The device's navigation app receives the final route information and displays it on the map.

[0188] Output: The user can check the final route information.

[0189] In-train sales

[0190] Step 1:

[0191] The user uses the in-car sales application on their terminal to request, "I would like to order a bento box."

[0192] Input: User's in-car sales request

[0193] Specific action: The user selects "In-car sales" from the app's menu and presses the "Order a bento box" button.

[0194] Output: An in-car sales request is set on the terminal.

[0195] Step 2:

[0196] The terminal sends the user's in-car sales request to the server.

[0197] Input: User's in-car sales request

[0198] Specific action: The terminal creates an HTTP request and sends it to the server.

[0199] Output: The server receives an in-vehicle sales request.

[0200] Step 3:

[0201] The server retrieves the in-car sales menu and inventory information based on the request and sends it to the terminal.

[0202] Input: In-car sales request

[0203] Specific operation: The server retrieves the product list and inventory information from the database and sends it to the terminal.

[0204] Output: The in-car sales menu and inventory information are sent to the terminal.

[0205] Step 4:

[0206] The terminal displays the menu information it has acquired to the user, and the user selects the product they want (for example, "sushi bento").

[0207] Input: In-car sales menu and inventory information

[0208] Specific action: The app on the device displays a menu list, and the user taps on "Sushi Bento".

[0209] Output: User selection information is set on the terminal.

[0210] Step 5:

[0211] The device sends the user's selection information to the server.

[0212] Input: User's selected information

[0213] Specific action: The terminal creates an HTTP request containing the selection information and sends it to the server.

[0214] Output: The server receives the user's selection information.

[0215] Step 6:

[0216] The server checks the inventory of the selected product, generates a purchase confirmation message, and sends it to the terminal.

[0217] Input: User's selected information

[0218] Specific operation: The server retrieves the inventory quantity of the product from the database, and if it is in stock, it generates a purchase confirmation message.

[0219] Output: A purchase confirmation message is generated on the server and sent to the device.

[0220] Step 7:

[0221] The device displays a purchase confirmation message to the user, and once the user confirms it, the order process is completed.

[0222] Input: Purchase confirmation message

[0223] Specific action: The user presses the confirmation button, and the device sends the order information to the server.

[0224] Output: The order has been confirmed and the shipping process has begun.

[0225] Ticket change

[0226] Step 1:

[0227] The user requests a "ticket change" using the ticket management application on their device.

[0228] Input: User's ticket change request

[0229] Specific action: The user presses the "Change Ticket" button on the ticket management screen in the app.

[0230] Output: A ticket change request is set on the terminal.

[0231] Step 2:

[0232] The device sends a user change request to the server.

[0233] Input: User's ticket change request

[0234] Specific action: The terminal creates an HTTP request and sends it to the server.

[0235] Output: The server receives a ticket change request.

[0236] Step 3:

[0237] The server calculates the modifiable options based on the request and sends that information to the terminal.

[0238] Input: Ticket change request

[0239] Specific operation: The server retrieves the current reservation information from the database and calculates the options that can be changed.

[0240] Output: Modifiable option information is generated on the server and sent to the terminal.

[0241] Step 4:

[0242] The device presents the user with available modification options, and the user selects the desired modification option.

[0243] Input: Changeable option information

[0244] Specific operation: The app on the device displays change options, and the user selects one.

[0245] Output: User selection information is set on the terminal.

[0246] Step 5:

[0247] The device sends the user's selection information to the server.

[0248] Input: User's selected information

[0249] Specific action: The terminal creates an HTTP request containing the selection information and sends it to the server.

[0250] Output: The server receives the user's selection information.

[0251] Step 6:

[0252] The server generates new ticket information based on the selections and sends it to the terminal.

[0253] Input: User's selected information

[0254] Specific operation: The server generates new ticket information and sends it to the terminal.

[0255] Output: New ticket information is generated on the server and sent to the terminal.

[0256] Step 7:

[0257] The terminal displays the new ticket information to the user.

[0258] Input: New ticket information

[0259] Specific action: The app on the device displays the new ticket information, and the user confirms it.

[0260] Output: The user can view the new ticket information.

[0261] The above steps enable a system where users, terminals, and servers work together to provide efficient and user-friendly services. Including specific actions in each processing step makes the program flow clearer.

[0262] (Application Example 1)

[0263] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0264] The present invention aims to provide a method for providing food delivery services more seamlessly and efficiently. Specifically, the objective is to provide a system that allows users to order food from within an autonomous vehicle, track the delivery status in real time, and automatically generate appropriate responses to user inquiries and requests.

[0265] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0266] In this invention, the server includes means for receiving user input and generating departure point and destination information; means for calculating an optimal route using the received departure point and destination information; means for presenting the calculated route information to the user and confirming the availability of highways; means for optimizing the route based on the user's response and generating final route information; means for receiving user order information and providing food delivery services; means for providing the user with real-time delivery status and estimated arrival time; and means for analyzing user inquiries and providing appropriate answers using a generative AI model. This enables a seamless experience and rapid response when users utilize food delivery services in an autonomous vehicle.

[0267] "User input" refers to the information and instructions that a user provides through their device.

[0268] "Departure point" refers to the location where the user boards or the starting point.

[0269] "Destination information" refers to information about the place or destination that the user intends to go.

[0270] The "optimal route" refers to the most suitable travel route calculated considering the efficiency and convenience of the path.

[0271] "Availability of using expressways" refers to the user's choice of whether or not to use expressways.

[0272] "Final route information" refers to the final guidance information based on an optimized route.

[0273] "Order information" refers to the detailed information that users enter when purchasing food or products.

[0274] A "food delivery service" refers to a service that delivers food to a location specified by the user.

[0275] "Delivery status" refers to the current progress and status in the delivery process of food and goods.

[0276] "Expected arrival time" refers to the estimated time when the delivered item reaches the destination.

[0277] "Inquiry" refers to information regarding questions and requests from users.

[0278] "Generated AI model" refers to a model that automatically creates answers and information for user inquiries using artificial intelligence technology.

[0279] "Appropriate answer" refers to a response that provides accurate and appropriate information to user inquiries and requests.

[0280] The present invention is a system that supports user operations in an autonomous vehicle in order to efficiently provide a food delivery service. This system is composed of a user, a terminal, and a server. The specific operations of each component will be described below.

[0281] System Configuration

[0282] In this system, the user operates through the terminal and cooperates with the server to provide the service. The terminal uses a smartphone or tablet, and the server uses a cloud server (for example, AWS (registered trademark) or Azure (registered trademark)).

[0283] Setting the Destination and Route

[0284] First, the user inputs the departure point and destination information into the terminal. Using the map application on the terminal, the user specifies the departure point and destination. This information is sent from the terminal to the server, and the server calculates the optimal route. The calculated route information is presented to the user, and the user selects whether to use the highway. The server generates an optimized final route based on that information and sends it to the terminal.

[0285] Food delivery service

[0286] When a user places an order for a meal from a terminal, the order information is sent to the server. The server provides a food delivery service and offers the user the delivery status and expected arrival time in real time. The delivery status is tracked using GPS, and the expected arrival time is calculated dynamically.

[0287] Customer support

[0288] The user's inquiries are sent directly from the terminal to the server. The server analyzes the inquiry content using a generative AI model (e.g., GPT-4 (registered trademark)) and generates an appropriate answer. The generated answer is sent to the terminal and displayed to the user.

[0289] Hardware and software used

[0290] Hardware: Smartphones, tablets, cloud servers (AWS, Azure)

[0291] Software: Native app development tools for iOS or ANDROID (registered trademark) (Swift, Kotlin, etc.), server-side frameworks (Node.js, Flask, etc.), SQL databases (MySQL (registered trademark), PostgreSQL, etc.), generative AI models (GPT-4)

[0292] Specific example

[0293] The case where a user orders pizza inside an autonomous vehicle and monitors the delivery status in real time will be described. When the user operates the terminal to order pizza, the order information is sent to the server. The server processes the order information, and the delivery person departs with the pizza. The delivery status is tracked by GPS, the expected arrival time is calculated, and provided to the user in real time. Furthermore, when the user makes an inquiry, the generative AI model generates an appropriate answer, and the answer is displayed to the user.

[0294] Example of a prompt:

[0295] "A user has ordered a pizza. Please provide the delivery person's current location in real time and the estimated arrival time."

[0296] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0297] Step 1:

[0298] The user operates the terminal and enters departure and destination information. The entered information is temporarily stored within the terminal. The terminal then sends the entered departure and destination information as data to the server.

[0299] Input: Departure point and destination information entered by the user on the device.

[0300] Output: Departure point and destination information sent to the server

[0301] Step 2:

[0302] The server calculates the optimal route based on the received origin and destination information. It uses a route calculation algorithm (e.g., Dijkstra's algorithm) to identify the most efficient route. The calculated route information is stored internally by the server.

[0303] Input: Departure point and destination information

[0304] Output: Calculated optimal route information

[0305] Step 3:

[0306] The server sends the calculated route information to the terminal, which then presents the information to the user. The user reviews the route information and selects whether or not to use highways. The selection information is then sent back to the server from the terminal.

[0307] Input: Optimal route information

[0308] Output: User's selection information (availability of highway usage)

[0309] Step 4:

[0310] Based on the user's selection information, the server generates an optimized final route. The route is recalculated to generate new route information. The final route information is sent from the server to the terminal and presented to the user.

[0311] Input: User's selection information

[0312] Output: Final route information

[0313] Step 5:

[0314] The user orders food from the terminal. The terminal sends the order information to the server. The order information includes the type and quantity of food and the delivery destination information.

[0315] Input: User's order information

[0316] Output: Order information sent to the server

[0317] Step 6:

[0318] The server receives the order information and processes it. Inventory checks and delivery arrangements for the order are made. The server generates an order confirmation message and sends it to the terminal.

[0319] Input: User's order information

[0320] Output: Order confirmation message

[0321] Step 7:

[0322] The server tracks the progress of food delivery in real time. It uses GPS to obtain the location information of the delivery staff and calculates the estimated arrival time. This information is processed within the server.

[0323] Input: Delivery driver's location information

[0324] Output: Estimated arrival time and delivery status

[0325] Step 8:

[0326] The server sends the delivery status and estimated arrival time to the terminal. The terminal displays this information to the user, allowing them to check the delivery status in real time.

[0327] Input: Estimated arrival time and delivery status

[0328] Output: Delivery status and estimated arrival time presented to the user.

[0329] Step 9:

[0330] When a user submits an inquiry, the device sends the inquiry as text data to the server. The server uses a generative AI model to analyze the inquiry and generate an appropriate response.

[0331] Input: Inquiry details

[0332] Output: Generated answer

[0333] Step 10:

[0334] The server sends the response generated by the AI ​​model to the terminal. The terminal displays the response to the user, and appropriate action is taken.

[0335] Input: Generated answer

[0336] Output: The answer presented to the user

[0337] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0338] This invention combines a system that provides various services in autonomous vehicles and public transportation as virtual crew members with an emotion engine that recognizes user emotions, thereby achieving more advanced and user-friendly service provision. The specific system configuration and operation are described below.

[0339] System Configuration

[0340] This system consists of a user, a terminal, a server, and an emotion engine. The user operates a terminal such as a tablet or smartphone to use autonomous vehicles or public transportation. The terminal receives user input, processes it, and sends it to the server. The emotion engine recognizes the user's emotions in real time and sends that information to the server. The server performs calculations and data processing based on the received information and sends the results to the terminal.

[0341] Setting the destination and route

[0342] First, the user specifies their pick-up and destination. For example, the user sets the departure point to "Tokyo Station" and the destination to "Shinagawa Station" on their device's map application. This information is sent from the device to the server. The server calculates the optimal route based on the received departure and destination information. This calculation also takes into account sentiment data from the sentiment engine. The calculated route information (distance, travel time, whether or not highways can be used, etc.) is sent to the device and displayed to the user. Based on the user's sentiment, the choice of whether or not to use highways is optimized. For example, if the sentiment engine determines that the user is in a hurry, it will recommend using highways. The user's response information is sent back to the server, and the final route is determined.

[0343] In-train sales

[0344] Next, let's consider the case where a user requests an in-train purchase. The user operates the in-train sales application on their terminal and requests, "I want to order a bento box." This request is sent from the terminal to the server. Based on the specified request, the server retrieves the in-train sales menu and inventory information and sends it to the terminal. The emotion engine can also recognize the user's emotions and suggest products that match the user's mood. For example, if it determines that the user is tired, it will suggest a relaxing tea. The user selects the product they want, and this selection information is sent from the terminal to the server. The server checks the inventory of the selected product and sends a purchase confirmation message to the terminal. The terminal displays this purchase confirmation message to the user, and once the user confirms it, the order is finalized.

[0345] Ticket change

[0346] Finally, consider the case where a user requests a change to their ticket. The user operates the ticket management application on their terminal and requests a "change ticket." This request is sent from the terminal to the server. The server calculates the available options based on the request and sends that information to the terminal. The emotion engine analyzes the user's emotions and can suggest the most suitable change option based on the user's mood. For example, if it is determined that the user wants to relax, it will suggest a quiet carriage. The user selects their desired change option, and this selection information is sent from the terminal to the server. The server generates new ticket information based on the selection and sends it to the terminal. The terminal displays the new ticket information to the user.

[0347] As described above, the present invention is a system in which a user, terminal, server, and emotion engine work together to provide efficient and user-friendly services.

[0348] The following describes the processing flow.

[0349] Setting the destination and route

[0350] Step 1:

[0351] The user operates the device and enters the starting point and destination. For example, going from "Tokyo Station" to "Shinagawa Station".

[0352] Step 2:

[0353] The terminal receives user input information and sends it to the server. Specifically, data on the departure point and destination is sent to the server.

[0354] Step 3:

[0355] The server calculates the optimal route based on the information it receives. This calculation uses external map APIs and internal route calculation algorithms.

[0356] Step 4:

[0357] The server sends calculated route information to the terminal. This includes distance, estimated travel time, and whether highways can be used.

[0358] Step 5:

[0359] The terminal displays the route information it has received to the user. The user can then choose whether or not to use the highway.

[0360] Step 6:

[0361] The emotion engine analyzes the user's facial expressions and tone of voice to determine their emotional state. For example, if it determines that the user is anxious, it will recommend using the highway.

[0362] Step 7:

[0363] The user selects whether or not to use the highway, and the terminal sends this selection information to the server.

[0364] Step 8:

[0365] The server recalculates the route based on the user's selection information and sentiment data, and generates the final route information.

[0366] Step 9:

[0367] The server sends the final route information to the terminal. The terminal displays the final route on its navigation system.

[0368] In-train sales

[0369] Step 1:

[0370] The user operates the terminal and selects a request for in-car sales. For example, "Order a bento box."

[0371] Step 2:

[0372] The terminal sends the user's request to the server. Specifically, this includes the user ID and the request details.

[0373] Step 3:

[0374] The server retrieves the in-vehicle sales menu and inventory information. For example, it reads the latest menu and inventory numbers from the database.

[0375] Step 4:

[0376] The server sends the retrieved menu information to the terminal. This includes the product name, price, and stock quantity.

[0377] Step 5:

[0378] The terminal displays menu information to the user. The user selects the desired product.

[0379] Step 6:

[0380] The emotion engine analyzes the user's emotions and suggests products that match their mood. For example, it might suggest a relaxing tea to a tired user.

[0381] Step 7:

[0382] The user selects an item, and the device sends that selection information to the server. Specifically, this includes the item ID and quantity.

[0383] Step 8:

[0384] The server checks the inventory of the selected product. If it is in stock, it generates a purchase confirmation message and sends it to the terminal.

[0385] Step 9:

[0386] The device displays a purchase confirmation message to the user. Once the user confirms, the order is finalized.

[0387] Ticket change

[0388] Step 1:

[0389] The user operates the terminal and selects a request to change their ticket. For example, "Change departure time."

[0390] Step 2:

[0391] The terminal sends the user's request to the server. Specifically, this includes current ticket information and conditions for change.

[0392] Step 3:

[0393] The server calculates the options that can be changed. For example, it searches for available trains and seats.

[0394] Step 4:

[0395] The server sends the calculation results to the terminal. This includes information about trains and seats that can be changed.

[0396] Step 5:

[0397] The device displays change options to the user. The user selects the desired change option.

[0398] Step 6:

[0399] The emotion engine analyzes the user's emotions and suggests the optimal change options based on the user's mood. For example, it suggests a quiet vehicle to a user who wants to relax.

[0400] Step 7:

[0401] The user selects a change option, and the device sends that selection information to the server. Specifically, this includes the changed train ID and seat number.

[0402] Step 8:

[0403] The server generates new ticket information, which includes new train and seat information.

[0404] Step 9:

[0405] The server sends the new ticket information to the terminal. The terminal displays the new ticket information to the user.

[0406] (Example 2)

[0407] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0408] In providing services in autonomous vehicles and public transportation, traditional methods failed to consider user emotions, making it difficult to enhance user satisfaction. Furthermore, it was challenging to suggest optimal services and routes when users were in a hurry or experiencing specific emotional states. This resulted in a decline in service quality and a compromised user experience.

[0409] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0410] In this invention, the server includes means for receiving user input and generating departure point and destination information; means for calculating the optimal route using the received departure point and destination information; means for presenting the calculated route information to the user and confirming the availability of expressways; means for optimizing the route based on the user's response and generating final route information; means for recognizing the user's emotions in real time and transmitting that information to the server; means for optimizing the calculation results based on the information received from the emotion recognition means; and means for feeding the calculation results back to the user. This makes it possible to provide the optimal route and services based on the user's emotions, thereby improving the quality of the user experience.

[0411] "An emotion recognition method that recognizes user emotions in real time" refers to a means that analyzes the user's facial expressions, voice, behavioral patterns, etc., through a terminal operated by the user, and has the function of identifying the emotional state the user is currently feeling in real time and transmitting the information to a server.

[0412] A "means for calculating the optimal route" is a means that, based on information about the starting point and destination, takes into account traffic conditions and user sentiment data to calculate the most efficient and user-friendly route.

[0413] "Means for receiving user input and generating departure point and destination information" refers to means that have the function of receiving departure point and destination information entered by the user through a terminal and processing it as digital information.

[0414] "A means of presenting calculated route information to the user and confirming whether or not to use expressways" refers to a means that displays route information calculated by the server on the user's terminal and has a function to confirm with the user whether or not to use expressways.

[0415] "Means for optimizing routes based on user responses and generating final route information" refers to means that have the function of recalculating routes based on user selections and responses to determine the most appropriate final route.

[0416] "Means for providing feedback of calculation results to the user" refers to means that have the function of transmitting the results of various processes performed by the server to the user's terminal and displaying them so that the user can confirm them.

[0417] "Means for obtaining in-vehicle sales menus and inventory information" refers to means that a server manages the in-vehicle sales menu and its inventory status, and provides this information to the user terminal.

[0418] "Means for generating and presenting new ticket information to the user" refers to means that a server has the function of creating new ticket information based on a user's request and displaying it on the user's terminal.

[0419] Modes for carrying out the invention

[0420] This invention combines a system that provides various services in autonomous vehicles and public transportation as virtual crew members with an emotion engine that recognizes user emotions, thereby achieving more advanced and user-friendly service provision. The specific system configuration and operation are described below.

[0421] System Configuration

[0422] This system consists of a user, a terminal, a server, and an emotion engine. When using autonomous vehicles or public transportation, the user operates a terminal such as a tablet or smartphone. The terminal receives user input, processes it, and sends it to the server. The emotion engine recognizes the user's emotions in real time and sends that information to the server. The server performs calculations and data processing based on the received departure point, destination information, and user emotion data, and sends the results to the terminal.

[0423] Setting the destination and route

[0424] First, the user specifies their starting point and destination using their device. For example, they might set the starting point to "Tokyo Station" and the destination to "Shinagawa Station" on a map application. This information is sent from the device to the server. The server calculates the optimal route based on the starting point and destination data. This calculation also takes into account sentiment data from the sentiment engine; for example, if it determines that the user is in a hurry, it will suggest a route that prioritizes highways. The calculation results include distance, travel time, and whether highways are available. This information is sent to the device and displayed to the user. The user confirms the route and resends the response information from the device to the server, which then determines the final route and sends that information back to the device.

[0425] Example prompt:

[0426] "My departure point is Tokyo Station, and my destination is Shinagawa Station. I'm in a hurry."

[0427] In-train sales

[0428] This section explains how a user requests in-train sales. The user operates the in-train sales application on their terminal and sends a request such as "I want to order a bento box." The terminal sends this request to the server. Based on the request, the server retrieves the in-train sales menu and inventory information and sends it to the terminal. The emotion engine recognizes the user's emotions and, for example, if it determines that the user is tired, it can suggest relaxing tea or other items. Once the user selects the desired items, the selection information is sent from the terminal to the server, and after the server checks the inventory, it sends a purchase confirmation message to the terminal. The terminal displays the purchase confirmation message to the user, and the order is finalized upon user confirmation.

[0429] Example prompt:

[0430] "I'd like to order a bento box. I'm very tired."

[0431] Ticket change

[0432] This section describes a scenario where a user requests a change to their train ticket. The user operates the ticket management application on their terminal and requests a "change of ticket." This request is sent from the terminal to the server. The server calculates the available options and suggests the best option based on emotional data. For example, if it is determined that the user wants to relax, it will suggest a quiet carriage. The user selects their desired change option, and this information is sent from the terminal to the server. The server generates new ticket information, sends it to the terminal, and displays it to the user.

[0433] Example prompt:

[0434] "I want to change my ticket. I want to relax."

[0435] As described above, the present invention is a system in which the user, terminal, server, and emotion engine work together to provide efficient and user-friendly services. By considering the user's emotions when providing various services, the quality of the user experience can be improved.

[0436] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0437] Step 1:

[0438] The user uses a device to specify the starting point and destination.

[0439] Specific operation: The user opens a map application and enters "Tokyo Station" as the starting point and "Shinagawa Station" as the destination. The terminal receives this input information and stores it as input data.

[0440] Input: Departure point and destination information

[0441] Output: Data on departure point and destination.

[0442] Step 2:

[0443] The terminal sends the entered information to the server.

[0444] Specific operation: The terminal converts the user's entered departure point and destination information into a digital format and sends it to the server.

[0445] Input: User-entered data for departure point and destination.

[0446] Output: Departure and destination data sent to the server

[0447] Step 3:

[0448] The emotion engine recognizes the user's emotions in real time and sends that information to the server.

[0449] Specific operation: The emotion engine analyzes the user's facial expressions, voice, and actions to recognize their emotional state, such as whether they are in a hurry or relaxed. The recognized emotional information is then sent to the server.

[0450] Input: Real-time user sentiment information

[0451] Output: Emotion data sent to the server

[0452] Step 4:

[0453] The server calculates the optimal route based on the departure point, destination information, and sentiment data it receives.

[0454] Specific operation: The server considers the user's sentiment data in addition to the origin and destination data, and uses an optimal route algorithm to perform calculations. For example, if the user is in a hurry, it will calculate a route that prioritizes highways.

[0455] Input: Origin, destination, and sentiment data

[0456] Output: Optimal route information (distance, travel time, availability of highways, etc.)

[0457] Step 5:

[0458] The device receives the calculated route information and presents it to the user.

[0459] Specific operation: The server sends the calculated optimal route information to the terminal. The terminal displays this information and prompts the user for confirmation. For example, the displayed route information might say "From Tokyo Station to Shinagawa Station, travel time 20 minutes, expressway available."

[0460] Input: Optimal route information from the server

[0461] Output: Route information presented to the user

[0462] Step 6:

[0463] The user checks route information on their device and responds regarding the availability of highway access.

[0464] Specific operation: The user checks the route information displayed on the device and selects whether to "use the highway" or "do not use the highway." The selected information is sent from the device to the server.

[0465] Input: User response information (whether or not highway use is permitted)

[0466] Output: Response information sent to the server

[0467] Step 7:

[0468] The server optimizes the route based on the user's response information, generates the final route information, and sends it to the terminal.

[0469] Specific operation: The server makes final adjustments to the route based on the user's response information and generates the final route information. The terminal receives this and presents the final route to the user.

[0470] Input: User response information

[0471] Output: Final route information

[0472] Step 8:

[0473] The user checks the in-car sales menu and selects an item.

[0474] Specific operation: The user checks the in-vehicle sales menu on their terminal and selects the desired product (e.g., "bento box"). The selection information is sent from the terminal to the server.

[0475] Input: User's selection information (product)

[0476] Output: Selection information sent to the server

[0477] Step 9:

[0478] The server checks the inventory of the selected product and sends a purchase confirmation message to the terminal.

[0479] Specific operation: The server checks if the product selected by the user is in stock, and if it is, it generates a purchase confirmation message and sends it to the terminal.

[0480] Input: Server selection information (product)

[0481] Output: Purchase confirmation message sent to the device

[0482] Step 10:

[0483] The device displays a purchase confirmation message to the user, who then confirms it.

[0484] Specific operation: The terminal displays a purchase confirmation message received from the server to the user. The order is confirmed when the user confirms it and presses the confirm button.

[0485] Input: Purchase confirmation message

[0486] Output: Confirmation message and confirmation information displayed to the user

[0487] Step 11:

[0488] The user requests a change to their ticket using the ticket management application.

[0489] Specific operation: The user requests a "ticket change" on their terminal, for example, by entering "I would like a quiet carriage." The terminal sends the request to the server.

[0490] Input: User's ticket change request

[0491] Output: Request sent to the server

[0492] Step 12:

[0493] The server calculates the available options, and the emotion engine suggests the optimal option.

[0494] Specific operation: The server calculates the modifiable options based on the input request. The emotion engine evaluates the user's emotional state and, for example, suggests a quiet vehicle in response to a request to relax.

[0495] Input: User change requests and sentiment data

[0496] Output: Optimal change options

[0497] Step 13:

[0498] The terminal receives the new ticket information and presents it to the user.

[0499] Specific operation: The server sends the calculated new ticket information to the terminal. The terminal displays this information to the user and prompts them to confirm.

[0500] Input: Ticket information from the server

[0501] Output: New ticket information displayed to the user

[0502] (Application Example 2)

[0503] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0504] Services in modern autonomous vehicles often provide a uniform service without considering the mood or emotions of passengers. This makes it difficult to meet the diverse needs of users, resulting in a diminished user experience. Furthermore, there is a lack of optimal suggestions that take user feelings into account, even when it comes to in-vehicle sales or ticket changes. This leads to the problem of services being provided without passengers feeling comfortable or satisfied.

[0505] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving user input and generating departure point and destination information, means for calculating the optimal route using the received departure point and destination information, means for presenting the calculated route information to the user and confirming whether or not highways can be used, means for optimizing the route based on the user's response and generating final route information, means for recognizing the user's emotions in real time and transmitting emotion data to the server, and means for deciding whether or not to use highways considering the user's emotion information. This makes it possible to provide individually optimal routes and services based on the user's emotions.

[0506] An "autonomous vehicle" is a vehicle that can drive automatically without human intervention, using artificial intelligence and sensor technology.

[0507] "User input" refers to instructions or information that a user provides to a system through a terminal or device.

[0508] The "starting point" refers to the location where the user begins using the autonomous vehicle.

[0509] "Destination information" refers to information about the location that the user designates as their destination.

[0510] The "optimal route" is the most efficient path from the starting point to the destination. Efficiency is calculated by considering factors such as time, distance, and traffic conditions.

[0511] An "emotion engine" is a system that analyzes a user's facial expressions, tone of voice, and other factors to recognize their emotions in real time.

[0512] "Emotional data" refers to information about a user's emotions as recognized by the emotion engine.

[0513] "Whether or not to use expressways" refers to the user's choice of whether or not to use expressways to reach their destination.

[0514] "In-vehicle sales" refers to food, beverages, and other goods offered inside autonomous vehicles.

[0515] "Inventory information" refers to information regarding the current stock status of products offered for sale on board the train.

[0516] "Ticket change" refers to changing the type or details of a ticket that a user has already obtained.

[0517] "Selected information" refers to the specific information that the user has chosen from the options presented to them.

[0518] This invention is a system for making passenger services in autonomous vehicles more advanced and user-friendly. This system is composed of a user, a terminal, a server, and an emotion engine, which recognize the user's emotions in real time and provide optimal services in response to those emotions.

[0519] System Configuration

[0520] User

[0521] The users are passengers using the autonomous vehicle and operate the system through devices such as tablets and smartphones.

[0522] terminal

[0523] The terminal is a device that receives user instructions and collects data for emotion recognition. The terminal uses its camera and microphone to transmit the user's facial expressions and voice to the emotion engine.

[0524] server

[0525] A server primarily has three roles:

[0526] 1. Calculate the optimal route based on user input, originating point, and destination information.

[0527] 2. Receive requests for in-vehicle sales and manage product inventory information.

[0528] 3. Accept ticket changes and provide changeable options.

[0529] Emotional Engine

[0530] The emotion engine analyzes emotional data in real time from the user's facial expressions and voice, and sends that data to the server. Examples of technologies used include Amazon Rekognition and Microsoft® Azure Emotion API.

[0531] Destination setting

[0532] The user first enters their starting point and destination into the terminal. For example, if the user sets the starting point to "Tokyo Station" and the destination to "Shinagawa Station," this information is sent from the terminal to the server. The server calculates the optimal route and presents the user with route information, including whether or not highways can be used. Furthermore, the emotion engine analyzes whether the user is in a hurry and may recommend using highways.

[0533] In-train sales

[0534] When a user wants to use the in-train sales service, they request to "order a bento box" through the application on their terminal. This request is sent to the server, which then suggests products based on inventory information. If the emotion engine determines that the user is tired, it also suggests a relaxing tea.

[0535] Ticket change

[0536] When a user requests a change to their ticket, that information is sent from the terminal to the server. The server also takes the user's emotional state into consideration; for example, if the user wants to relax, it might suggest a quieter carriage.

[0537] Specific examples and prompt statements

[0538] Specific examples of route acquisition:

[0539] Departure point: "Tokyo Station"

[0540] Destination: Shinagawa Station

[0541] Emotion: "When the user is in a hurry"

[0542] Result: "Proposal for the fastest route using expressways"

[0543] Please calculate the route based on the following information:

[0544] Departure point: Tokyo Station

[0545] Destination: Shinagawa Station

[0546] User's emotion: In a hurry

[0547] Examples of in-train sales:

[0548] Order: "Bento"

[0549] Emotion: "When the user is tired"

[0550] Result: "Suggestion for tea with relaxing effects"

[0551] Please order the following items:

[0552] Order: Bento

[0553] User's emotion: tired

[0554] Additional suggestion: Tea with relaxing effects

[0555] Specific examples of ticket changes:

[0556] Change option: "Quiet vehicle"

[0557] Emotion: "When the user wants to relax"

[0558] Result: "Propose a quiet vehicle."

[0559] Please change your ticket using one of the following options:

[0560] Change options: Quiet vehicle

[0561] User's emotion: Wants to relax

[0562] As described above, the system of the present invention can provide individually optimized routes and services while taking into account the user's emotions. This is expected to significantly improve passenger comfort and satisfaction.

[0563] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0564] Step 1:

[0565] The user enters their departure point and destination information into the device. For example, they might enter "Tokyo Station" as the departure point and "Shinagawa Station" as the destination. This input information is retrieved by the map application on the device.

[0566] Step 2:

[0567] The terminal sends the entered departure and destination information to the server. At this time, the emotion engine analyzes the user's facial expressions and voice, and emotion data is generated. This emotion data is also sent to the server.

[0568] Step 3:

[0569] The server receives input information and sentiment data. It calculates the optimal route based on the departure and destination information, and further optimizes route selection by considering sentiment data. The decision of whether or not to use highways is also made at this stage.

[0570] Step 4:

[0571] The server sends the calculated optimal route information and a suggestion regarding the availability of highways to the terminal. The terminal displays this information to the user and receives the user's response again.

[0572] Step 5:

[0573] The user enters response information into the terminal in response to the presented route information and the suggestion regarding the use of expressways. For example, they might select "Use expressways."

[0574] Step 6:

[0575] The terminal sends the user's response information back to the server. The server optimizes the final route based on the response information and generates the final route information.

[0576] Step 7:

[0577] The server sends the final route information to the terminal, which then displays it to the user. At this stage, the user can verify the optimized route information.

[0578] Step 8:

[0579] If a user wants to use the in-train sales service, they enter their order information through the application on their terminal. For example, they might enter "I want to order a bento box." This information is then sent from the terminal to the server.

[0580] Step 9:

[0581] The server retrieves in-car sales menus and inventory information based on the received order information, and also takes into account the user's emotional state to make appropriate product suggestions. For example, if the user is tired, it will suggest a relaxing tea.

[0582] Step 10:

[0583] The server sends the suggested menu information to the terminal, which then displays it to the user. The user selects an item from the presented menu and enters the selection information into the terminal.

[0584] Step 11:

[0585] The terminal sends the selection information to the server. The server checks the inventory of the selected items, generates a purchase confirmation message, and sends it to the terminal. The terminal displays this message to the user, who then confirms it.

[0586] Step 12:

[0587] If a user wishes to change their ticket, they enter a change request through the ticket management application on their terminal. For example, they might enter, "I would like to change to a quieter carriage." This request is then sent from the terminal to the server.

[0588] Step 13:

[0589] The server generates information on modifiable options based on the received change request. Sentimental data is also taken into consideration, and options suitable for, for example, a user who wants to relax are presented.

[0590] Step 14:

[0591] The server sends information about the configurable options to the terminal, which then displays this information to the user. The user selects the desired options and enters that information into the terminal.

[0592] Step 15:

[0593] The terminal sends the selection information to the server. The server generates new ticket information and sends that information to the terminal. The terminal displays the new ticket information to the user, allowing the user to confirm the details of the new ticket.

[0594] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0595] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0596] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0597] [Second Embodiment]

[0598] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0599] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0600] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0601] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0602] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0603] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0604] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0605] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0606] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0607] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0608] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0609] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0610] This invention is a system that provides various services in autonomous vehicles and public transportation by having virtual crew members. The specific system configuration and operation are described below.

[0611] System Configuration

[0612] This system consists of users, terminals, and servers. Users operate terminals such as tablets and smartphones to use autonomous vehicles and public transportation. The terminal receives user input, processes it, and sends it to the server. The server performs calculations and data processing based on the received information and sends the results to the terminal.

[0613] Setting the destination and route

[0614] First, the user specifies their starting point and destination. For example, the user sets the starting point to "Tokyo Station" and the destination to "Shinagawa Station" on the map application on their device. This information is sent from the device to the server. The server calculates the optimal route based on the received starting point and destination information. The calculated route information (distance, travel time, whether or not highways can be used, etc.) is sent to the device and displayed to the user. The user then selects whether or not to use highways. For example, if the user selects "Use highways," this information is sent from the device to the server. The server optimizes the route based on this selection, generates the final route information, and sends it to the device. The device then displays the final route on its navigation system.

[0615] In-train sales

[0616] Next, let's consider the case where a user requests an in-train purchase. The user operates the in-train sales application on their terminal and requests, "I want to order a bento box." This request is sent from the terminal to the server. The server retrieves the in-train sales menu and inventory information based on the specified request and sends it to the terminal. The terminal displays the retrieved menu information to the user. The user selects the desired product (for example, "sushi bento"). This selection information is sent from the terminal to the server. The server checks the inventory of the selected product and sends a purchase confirmation message to the terminal. The terminal displays this purchase confirmation message to the user. Once the user confirms, the ordering process is complete.

[0617] Ticket change

[0618] Finally, consider the case where a user requests a change to their ticket. The user operates the ticket management application on their terminal and requests a "change ticket." This request is sent from the terminal to the server. The server calculates the available change options based on the request and sends that information to the terminal. The terminal presents the user with the available change options (e.g., change time, change seat). The user selects the desired change option. This selection information is sent from the terminal to the server. The server generates new ticket information based on the selection and sends it to the terminal. The terminal displays the new ticket information to the user.

[0619] As described above, the present invention is a system in which users, terminals, and servers cooperate to provide efficient and user-friendly services.

[0620] The following describes the processing flow.

[0621] Setting the destination and route

[0622] Step 1:

[0623] The user operates the terminal and enters the departure point and destination. For example, the user enters "Tokyo Station" as the departure point and "Shinagawa Station" as the destination.

[0624] Step 2:

[0625] The terminal receives user input information and sends it to the server. Specifically, it sends data on the departure point and destination to the server.

[0626] Step 3:

[0627] The server calculates the optimal route based on the information it receives. This calculation uses external map APIs or the company's own database.

[0628] Step 4:

[0629] The server sends calculated route information to the terminal, including distance, estimated travel time, and whether highways can be used.

[0630] Step 5:

[0631] The terminal displays the route information it has received to the user. The user can then choose whether or not to use the highway.

[0632] Step 6:

[0633] The user selects whether or not to use the highway, and the terminal sends this selection information to the server.

[0634] Step 7:

[0635] The server recalculates the route based on the user's selection information and generates the final route information.

[0636] Step 8:

[0637] The server sends the final route information to the terminal. The terminal displays the final route on its navigation system.

[0638] In-train sales

[0639] Step 1:

[0640] The user operates the terminal and selects a request for in-car sales. For example, they might select the "Order a bento box" option.

[0641] Step 2:

[0642] The terminal sends the user's request to the server. Specifically, this includes the user ID and the request details.

[0643] Step 3:

[0644] The server retrieves the in-vehicle sales menu and inventory information. For example, it reads the latest menu and inventory numbers from the database.

[0645] Step 4:

[0646] The server sends the retrieved menu information to the terminal. This includes the product name, price, and stock quantity.

[0647] Step 5:

[0648] The terminal displays menu information to the user. The user selects the desired product.

[0649] Step 6:

[0650] The user selects an item, and the device sends that selection information to the server. Specifically, this includes the item ID and quantity.

[0651] Step 7:

[0652] The server checks the inventory of the selected product. If it is in stock, it generates a purchase confirmation message and sends it to the terminal.

[0653] Step 8:

[0654] The device displays a purchase confirmation message to the user. The order is finalized once the user confirms it.

[0655] Ticket change

[0656] Step 1:

[0657] The user operates the terminal and selects a request to change their ticket. For example, they might select the "Change boarding time" menu.

[0658] Step 2:

[0659] The terminal sends the user's request to the server. Specifically, this includes current ticket information and conditions for change.

[0660] Step 3:

[0661] The server calculates the options that can be changed. For example, it searches the database for available trains and seats.

[0662] Step 4:

[0663] The server sends the calculation results to the terminal. This includes information about trains and seats that can be changed.

[0664] Step 5:

[0665] The device displays change options to the user. The user selects the desired change option.

[0666] Step 6:

[0667] The user selects a change option, and the device sends that selection information to the server. Specifically, this includes the changed train ID and seat number.

[0668] Step 7:

[0669] The server generates new ticket information, which includes new train and seat information.

[0670] Step 8:

[0671] The server sends the new ticket information to the terminal. The terminal displays the new ticket information to the user.

[0672] (Example 1)

[0673] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0674] To improve user convenience and comfort in autonomous vehicles and public transportation, it is necessary to efficiently provide a wide range of services, such as destination setting, in-vehicle sales, and ticket changes. However, in conventional systems, these services are often fragmented, making it difficult to provide a unified experience. Furthermore, advanced data processing technology is required to process user input information quickly and accurately and provide users with the necessary information.

[0675] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0676] In this invention, the server includes means for receiving user input and generating departure point and destination information; means for calculating an optimal route using the received departure point and destination information; means for presenting the calculated route information to the user and confirming the availability of highways; means for optimizing the route based on the user's response and generating final route information; means for displaying the final route information on a navigation system; means for receiving the user's in-vehicle sales request and transmitting it to the server; means for acquiring in-vehicle sales menus and inventory information; means for presenting the acquired menu information to the user; means for receiving the user's selection information and transmitting it to the server; means for confirming the inventory of selected items and presenting a purchase confirmation message to the user; means for receiving user confirmation and completing the order process; means for receiving the user's ticket change request and transmitting it to the server; means for presenting the user with changeable option information calculated by the server; means for receiving the user's selection information and transmitting it to the server; means for generating new ticket information and presenting it to the user; and means for displaying the new ticket information on a navigation system. This enables the provision of a consistent user experience and the efficient execution of various services.

[0677] "User" refers to a person who operates this system in order to use autonomous vehicles or public transportation.

[0678] A "device" is an electronic device used by a user, and includes tablets, smartphones, and other similar devices.

[0679] A "server" refers to a central system that receives information from users, performs data processing and calculations, and sends the results to terminals.

[0680] The "starting point" refers to the initial location from which the user boards the vehicle, and is one of the location information settings in a map application.

[0681] "Destination information" refers to information about the location that the user sets as their destination.

[0682] "Optimal route" refers to the shortest or most efficient route information calculated by the server based on the origin and destination information received.

[0683] A "navigation system" is a system that displays optimal route information and final route information to guide the user to their destination.

[0684] An "in-car sales request" refers to a request sent from a terminal to a server when a user wishes to purchase goods while in a vehicle.

[0685] The "in-train sales menu" refers to a list of products offered on board the train.

[0686] "Inventory information" refers to information showing the availability of each product included in the in-train sales menu.

[0687] "Selected information" refers to information about products or services that the user has selected on their device.

[0688] A "purchase confirmation message" refers to a confirmation message sent by the server to the user to complete the purchase process for the selected product.

[0689] A "ticket change request" refers to a request sent from a user's device to a server when the user wishes to change the details of a ticket they have already purchased.

[0690] "Changeable option information" refers to the ticket modification options that the server calculates and that the user can select.

[0691] "New ticket information" refers to information about a new ticket generated based on the user's selection.

[0692] This invention relates to a virtual crew system that provides a unified range of services in autonomous vehicles and public transportation. The following describes embodiments of this system.

[0693] System Configuration

[0694] This system consists of users, terminals, and servers. Users operate terminals such as tablets and smartphones to use autonomous vehicles and public transportation. The terminal receives user input, processes it, and sends it to the server. The server performs calculations and data processing based on the received information and sends the results to the terminal.

[0695] Setting the destination and route

[0696] First, the user specifies their pick-up and destination. For example, the user uses their device's map application to set the departure point to "Tokyo Station" and the destination to "Shinagawa Station." This information is sent from the device to the server. The server uses the Google Maps API to calculate the optimal route based on the received departure and destination information. The calculated route information (distance, travel time, whether highways can be used, etc.) is sent to the device and displayed to the user. The user then chooses whether or not to use highways. For example, if they select "Use highways," this information is sent from the device to the server. The server optimizes the route based on this selection, generates the final route information, and sends it to the device. The device then displays the final route on its navigation system.

[0697] Specific example:

[0698] Prompt message:

[0699] "Please set the departure point to Tokyo Station and the destination to Shinagawa Station."

[0700] In-train sales

[0701] Next, let's consider the case where a user requests an in-train purchase. The user uses the in-train sales application on their terminal to request, "I want to order a bento box." This request is sent from the terminal to the server. The server retrieves the in-train sales menu and inventory information based on the specified request and sends it to the terminal. The terminal displays the retrieved menu information to the user. The user selects the desired item (for example, "sushi bento"). This selection information is sent from the terminal to the server. The server checks the inventory of the selected item and sends a purchase confirmation message to the terminal. The terminal displays this purchase confirmation message to the user. Once the user confirms, the ordering process is complete.

[0702] Specific example:

[0703] Prompt message:

[0704] "Please select and order your bento box."

[0705] Ticket change

[0706] Finally, consider the case where a user requests a change to their ticket. The user requests a "change ticket" by operating the ticket management application on their terminal. This request is sent from the terminal to the server. The server calculates the available change options based on the request and sends that information to the terminal. The server uses its own backend system and reservation management system. It presents the user with the available change options (e.g., change time, change seat). The user selects the desired change option and sends the selection information from the terminal to the server. The server generates new ticket information based on the selection and sends it to the terminal. The terminal displays the new ticket information to the user.

[0707] Specific example:

[0708] Prompt message:

[0709] "Please view and select the ticket change options."

[0710] As described above, this system enables efficient and user-friendly services through the collaboration of users, terminals, and servers. By utilizing data processing technologies, including the Google Maps API, this system makes it possible to provide advanced services.

[0711] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0712] Setting the destination and route

[0713] Step 1:

[0714] The user operates their device (tablet or smartphone) and opens a map application. The user sets the starting point to "Tokyo Station" and the destination to "Shinagawa Station," and then presses the send button.

[0715] Input: Departure point and destination information (Tokyo Station, Shinagawa Station)

[0716] Specific operation: The user specifies the starting point and destination by dragging and dropping pins on the map using their device.

[0717] Output: The departure point and destination information are set on the terminal.

[0718] Step 2:

[0719] The device sends its departure point and destination information to the server.

[0720] Input: Departure point and destination information set by the user.

[0721] Specific operation: The app on the device creates an HTTP request and sends it to the server.

[0722] Output: The server receives the departure and destination information.

[0723] Step 3:

[0724] The server calculates the optimal route based on the information it receives. The server uses the Google Maps API to obtain the optimal route (distance, travel time, whether highways can be used, etc.).

[0725] Input: Departure point and destination information

[0726] Specific operation: The server accesses the Google Maps API route search endpoint and sends the starting point and destination parameters.

[0727] Output: Optimal route information is generated on the server.

[0728] Step 4:

[0729] The server sends the calculated optimal route information to the terminal.

[0730] Input: Optimal route information

[0731] Specific operation: The server sends data in JSON format to the terminal.

[0732] Output: The terminal receives optimal route information.

[0733] Step 5:

[0734] The terminal displays the route information it has received to the user. The user can choose whether or not to use highways.

[0735] Input: Optimal route information

[0736] Specific action: The app on the device displays route information on the map and shows a checkbox indicating whether to "use highways."

[0737] Output: The user's highway usage selection information is determined.

[0738] Step 6:

[0739] The user selects the option to use the highway and presses the submit button. The terminal sends the selection information to the server.

[0740] Input: User's highway usage selection information

[0741] Specific operation: The user checks / unchecks a checkbox within the app on their device and presses the submit button. The device sends the selection information to the server via an HTTP request.

[0742] Output: The server receives the selection information.

[0743] Step 7:

[0744] The server recalculates the route based on the selected information and generates the final route information. It then sends the final route information to the terminal.

[0745] Input: User's highway usage selection information

[0746] Specific operation: The server uses the selected information to re-execute the optimal route calculation algorithm and generate the final route information.

[0747] Output: The final route information is generated on the server and sent to the terminal.

[0748] Step 8:

[0749] The device displays the final route information on the navigation system.

[0750] Input: Final route information

[0751] Specific operation: The device's navigation app receives the final route information and displays it on the map.

[0752] Output: The user can check the final route information.

[0753] In-train sales

[0754] Step 1:

[0755] The user uses the in-car sales application on their terminal to request, "I would like to order a bento box."

[0756] Input: User's in-car sales request

[0757] Specific action: The user selects "In-car sales" from the app's menu and presses the "Order a bento box" button.

[0758] Output: An in-car sales request is set on the terminal.

[0759] Step 2:

[0760] The terminal sends the user's in-car sales request to the server.

[0761] Input: User's in-car sales request

[0762] Specific action: The terminal creates an HTTP request and sends it to the server.

[0763] Output: The server receives an in-vehicle sales request.

[0764] Step 3:

[0765] The server retrieves the in-car sales menu and inventory information based on the request and sends it to the terminal.

[0766] Input: In-car sales request

[0767] Specific operation: The server retrieves the product list and inventory information from the database and sends it to the terminal.

[0768] Output: The in-car sales menu and inventory information are sent to the terminal.

[0769] Step 4:

[0770] The terminal displays the menu information it has acquired to the user, and the user selects the product they want (for example, "sushi bento").

[0771] Input: In-car sales menu and inventory information

[0772] Specific action: The app on the device displays a menu list, and the user taps on "Sushi Bento".

[0773] Output: User selection information is set on the terminal.

[0774] Step 5:

[0775] The device sends the user's selection information to the server.

[0776] Input: User's selected information

[0777] Specific action: The terminal creates an HTTP request containing the selection information and sends it to the server.

[0778] Output: The server receives the user's selection information.

[0779] Step 6:

[0780] The server checks the inventory of the selected product, generates a purchase confirmation message, and sends it to the terminal.

[0781] Input: User's selected information

[0782] Specific operation: The server retrieves the inventory quantity of the product from the database, and if it is in stock, it generates a purchase confirmation message.

[0783] Output: A purchase confirmation message is generated on the server and sent to the device.

[0784] Step 7:

[0785] The device displays a purchase confirmation message to the user, and once the user confirms it, the order process is completed.

[0786] Input: Purchase confirmation message

[0787] Specific action: The user presses the confirmation button, and the device sends the order information to the server.

[0788] Output: The order has been confirmed and the shipping process has begun.

[0789] Ticket change

[0790] Step 1:

[0791] The user requests a "ticket change" using the ticket management application on their device.

[0792] Input: User's ticket change request

[0793] Specific action: The user presses the "Change Ticket" button on the ticket management screen in the app.

[0794] Output: A ticket change request is set on the terminal.

[0795] Step 2:

[0796] The device sends a user change request to the server.

[0797] Input: User's ticket change request

[0798] Specific action: The terminal creates an HTTP request and sends it to the server.

[0799] Output: The server receives a ticket change request.

[0800] Step 3:

[0801] The server calculates the modifiable options based on the request and sends that information to the terminal.

[0802] Input: Ticket change request

[0803] Specific operation: The server retrieves the current reservation information from the database and calculates the options that can be changed.

[0804] Output: Modifiable option information is generated on the server and sent to the terminal.

[0805] Step 4:

[0806] The device presents the user with available modification options, and the user selects the desired modification option.

[0807] Input: Changeable option information

[0808] Specific operation: The app on the device displays change options, and the user selects one.

[0809] Output: User selection information is set on the terminal.

[0810] Step 5:

[0811] The device sends the user's selection information to the server.

[0812] Input: User's selected information

[0813] Specific action: The terminal creates an HTTP request containing the selection information and sends it to the server.

[0814] Output: The server receives the user's selection information.

[0815] Step 6:

[0816] The server generates new ticket information based on the selections and sends it to the terminal.

[0817] Input: User's selected information

[0818] Specific operation: The server generates new ticket information and sends it to the terminal.

[0819] Output: New ticket information is generated on the server and sent to the terminal.

[0820] Step 7:

[0821] The terminal displays the new ticket information to the user.

[0822] Input: New ticket information

[0823] Specific action: The app on the device displays the new ticket information, and the user confirms it.

[0824] Output: The user can view the new ticket information.

[0825] The above steps enable a system where users, terminals, and servers work together to provide efficient and user-friendly services. Including specific actions in each processing step makes the program flow clearer.

[0826] (Application Example 1)

[0827] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0828] The present invention aims to provide a method for providing food delivery services more seamlessly and efficiently. Specifically, the objective is to provide a system that allows users to order food from within an autonomous vehicle, track the delivery status in real time, and automatically generate appropriate responses to user inquiries and requests.

[0829] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0830] In this invention, the server includes means for receiving user input and generating departure point and destination information; means for calculating an optimal route using the received departure point and destination information; means for presenting the calculated route information to the user and confirming the availability of highways; means for optimizing the route based on the user's response and generating final route information; means for receiving user order information and providing food delivery services; means for providing the user with real-time delivery status and estimated arrival time; and means for analyzing user inquiries and providing appropriate answers using a generative AI model. This enables a seamless experience and rapid response when users utilize food delivery services in an autonomous vehicle.

[0831] "User input" refers to the information and instructions that a user provides through their device.

[0832] "Departure point" refers to the location where the user boards or the starting point.

[0833] "Destination information" refers to information about the place or destination that the user intends to go.

[0834] The "optimal route" refers to the most suitable travel route calculated considering the efficiency and convenience of the path.

[0835] "Availability of using expressways" refers to the user's choice of whether or not to use expressways.

[0836] "Final route information" refers to the final guidance information based on an optimized route.

[0837] "Order information" refers to the detailed information that users enter when purchasing food or products.

[0838] A "food delivery service" refers to a service that delivers food to a location specified by the user.

[0839] "Delivery status" refers to the current progress and status of the food or product delivery process.

[0840] "Estimated arrival time" refers to the estimated time it will take for a delivery to reach its destination.

[0841] "Inquiries" refer to information related to questions and requests from users.

[0842] A "generative AI model" refers to a model that uses artificial intelligence technology to automatically create answers and information in response to user inquiries.

[0843] An "appropriate response" refers to a response that provides accurate and relevant information in response to a user's inquiry or request.

[0844] This invention is a system that supports user operation within an autonomous vehicle in order to efficiently provide food delivery services. This system consists of a user, a terminal, and a server. The specific operation of each component is described below.

[0845] System Configuration

[0846] This system allows users to operate it through a device, which then interacts with a server to provide services. The device can be a smartphone or tablet, and the server is a cloud server (such as AWS or Azure).

[0847] Setting the destination and route

[0848] First, the user enters their starting point and destination information into the terminal. Using a map application on the terminal, the user specifies the starting point and destination. This information is sent from the terminal to the server, which calculates the optimal route. The calculated route information is presented to the user, and the user selects whether or not to use highways. Based on this information, the server generates an optimized final route and sends it to the terminal.

[0849] Food delivery service

[0850] When a user orders food from their device, the order information is sent to a server. The server provides the food delivery service and gives the user real-time delivery status and estimated arrival time. Delivery status is tracked using GPS, and the estimated arrival time is calculated dynamically.

[0851] Customer Support

[0852] User inquiries are sent directly from the terminal to the server. The server analyzes the inquiry using a generative AI model (e.g., GPT-4) and generates an appropriate response. The generated response is sent to the terminal and displayed to the user.

[0853] Hardware and software to be used

[0854] Hardware: Smartphones, tablets, cloud servers (AWS, Azure)

[0855] Software: iOS or Android native app development tools (Swift, Kotlin, etc.), server-side frameworks (Node.js, Flask, etc.), SQL databases (MySQL, PostgreSQL, etc.), generative AI models (GPT-4)

[0856] Specific example

[0857] This describes a scenario where a user orders a pizza from inside an autonomous vehicle and monitors the delivery status in real time. When the user operates a terminal and orders a pizza, the order information is sent to a server. The server processes the order information, and a delivery person departs with the pizza. The delivery status is tracked via GPS, the estimated arrival time is calculated, and this information is provided to the user in real time. Furthermore, if the user makes a question, a generative AI model generates an appropriate answer, which is then displayed to the user.

[0858] Example of a prompt:

[0859] "A user has ordered a pizza. Please provide the delivery person's current location in real time and the estimated arrival time."

[0860] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0861] Step 1:

[0862] The user operates the terminal and enters departure and destination information. The entered information is temporarily stored within the terminal. The terminal then sends the entered departure and destination information as data to the server.

[0863] Input: Departure point and destination information entered by the user on the device.

[0864] Output: Departure point and destination information sent to the server

[0865] Step 2:

[0866] The server calculates the optimal route based on the received origin and destination information. It uses a route calculation algorithm (e.g., Dijkstra's algorithm) to identify the most efficient route. The calculated route information is stored internally by the server.

[0867] Input: Departure point and destination information

[0868] Output: Calculated optimal route information

[0869] Step 3:

[0870] The server sends the calculated route information to the terminal, which then presents the information to the user. The user reviews the route information and selects whether or not to use highways. The selection information is then sent back to the server from the terminal.

[0871] Input: Optimal route information

[0872] Output: User selection information (whether or not highways can be used)

[0873] Step 4:

[0874] The server generates an optimized final route based on the user's selection information. It recalculates the route and generates new route information. The final route information is sent from the server to the terminal and presented to the user.

[0875] Input: User's selected information

[0876] Output: Final route information

[0877] Step 5:

[0878] The user orders food from their device. The device sends the order information to the server. The order information includes the type and quantity of food, and the delivery address.

[0879] Input: User's order information

[0880] Output: Order information sent to the server

[0881] Step 6:

[0882] The server receives and processes the order information. This includes checking order availability and arranging delivery. The server then generates an order confirmation message and sends it to the terminal.

[0883] Input: User's order information

[0884] Output: Order confirmation message

[0885] Step 7:

[0886] The server tracks the progress of food deliveries in real time. It uses GPS to obtain the delivery person's location and calculates the estimated arrival time. This information is processed within the server.

[0887] Input: Delivery driver's location information

[0888] Output: Estimated arrival time and delivery status

[0889] Step 8:

[0890] The server sends the delivery status and estimated arrival time to the terminal. The terminal displays this information to the user, allowing them to check the delivery status in real time.

[0891] Input: Estimated arrival time and delivery status

[0892] Output: Delivery status and estimated arrival time presented to the user.

[0893] Step 9:

[0894] When a user submits an inquiry, the device sends the inquiry as text data to the server. The server uses a generative AI model to analyze the inquiry and generate an appropriate response.

[0895] Input: Inquiry details

[0896] Output: Generated answer

[0897] Step 10:

[0898] The server sends the response generated by the AI ​​model to the terminal. The terminal displays the response to the user, and appropriate action is taken.

[0899] Input: Generated answer

[0900] Output: The answer presented to the user

[0901] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0902] This invention combines a system that provides various services in autonomous vehicles and public transportation as virtual crew members with an emotion engine that recognizes user emotions, thereby achieving more advanced and user-friendly service provision. The specific system configuration and operation are described below.

[0903] System Configuration

[0904] This system consists of a user, a terminal, a server, and an emotion engine. The user operates a terminal such as a tablet or smartphone to use autonomous vehicles or public transportation. The terminal receives user input, processes it, and sends it to the server. The emotion engine recognizes the user's emotions in real time and sends that information to the server. The server performs calculations and data processing based on the received information and sends the results to the terminal.

[0905] Setting the destination and route

[0906] First, the user specifies their pick-up and destination. For example, the user sets the departure point to "Tokyo Station" and the destination to "Shinagawa Station" on their device's map application. This information is sent from the device to the server. The server calculates the optimal route based on the received departure and destination information. This calculation also takes into account sentiment data from the sentiment engine. The calculated route information (distance, travel time, whether or not highways can be used, etc.) is sent to the device and displayed to the user. Based on the user's sentiment, the choice of whether or not to use highways is optimized. For example, if the sentiment engine determines that the user is in a hurry, it will recommend using highways. The user's response information is sent back to the server, and the final route is determined.

[0907] In-train sales

[0908] Next, let's consider the case where a user requests an in-train purchase. The user operates the in-train sales application on their terminal and requests, "I want to order a bento box." This request is sent from the terminal to the server. Based on the specified request, the server retrieves the in-train sales menu and inventory information and sends it to the terminal. The emotion engine can also recognize the user's emotions and suggest products that match the user's mood. For example, if it determines that the user is tired, it will suggest a relaxing tea. The user selects the product they want, and this selection information is sent from the terminal to the server. The server checks the inventory of the selected product and sends a purchase confirmation message to the terminal. The terminal displays this purchase confirmation message to the user, and once the user confirms it, the order is finalized.

[0909] Ticket change

[0910] Finally, consider the case where a user requests a change to their ticket. The user operates the ticket management application on their terminal and requests a "change ticket." This request is sent from the terminal to the server. The server calculates the available options based on the request and sends that information to the terminal. The emotion engine analyzes the user's emotions and can suggest the most suitable change option based on the user's mood. For example, if it is determined that the user wants to relax, it will suggest a quiet carriage. The user selects their desired change option, and this selection information is sent from the terminal to the server. The server generates new ticket information based on the selection and sends it to the terminal. The terminal displays the new ticket information to the user.

[0911] As described above, the present invention is a system in which a user, terminal, server, and emotion engine work together to provide efficient and user-friendly services.

[0912] The following describes the processing flow.

[0913] Setting the destination and route

[0914] Step 1:

[0915] The user operates the device and enters the starting point and destination. For example, going from "Tokyo Station" to "Shinagawa Station".

[0916] Step 2:

[0917] The terminal receives user input information and sends it to the server. Specifically, data on the departure point and destination is sent to the server.

[0918] Step 3:

[0919] The server calculates the optimal route based on the information it receives. This calculation uses external map APIs and internal route calculation algorithms.

[0920] Step 4:

[0921] The server sends calculated route information to the terminal. This includes distance, estimated travel time, and whether highways can be used.

[0922] Step 5:

[0923] The terminal displays the route information it has received to the user. The user can then choose whether or not to use the highway.

[0924] Step 6:

[0925] The emotion engine analyzes the user's facial expressions and tone of voice to determine their emotional state. For example, if it determines that the user is anxious, it will recommend using the highway.

[0926] Step 7:

[0927] The user selects whether or not to use the highway, and the terminal sends this selection information to the server.

[0928] Step 8:

[0929] The server recalculates the route based on the user's selection information and sentiment data, and generates the final route information.

[0930] Step 9:

[0931] The server sends the final route information to the terminal. The terminal displays the final route on its navigation system.

[0932] In-train sales

[0933] Step 1:

[0934] The user operates the terminal and selects a request for in-car sales. For example, "Order a bento box."

[0935] Step 2:

[0936] The terminal sends the user's request to the server. Specifically, this includes the user ID and the request details.

[0937] Step 3:

[0938] The server retrieves the in-vehicle sales menu and inventory information. For example, it reads the latest menu and inventory numbers from the database.

[0939] Step 4:

[0940] The server sends the retrieved menu information to the terminal. This includes the product name, price, and stock quantity.

[0941] Step 5:

[0942] The terminal displays menu information to the user. The user selects the desired product.

[0943] Step 6:

[0944] The emotion engine analyzes the user's emotions and suggests products that match their mood. For example, it might suggest a relaxing tea to a tired user.

[0945] Step 7:

[0946] The user selects an item, and the device sends that selection information to the server. Specifically, this includes the item ID and quantity.

[0947] Step 8:

[0948] The server checks the inventory of the selected product. If it is in stock, it generates a purchase confirmation message and sends it to the terminal.

[0949] Step 9:

[0950] The device displays a purchase confirmation message to the user. Once the user confirms, the order is finalized.

[0951] Ticket change

[0952] Step 1:

[0953] The user operates the terminal and selects a request to change their ticket. For example, "Change departure time."

[0954] Step 2:

[0955] The terminal sends the user's request to the server. Specifically, this includes current ticket information and conditions for change.

[0956] Step 3:

[0957] The server calculates the options that can be changed. For example, it searches for available trains and seats.

[0958] Step 4:

[0959] The server sends the calculation results to the terminal. This includes information about trains and seats that can be changed.

[0960] Step 5:

[0961] The device displays change options to the user. The user selects the desired change option.

[0962] Step 6:

[0963] The emotion engine analyzes the user's emotions and suggests the optimal change options based on the user's mood. For example, it suggests a quiet vehicle to a user who wants to relax.

[0964] Step 7:

[0965] The user selects a change option, and the device sends that selection information to the server. Specifically, this includes the changed train ID and seat number.

[0966] Step 8:

[0967] The server generates new ticket information, which includes new train and seat information.

[0968] Step 9:

[0969] The server sends the new ticket information to the terminal. The terminal displays the new ticket information to the user.

[0970] (Example 2)

[0971] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0972] In providing services in autonomous vehicles and public transportation, traditional methods failed to consider user emotions, making it difficult to enhance user satisfaction. Furthermore, it was challenging to suggest optimal services and routes when users were in a hurry or experiencing specific emotional states. This resulted in a decline in service quality and a compromised user experience.

[0973] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0974] In this invention, the server includes means for receiving user input and generating departure point and destination information; means for calculating the optimal route using the received departure point and destination information; means for presenting the calculated route information to the user and confirming the availability of expressways; means for optimizing the route based on the user's response and generating final route information; means for recognizing the user's emotions in real time and transmitting that information to the server; means for optimizing the calculation results based on the information received from the emotion recognition means; and means for feeding the calculation results back to the user. This makes it possible to provide the optimal route and services based on the user's emotions, thereby improving the quality of the user experience.

[0975] "An emotion recognition method that recognizes user emotions in real time" refers to a means that analyzes the user's facial expressions, voice, behavioral patterns, etc., through a terminal operated by the user, and has the function of identifying the emotional state the user is currently feeling in real time and transmitting the information to a server.

[0976] A "means for calculating the optimal route" is a means that, based on information about the starting point and destination, takes into account traffic conditions and user sentiment data to calculate the most efficient and user-friendly route.

[0977] "Means for receiving user input and generating departure point and destination information" refers to means that have the function of receiving departure point and destination information entered by the user through a terminal and processing it as digital information.

[0978] "A means of presenting calculated route information to the user and confirming whether or not to use expressways" refers to a means that displays route information calculated by the server on the user's terminal and has a function to confirm with the user whether or not to use expressways.

[0979] "Means for optimizing routes based on user responses and generating final route information" refers to means that have the function of recalculating routes based on user selections and responses to determine the most appropriate final route.

[0980] "Means for providing feedback of calculation results to the user" refers to means that have the function of transmitting the results of various processes performed by the server to the user's terminal and displaying them so that the user can confirm them.

[0981] "Means for obtaining in-vehicle sales menus and inventory information" refers to means that a server manages the in-vehicle sales menu and its inventory status, and provides this information to the user terminal.

[0982] "Means for generating and presenting new ticket information to the user" refers to means that a server has the function of creating new ticket information based on a user's request and displaying it on the user's terminal.

[0983] Modes for carrying out the invention

[0984] This invention combines a system that provides various services in autonomous vehicles and public transportation as virtual crew members with an emotion engine that recognizes user emotions, thereby achieving more advanced and user-friendly service provision. The specific system configuration and operation are described below.

[0985] System Configuration

[0986] This system consists of a user, a terminal, a server, and an emotion engine. When using autonomous vehicles or public transportation, the user operates a terminal such as a tablet or smartphone. The terminal receives user input, processes it, and sends it to the server. The emotion engine recognizes the user's emotions in real time and sends that information to the server. The server performs calculations and data processing based on the received departure point, destination information, and user emotion data, and sends the results to the terminal.

[0987] Setting the destination and route

[0988] First, the user specifies their starting point and destination using their device. For example, they might set the starting point to "Tokyo Station" and the destination to "Shinagawa Station" on a map application. This information is sent from the device to the server. The server calculates the optimal route based on the starting point and destination data. This calculation also takes into account sentiment data from the sentiment engine; for example, if it determines that the user is in a hurry, it will suggest a route that prioritizes highways. The calculation results include distance, travel time, and whether highways are available. This information is sent to the device and displayed to the user. The user confirms the route and resends the response information from the device to the server, which then determines the final route and sends that information back to the device.

[0989] Example prompt:

[0990] "My departure point is Tokyo Station, and my destination is Shinagawa Station. I'm in a hurry."

[0991] In-train sales

[0992] This section explains how a user requests in-train sales. The user operates the in-train sales application on their terminal and sends a request such as "I want to order a bento box." The terminal sends this request to the server. Based on the request, the server retrieves the in-train sales menu and inventory information and sends it to the terminal. The emotion engine recognizes the user's emotions and, for example, if it determines that the user is tired, it can suggest relaxing tea or other items. Once the user selects the desired items, the selection information is sent from the terminal to the server, and after the server checks the inventory, it sends a purchase confirmation message to the terminal. The terminal displays the purchase confirmation message to the user, and the order is finalized upon user confirmation.

[0993] Example prompt:

[0994] "I'd like to order a bento box. I'm very tired."

[0995] Ticket change

[0996] This section describes a scenario where a user requests a change to their train ticket. The user operates the ticket management application on their terminal and requests a "change of ticket." This request is sent from the terminal to the server. The server calculates the available options and suggests the best option based on emotional data. For example, if it is determined that the user wants to relax, it will suggest a quiet carriage. The user selects their desired change option, and this information is sent from the terminal to the server. The server generates new ticket information, sends it to the terminal, and displays it to the user.

[0997] Example prompt:

[0998] "I want to change my ticket. I want to relax."

[0999] As described above, the present invention is a system in which the user, terminal, server, and emotion engine work together to provide efficient and user-friendly services. By considering the user's emotions when providing various services, the quality of the user experience can be improved.

[1000] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1001] Step 1:

[1002] The user uses a device to specify the starting point and destination.

[1003] Specific operation: The user opens a map application and enters "Tokyo Station" as the starting point and "Shinagawa Station" as the destination. The terminal receives this input information and stores it as input data.

[1004] Input: Departure point and destination information

[1005] Output: Data on departure point and destination.

[1006] Step 2:

[1007] The terminal sends the entered information to the server.

[1008] Specific operation: The terminal converts the user's entered departure point and destination information into a digital format and sends it to the server.

[1009] Input: User-entered data for departure point and destination.

[1010] Output: Departure and destination data sent to the server

[1011] Step 3:

[1012] The emotion engine recognizes the user's emotions in real time and sends that information to the server.

[1013] Specific operation: The emotion engine analyzes the user's facial expressions, voice, and actions to recognize their emotional state, such as whether they are in a hurry or relaxed. The recognized emotional information is then sent to the server.

[1014] Input: Real-time user sentiment information

[1015] Output: Emotion data sent to the server

[1016] Step 4:

[1017] The server calculates the optimal route based on the departure point, destination information, and sentiment data it receives.

[1018] Specific operation: The server considers the user's sentiment data in addition to the origin and destination data, and uses an optimal route algorithm to perform calculations. For example, if the user is in a hurry, it will calculate a route that prioritizes highways.

[1019] Input: Origin, destination, and sentiment data

[1020] Output: Optimal route information (distance, travel time, availability of highways, etc.)

[1021] Step 5:

[1022] The device receives the calculated route information and presents it to the user.

[1023] Specific operation: The server sends the calculated optimal route information to the terminal. The terminal displays this information and prompts the user for confirmation. For example, the displayed route information might say "From Tokyo Station to Shinagawa Station, travel time 20 minutes, expressway available."

[1024] Input: Optimal route information from the server

[1025] Output: Route information presented to the user

[1026] Step 6:

[1027] The user checks route information on their device and responds regarding the availability of highway access.

[1028] Specific operation: The user checks the route information displayed on the device and selects whether to "use the highway" or "do not use the highway." The selected information is sent from the device to the server.

[1029] Input: User response information (whether or not highway use is permitted)

[1030] Output: Response information sent to the server

[1031] Step 7:

[1032] The server optimizes the route based on the user's response information, generates the final route information, and sends it to the terminal.

[1033] Specific operation: The server makes final adjustments to the route based on the user's response information and generates the final route information. The terminal receives this and presents the final route to the user.

[1034] Input: User response information

[1035] Output: Final route information

[1036] Step 8:

[1037] The user checks the in-car sales menu and selects an item.

[1038] Specific operation: The user checks the in-vehicle sales menu on their terminal and selects the desired product (e.g., "bento box"). The selection information is sent from the terminal to the server.

[1039] Input: User's selection information (product)

[1040] Output: Selection information sent to the server

[1041] Step 9:

[1042] The server checks the inventory of the selected product and sends a purchase confirmation message to the terminal.

[1043] Specific operation: The server checks if the product selected by the user is in stock, and if it is, it generates a purchase confirmation message and sends it to the terminal.

[1044] Input: Server selection information (product)

[1045] Output: Purchase confirmation message sent to the device

[1046] Step 10:

[1047] The device displays a purchase confirmation message to the user, who then confirms it.

[1048] Specific operation: The terminal displays a purchase confirmation message received from the server to the user. The order is confirmed when the user confirms it and presses the confirm button.

[1049] Input: Purchase confirmation message

[1050] Output: Confirmation message and confirmation information displayed to the user

[1051] Step 11:

[1052] The user requests a change to their ticket using the ticket management application.

[1053] Specific operation: The user requests a "ticket change" on their terminal, for example, by entering "I would like a quiet carriage." The terminal sends the request to the server.

[1054] Input: User's ticket change request

[1055] Output: Request sent to the server

[1056] Step 12:

[1057] The server calculates the available options, and the emotion engine suggests the optimal option.

[1058] Specific operation: The server calculates the modifiable options based on the input request. The emotion engine evaluates the user's emotional state and, for example, suggests a quiet vehicle in response to a request to relax.

[1059] Input: User change requests and sentiment data

[1060] Output: Optimal change options

[1061] Step 13:

[1062] The terminal receives the new ticket information and presents it to the user.

[1063] Specific operation: The server sends the calculated new ticket information to the terminal. The terminal displays this information to the user and prompts them to confirm.

[1064] Input: Ticket information from the server

[1065] Output: New ticket information displayed to the user

[1066] (Application Example 2)

[1067] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1068] Services in modern autonomous vehicles often provide a uniform service without considering the mood or emotions of passengers. This makes it difficult to meet the diverse needs of users, resulting in a diminished user experience. Furthermore, there is a lack of optimal suggestions that take user feelings into account, even when it comes to in-vehicle sales or ticket changes. This leads to the problem of services being provided without passengers feeling comfortable or satisfied.

[1069] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving user input and generating departure point and destination information, means for calculating the optimal route using the received departure point and destination information, means for presenting the calculated route information to the user and confirming whether or not highways can be used, means for optimizing the route based on the user's response and generating final route information, means for recognizing the user's emotions in real time and transmitting emotion data to the server, and means for deciding whether or not to use highways considering the user's emotion information. This makes it possible to provide individually optimal routes and services based on the user's emotions.

[1070] An "autonomous vehicle" is a vehicle that can drive automatically without human intervention, using artificial intelligence and sensor technology.

[1071] "User input" refers to instructions or information that a user provides to a system through a terminal or device.

[1072] The "starting point" refers to the location where the user begins using the autonomous vehicle.

[1073] "Destination information" refers to information about the location that the user designates as their destination.

[1074] The "optimal route" is the most efficient path from the starting point to the destination. Efficiency is calculated by considering factors such as time, distance, and traffic conditions.

[1075] An "emotion engine" is a system that analyzes a user's facial expressions, tone of voice, and other factors to recognize their emotions in real time.

[1076] "Emotional data" refers to information about a user's emotions as recognized by the emotion engine.

[1077] "Whether or not to use expressways" refers to the user's choice of whether or not to use expressways to reach their destination.

[1078] "In-vehicle sales" refers to food, beverages, and other goods offered inside autonomous vehicles.

[1079] "Inventory information" refers to information regarding the current stock status of products offered for sale on board the train.

[1080] "Ticket change" refers to changing the type or details of a ticket that a user has already obtained.

[1081] "Selected information" refers to the specific information that the user has chosen from the options presented to them.

[1082] This invention is a system for making passenger services in autonomous vehicles more advanced and user-friendly. This system is composed of a user, a terminal, a server, and an emotion engine, which recognize the user's emotions in real time and provide optimal services in response to those emotions.

[1083] System Configuration

[1084] User

[1085] The users are passengers using the autonomous vehicle and operate the system through devices such as tablets and smartphones.

[1086] terminal

[1087] The terminal is a device that receives user instructions and collects data for emotion recognition. The terminal uses its camera and microphone to transmit the user's facial expressions and voice to the emotion engine.

[1088] server

[1089] A server primarily has three roles:

[1090] 1. Calculate the optimal route based on user input, originating point, and destination information.

[1091] 2. Receive requests for in-vehicle sales and manage product inventory information.

[1092] 3. Accept ticket changes and provide changeable options.

[1093] Emotional Engine

[1094] The emotion engine analyzes emotional data in real time from the user's facial expressions and voice, and sends that data to the server. Examples of technologies used include Amazon Rekognition and Microsoft Azure Emotion API.

[1095] Destination setting

[1096] The user first enters their starting point and destination into the terminal. For example, if the user sets the starting point to "Tokyo Station" and the destination to "Shinagawa Station," this information is sent from the terminal to the server. The server calculates the optimal route and presents the user with route information, including whether or not highways can be used. Furthermore, the emotion engine analyzes whether the user is in a hurry and may recommend using highways.

[1097] In-train sales

[1098] When a user wants to use the in-train sales service, they request to "order a bento box" through the application on their terminal. This request is sent to the server, which then suggests products based on inventory information. If the emotion engine determines that the user is tired, it also suggests a relaxing tea.

[1099] Ticket change

[1100] When a user requests a change to their ticket, that information is sent from the terminal to the server. The server also takes the user's emotional state into consideration; for example, if the user wants to relax, it might suggest a quieter carriage.

[1101] Specific examples and prompt statements

[1102] Specific examples of route acquisition:

[1103] Departure point: "Tokyo Station"

[1104] Destination: Shinagawa Station

[1105] Emotion: "When the user is in a hurry"

[1106] Result: "Proposal for the fastest route using expressways"

[1107] Please calculate the route based on the following information:

[1108] Departure point: Tokyo Station

[1109] Destination: Shinagawa Station

[1110] User's emotion: In a hurry

[1111] Examples of in-train sales:

[1112] Order: "Bento"

[1113] Emotion: "When the user is tired"

[1114] Result: "Suggestion for tea with relaxing effects"

[1115] Please order the following items:

[1116] Order: Bento

[1117] User's emotion: tired

[1118] Additional suggestion: Tea with relaxing effects

[1119] Specific examples of ticket changes:

[1120] Change option: "Quiet vehicle"

[1121] Emotion: "When the user wants to relax"

[1122] Result: "Propose a quiet vehicle."

[1123] Please change your ticket using one of the following options:

[1124] Change options: Quiet vehicle

[1125] User's emotion: Wants to relax

[1126] As described above, the system of the present invention can provide individually optimized routes and services while taking into account the user's emotions. This is expected to significantly improve passenger comfort and satisfaction.

[1127] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1128] Step 1:

[1129] The user enters their departure point and destination information into the device. For example, they might enter "Tokyo Station" as the departure point and "Shinagawa Station" as the destination. This input information is retrieved by the map application on the device.

[1130] Step 2:

[1131] The terminal sends the entered departure and destination information to the server. At this time, the emotion engine analyzes the user's facial expressions and voice, and emotion data is generated. This emotion data is also sent to the server.

[1132] Step 3:

[1133] The server receives input information and sentiment data. It calculates the optimal route based on the departure and destination information, and further optimizes route selection by considering sentiment data. The decision of whether or not to use highways is also made at this stage.

[1134] Step 4:

[1135] The server sends the calculated optimal route information and a suggestion regarding the availability of highways to the terminal. The terminal displays this information to the user and receives the user's response again.

[1136] Step 5:

[1137] The user enters response information into the terminal in response to the presented route information and the suggestion regarding the use of expressways. For example, they might select "Use expressways."

[1138] Step 6:

[1139] The terminal sends the user's response information back to the server. The server optimizes the final route based on the response information and generates the final route information.

[1140] Step 7:

[1141] The server sends the final route information to the terminal, which then displays it to the user. At this stage, the user can verify the optimized route information.

[1142] Step 8:

[1143] If a user wants to use the in-train sales service, they enter their order information through the application on their terminal. For example, they might enter "I want to order a bento box." This information is then sent from the terminal to the server.

[1144] Step 9:

[1145] The server retrieves in-car sales menus and inventory information based on the received order information, and also takes into account the user's emotional state to make appropriate product suggestions. For example, if the user is tired, it will suggest a relaxing tea.

[1146] Step 10:

[1147] The server sends the suggested menu information to the terminal, which then displays it to the user. The user selects an item from the presented menu and enters the selection information into the terminal.

[1148] Step 11:

[1149] The terminal sends the selection information to the server. The server checks the inventory of the selected items, generates a purchase confirmation message, and sends it to the terminal. The terminal displays this message to the user, who then confirms it.

[1150] Step 12:

[1151] If a user wishes to change their ticket, they enter a change request through the ticket management application on their terminal. For example, they might enter, "I would like to change to a quieter carriage." This request is then sent from the terminal to the server.

[1152] Step 13:

[1153] The server generates information on modifiable options based on the received change request. Sentimental data is also taken into consideration, and options suitable for, for example, a user who wants to relax are presented.

[1154] Step 14:

[1155] The server sends information about the configurable options to the terminal, which then displays this information to the user. The user selects the desired options and enters that information into the terminal.

[1156] Step 15:

[1157] The terminal sends the selection information to the server. The server generates new ticket information and sends that information to the terminal. The terminal displays the new ticket information to the user, allowing the user to confirm the details of the new ticket.

[1158] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1159] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1160] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1161] [Third Embodiment]

[1162] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1163] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1164] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1165] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1166] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1167] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1168] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1169] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1170] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1171] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1172] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1173] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1174] This invention is a system that provides various services in autonomous vehicles and public transportation by having virtual crew members. The specific system configuration and operation are described below.

[1175] System Configuration

[1176] This system consists of users, terminals, and servers. Users operate terminals such as tablets and smartphones to use autonomous vehicles and public transportation. The terminal receives user input, processes it, and sends it to the server. The server performs calculations and data processing based on the received information and sends the results to the terminal.

[1177] Setting the destination and route

[1178] First, the user specifies their starting point and destination. For example, the user sets the starting point to "Tokyo Station" and the destination to "Shinagawa Station" on the map application on their device. This information is sent from the device to the server. The server calculates the optimal route based on the received starting point and destination information. The calculated route information (distance, travel time, whether or not highways can be used, etc.) is sent to the device and displayed to the user. The user then selects whether or not to use highways. For example, if the user selects "Use highways," this information is sent from the device to the server. The server optimizes the route based on this selection, generates the final route information, and sends it to the device. The device then displays the final route on its navigation system.

[1179] In-train sales

[1180] Next, let's consider the case where a user requests an in-train purchase. The user operates the in-train sales application on their terminal and requests, "I want to order a bento box." This request is sent from the terminal to the server. The server retrieves the in-train sales menu and inventory information based on the specified request and sends it to the terminal. The terminal displays the retrieved menu information to the user. The user selects the desired product (for example, "sushi bento"). This selection information is sent from the terminal to the server. The server checks the inventory of the selected product and sends a purchase confirmation message to the terminal. The terminal displays this purchase confirmation message to the user. Once the user confirms, the ordering process is complete.

[1181] Ticket change

[1182] Finally, consider the case where a user requests a change to their ticket. The user operates the ticket management application on their terminal and requests a "change ticket." This request is sent from the terminal to the server. The server calculates the available change options based on the request and sends that information to the terminal. The terminal presents the user with the available change options (e.g., change time, change seat). The user selects the desired change option. This selection information is sent from the terminal to the server. The server generates new ticket information based on the selection and sends it to the terminal. The terminal displays the new ticket information to the user.

[1183] As described above, the present invention is a system in which users, terminals, and servers cooperate to provide efficient and user-friendly services.

[1184] The following describes the processing flow.

[1185] Setting the destination and route

[1186] Step 1:

[1187] The user operates the terminal and enters the departure point and destination. For example, the user enters "Tokyo Station" as the departure point and "Shinagawa Station" as the destination.

[1188] Step 2:

[1189] The terminal receives user input information and sends it to the server. Specifically, it sends data on the departure point and destination to the server.

[1190] Step 3:

[1191] The server calculates the optimal route based on the information it receives. This calculation uses external map APIs or the company's own database.

[1192] Step 4:

[1193] The server sends calculated route information to the terminal, including distance, estimated travel time, and whether highways can be used.

[1194] Step 5:

[1195] The terminal displays the route information it has received to the user. The user can then choose whether or not to use the highway.

[1196] Step 6:

[1197] The user selects whether or not to use the highway, and the terminal sends this selection information to the server.

[1198] Step 7:

[1199] The server recalculates the route based on the user's selection information and generates the final route information.

[1200] Step 8:

[1201] The server sends the final route information to the terminal. The terminal displays the final route on its navigation system.

[1202] In-train sales

[1203] Step 1:

[1204] The user operates the terminal and selects a request for in-car sales. For example, they might select the "Order a bento box" option.

[1205] Step 2:

[1206] The terminal sends the user's request to the server. Specifically, this includes the user ID and the request details.

[1207] Step 3:

[1208] The server retrieves the in-vehicle sales menu and inventory information. For example, it reads the latest menu and inventory numbers from the database.

[1209] Step 4:

[1210] The server sends the retrieved menu information to the terminal. This includes the product name, price, and stock quantity.

[1211] Step 5:

[1212] The terminal displays menu information to the user. The user selects the desired product.

[1213] Step 6:

[1214] The user selects an item, and the device sends that selection information to the server. Specifically, this includes the item ID and quantity.

[1215] Step 7:

[1216] The server checks the inventory of the selected product. If it is in stock, it generates a purchase confirmation message and sends it to the terminal.

[1217] Step 8:

[1218] The device displays a purchase confirmation message to the user. The order is finalized once the user confirms it.

[1219] Ticket change

[1220] Step 1:

[1221] The user operates the terminal and selects a request to change their ticket. For example, they might select the "Change boarding time" menu.

[1222] Step 2:

[1223] The terminal sends the user's request to the server. Specifically, this includes current ticket information and conditions for change.

[1224] Step 3:

[1225] The server calculates the options that can be changed. For example, it searches the database for available trains and seats.

[1226] Step 4:

[1227] The server sends the calculation results to the terminal. This includes information about trains and seats that can be changed.

[1228] Step 5:

[1229] The device displays change options to the user. The user selects the desired change option.

[1230] Step 6:

[1231] The user selects a change option, and the device sends that selection information to the server. Specifically, this includes the changed train ID and seat number.

[1232] Step 7:

[1233] The server generates new ticket information, which includes new train and seat information.

[1234] Step 8:

[1235] The server sends the new ticket information to the terminal. The terminal displays the new ticket information to the user.

[1236] (Example 1)

[1237] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1238] To improve user convenience and comfort in autonomous vehicles and public transportation, it is necessary to efficiently provide a wide range of services, such as destination setting, in-vehicle sales, and ticket changes. However, in conventional systems, these services are often fragmented, making it difficult to provide a unified experience. Furthermore, advanced data processing technology is required to process user input information quickly and accurately and provide users with the necessary information.

[1239] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1240] In this invention, the server includes means for receiving user input and generating departure point and destination information; means for calculating an optimal route using the received departure point and destination information; means for presenting the calculated route information to the user and confirming the availability of highways; means for optimizing the route based on the user's response and generating final route information; means for displaying the final route information on a navigation system; means for receiving the user's in-vehicle sales request and transmitting it to the server; means for acquiring in-vehicle sales menus and inventory information; means for presenting the acquired menu information to the user; means for receiving the user's selection information and transmitting it to the server; means for confirming the inventory of selected items and presenting a purchase confirmation message to the user; means for receiving user confirmation and completing the order process; means for receiving the user's ticket change request and transmitting it to the server; means for presenting the user with changeable option information calculated by the server; means for receiving the user's selection information and transmitting it to the server; means for generating new ticket information and presenting it to the user; and means for displaying the new ticket information on a navigation system. This enables the provision of a consistent user experience and the efficient execution of various services.

[1241] "User" refers to a person who operates this system in order to use autonomous vehicles or public transportation.

[1242] A "device" is an electronic device used by a user, and includes tablets, smartphones, and other similar devices.

[1243] A "server" refers to a central system that receives information from users, performs data processing and calculations, and sends the results to terminals.

[1244] The "starting point" refers to the initial location from which the user boards the vehicle, and is one of the location information settings in a map application.

[1245] "Destination information" refers to information about the location that the user sets as their destination.

[1246] "Optimal route" refers to the shortest or most efficient route information calculated by the server based on the origin and destination information received.

[1247] A "navigation system" is a system that displays optimal route information and final route information to guide the user to their destination.

[1248] An "in-car sales request" refers to a request sent from a terminal to a server when a user wishes to purchase goods while in a vehicle.

[1249] The "in-train sales menu" refers to a list of products offered on board the train.

[1250] "Inventory information" refers to information showing the availability of each product included in the in-train sales menu.

[1251] "Selected information" refers to information about products or services that the user has selected on their device.

[1252] A "purchase confirmation message" refers to a confirmation message sent by the server to the user to complete the purchase process for the selected product.

[1253] A "ticket change request" refers to a request sent from a user's device to a server when the user wishes to change the details of a ticket they have already purchased.

[1254] "Changeable option information" refers to the ticket modification options that the server calculates and that the user can select.

[1255] "New ticket information" refers to information about a new ticket generated based on the user's selection.

[1256] This invention relates to a virtual crew system that provides a unified range of services in autonomous vehicles and public transportation. The following describes embodiments of this system.

[1257] System Configuration

[1258] This system consists of users, terminals, and servers. Users operate terminals such as tablets and smartphones to use autonomous vehicles and public transportation. The terminal receives user input, processes it, and sends it to the server. The server performs calculations and data processing based on the received information and sends the results to the terminal.

[1259] Setting the destination and route

[1260] First, the user specifies their pick-up and destination. For example, the user uses their device's map application to set the departure point to "Tokyo Station" and the destination to "Shinagawa Station." This information is sent from the device to the server. The server uses the Google Maps API to calculate the optimal route based on the received departure and destination information. The calculated route information (distance, travel time, whether highways can be used, etc.) is sent to the device and displayed to the user. The user then chooses whether or not to use highways. For example, if they select "Use highways," this information is sent from the device to the server. The server optimizes the route based on this selection, generates the final route information, and sends it to the device. The device then displays the final route on its navigation system.

[1261] Specific example:

[1262] Prompt message:

[1263] "Please set the departure point to Tokyo Station and the destination to Shinagawa Station."

[1264] In-train sales

[1265] Next, let's consider the case where a user requests an in-train purchase. The user uses the in-train sales application on their terminal to request, "I want to order a bento box." This request is sent from the terminal to the server. The server retrieves the in-train sales menu and inventory information based on the specified request and sends it to the terminal. The terminal displays the retrieved menu information to the user. The user selects the desired item (for example, "sushi bento"). This selection information is sent from the terminal to the server. The server checks the inventory of the selected item and sends a purchase confirmation message to the terminal. The terminal displays this purchase confirmation message to the user. Once the user confirms, the ordering process is complete.

[1266] Specific example:

[1267] Prompt message:

[1268] "Please select and order your bento box."

[1269] Ticket change

[1270] Finally, consider the case where a user requests a change to their ticket. The user requests a "change ticket" by operating the ticket management application on their terminal. This request is sent from the terminal to the server. The server calculates the available change options based on the request and sends that information to the terminal. The server uses its own backend system and reservation management system. It presents the user with the available change options (e.g., change time, change seat). The user selects the desired change option and sends the selection information from the terminal to the server. The server generates new ticket information based on the selection and sends it to the terminal. The terminal displays the new ticket information to the user.

[1271] Specific example:

[1272] Prompt message:

[1273] "Please view and select the ticket change options."

[1274] As described above, this system enables efficient and user-friendly services through the collaboration of users, terminals, and servers. By utilizing data processing technologies, including the Google Maps API, this system makes it possible to provide advanced services.

[1275] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1276] Setting the destination and route

[1277] Step 1:

[1278] The user operates their device (tablet or smartphone) and opens a map application. The user sets the starting point to "Tokyo Station" and the destination to "Shinagawa Station," and then presses the send button.

[1279] Input: Departure point and destination information (Tokyo Station, Shinagawa Station)

[1280] Specific operation: The user specifies the starting point and destination by dragging and dropping pins on the map using their device.

[1281] Output: The departure point and destination information are set on the terminal.

[1282] Step 2:

[1283] The device sends its departure point and destination information to the server.

[1284] Input: Departure point and destination information set by the user.

[1285] Specific operation: The app on the device creates an HTTP request and sends it to the server.

[1286] Output: The server receives the departure and destination information.

[1287] Step 3:

[1288] The server calculates the optimal route based on the information it receives. The server uses the Google Maps API to obtain the optimal route (distance, travel time, whether highways can be used, etc.).

[1289] Input: Departure point and destination information

[1290] Specific operation: The server accesses the Google Maps API route search endpoint and sends the starting point and destination parameters.

[1291] Output: Optimal route information is generated on the server.

[1292] Step 4:

[1293] The server sends the calculated optimal route information to the terminal.

[1294] Input: Optimal route information

[1295] Specific operation: The server sends data in JSON format to the terminal.

[1296] Output: The terminal receives optimal route information.

[1297] Step 5:

[1298] The terminal displays the route information it has received to the user. The user can choose whether or not to use highways.

[1299] Input: Optimal route information

[1300] Specific action: The app on the device displays route information on the map and shows a checkbox indicating whether to "use highways."

[1301] Output: The user's highway usage selection information is determined.

[1302] Step 6:

[1303] The user selects the option to use the highway and presses the submit button. The terminal sends the selection information to the server.

[1304] Input: User's highway usage selection information

[1305] Specific operation: The user checks / unchecks a checkbox within the app on their device and presses the submit button. The device sends the selection information to the server via an HTTP request.

[1306] Output: The server receives the selection information.

[1307] Step 7:

[1308] The server recalculates the route based on the selected information and generates the final route information. It then sends the final route information to the terminal.

[1309] Input: User's highway usage selection information

[1310] Specific operation: The server uses the selected information to re-execute the optimal route calculation algorithm and generate the final route information.

[1311] Output: The final route information is generated on the server and sent to the terminal.

[1312] Step 8:

[1313] The device displays the final route information on the navigation system.

[1314] Input: Final route information

[1315] Specific operation: The device's navigation app receives the final route information and displays it on the map.

[1316] Output: The user can check the final route information.

[1317] In-train sales

[1318] Step 1:

[1319] The user uses the in-car sales application on their terminal to request, "I would like to order a bento box."

[1320] Input: User's in-car sales request

[1321] Specific action: The user selects "In-car sales" from the app's menu and presses the "Order a bento box" button.

[1322] Output: An in-car sales request is set on the terminal.

[1323] Step 2:

[1324] The terminal sends the user's in-car sales request to the server.

[1325] Input: User's in-car sales request

[1326] Specific action: The terminal creates an HTTP request and sends it to the server.

[1327] Output: The server receives an in-vehicle sales request.

[1328] Step 3:

[1329] The server retrieves the in-car sales menu and inventory information based on the request and sends it to the terminal.

[1330] Input: In-car sales request

[1331] Specific operation: The server retrieves the product list and inventory information from the database and sends it to the terminal.

[1332] Output: The in-car sales menu and inventory information are sent to the terminal.

[1333] Step 4:

[1334] The terminal displays the menu information it has acquired to the user, and the user selects the product they want (for example, "sushi bento").

[1335] Input: In-car sales menu and inventory information

[1336] Specific action: The app on the device displays a menu list, and the user taps on "Sushi Bento".

[1337] Output: User selection information is set on the terminal.

[1338] Step 5:

[1339] The device sends the user's selection information to the server.

[1340] Input: User's selected information

[1341] Specific action: The terminal creates an HTTP request containing the selection information and sends it to the server.

[1342] Output: The server receives the user's selection information.

[1343] Step 6:

[1344] The server checks the inventory of the selected product, generates a purchase confirmation message, and sends it to the terminal.

[1345] Input: User's selected information

[1346] Specific operation: The server retrieves the inventory quantity of the product from the database, and if it is in stock, it generates a purchase confirmation message.

[1347] Output: A purchase confirmation message is generated on the server and sent to the device.

[1348] Step 7:

[1349] The device displays a purchase confirmation message to the user, and once the user confirms it, the order process is completed.

[1350] Input: Purchase confirmation message

[1351] Specific action: The user presses the confirmation button, and the device sends the order information to the server.

[1352] Output: The order has been confirmed and the shipping process has begun.

[1353] Ticket change

[1354] Step 1:

[1355] The user requests a "ticket change" using the ticket management application on their device.

[1356] Input: User's ticket change request

[1357] Specific action: The user presses the "Change Ticket" button on the ticket management screen in the app.

[1358] Output: A ticket change request is set on the terminal.

[1359] Step 2:

[1360] The device sends a user change request to the server.

[1361] Input: User's ticket change request

[1362] Specific action: The terminal creates an HTTP request and sends it to the server.

[1363] Output: The server receives a ticket change request.

[1364] Step 3:

[1365] The server calculates the modifiable options based on the request and sends that information to the terminal.

[1366] Input: Ticket change request

[1367] Specific operation: The server retrieves the current reservation information from the database and calculates the options that can be changed.

[1368] Output: Modifiable option information is generated on the server and sent to the terminal.

[1369] Step 4:

[1370] The device presents the user with available modification options, and the user selects the desired modification option.

[1371] Input: Changeable option information

[1372] Specific operation: The app on the device displays change options, and the user selects one.

[1373] Output: User selection information is set on the terminal.

[1374] Step 5:

[1375] The device sends the user's selection information to the server.

[1376] Input: User's selected information

[1377] Specific action: The terminal creates an HTTP request containing the selection information and sends it to the server.

[1378] Output: The server receives the user's selection information.

[1379] Step 6:

[1380] The server generates new ticket information based on the selections and sends it to the terminal.

[1381] Input: User's selected information

[1382] Specific operation: The server generates new ticket information and sends it to the terminal.

[1383] Output: New ticket information is generated on the server and sent to the terminal.

[1384] Step 7:

[1385] The terminal displays the new ticket information to the user.

[1386] Input: New ticket information

[1387] Specific action: The app on the device displays the new ticket information, and the user confirms it.

[1388] Output: The user can view the new ticket information.

[1389] The above steps enable a system where users, terminals, and servers work together to provide efficient and user-friendly services. Including specific actions in each processing step makes the program flow clearer.

[1390] (Application Example 1)

[1391] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1392] The present invention aims to provide a method for providing food delivery services more seamlessly and efficiently. Specifically, the objective is to provide a system that allows users to order food from within an autonomous vehicle, track the delivery status in real time, and automatically generate appropriate responses to user inquiries and requests.

[1393] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1394] In this invention, the server includes means for receiving user input and generating departure point and destination information; means for calculating an optimal route using the received departure point and destination information; means for presenting the calculated route information to the user and confirming the availability of highways; means for optimizing the route based on the user's response and generating final route information; means for receiving user order information and providing food delivery services; means for providing the user with real-time delivery status and estimated arrival time; and means for analyzing user inquiries and providing appropriate answers using a generative AI model. This enables a seamless experience and rapid response when users utilize food delivery services in an autonomous vehicle.

[1395] "User input" refers to the information and instructions that a user provides through their device.

[1396] "Departure point" refers to the location where the user boards or the starting point.

[1397] "Destination information" refers to information about the place or destination that the user intends to go.

[1398] The "optimal route" refers to the most suitable travel route calculated considering the efficiency and convenience of the path.

[1399] "Availability of using expressways" refers to the user's choice of whether or not to use expressways.

[1400] "Final route information" refers to the final guidance information based on an optimized route.

[1401] "Order information" refers to the detailed information that users enter when purchasing food or products.

[1402] A "food delivery service" refers to a service that delivers food to a location specified by the user.

[1403] "Delivery status" refers to the current progress and status of the food or product delivery process.

[1404] "Estimated arrival time" refers to the estimated time it will take for a delivery to reach its destination.

[1405] "Inquiries" refer to information related to questions and requests from users.

[1406] A "generative AI model" refers to a model that uses artificial intelligence technology to automatically create answers and information in response to user inquiries.

[1407] An "appropriate response" refers to a response that provides accurate and relevant information in response to a user's inquiry or request.

[1408] This invention is a system that supports user operation within an autonomous vehicle in order to efficiently provide food delivery services. This system consists of a user, a terminal, and a server. The specific operation of each component is described below.

[1409] System Configuration

[1410] This system allows users to operate it through a device, which then interacts with a server to provide services. The device can be a smartphone or tablet, and the server is a cloud server (such as AWS or Azure).

[1411] Setting the destination and route

[1412] First, the user enters their starting point and destination information into the terminal. Using a map application on the terminal, the user specifies the starting point and destination. This information is sent from the terminal to the server, which calculates the optimal route. The calculated route information is presented to the user, and the user selects whether or not to use highways. Based on this information, the server generates an optimized final route and sends it to the terminal.

[1413] Food delivery service

[1414] When a user orders food from their device, the order information is sent to a server. The server provides the food delivery service and gives the user real-time delivery status and estimated arrival time. Delivery status is tracked using GPS, and the estimated arrival time is calculated dynamically.

[1415] Customer Support

[1416] User inquiries are sent directly from the terminal to the server. The server analyzes the inquiry using a generative AI model (e.g., GPT-4) and generates an appropriate response. The generated response is sent to the terminal and displayed to the user.

[1417] Hardware and software to be used

[1418] Hardware: Smartphones, tablets, cloud servers (AWS, Azure)

[1419] Software: iOS or Android native app development tools (Swift, Kotlin, etc.), server-side frameworks (Node.js, Flask, etc.), SQL databases (MySQL, PostgreSQL, etc.), generative AI models (GPT-4)

[1420] Specific example

[1421] This describes a scenario where a user orders a pizza from inside an autonomous vehicle and monitors the delivery status in real time. When the user operates a terminal and orders a pizza, the order information is sent to a server. The server processes the order information, and a delivery person departs with the pizza. The delivery status is tracked via GPS, the estimated arrival time is calculated, and this information is provided to the user in real time. Furthermore, if the user makes a question, a generative AI model generates an appropriate answer, which is then displayed to the user.

[1422] Example of a prompt:

[1423] "A user has ordered a pizza. Please provide the delivery person's current location in real time and the estimated arrival time."

[1424] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1425] Step 1:

[1426] The user operates the terminal and enters departure and destination information. The entered information is temporarily stored within the terminal. The terminal then sends the entered departure and destination information as data to the server.

[1427] Input: Departure point and destination information entered by the user on the device.

[1428] Output: Departure point and destination information sent to the server

[1429] Step 2:

[1430] The server calculates the optimal route based on the received origin and destination information. It uses a route calculation algorithm (e.g., Dijkstra's algorithm) to identify the most efficient route. The calculated route information is stored internally by the server.

[1431] Input: Departure point and destination information

[1432] Output: Calculated optimal route information

[1433] Step 3:

[1434] The server sends the calculated route information to the terminal, which then presents the information to the user. The user reviews the route information and selects whether or not to use highways. The selection information is then sent back to the server from the terminal.

[1435] Input: Optimal route information

[1436] Output: User selection information (whether or not highways can be used)

[1437] Step 4:

[1438] The server generates an optimized final route based on the user's selection information. It recalculates the route and generates new route information. The final route information is sent from the server to the terminal and presented to the user.

[1439] Input: User's selected information

[1440] Output: Final route information

[1441] Step 5:

[1442] The user orders food from their device. The device sends the order information to the server. The order information includes the type and quantity of food, and the delivery address.

[1443] Input: User's order information

[1444] Output: Order information sent to the server

[1445] Step 6:

[1446] The server receives and processes the order information. This includes checking order availability and arranging delivery. The server then generates an order confirmation message and sends it to the terminal.

[1447] Input: User's order information

[1448] Output: Order confirmation message

[1449] Step 7:

[1450] The server tracks the progress of food deliveries in real time. It uses GPS to obtain the delivery person's location and calculates the estimated arrival time. This information is processed within the server.

[1451] Input: Delivery driver's location information

[1452] Output: Estimated arrival time and delivery status

[1453] Step 8:

[1454] The server sends the delivery status and estimated arrival time to the terminal. The terminal displays this information to the user, allowing them to check the delivery status in real time.

[1455] Input: Estimated arrival time and delivery status

[1456] Output: Delivery status and estimated arrival time presented to the user.

[1457] Step 9:

[1458] When a user submits an inquiry, the device sends the inquiry as text data to the server. The server uses a generative AI model to analyze the inquiry and generate an appropriate response.

[1459] Input: Inquiry details

[1460] Output: Generated answer

[1461] Step 10:

[1462] The server sends the response generated by the AI ​​model to the terminal. The terminal displays the response to the user, and appropriate action is taken.

[1463] Input: Generated answer

[1464] Output: The answer presented to the user

[1465] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1466] This invention combines a system that provides various services in autonomous vehicles and public transportation as virtual crew members with an emotion engine that recognizes user emotions, thereby achieving more advanced and user-friendly service provision. The specific system configuration and operation are described below.

[1467] System Configuration

[1468] This system consists of a user, a terminal, a server, and an emotion engine. The user operates a terminal such as a tablet or smartphone to use autonomous vehicles or public transportation. The terminal receives user input, processes it, and sends it to the server. The emotion engine recognizes the user's emotions in real time and sends that information to the server. The server performs calculations and data processing based on the received information and sends the results to the terminal.

[1469] Setting the destination and route

[1470] First, the user specifies their pick-up and destination. For example, the user sets the departure point to "Tokyo Station" and the destination to "Shinagawa Station" on their device's map application. This information is sent from the device to the server. The server calculates the optimal route based on the received departure and destination information. This calculation also takes into account sentiment data from the sentiment engine. The calculated route information (distance, travel time, whether or not highways can be used, etc.) is sent to the device and displayed to the user. Based on the user's sentiment, the choice of whether or not to use highways is optimized. For example, if the sentiment engine determines that the user is in a hurry, it will recommend using highways. The user's response information is sent back to the server, and the final route is determined.

[1471] In-train sales

[1472] Next, let's consider the case where a user requests an in-train purchase. The user operates the in-train sales application on their terminal and requests, "I want to order a bento box." This request is sent from the terminal to the server. Based on the specified request, the server retrieves the in-train sales menu and inventory information and sends it to the terminal. The emotion engine can also recognize the user's emotions and suggest products that match the user's mood. For example, if it determines that the user is tired, it will suggest a relaxing tea. The user selects the product they want, and this selection information is sent from the terminal to the server. The server checks the inventory of the selected product and sends a purchase confirmation message to the terminal. The terminal displays this purchase confirmation message to the user, and once the user confirms it, the order is finalized.

[1473] Ticket change

[1474] Finally, consider the case where a user requests a change to their ticket. The user operates the ticket management application on their terminal and requests a "change ticket." This request is sent from the terminal to the server. The server calculates the available options based on the request and sends that information to the terminal. The emotion engine analyzes the user's emotions and can suggest the most suitable change option based on the user's mood. For example, if it is determined that the user wants to relax, it will suggest a quiet carriage. The user selects their desired change option, and this selection information is sent from the terminal to the server. The server generates new ticket information based on the selection and sends it to the terminal. The terminal displays the new ticket information to the user.

[1475] As described above, the present invention is a system in which a user, terminal, server, and emotion engine work together to provide efficient and user-friendly services.

[1476] The following describes the processing flow.

[1477] Setting the destination and route

[1478] Step 1:

[1479] The user operates the device and enters the starting point and destination. For example, going from "Tokyo Station" to "Shinagawa Station".

[1480] Step 2:

[1481] The terminal receives user input information and sends it to the server. Specifically, data on the departure point and destination is sent to the server.

[1482] Step 3:

[1483] The server calculates the optimal route based on the information it receives. This calculation uses external map APIs and internal route calculation algorithms.

[1484] Step 4:

[1485] The server sends calculated route information to the terminal. This includes distance, estimated travel time, and whether highways can be used.

[1486] Step 5:

[1487] The terminal displays the route information it has received to the user. The user can then choose whether or not to use the highway.

[1488] Step 6:

[1489] The emotion engine analyzes the user's facial expressions and tone of voice to determine their emotional state. For example, if it determines that the user is anxious, it will recommend using the highway.

[1490] Step 7:

[1491] The user selects whether or not to use the highway, and the terminal sends this selection information to the server.

[1492] Step 8:

[1493] The server recalculates the route based on the user's selection information and sentiment data, and generates the final route information.

[1494] Step 9:

[1495] The server sends the final route information to the terminal. The terminal displays the final route on its navigation system.

[1496] In-train sales

[1497] Step 1:

[1498] The user operates the terminal and selects a request for in-car sales. For example, "Order a bento box."

[1499] Step 2:

[1500] The terminal sends the user's request to the server. Specifically, this includes the user ID and the request details.

[1501] Step 3:

[1502] The server retrieves the in-vehicle sales menu and inventory information. For example, it reads the latest menu and inventory numbers from the database.

[1503] Step 4:

[1504] The server sends the retrieved menu information to the terminal. This includes the product name, price, and stock quantity.

[1505] Step 5:

[1506] The terminal displays menu information to the user. The user selects the desired product.

[1507] Step 6:

[1508] The emotion engine analyzes the user's emotions and suggests products that match their mood. For example, it might suggest a relaxing tea to a tired user.

[1509] Step 7:

[1510] The user selects an item, and the device sends that selection information to the server. Specifically, this includes the item ID and quantity.

[1511] Step 8:

[1512] The server checks the inventory of the selected product. If it is in stock, it generates a purchase confirmation message and sends it to the terminal.

[1513] Step 9:

[1514] The device displays a purchase confirmation message to the user. Once the user confirms, the order is finalized.

[1515] Ticket change

[1516] Step 1:

[1517] The user operates the terminal and selects a request to change their ticket. For example, "Change departure time."

[1518] Step 2:

[1519] The terminal sends the user's request to the server. Specifically, this includes current ticket information and conditions for change.

[1520] Step 3:

[1521] The server calculates the options that can be changed. For example, it searches for available trains and seats.

[1522] Step 4:

[1523] The server sends the calculation results to the terminal. This includes information about trains and seats that can be changed.

[1524] Step 5:

[1525] The device displays change options to the user. The user selects the desired change option.

[1526] Step 6:

[1527] The emotion engine analyzes the user's emotions and suggests the optimal change options based on the user's mood. For example, it suggests a quiet vehicle to a user who wants to relax.

[1528] Step 7:

[1529] The user selects a change option, and the device sends that selection information to the server. Specifically, this includes the changed train ID and seat number.

[1530] Step 8:

[1531] The server generates new ticket information, which includes new train and seat information.

[1532] Step 9:

[1533] The server sends the new ticket information to the terminal. The terminal displays the new ticket information to the user.

[1534] (Example 2)

[1535] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1536] In providing services in autonomous vehicles and public transportation, traditional methods failed to consider user emotions, making it difficult to enhance user satisfaction. Furthermore, it was challenging to suggest optimal services and routes when users were in a hurry or experiencing specific emotional states. This resulted in a decline in service quality and a compromised user experience.

[1537] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1538] In this invention, the server includes means for receiving user input and generating departure point and destination information; means for calculating the optimal route using the received departure point and destination information; means for presenting the calculated route information to the user and confirming the availability of expressways; means for optimizing the route based on the user's response and generating final route information; means for recognizing the user's emotions in real time and transmitting that information to the server; means for optimizing the calculation results based on the information received from the emotion recognition means; and means for feeding the calculation results back to the user. This makes it possible to provide the optimal route and services based on the user's emotions, thereby improving the quality of the user experience.

[1539] "An emotion recognition method that recognizes user emotions in real time" refers to a means that analyzes the user's facial expressions, voice, behavioral patterns, etc., through a terminal operated by the user, and has the function of identifying the emotional state the user is currently feeling in real time and transmitting the information to a server.

[1540] A "means for calculating the optimal route" is a means that, based on information about the starting point and destination, takes into account traffic conditions and user sentiment data to calculate the most efficient and user-friendly route.

[1541] "Means for receiving user input and generating departure point and destination information" refers to means that have the function of receiving departure point and destination information entered by the user through a terminal and processing it as digital information.

[1542] "A means of presenting calculated route information to the user and confirming whether or not to use expressways" refers to a means that displays route information calculated by the server on the user's terminal and has a function to confirm with the user whether or not to use expressways.

[1543] "Means for optimizing routes based on user responses and generating final route information" refers to means that have the function of recalculating routes based on user selections and responses to determine the most appropriate final route.

[1544] "Means for providing feedback of calculation results to the user" refers to means that have the function of transmitting the results of various processes performed by the server to the user's terminal and displaying them so that the user can confirm them.

[1545] "Means for obtaining in-vehicle sales menus and inventory information" refers to means that a server manages the in-vehicle sales menu and its inventory status, and provides this information to the user terminal.

[1546] "Means for generating and presenting new ticket information to the user" refers to means that a server has the function of creating new ticket information based on a user's request and displaying it on the user's terminal.

[1547] Modes for carrying out the invention

[1548] This invention combines a system that provides various services in autonomous vehicles and public transportation as virtual crew members with an emotion engine that recognizes user emotions, thereby achieving more advanced and user-friendly service provision. The specific system configuration and operation are described below.

[1549] System Configuration

[1550] This system consists of a user, a terminal, a server, and an emotion engine. When using autonomous vehicles or public transportation, the user operates a terminal such as a tablet or smartphone. The terminal receives user input, processes it, and sends it to the server. The emotion engine recognizes the user's emotions in real time and sends that information to the server. The server performs calculations and data processing based on the received departure point, destination information, and user emotion data, and sends the results to the terminal.

[1551] Setting the destination and route

[1552] First, the user specifies their starting point and destination using their device. For example, they might set the starting point to "Tokyo Station" and the destination to "Shinagawa Station" on a map application. This information is sent from the device to the server. The server calculates the optimal route based on the starting point and destination data. This calculation also takes into account sentiment data from the sentiment engine; for example, if it determines that the user is in a hurry, it will suggest a route that prioritizes highways. The calculation results include distance, travel time, and whether highways are available. This information is sent to the device and displayed to the user. The user confirms the route and resends the response information from the device to the server, which then determines the final route and sends that information back to the device.

[1553] Example prompt:

[1554] "My departure point is Tokyo Station, and my destination is Shinagawa Station. I'm in a hurry."

[1555] In-train sales

[1556] This section explains how a user requests in-train sales. The user operates the in-train sales application on their terminal and sends a request such as "I want to order a bento box." The terminal sends this request to the server. Based on the request, the server retrieves the in-train sales menu and inventory information and sends it to the terminal. The emotion engine recognizes the user's emotions and, for example, if it determines that the user is tired, it can suggest relaxing tea or other items. Once the user selects the desired items, the selection information is sent from the terminal to the server, and after the server checks the inventory, it sends a purchase confirmation message to the terminal. The terminal displays the purchase confirmation message to the user, and the order is finalized upon user confirmation.

[1557] Example prompt:

[1558] "I'd like to order a bento box. I'm very tired."

[1559] Ticket change

[1560] This section describes a scenario where a user requests a change to their train ticket. The user operates the ticket management application on their terminal and requests a "change of ticket." This request is sent from the terminal to the server. The server calculates the available options and suggests the best option based on emotional data. For example, if it is determined that the user wants to relax, it will suggest a quiet carriage. The user selects their desired change option, and this information is sent from the terminal to the server. The server generates new ticket information, sends it to the terminal, and displays it to the user.

[1561] Example prompt:

[1562] "I want to change my ticket. I want to relax."

[1563] As described above, the present invention is a system in which the user, terminal, server, and emotion engine work together to provide efficient and user-friendly services. By considering the user's emotions when providing various services, the quality of the user experience can be improved.

[1564] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1565] Step 1:

[1566] The user uses a device to specify the starting point and destination.

[1567] Specific operation: The user opens a map application and enters "Tokyo Station" as the starting point and "Shinagawa Station" as the destination. The terminal receives this input information and stores it as input data.

[1568] Input: Departure point and destination information

[1569] Output: Data on departure point and destination.

[1570] Step 2:

[1571] The terminal sends the entered information to the server.

[1572] Specific operation: The terminal converts the user's entered departure point and destination information into a digital format and sends it to the server.

[1573] Input: User-entered data for departure point and destination.

[1574] Output: Departure and destination data sent to the server

[1575] Step 3:

[1576] The emotion engine recognizes the user's emotions in real time and sends that information to the server.

[1577] Specific operation: The emotion engine analyzes the user's facial expressions, voice, and actions to recognize their emotional state, such as whether they are in a hurry or relaxed. The recognized emotional information is then sent to the server.

[1578] Input: Real-time user sentiment information

[1579] Output: Emotion data sent to the server

[1580] Step 4:

[1581] The server calculates the optimal route based on the departure point, destination information, and sentiment data it receives.

[1582] Specific operation: The server considers the user's sentiment data in addition to the origin and destination data, and uses an optimal route algorithm to perform calculations. For example, if the user is in a hurry, it will calculate a route that prioritizes highways.

[1583] Input: Origin, destination, and sentiment data

[1584] Output: Optimal route information (distance, travel time, availability of highways, etc.)

[1585] Step 5:

[1586] The device receives the calculated route information and presents it to the user.

[1587] Specific operation: The server sends the calculated optimal route information to the terminal. The terminal displays this information and prompts the user for confirmation. For example, the displayed route information might say "From Tokyo Station to Shinagawa Station, travel time 20 minutes, expressway available."

[1588] Input: Optimal route information from the server

[1589] Output: Route information presented to the user

[1590] Step 6:

[1591] The user checks route information on their device and responds regarding the availability of highway access.

[1592] Specific operation: The user checks the route information displayed on the device and selects whether to "use the highway" or "do not use the highway." The selected information is sent from the device to the server.

[1593] Input: User response information (whether or not highway use is permitted)

[1594] Output: Response information sent to the server

[1595] Step 7:

[1596] The server optimizes the route based on the user's response information, generates the final route information, and sends it to the terminal.

[1597] Specific operation: The server makes final adjustments to the route based on the user's response information and generates the final route information. The terminal receives this and presents the final route to the user.

[1598] Input: User response information

[1599] Output: Final route information

[1600] Step 8:

[1601] The user checks the in-car sales menu and selects an item.

[1602] Specific operation: The user checks the in-vehicle sales menu on their terminal and selects the desired product (e.g., "bento box"). The selection information is sent from the terminal to the server.

[1603] Input: User's selection information (product)

[1604] Output: Selection information sent to the server

[1605] Step 9:

[1606] The server checks the inventory of the selected product and sends a purchase confirmation message to the terminal.

[1607] Specific operation: The server checks if the product selected by the user is in stock, and if it is, it generates a purchase confirmation message and sends it to the terminal.

[1608] Input: Server selection information (product)

[1609] Output: Purchase confirmation message sent to the device

[1610] Step 10:

[1611] The device displays a purchase confirmation message to the user, who then confirms it.

[1612] Specific operation: The terminal displays a purchase confirmation message received from the server to the user. The order is confirmed when the user confirms it and presses the confirm button.

[1613] Input: Purchase confirmation message

[1614] Output: Confirmation message and confirmation information displayed to the user

[1615] Step 11:

[1616] The user requests a change to their ticket using the ticket management application.

[1617] Specific operation: The user requests a "ticket change" on their terminal, for example, by entering "I would like a quiet carriage." The terminal sends the request to the server.

[1618] Input: User's ticket change request

[1619] Output: Request sent to the server

[1620] Step 12:

[1621] The server calculates the available options, and the emotion engine suggests the optimal option.

[1622] Specific operation: The server calculates the modifiable options based on the input request. The emotion engine evaluates the user's emotional state and, for example, suggests a quiet vehicle in response to a request to relax.

[1623] Input: User change requests and sentiment data

[1624] Output: Optimal change options

[1625] Step 13:

[1626] The terminal receives the new ticket information and presents it to the user.

[1627] Specific operation: The server sends the calculated new ticket information to the terminal. The terminal displays this information to the user and prompts them to confirm.

[1628] Input: Ticket information from the server

[1629] Output: New ticket information displayed to the user

[1630] (Application Example 2)

[1631] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1632] Services in modern autonomous vehicles often provide a uniform service without considering the mood or emotions of passengers. This makes it difficult to meet the diverse needs of users, resulting in a diminished user experience. Furthermore, there is a lack of optimal suggestions that take user feelings into account, even when it comes to in-vehicle sales or ticket changes. This leads to the problem of services being provided without passengers feeling comfortable or satisfied.

[1633] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving user input and generating departure point and destination information, means for calculating the optimal route using the received departure point and destination information, means for presenting the calculated route information to the user and confirming whether or not highways can be used, means for optimizing the route based on the user's response and generating final route information, means for recognizing the user's emotions in real time and transmitting emotion data to the server, and means for deciding whether or not to use highways considering the user's emotion information. This makes it possible to provide individually optimal routes and services based on the user's emotions.

[1634] An "autonomous vehicle" is a vehicle that can drive automatically without human intervention, using artificial intelligence and sensor technology.

[1635] "User input" refers to instructions or information that a user provides to a system through a terminal or device.

[1636] The "starting point" refers to the location where the user begins using the autonomous vehicle.

[1637] "Destination information" refers to information about the location that the user designates as their destination.

[1638] The "optimal route" is the most efficient path from the starting point to the destination. Efficiency is calculated by considering factors such as time, distance, and traffic conditions.

[1639] An "emotion engine" is a system that analyzes a user's facial expressions, tone of voice, and other factors to recognize their emotions in real time.

[1640] "Emotional data" refers to information about a user's emotions as recognized by the emotion engine.

[1641] "Whether or not to use expressways" refers to the user's choice of whether or not to use expressways to reach their destination.

[1642] "In-vehicle sales" refers to food, beverages, and other goods offered inside autonomous vehicles.

[1643] "Inventory information" refers to information regarding the current stock status of products offered for sale on board the train.

[1644] "Ticket change" refers to changing the type or details of a ticket that a user has already obtained.

[1645] "Selected information" refers to the specific information that the user has chosen from the options presented to them.

[1646] This invention is a system for making passenger services in autonomous vehicles more advanced and user-friendly. This system is composed of a user, a terminal, a server, and an emotion engine, which recognize the user's emotions in real time and provide optimal services in response to those emotions.

[1647] System Configuration

[1648] User

[1649] The users are passengers using the autonomous vehicle and operate the system through devices such as tablets and smartphones.

[1650] terminal

[1651] The terminal is a device that receives user instructions and collects data for emotion recognition. The terminal uses its camera and microphone to transmit the user's facial expressions and voice to the emotion engine.

[1652] server

[1653] A server primarily has three roles:

[1654] 1. Calculate the optimal route based on user input, originating point, and destination information.

[1655] 2. Receive requests for in-vehicle sales and manage product inventory information.

[1656] 3. Accept ticket changes and provide changeable options.

[1657] Emotional Engine

[1658] The emotion engine analyzes emotional data in real time from the user's facial expressions and voice, and sends that data to the server. Examples of technologies used include Amazon Rekognition and Microsoft Azure Emotion API.

[1659] Destination setting

[1660] The user first enters their starting point and destination into the terminal. For example, if the user sets the starting point to "Tokyo Station" and the destination to "Shinagawa Station," this information is sent from the terminal to the server. The server calculates the optimal route and presents the user with route information, including whether or not highways can be used. Furthermore, the emotion engine analyzes whether the user is in a hurry and may recommend using highways.

[1661] In-train sales

[1662] When a user wants to use the in-train sales service, they request to "order a bento box" through the application on their terminal. This request is sent to the server, which then suggests products based on inventory information. If the emotion engine determines that the user is tired, it also suggests a relaxing tea.

[1663] Ticket change

[1664] When a user requests a change to their ticket, that information is sent from the terminal to the server. The server also takes the user's emotional state into consideration; for example, if the user wants to relax, it might suggest a quieter carriage.

[1665] Specific examples and prompt statements

[1666] Specific examples of route acquisition:

[1667] Departure point: "Tokyo Station"

[1668] Destination: Shinagawa Station

[1669] Emotion: "When the user is in a hurry"

[1670] Result: "Proposal for the fastest route using expressways"

[1671] Please calculate the route based on the following information:

[1672] Departure point: Tokyo Station

[1673] Destination: Shinagawa Station

[1674] User's emotion: In a hurry

[1675] Examples of in-train sales:

[1676] Order: "Bento"

[1677] Emotion: "When the user is tired"

[1678] Result: "Suggestion for tea with relaxing effects"

[1679] Please order the following items:

[1680] Order: Bento

[1681] User's emotion: tired

[1682] Additional suggestion: Tea with relaxing effects

[1683] Specific examples of ticket changes:

[1684] Change option: "Quiet vehicle"

[1685] Emotion: "When the user wants to relax"

[1686] Result: "Propose a quiet vehicle."

[1687] Please change your ticket using one of the following options:

[1688] Change options: Quiet vehicle

[1689] User's emotion: Wants to relax

[1690] As described above, the system of the present invention can provide individually optimized routes and services while taking into account the user's emotions. This is expected to significantly improve passenger comfort and satisfaction.

[1691] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1692] Step 1:

[1693] The user enters their departure point and destination information into the device. For example, they might enter "Tokyo Station" as the departure point and "Shinagawa Station" as the destination. This input information is retrieved by the map application on the device.

[1694] Step 2:

[1695] The terminal sends the entered departure and destination information to the server. At this time, the emotion engine analyzes the user's facial expressions and voice, and emotion data is generated. This emotion data is also sent to the server.

[1696] Step 3:

[1697] The server receives input information and sentiment data. It calculates the optimal route based on the departure and destination information, and further optimizes route selection by considering sentiment data. The decision of whether or not to use highways is also made at this stage.

[1698] Step 4:

[1699] The server sends the calculated optimal route information and a suggestion regarding the availability of highways to the terminal. The terminal displays this information to the user and receives the user's response again.

[1700] Step 5:

[1701] The user enters response information into the terminal in response to the presented route information and the suggestion regarding the use of expressways. For example, they might select "Use expressways."

[1702] Step 6:

[1703] The terminal sends the user's response information back to the server. The server optimizes the final route based on the response information and generates the final route information.

[1704] Step 7:

[1705] The server sends the final route information to the terminal, which then displays it to the user. At this stage, the user can verify the optimized route information.

[1706] Step 8:

[1707] If a user wants to use the in-train sales service, they enter their order information through the application on their terminal. For example, they might enter "I want to order a bento box." This information is then sent from the terminal to the server.

[1708] Step 9:

[1709] The server retrieves in-car sales menus and inventory information based on the received order information, and also takes into account the user's emotional state to make appropriate product suggestions. For example, if the user is tired, it will suggest a relaxing tea.

[1710] Step 10:

[1711] The server sends the suggested menu information to the terminal, which then displays it to the user. The user selects an item from the presented menu and enters the selection information into the terminal.

[1712] Step 11:

[1713] The terminal sends the selection information to the server. The server checks the inventory of the selected items, generates a purchase confirmation message, and sends it to the terminal. The terminal displays this message to the user, who then confirms it.

[1714] Step 12:

[1715] If a user wishes to change their ticket, they enter a change request through the ticket management application on their terminal. For example, they might enter, "I would like to change to a quieter carriage." This request is then sent from the terminal to the server.

[1716] Step 13:

[1717] The server generates information on modifiable options based on the received change request. Sentimental data is also taken into consideration, and options suitable for, for example, a user who wants to relax are presented.

[1718] Step 14:

[1719] The server sends information about the configurable options to the terminal, which then displays this information to the user. The user selects the desired options and enters that information into the terminal.

[1720] Step 15:

[1721] The terminal sends the selection information to the server. The server generates new ticket information and sends that information to the terminal. The terminal displays the new ticket information to the user, allowing the user to confirm the details of the new ticket.

[1722] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1723] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1724] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1725] [Fourth Embodiment]

[1726] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1727] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1728] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1729] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1730] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1731] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1732] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1733] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1734] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1735] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1736] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1737] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1738] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1739] This invention is a system that provides various services in autonomous vehicles and public transportation by having virtual crew members. The specific system configuration and operation are described below.

[1740] System Configuration

[1741] This system consists of users, terminals, and servers. Users operate terminals such as tablets and smartphones to use autonomous vehicles and public transportation. The terminal receives user input, processes it, and sends it to the server. The server performs calculations and data processing based on the received information and sends the results to the terminal.

[1742] Setting the destination and route

[1743] First, the user specifies their starting point and destination. For example, the user sets the starting point to "Tokyo Station" and the destination to "Shinagawa Station" on the map application on their device. This information is sent from the device to the server. The server calculates the optimal route based on the received starting point and destination information. The calculated route information (distance, travel time, whether or not highways can be used, etc.) is sent to the device and displayed to the user. The user then selects whether or not to use highways. For example, if the user selects "Use highways," this information is sent from the device to the server. The server optimizes the route based on this selection, generates the final route information, and sends it to the device. The device then displays the final route on its navigation system.

[1744] In-train sales

[1745] Next, let's consider the case where a user requests an in-train purchase. The user operates the in-train sales application on their terminal and requests, "I want to order a bento box." This request is sent from the terminal to the server. The server retrieves the in-train sales menu and inventory information based on the specified request and sends it to the terminal. The terminal displays the retrieved menu information to the user. The user selects the desired product (for example, "sushi bento"). This selection information is sent from the terminal to the server. The server checks the inventory of the selected product and sends a purchase confirmation message to the terminal. The terminal displays this purchase confirmation message to the user. Once the user confirms, the ordering process is complete.

[1746] Ticket change

[1747] Finally, consider the case where a user requests a change to their ticket. The user operates the ticket management application on their terminal and requests a "change ticket." This request is sent from the terminal to the server. The server calculates the available change options based on the request and sends that information to the terminal. The terminal presents the user with the available change options (e.g., change time, change seat). The user selects the desired change option. This selection information is sent from the terminal to the server. The server generates new ticket information based on the selection and sends it to the terminal. The terminal displays the new ticket information to the user.

[1748] As described above, the present invention is a system in which users, terminals, and servers cooperate to provide efficient and user-friendly services.

[1749] The following describes the processing flow.

[1750] Setting the destination and route

[1751] Step 1:

[1752] The user operates the terminal and enters the departure point and destination. For example, the user enters "Tokyo Station" as the departure point and "Shinagawa Station" as the destination.

[1753] Step 2:

[1754] The terminal receives user input information and sends it to the server. Specifically, it sends data on the departure point and destination to the server.

[1755] Step 3:

[1756] The server calculates the optimal route based on the information it receives. This calculation uses external map APIs or the company's own database.

[1757] Step 4:

[1758] The server sends calculated route information to the terminal, including distance, estimated travel time, and whether highways can be used.

[1759] Step 5:

[1760] The terminal displays the route information it has received to the user. The user can then choose whether or not to use the highway.

[1761] Step 6:

[1762] The user selects whether or not to use the highway, and the terminal sends this selection information to the server.

[1763] Step 7:

[1764] The server recalculates the route based on the user's selection information and generates the final route information.

[1765] Step 8:

[1766] The server sends the final route information to the terminal. The terminal displays the final route on its navigation system.

[1767] In-train sales

[1768] Step 1:

[1769] The user operates the terminal and selects a request for in-car sales. For example, they might select the "Order a bento box" option.

[1770] Step 2:

[1771] The terminal sends the user's request to the server. Specifically, this includes the user ID and the request details.

[1772] Step 3:

[1773] The server retrieves the in-vehicle sales menu and inventory information. For example, it reads the latest menu and inventory numbers from the database.

[1774] Step 4:

[1775] The server sends the retrieved menu information to the terminal. This includes the product name, price, and stock quantity.

[1776] Step 5:

[1777] The terminal displays menu information to the user. The user selects the desired product.

[1778] Step 6:

[1779] The user selects an item, and the device sends that selection information to the server. Specifically, this includes the item ID and quantity.

[1780] Step 7:

[1781] The server checks the inventory of the selected product. If it is in stock, it generates a purchase confirmation message and sends it to the terminal.

[1782] Step 8:

[1783] The device displays a purchase confirmation message to the user. The order is finalized once the user confirms it.

[1784] Ticket change

[1785] Step 1:

[1786] The user operates the terminal and selects a request to change their ticket. For example, they might select the "Change boarding time" menu.

[1787] Step 2:

[1788] The terminal sends the user's request to the server. Specifically, this includes current ticket information and conditions for change.

[1789] Step 3:

[1790] The server calculates the options that can be changed. For example, it searches the database for available trains and seats.

[1791] Step 4:

[1792] The server sends the calculation results to the terminal. This includes information about trains and seats that can be changed.

[1793] Step 5:

[1794] The device displays change options to the user. The user selects the desired change option.

[1795] Step 6:

[1796] The user selects a change option, and the device sends that selection information to the server. Specifically, this includes the changed train ID and seat number.

[1797] Step 7:

[1798] The server generates new ticket information, which includes new train and seat information.

[1799] Step 8:

[1800] The server sends the new ticket information to the terminal. The terminal displays the new ticket information to the user.

[1801] (Example 1)

[1802] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1803] To improve user convenience and comfort in autonomous vehicles and public transportation, it is necessary to efficiently provide a wide range of services, such as destination setting, in-vehicle sales, and ticket changes. However, in conventional systems, these services are often fragmented, making it difficult to provide a unified experience. Furthermore, advanced data processing technology is required to process user input information quickly and accurately and provide users with the necessary information.

[1804] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1805] In this invention, the server includes means for receiving user input and generating departure point and destination information; means for calculating an optimal route using the received departure point and destination information; means for presenting the calculated route information to the user and confirming the availability of highways; means for optimizing the route based on the user's response and generating final route information; means for displaying the final route information on a navigation system; means for receiving the user's in-vehicle sales request and transmitting it to the server; means for acquiring in-vehicle sales menus and inventory information; means for presenting the acquired menu information to the user; means for receiving the user's selection information and transmitting it to the server; means for confirming the inventory of selected items and presenting a purchase confirmation message to the user; means for receiving user confirmation and completing the order process; means for receiving the user's ticket change request and transmitting it to the server; means for presenting the user with changeable option information calculated by the server; means for receiving the user's selection information and transmitting it to the server; means for generating new ticket information and presenting it to the user; and means for displaying the new ticket information on a navigation system. This enables the provision of a consistent user experience and the efficient execution of various services.

[1806] "User" refers to a person who operates this system in order to use autonomous vehicles or public transportation.

[1807] A "device" is an electronic device used by a user, and includes tablets, smartphones, and other similar devices.

[1808] A "server" refers to a central system that receives information from users, performs data processing and calculations, and sends the results to terminals.

[1809] The "starting point" refers to the initial location from which the user boards the vehicle, and is one of the location information settings in a map application.

[1810] "Destination information" refers to information about the location that the user sets as their destination.

[1811] "Optimal route" refers to the shortest or most efficient route information calculated by the server based on the origin and destination information received.

[1812] A "navigation system" is a system that displays optimal route information and final route information to guide the user to their destination.

[1813] An "in-car sales request" refers to a request sent from a terminal to a server when a user wishes to purchase goods while in a vehicle.

[1814] The "in-train sales menu" refers to a list of products offered on board the train.

[1815] "Inventory information" refers to information showing the availability of each product included in the in-train sales menu.

[1816] "Selected information" refers to information about products or services that the user has selected on their device.

[1817] A "purchase confirmation message" refers to a confirmation message sent by the server to the user to complete the purchase process for the selected product.

[1818] A "ticket change request" refers to a request sent from a user's device to a server when the user wishes to change the details of a ticket they have already purchased.

[1819] "Changeable option information" refers to the ticket modification options that the server calculates and that the user can select.

[1820] "New ticket information" refers to information about a new ticket generated based on the user's selection.

[1821] This invention relates to a virtual crew system that provides a unified range of services in autonomous vehicles and public transportation. The following describes embodiments of this system.

[1822] System Configuration

[1823] This system consists of users, terminals, and servers. Users operate terminals such as tablets and smartphones to use autonomous vehicles and public transportation. The terminal receives user input, processes it, and sends it to the server. The server performs calculations and data processing based on the received information and sends the results to the terminal.

[1824] Setting the destination and route

[1825] First, the user specifies their pick-up and destination. For example, the user uses their device's map application to set the departure point to "Tokyo Station" and the destination to "Shinagawa Station." This information is sent from the device to the server. The server uses the Google Maps API to calculate the optimal route based on the received departure and destination information. The calculated route information (distance, travel time, whether highways can be used, etc.) is sent to the device and displayed to the user. The user then chooses whether or not to use highways. For example, if they select "Use highways," this information is sent from the device to the server. The server optimizes the route based on this selection, generates the final route information, and sends it to the device. The device then displays the final route on its navigation system.

[1826] Specific example:

[1827] Prompt message:

[1828] "Please set the departure point to Tokyo Station and the destination to Shinagawa Station."

[1829] In-train sales

[1830] Next, let's consider the case where a user requests an in-train purchase. The user uses the in-train sales application on their terminal to request, "I want to order a bento box." This request is sent from the terminal to the server. The server retrieves the in-train sales menu and inventory information based on the specified request and sends it to the terminal. The terminal displays the retrieved menu information to the user. The user selects the desired item (for example, "sushi bento"). This selection information is sent from the terminal to the server. The server checks the inventory of the selected item and sends a purchase confirmation message to the terminal. The terminal displays this purchase confirmation message to the user. Once the user confirms, the ordering process is complete.

[1831] Specific example:

[1832] Prompt message:

[1833] "Please select and order your bento box."

[1834] Ticket change

[1835] Finally, consider the case where a user requests a change to their ticket. The user requests a "change ticket" by operating the ticket management application on their terminal. This request is sent from the terminal to the server. The server calculates the available change options based on the request and sends that information to the terminal. The server uses its own backend system and reservation management system. It presents the user with the available change options (e.g., change time, change seat). The user selects the desired change option and sends the selection information from the terminal to the server. The server generates new ticket information based on the selection and sends it to the terminal. The terminal displays the new ticket information to the user.

[1836] Specific example:

[1837] Prompt message:

[1838] "Please view and select the ticket change options."

[1839] As described above, this system enables efficient and user-friendly services through the collaboration of users, terminals, and servers. By utilizing data processing technologies, including the Google Maps API, this system makes it possible to provide advanced services.

[1840] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1841] Setting the destination and route

[1842] Step 1:

[1843] The user operates their device (tablet or smartphone) and opens a map application. The user sets the starting point to "Tokyo Station" and the destination to "Shinagawa Station," and then presses the send button.

[1844] Input: Departure point and destination information (Tokyo Station, Shinagawa Station)

[1845] Specific operation: The user specifies the starting point and destination by dragging and dropping pins on the map using their device.

[1846] Output: The departure point and destination information are set on the terminal.

[1847] Step 2:

[1848] The device sends its departure point and destination information to the server.

[1849] Input: Departure point and destination information set by the user.

[1850] Specific operation: The app on the device creates an HTTP request and sends it to the server.

[1851] Output: The server receives the departure and destination information.

[1852] Step 3:

[1853] The server calculates the optimal route based on the information it receives. The server uses the Google Maps API to obtain the optimal route (distance, travel time, whether highways can be used, etc.).

[1854] Input: Departure point and destination information

[1855] Specific operation: The server accesses the Google Maps API route search endpoint and sends the starting point and destination parameters.

[1856] Output: Optimal route information is generated on the server.

[1857] Step 4:

[1858] The server sends the calculated optimal route information to the terminal.

[1859] Input: Optimal route information

[1860] Specific operation: The server sends data in JSON format to the terminal.

[1861] Output: The terminal receives optimal route information.

[1862] Step 5:

[1863] The terminal displays the route information it has received to the user. The user can choose whether or not to use highways.

[1864] Input: Optimal route information

[1865] Specific action: The app on the device displays route information on the map and shows a checkbox indicating whether to "use highways."

[1866] Output: The user's highway usage selection information is determined.

[1867] Step 6:

[1868] The user selects the option to use the highway and presses the submit button. The terminal sends the selection information to the server.

[1869] Input: User's highway usage selection information

[1870] Specific operation: The user checks / unchecks a checkbox within the app on their device and presses the submit button. The device sends the selection information to the server via an HTTP request.

[1871] Output: The server receives the selection information.

[1872] Step 7:

[1873] The server recalculates the route based on the selected information and generates the final route information. It then sends the final route information to the terminal.

[1874] Input: User's highway usage selection information

[1875] Specific operation: The server uses the selected information to re-execute the optimal route calculation algorithm and generate the final route information.

[1876] Output: The final route information is generated on the server and sent to the terminal.

[1877] Step 8:

[1878] The device displays the final route information on the navigation system.

[1879] Input: Final route information

[1880] Specific operation: The device's navigation app receives the final route information and displays it on the map.

[1881] Output: The user can check the final route information.

[1882] In-train sales

[1883] Step 1:

[1884] The user uses the in-car sales application on their terminal to request, "I would like to order a bento box."

[1885] Input: User's in-car sales request

[1886] Specific action: The user selects "In-car sales" from the app's menu and presses the "Order a bento box" button.

[1887] Output: An in-car sales request is set on the terminal.

[1888] Step 2:

[1889] The terminal sends the user's in-car sales request to the server.

[1890] Input: User's in-car sales request

[1891] Specific action: The terminal creates an HTTP request and sends it to the server.

[1892] Output: The server receives an in-vehicle sales request.

[1893] Step 3:

[1894] The server retrieves the in-car sales menu and inventory information based on the request and sends it to the terminal.

[1895] Input: In-car sales request

[1896] Specific operation: The server retrieves the product list and inventory information from the database and sends it to the terminal.

[1897] Output: The in-car sales menu and inventory information are sent to the terminal.

[1898] Step 4:

[1899] The terminal displays the menu information it has acquired to the user, and the user selects the product they want (for example, "sushi bento").

[1900] Input: In-car sales menu and inventory information

[1901] Specific action: The app on the device displays a menu list, and the user taps on "Sushi Bento".

[1902] Output: User selection information is set on the terminal.

[1903] Step 5:

[1904] The device sends the user's selection information to the server.

[1905] Input: User's selected information

[1906] Specific action: The terminal creates an HTTP request containing the selection information and sends it to the server.

[1907] Output: The server receives the user's selection information.

[1908] Step 6:

[1909] The server checks the inventory of the selected product, generates a purchase confirmation message, and sends it to the terminal.

[1910] Input: User's selected information

[1911] Specific operation: The server retrieves the inventory quantity of the product from the database, and if it is in stock, it generates a purchase confirmation message.

[1912] Output: A purchase confirmation message is generated on the server and sent to the device.

[1913] Step 7:

[1914] The device displays a purchase confirmation message to the user, and once the user confirms it, the order process is completed.

[1915] Input: Purchase confirmation message

[1916] Specific action: The user presses the confirmation button, and the device sends the order information to the server.

[1917] Output: The order has been confirmed and the shipping process has begun.

[1918] Ticket change

[1919] Step 1:

[1920] The user requests a "ticket change" using the ticket management application on their device.

[1921] Input: User's ticket change request

[1922] Specific action: The user presses the "Change Ticket" button on the ticket management screen in the app.

[1923] Output: A ticket change request is set on the terminal.

[1924] Step 2:

[1925] The device sends a user change request to the server.

[1926] Input: User's ticket change request

[1927] Specific action: The terminal creates an HTTP request and sends it to the server.

[1928] Output: The server receives a ticket change request.

[1929] Step 3:

[1930] The server calculates the modifiable options based on the request and sends that information to the terminal.

[1931] Input: Ticket change request

[1932] Specific operation: The server retrieves the current reservation information from the database and calculates the options that can be changed.

[1933] Output: Modifiable option information is generated on the server and sent to the terminal.

[1934] Step 4:

[1935] The device presents the user with available modification options, and the user selects the desired modification option.

[1936] Input: Changeable option information

[1937] Specific operation: The app on the device displays change options, and the user selects one.

[1938] Output: User selection information is set on the terminal.

[1939] Step 5:

[1940] The device sends the user's selection information to the server.

[1941] Input: User's selected information

[1942] Specific action: The terminal creates an HTTP request containing the selection information and sends it to the server.

[1943] Output: The server receives the user's selection information.

[1944] Step 6:

[1945] The server generates new ticket information based on the selections and sends it to the terminal.

[1946] Input: User's selected information

[1947] Specific operation: The server generates new ticket information and sends it to the terminal.

[1948] Output: New ticket information is generated on the server and sent to the terminal.

[1949] Step 7:

[1950] The terminal displays the new ticket information to the user.

[1951] Input: New ticket information

[1952] Specific action: The app on the device displays the new ticket information, and the user confirms it.

[1953] Output: The user can view the new ticket information.

[1954] The above steps enable a system where users, terminals, and servers work together to provide efficient and user-friendly services. Including specific actions in each processing step makes the program flow clearer.

[1955] (Application Example 1)

[1956] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1957] The present invention aims to provide a method for providing food delivery services more seamlessly and efficiently. Specifically, the objective is to provide a system that allows users to order food from within an autonomous vehicle, track the delivery status in real time, and automatically generate appropriate responses to user inquiries and requests.

[1958] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1959] In this invention, the server includes means for receiving user input and generating departure point and destination information; means for calculating an optimal route using the received departure point and destination information; means for presenting the calculated route information to the user and confirming the availability of highways; means for optimizing the route based on the user's response and generating final route information; means for receiving user order information and providing food delivery services; means for providing the user with real-time delivery status and estimated arrival time; and means for analyzing user inquiries and providing appropriate answers using a generative AI model. This enables a seamless experience and rapid response when users utilize food delivery services in an autonomous vehicle.

[1960] "User input" refers to the information and instructions that a user provides through their device.

[1961] "Departure point" refers to the location where the user boards or the starting point.

[1962] "Destination information" refers to information about the place or destination that the user intends to go.

[1963] The "optimal route" refers to the most suitable travel route calculated considering the efficiency and convenience of the path.

[1964] "Availability of using expressways" refers to the user's choice of whether or not to use expressways.

[1965] "Final route information" refers to the final guidance information based on an optimized route.

[1966] "Order information" refers to the detailed information that users enter when purchasing food or products.

[1967] A "food delivery service" refers to a service that delivers food to a location specified by the user.

[1968] "Delivery status" refers to the current progress and status of the food or product delivery process.

[1969] "Estimated arrival time" refers to the estimated time it will take for a delivery to reach its destination.

[1970] "Inquiries" refer to information related to questions and requests from users.

[1971] A "generative AI model" refers to a model that uses artificial intelligence technology to automatically create answers and information in response to user inquiries.

[1972] An "appropriate response" refers to a response that provides accurate and relevant information in response to a user's inquiry or request.

[1973] This invention is a system that supports user operation within an autonomous vehicle in order to efficiently provide food delivery services. This system consists of a user, a terminal, and a server. The specific operation of each component is described below.

[1974] System Configuration

[1975] This system allows users to operate it through a device, which then interacts with a server to provide services. The device can be a smartphone or tablet, and the server is a cloud server (such as AWS or Azure).

[1976] Setting the destination and route

[1977] First, the user enters their starting point and destination information into the terminal. Using a map application on the terminal, the user specifies the starting point and destination. This information is sent from the terminal to the server, which calculates the optimal route. The calculated route information is presented to the user, and the user selects whether or not to use highways. Based on this information, the server generates an optimized final route and sends it to the terminal.

[1978] Food delivery service

[1979] When a user orders food from their device, the order information is sent to a server. The server provides the food delivery service and gives the user real-time delivery status and estimated arrival time. Delivery status is tracked using GPS, and the estimated arrival time is calculated dynamically.

[1980] Customer Support

[1981] User inquiries are sent directly from the terminal to the server. The server analyzes the inquiry using a generative AI model (e.g., GPT-4) and generates an appropriate response. The generated response is sent to the terminal and displayed to the user.

[1982] Hardware and software to be used

[1983] Hardware: Smartphones, tablets, cloud servers (AWS, Azure)

[1984] Software: iOS or Android native app development tools (Swift, Kotlin, etc.), server-side frameworks (Node.js, Flask, etc.), SQL databases (MySQL, PostgreSQL, etc.), generative AI models (GPT-4)

[1985] Specific example

[1986] This describes a scenario where a user orders a pizza from inside an autonomous vehicle and monitors the delivery status in real time. When the user operates a terminal and orders a pizza, the order information is sent to a server. The server processes the order information, and a delivery person departs with the pizza. The delivery status is tracked via GPS, the estimated arrival time is calculated, and this information is provided to the user in real time. Furthermore, if the user makes a question, a generative AI model generates an appropriate answer, which is then displayed to the user.

[1987] Example of a prompt:

[1988] "A user has ordered a pizza. Please provide the delivery person's current location in real time and the estimated arrival time."

[1989] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1990] Step 1:

[1991] The user operates the terminal and enters departure and destination information. The entered information is temporarily stored within the terminal. The terminal then sends the entered departure and destination information as data to the server.

[1992] Input: Departure point and destination information entered by the user on the device.

[1993] Output: Departure point and destination information sent to the server

[1994] Step 2:

[1995] The server calculates the optimal route based on the received origin and destination information. It uses a route calculation algorithm (e.g., Dijkstra's algorithm) to identify the most efficient route. The calculated route information is stored internally by the server.

[1996] Input: Departure point and destination information

[1997] Output: Calculated optimal route information

[1998] Step 3:

[1999] The server sends the calculated route information to the terminal, which then presents the information to the user. The user reviews the route information and selects whether or not to use highways. The selection information is then sent back to the server from the terminal.

[2000] Input: Optimal route information

[2001] Output: User selection information (whether or not highways can be used)

[2002] Step 4:

[2003] The server generates an optimized final route based on the user's selection information. It recalculates the route and generates new route information. The final route information is sent from the server to the terminal and presented to the user.

[2004] Input: User's selected information

[2005] Output: Final route information

[2006] Step 5:

[2007] The user orders food from their device. The device sends the order information to the server. The order information includes the type and quantity of food, and the delivery address.

[2008] Input: User's order information

[2009] Output: Order information sent to the server

[2010] Step 6:

[2011] The server receives and processes the order information. This includes checking order availability and arranging delivery. The server then generates an order confirmation message and sends it to the terminal.

[2012] Input: User's order information

[2013] Output: Order confirmation message

[2014] Step 7:

[2015] The server tracks the progress of food deliveries in real time. It uses GPS to obtain the delivery person's location and calculates the estimated arrival time. This information is processed within the server.

[2016] Input: Delivery driver's location information

[2017] Output: Estimated arrival time and delivery status

[2018] Step 8:

[2019] The server sends the delivery status and estimated arrival time to the terminal. The terminal displays this information to the user, allowing them to check the delivery status in real time.

[2020] Input: Estimated arrival time and delivery status

[2021] Output: Delivery status and estimated arrival time presented to the user.

[2022] Step 9:

[2023] When a user submits an inquiry, the device sends the inquiry as text data to the server. The server uses a generative AI model to analyze the inquiry and generate an appropriate response.

[2024] Input: Inquiry details

[2025] Output: Generated answer

[2026] Step 10:

[2027] The server sends the response generated by the AI ​​model to the terminal. The terminal displays the response to the user, and appropriate action is taken.

[2028] Input: Generated answer

[2029] Output: The answer presented to the user

[2030] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[2031] This invention combines a system that provides various services in autonomous vehicles and public transportation as virtual crew members with an emotion engine that recognizes user emotions, thereby achieving more advanced and user-friendly service provision. The specific system configuration and operation are described below.

[2032] System Configuration

[2033] This system consists of a user, a terminal, a server, and an emotion engine. The user operates a terminal such as a tablet or smartphone to use autonomous vehicles or public transportation. The terminal receives user input, processes it, and sends it to the server. The emotion engine recognizes the user's emotions in real time and sends that information to the server. The server performs calculations and data processing based on the received information and sends the results to the terminal.

[2034] Setting the destination and route

[2035] First, the user specifies their pick-up and destination. For example, the user sets the departure point to "Tokyo Station" and the destination to "Shinagawa Station" on their device's map application. This information is sent from the device to the server. The server calculates the optimal route based on the received departure and destination information. This calculation also takes into account sentiment data from the sentiment engine. The calculated route information (distance, travel time, whether or not highways can be used, etc.) is sent to the device and displayed to the user. Based on the user's sentiment, the choice of whether or not to use highways is optimized. For example, if the sentiment engine determines that the user is in a hurry, it will recommend using highways. The user's response information is sent back to the server, and the final route is determined.

[2036] In-train sales

[2037] Next, let's consider the case where a user requests an in-train purchase. The user operates the in-train sales application on their terminal and requests, "I want to order a bento box." This request is sent from the terminal to the server. Based on the specified request, the server retrieves the in-train sales menu and inventory information and sends it to the terminal. The emotion engine can also recognize the user's emotions and suggest products that match the user's mood. For example, if it determines that the user is tired, it will suggest a relaxing tea. The user selects the product they want, and this selection information is sent from the terminal to the server. The server checks the inventory of the selected product and sends a purchase confirmation message to the terminal. The terminal displays this purchase confirmation message to the user, and once the user confirms it, the order is finalized.

[2038] Ticket change

[2039] Finally, consider the case where a user requests a change to their ticket. The user operates the ticket management application on their terminal and requests a "change ticket." This request is sent from the terminal to the server. The server calculates the available options based on the request and sends that information to the terminal. The emotion engine analyzes the user's emotions and can suggest the most suitable change option based on the user's mood. For example, if it is determined that the user wants to relax, it will suggest a quiet carriage. The user selects their desired change option, and this selection information is sent from the terminal to the server. The server generates new ticket information based on the selection and sends it to the terminal. The terminal displays the new ticket information to the user.

[2040] As described above, the present invention is a system in which a user, terminal, server, and emotion engine work together to provide efficient and user-friendly services.

[2041] The following describes the processing flow.

[2042] Setting the destination and route

[2043] Step 1:

[2044] The user operates the device and enters the starting point and destination. For example, going from "Tokyo Station" to "Shinagawa Station".

[2045] Step 2:

[2046] The terminal receives user input information and sends it to the server. Specifically, data on the departure point and destination is sent to the server.

[2047] Step 3:

[2048] The server calculates the optimal route based on the information it receives. This calculation uses external map APIs and internal route calculation algorithms.

[2049] Step 4:

[2050] The server sends calculated route information to the terminal. This includes distance, estimated travel time, and whether highways can be used.

[2051] Step 5:

[2052] The terminal displays the route information it has received to the user. The user can then choose whether or not to use the highway.

[2053] Step 6:

[2054] The emotion engine analyzes the user's facial expressions and tone of voice to determine their emotional state. For example, if it determines that the user is anxious, it will recommend using the highway.

[2055] Step 7:

[2056] The user selects whether or not to use the highway, and the terminal sends this selection information to the server.

[2057] Step 8:

[2058] The server recalculates the route based on the user's selection information and sentiment data, and generates the final route information.

[2059] Step 9:

[2060] The server sends the final route information to the terminal. The terminal displays the final route on its navigation system.

[2061] In-train sales

[2062] Step 1:

[2063] The user operates the terminal and selects a request for in-car sales. For example, "Order a bento box."

[2064] Step 2:

[2065] The terminal sends the user's request to the server. Specifically, this includes the user ID and the request details.

[2066] Step 3:

[2067] The server retrieves the in-vehicle sales menu and inventory information. For example, it reads the latest menu and inventory numbers from the database.

[2068] Step 4:

[2069] The server sends the retrieved menu information to the terminal. This includes the product name, price, and stock quantity.

[2070] Step 5:

[2071] The terminal displays menu information to the user. The user selects the desired product.

[2072] Step 6:

[2073] The emotion engine analyzes the user's emotions and suggests products that match their mood. For example, it might suggest a relaxing tea to a tired user.

[2074] Step 7:

[2075] The user selects an item, and the device sends that selection information to the server. Specifically, this includes the item ID and quantity.

[2076] Step 8:

[2077] The server checks the inventory of the selected product. If it is in stock, it generates a purchase confirmation message and sends it to the terminal.

[2078] Step 9:

[2079] The device displays a purchase confirmation message to the user. Once the user confirms, the order is finalized.

[2080] Ticket change

[2081] Step 1:

[2082] The user operates the terminal and selects a request to change their ticket. For example, "Change departure time."

[2083] Step 2:

[2084] The terminal sends the user's request to the server. Specifically, this includes current ticket information and conditions for change.

[2085] Step 3:

[2086] The server calculates the options that can be changed. For example, it searches for available trains and seats.

[2087] Step 4:

[2088] The server sends the calculation results to the terminal. This includes information about trains and seats that can be changed.

[2089] Step 5:

[2090] The device displays change options to the user. The user selects the desired change option.

[2091] Step 6:

[2092] The emotion engine analyzes the user's emotions and suggests the optimal change options based on the user's mood. For example, it suggests a quiet vehicle to a user who wants to relax.

[2093] Step 7:

[2094] The user selects a change option, and the device sends that selection information to the server. Specifically, this includes the changed train ID and seat number.

[2095] Step 8:

[2096] The server generates new ticket information, which includes new train and seat information.

[2097] Step 9:

[2098] The server sends the new ticket information to the terminal. The terminal displays the new ticket information to the user.

[2099] (Example 2)

[2100] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2101] In providing services in autonomous vehicles and public transportation, traditional methods failed to consider user emotions, making it difficult to enhance user satisfaction. Furthermore, it was challenging to suggest optimal services and routes when users were in a hurry or experiencing specific emotional states. This resulted in a decline in service quality and a compromised user experience.

[2102] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[2103] In this invention, the server includes means for receiving user input and generating departure point and destination information; means for calculating the optimal route using the received departure point and destination information; means for presenting the calculated route information to the user and confirming the availability of expressways; means for optimizing the route based on the user's response and generating final route information; means for recognizing the user's emotions in real time and transmitting that information to the server; means for optimizing the calculation results based on the information received from the emotion recognition means; and means for feeding the calculation results back to the user. This makes it possible to provide the optimal route and services based on the user's emotions, thereby improving the quality of the user experience.

[2104] "An emotion recognition method that recognizes user emotions in real time" refers to a means that analyzes the user's facial expressions, voice, behavioral patterns, etc., through a terminal operated by the user, and has the function of identifying the emotional state the user is currently feeling in real time and transmitting the information to a server.

[2105] A "means for calculating the optimal route" is a means that, based on information about the starting point and destination, takes into account traffic conditions and user sentiment data to calculate the most efficient and user-friendly route.

[2106] "Means for receiving user input and generating departure point and destination information" refers to means that have the function of receiving departure point and destination information entered by the user through a terminal and processing it as digital information.

[2107] "A means of presenting calculated route information to the user and confirming whether or not to use expressways" refers to a means that displays route information calculated by the server on the user's terminal and has a function to confirm with the user whether or not to use expressways.

[2108] "Means for optimizing routes based on user responses and generating final route information" refers to means that have the function of recalculating routes based on user selections and responses to determine the most appropriate final route.

[2109] "Means for providing feedback of calculation results to the user" refers to means that have the function of transmitting the results of various processes performed by the server to the user's terminal and displaying them so that the user can confirm them.

[2110] "Means for obtaining in-vehicle sales menus and inventory information" refers to means that a server manages the in-vehicle sales menu and its inventory status, and provides this information to the user terminal.

[2111] "Means for generating and presenting new ticket information to the user" refers to means that a server has the function of creating new ticket information based on a user's request and displaying it on the user's terminal.

[2112] Modes for carrying out the invention

[2113] This invention combines a system that provides various services in autonomous vehicles and public transportation as virtual crew members with an emotion engine that recognizes user emotions, thereby achieving more advanced and user-friendly service provision. The specific system configuration and operation are described below.

[2114] System Configuration

[2115] This system consists of a user, a terminal, a server, and an emotion engine. When using autonomous vehicles or public transportation, the user operates a terminal such as a tablet or smartphone. The terminal receives user input, processes it, and sends it to the server. The emotion engine recognizes the user's emotions in real time and sends that information to the server. The server performs calculations and data processing based on the received departure point, destination information, and user emotion data, and sends the results to the terminal.

[2116] Setting the destination and route

[2117] First, the user specifies their starting point and destination using their device. For example, they might set the starting point to "Tokyo Station" and the destination to "Shinagawa Station" on a map application. This information is sent from the device to the server. The server calculates the optimal route based on the starting point and destination data. This calculation also takes into account sentiment data from the sentiment engine; for example, if it determines that the user is in a hurry, it will suggest a route that prioritizes highways. The calculation results include distance, travel time, and whether highways are available. This information is sent to the device and displayed to the user. The user confirms the route and resends the response information from the device to the server, which then determines the final route and sends that information back to the device.

[2118] Example prompt:

[2119] "My departure point is Tokyo Station, and my destination is Shinagawa Station. I'm in a hurry."

[2120] In-train sales

[2121] This section explains how a user requests in-train sales. The user operates the in-train sales application on their terminal and sends a request such as "I want to order a bento box." The terminal sends this request to the server. Based on the request, the server retrieves the in-train sales menu and inventory information and sends it to the terminal. The emotion engine recognizes the user's emotions and, for example, if it determines that the user is tired, it can suggest relaxing tea or other items. Once the user selects the desired items, the selection information is sent from the terminal to the server, and after the server checks the inventory, it sends a purchase confirmation message to the terminal. The terminal displays the purchase confirmation message to the user, and the order is finalized upon user confirmation.

[2122] Example prompt:

[2123] "I'd like to order a bento box. I'm very tired."

[2124] Ticket change

[2125] This section describes a scenario where a user requests a change to their train ticket. The user operates the ticket management application on their terminal and requests a "change of ticket." This request is sent from the terminal to the server. The server calculates the available options and suggests the best option based on emotional data. For example, if it is determined that the user wants to relax, it will suggest a quiet carriage. The user selects their desired change option, and this information is sent from the terminal to the server. The server generates new ticket information, sends it to the terminal, and displays it to the user.

[2126] Example prompt:

[2127] "I want to change my ticket. I want to relax."

[2128] As described above, the present invention is a system in which the user, terminal, server, and emotion engine work together to provide efficient and user-friendly services. By considering the user's emotions when providing various services, the quality of the user experience can be improved.

[2129] The flow of the specific processing in Example 2 will be explained using Figure 13.

[2130] Step 1:

[2131] The user uses a device to specify the starting point and destination.

[2132] Specific operation: The user opens a map application and enters "Tokyo Station" as the starting point and "Shinagawa Station" as the destination. The terminal receives this input information and stores it as input data.

[2133] Input: Departure point and destination information

[2134] Output: Data on departure point and destination.

[2135] Step 2:

[2136] The terminal sends the entered information to the server.

[2137] Specific operation: The terminal converts the user's entered departure point and destination information into a digital format and sends it to the server.

[2138] Input: User-entered data for departure point and destination.

[2139] Output: Departure and destination data sent to the server

[2140] Step 3:

[2141] The emotion engine recognizes the user's emotions in real time and sends that information to the server.

[2142] Specific operation: The emotion engine analyzes the user's facial expressions, voice, and actions to recognize their emotional state, such as whether they are in a hurry or relaxed. The recognized emotional information is then sent to the server.

[2143] Input: Real-time user sentiment information

[2144] Output: Emotion data sent to the server

[2145] Step 4:

[2146] The server calculates the optimal route based on the departure point, destination information, and sentiment data it receives.

[2147] Specific operation: The server considers the user's sentiment data in addition to the origin and destination data, and uses an optimal route algorithm to perform calculations. For example, if the user is in a hurry, it will calculate a route that prioritizes highways.

[2148] Input: Origin, destination, and sentiment data

[2149] Output: Optimal route information (distance, travel time, availability of highways, etc.)

[2150] Step 5:

[2151] The device receives the calculated route information and presents it to the user.

[2152] Specific operation: The server sends the calculated optimal route information to the terminal. The terminal displays this information and prompts the user for confirmation. For example, the displayed route information might say "From Tokyo Station to Shinagawa Station, travel time 20 minutes, expressway available."

[2153] Input: Optimal route information from the server

[2154] Output: Route information presented to the user

[2155] Step 6:

[2156] The user checks route information on ...

Claims

1. For setting destinations in autonomous vehicles, A means for receiving user input and generating departure point and destination information, A means for calculating the optimal route using the received departure point and destination information, A means of presenting calculated route information to the user and confirming whether or not highways can be used, A system that includes means for optimizing routes based on user responses and generating final route information.

2. To provide in-car sales services, A means of receiving a user's in-vehicle sales request and sending it to the server, A means of obtaining in-car sales menus and inventory information, A means of presenting the acquired menu information to the user, A means of receiving user selection information and sending it to the server, A system that includes means for checking the stock of selected items and presenting a purchase confirmation message to the user.

3. To change your train ticket, A means of receiving a user's ticket change request and sending it to the server, A means of presenting the user with modifiable option information calculated by the server, A means of receiving user selection information and sending it to the server, A system that includes means for generating and presenting new ticket information to the user.

Images