system

Abstract

We provide the system. [Solution] An analysis means for acquiring email information and image information and analyzing the data contained therein, A means for classifying the acquired analytical data and displaying it to the user, A notification system that compares the user's set budget with their spending and notifies them when the budget is exceeded, An expense management system for managing expenses within a group and calculating shared expenses, A payment method that performs the remittance process based on the calculated shared cost, A tracking system that integrates scheduling and expense monitoring, and automatically tracks travel expenses, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventional expense management in travel is very complicated. Especially in group travel, it is difficult to settle accounts and share expenses. In addition, since budget management during travel is entrusted to self-management, expenses may exceed the budget. Moreover, after the travel, it is necessary to manually perform the settlement work, which burdens the users. Furthermore, there is a lack of means for effectively utilizing the expenditure data of travelers to ensure marketing by travel-related businesses and local governments. The purpose is to solve these problems and provide a system that can smoothly manage expenses.

Means for Solving the Problems

[0005] [[ID=*]] This invention provides an information acquisition means for acquiring emails and image data and analyzing the information contained therein with high accuracy. Furthermore, it includes an information classification means that automatically categorizes the analyzed information and presents it visually to the user. It also supports expense management by introducing a warning means that monitors expenses in real time based on a budget set by the user and issues a warning when the budget is exceeded. In addition, it provides an expense management means that aggregates the expenses of each individual in a group and calculates the amount to be split, and a settlement processing means that automatically processes settlements via online remittance. Furthermore, it adds an analysis means that proposes appropriate promotions to travel-related businesses and local governments based on the analyzed expense data, enabling more efficient marketing. As a result, expense management during and after travel becomes easier, and the burden on the user can be significantly reduced.

[0006] "Email and image data" refers to message information sent and received electronically, and visual information stored in digital format.

[0007] "Information acquisition means" refers to a device or method that has the function of collecting various data provided by users and extracting it in an analyzable format.

[0008] "Information classification means" refers to a device or method that has the function of grouping acquired data according to specific criteria and presenting it to the user in a visually user-friendly format.

[0009] "Warning device" refers to a device or method that has the function of issuing a message or notification to alert the user when a set condition is exceeded.

[0010] "Expense management means" refers to a device or method that has the function of allocating and calculating expenses by recording and analyzing the user's spending.

[0011] "Settlement processing means" refers to a device or method that has the function of settling expenses between groups, for example, by using an online money transfer service.

[0012] "Analytical means" refers to a device or method that has the function of analyzing acquired and classified data and making optimal suggestions for a specific purpose, such as marketing or promotion, based on the results. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiment for Carrying Out the Invention

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention is a system for automating expense management during and after travel, and is realized through the cooperation of a server, terminals, and users.

[0035] The server retrieves travel-related emails and image data from the user's mailbox or dedicated app. The retrieved data is then analyzed, for example, to extract information such as travel itinerary, accommodation, transportation, and expenses.

[0036] Next, the terminal automatically organizes the extracted information into categories using an information classification system and displays it on the interface for easy viewing by the user. Specifically, it is categorized into categories such as accommodation expenses, transportation expenses, and food expenses.

[0037] Users set their planned budget on the device before their trip. Based on this setting, the device monitors the user's spending in real time and issues an alert if the set budget is exceeded. This allows users to effectively manage their spending during their trip.

[0038] Furthermore, at the end of the trip, the server aggregates the expenses of the users who participated in the group trip and calculates the fair share. Based on the calculated amount, a settlement processing mechanism is used to automatically settle the payment, for example, via an online money transfer service.

[0039] Furthermore, the server uses the acquired and categorized spending data to propose promotions tailored to user preferences to travel-related businesses and local governments. This enables effective data-driven marketing.

[0040] As a concrete example, when a user books a hotel at their travel destination, the booking confirmation email is analyzed by the server, and details of the accommodation and price are automatically recorded as subordinate data. Subsequently, by taking photos of receipts from taxis and restaurants used during the trip, the device appropriately categorizes them as transportation and food expenses. After the trip, the server uses this data to distribute expenses among group members, and each person's share is settled using an online money transfer service. Through this series of processes, users are freed from the hassle of expense management, allowing them to enjoy their trip more freely.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] Users save travel-related emails and receipt images to a designated mailbox or a dedicated app.

[0044] Step 2:

[0045] The server accesses the specified mailbox or app folder to retrieve travel-related emails and image data. This data includes itinerary, fares, and booking information.

[0046] Step 3:

[0047] The server analyzes the content of the retrieved emails using natural language processing to extract important information, such as the departure date and time and the reservation amount.

[0048] Step 4:

[0049] The server uses OCR technology to extract text data from the receipt image and uses this information to verify the expenses.

[0050] Step 5:

[0051] The terminal receives the analyzed data sent from the server and automatically classifies it into categories (e.g., transportation expenses, accommodation expenses, food expenses) using an information classification system.

[0052] Step 6:

[0053] The user enters their travel budget into the device and begins budget management. This information is recorded on the device and used later for monitoring expenses.

[0054] Step 7:

[0055] The device tracks spending in real time based on user input and displays a warning to the user if the budget is exceeded.

[0056] Step 8:

[0057] After the trip ends, the server aggregates spending data from the group participants and uses an expense management system to calculate each person's share of the expenses.

[0058] Step 9:

[0059] The terminal displays the calculated split amount to the user and offers the option to settle the payment via an online money transfer service.

[0060] Step 10:

[0061] The server will process the payment using the selected payment method, based on instructions from the consenting user. This completes the entire payment process.

[0062] Step 11:

[0063] The server analyzes the expenditure data to make promotional proposals to travel-related businesses and local governments based on the analyzed data.

[0064] This series of steps allows users to efficiently manage and settle expenses during and after their trip.

[0065] (Example 1)

[0066] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0067] Managing travel expenses during and after a trip is often cumbersome and time-consuming, and fair cost sharing is particularly difficult in group travel. Furthermore, effective marketing activities for travel-related businesses require the collection and analysis of information based on user preferences, but there are limitations to doing this manually. To solve these problems, the present invention aims to provide an efficient system that integrates data acquisition, analysis, classification, expenditure management, settlement, and analysis.

[0068] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0069] In this invention, the server includes: information acquisition means for acquiring data using communication means and analyzing the information contained therein; information classification means for classifying the acquired information into higher-level categories and displaying them to the user; warning means for comparing the budget set by the user with expenditures and issuing a warning if the budget is exceeded; expenditure management means for managing expenditures within the community and calculating the amount to be shared; settlement processing means for processing remittances based on the calculated amount to be shared; analysis means for making proposals to stakeholders based on the analyzed expenditure data; means for extracting textual information from image data using visual recognition technology; and means for performing predictive analysis based on the analyzed data and generating a proposal model. This enables efficient expense management and marketing activities through the automated management and analysis of travel-related information.

[0070] "Communication means" is a general term for technologies or devices used to acquire and transmit electronic data.

[0071] "Information acquisition means" refers to the technology or process for analyzing and extracting necessary information from collected data.

[0072] "Information classification means" refers to a technology or algorithm for organizing and classifying acquired information into categories.

[0073] "Warning mechanisms" refer to functions that notify or alert users when specific conditions are met.

[0074] "Expenditure management tools" are techniques or processes for tracking expenditures within a community and calculating fair share of those expenditures.

[0075] "Settlement processing means" refers to the process of sending and receiving money based on the calculated share of the cost.

[0076] "Analysis tools" refer to techniques or algorithms used to analyze collected data and make suggestions or predictions tailored to specific purposes.

[0077] "Visual recognition technology" refers to technologies for extracting and recognizing text and other information from image data.

[0078] "Predictive analytics" is an analytical method used to predict future trends and behaviors based on past and present data.

[0079] A "proposal model" refers to a computational model that presents the optimal actions or choices based on the analysis results.

[0080] This travel expense management system is a comprehensive expense management tool realized through the collaboration of a server, terminals, and users. Specifically, the server uses communication methods to retrieve travel-related data from the user's email inbox or dedicated application. For example, when retrieving emails, a general mail server API is used. After retrieving the data, the server uses information retrieval methods to analyze the text information within the emails and the image data retrieved from cloud storage.

[0081] Visual recognition technology is used to analyze image data. This technology allows for the extraction of textual information from images such as receipts and reservation documents. Furthermore, the server uses information classification means to classify the extracted information into higher-level categories and transmits that information to the terminal.

[0082] The terminal receives information sent from the server and displays it to the user on its interface. Based on this information, the terminal has a function to compare the budget set by the user with actual spending, and notifies the user using a warning mechanism if the budget is exceeded. This means that the warning function implemented in the terminal is properly managed.

[0083] Users set a budget using an application on their device and monitor and manage expenses incurred during their trip. Upon completion of the trip, the server uses an expense management system to aggregate expenses within the group and calculate a fair share. A settlement processing system then automatically processes the payment based on the calculated share.

[0084] The analyzed spending data is further processed by analytical tools to support effective marketing activities, for example, by creating promotional proposals for travel-related businesses and relevant organizations.

[0085] For example, when a user books a hotel at their travel destination, the booking confirmation email is retrieved and analyzed by the server, and the accommodation information is displayed on the device. During the trip, simply taking a picture of the taxi receipt with the device automatically categorizes it as a transportation expense using visual recognition technology. This eliminates the need for cumbersome expense entry, allowing users to enjoy their trips more freely.

[0086] An example of a prompt for a generated AI model would be: "I would like to propose a system to help manage travel expenses. I want to develop an app that extracts necessary information from travel-related emails, categorizes expenses, and includes budget management functions. Please advise on how to implement this system."

[0087] This system allows travelers to manage and settle their expenses efficiently and easily, reducing financial stress during their trip and allowing them to enjoy their travels to the fullest.

[0088] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0089] Step 1:

[0090] The server uses communication methods to retrieve travel-related data from the user's mailbox or dedicated app. The input at this stage consists of the user's emails and image data in cloud storage. The server filters this data, identifying and collecting travel-related information. The collection process utilizes email server APIs, generating a dataset that is then passed to the information retrieval system.

[0091] Step 2:

[0092] The server uses information acquisition methods to analyze the text and image data of the acquired emails. The input consists of the email text and image files collected in step 1. The server uses natural language processing and visual recognition technologies to extract information such as travel itinerary, accommodation, and transportation, and generates structured data as output. Specifically, the Natural Language Understanding API and image recognition algorithms are executed.

[0093] Step 3:

[0094] The terminal receives structured data sent from the server and automatically organizes it into categories using an information classification system. The input for this step is the parsed information received from the server. The terminal generates a database of categories such as accommodation, transportation, and food expenses, and this is reflected in the interface as output. The information is displayed visually using a user interface such as React Native.

[0095] Step 4:

[0096] The user sets a planned budget on the device before their trip. The input is the budget amount set by the user in the application. The device monitors this setting in real time and triggers a warning function if spending exceeds the entered budget. The output is a push notification or alert triggered to the user.

[0097] Step 5:

[0098] At the end of the trip, the server uses an expense management system to aggregate the group's expenses and calculate a fair share. The input here is expense data from all participants sent from their terminals. The server runs a bill-splitting algorithm, calculates each participant's share as output, and passes it to the settlement processing system.

[0099] Step 6:

[0100] The server settles the calculated share amount using a settlement processing method via an online money transfer service. This process uses the calculated share amount as input. In actual operation, it calls a money transfer API and completes the transfer process as output.

[0101] Step 7:

[0102] The server uses analytical tools based on the analyzed data to propose promotions to stakeholders. Inputs include user preference information and spending trends. The server performs predictive analysis using an AI model, generates promotion strategies as output, and provides feedback to travel-related businesses. When using the generating AI model, these proposals are realized by appropriately creating prompt statements and inputting them into the model.

[0103] (Application Example 1)

[0104] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0105] Managing expenses during and after travel is cumbersome for many people, and group travel, in particular, presents challenges due to the complexity of organizing expenses, splitting costs, and making payments. Furthermore, individual budget management is not easy, and there is a risk of exceeding budgets, so real-time spending monitoring is required. Another challenge is to utilize the large amount of data generated during travel and provide feedback to travel-related businesses and the local economy.

[0106] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0107] This invention includes an analysis means for acquiring email and image information and analyzing the data contained therein; a display means for classifying the acquired analysis data and displaying it to the user; a notification means for comparing the budget set by the user with the expenditure and notifying the user when the budget is exceeded; an expense management means for managing expenses within the group and calculating shared expenses; a settlement means for executing remittance processing based on the calculated shared expenses; and a tracking means for integrating schedule and expense monitoring and automatically tracking expenses during travel. This makes it possible to efficiently manage expenses during travel, support the user's budget management, and automate the calculation and payment of split expenses during group travel. It can also provide effective feedback to travel-related businesses and the local economy.

[0108] "Analysis means" refers to a means that acquires email information and image information, analyzes the data contained therein, and extracts meaningful information.

[0109] A "display means" is a means that has the function of showing the acquired analysis data on the screen in a way that is easy for the user to understand.

[0110] A "notification mechanism" is a function that issues an alert when the user's set budget is exceeded, serving as a means to draw the user's attention.

[0111] An "expense management system" is a means of organizing expenses within a group and calculating shared expenses.

[0112] A "payment method" is a means of automatically processing a transfer based on the calculated shared cost and completing the payment.

[0113] A "tracking system" is a means of monitoring schedules and expenses, and recording and managing travel expenses in real time.

[0114] This invention is a system for streamlining expense management during and after travel. The server retrieves travel-related email and image information using the user's mailbox or a dedicated app. This data is analyzed using Google® Cloud Vision API and natural language processing (NLP) technology to extract information such as travel itinerary, accommodation, transportation, and expenses.

[0115] The device organizes the analyzed information into categories, classifying it into accommodation, transportation, and food expenses, and displays it on the interface. Users set a budget on the device before traveling, and the device monitors spending in real time. A feature that immediately notifies users if they exceed their budget makes it easy for them to manage their spending.

[0116] Furthermore, the server collects expenses from users participating in the group trip and calculates a fair and shared cost. Based on this calculation, it automatically settles payments using online payment services such as the Stripe API.

[0117] For example, if a user scans a restaurant receipt they took during their trip, the system can automatically categorize the restaurant expense as food expenses, helping them stay within their budget. Furthermore, it utilizes a generative AI model to create promotional suggestions tailored to the user's preferences in real time, for travel-related businesses and regions.

[0118] An example of a prompt message would be: "Set a budget for your next trip and track expenses in real time. You will be notified if you exceed your budget. After the trip, divide the group travel expenses equally and settle the accounts online."

[0119] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0120] Step 1:

[0121] The server retrieves travel-related email and image information from the user's mailbox or dedicated app. The input consists of the user's emails and image data, and the output is a collection of this data. Email information is retrieved using the POP3 or IMAP protocol, while image information is uploaded directly via the app.

[0122] Step 2:

[0123] The server processes image data using the Google Cloud Vision API and parses email data using natural language processing (NLP) techniques. The input is the email and image data obtained in step 1, and the output is extracted travel itinerary, accommodation, transportation, and cost information. Specifically, it converts receipt information from image data into text and extracts travel booking details from emails.

[0124] Step 3:

[0125] The terminal receives analysis information sent from the server and classifies it into categories such as accommodation, transportation, and food expenses. The input is the analyzed information, and the output is information organized by category. A graphical user interface (GUI) is provided to display the classification results on the user interface for easier operation.

[0126] Step 4:

[0127] The user registers their budget information, set before their trip, on their device. The input is the budget amount and its details, while the output is the budget data stored within the device. Specifically, this is achieved by the user entering the budget figures into a smartphone application.

[0128] Step 5:

[0129] The device monitors travel expenses in real time and notifies the user if an overspending is detected. Input is real-time spending data, and output is an alert for budget overruns. Notifications are sent to the smartphone as push notifications.

[0130] Step 6:

[0131] The server aggregates the expenses of all group members at the end of the trip and calculates the split amount. The input is the expense data of each individual user, and the output is the calculated split amount. Specifically, the calculation is performed using a spreadsheet-like data structure.

[0132] Step 7:

[0133] The server automatically performs settlement using an online payment service based on the calculation results. The input is the calculated split amount and each user's payment information, and the output is the completed payment result. The payment is processed using the Stripe API, and the user receives a notification.

[0134] Step 8:

[0135] The server generates reports that propose promotions to travel-related businesses and local economies based on analyzed spending data. The input is spending data and information about user preferences, and the output is a promotional report customized by a generating AI model. An example of a prompt message might be, "Set a budget for your next trip and track your expenses in real time..."

[0136] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0137] This invention is a system that automates expense management during and after travel and aims to improve the user experience based on emotions, and is realized through the cooperation of a server, terminal, emotion engine, and user.

[0138] The server retrieves travel-related emails and image data from the user's mailbox or dedicated app. This includes information such as travel itinerary, accommodation reservations, transportation methods, and expenses. This data is then analyzed using natural language processing and optical character recognition technology to extract important information.

[0139] The terminal receives analyzed data sent from the server and automatically categorizes it using an information classification system. Users set a budget before traveling, and this information is recorded on the terminal. The terminal tracks spending in real time and notifies the user using an alert system if the budget is exceeded.

[0140] The emotion engine analyzes user data, such as voice, text, and facial expressions, to recognize the user's emotions in real time. This emotion data includes items such as joy, anxiety, and stress. Based on this information, the emotion engine presents the user with the most appropriate travel plan and activities for the situation, helping to improve the experience.

[0141] If users are traveling in a group, the server aggregates spending data within the group and calculates the amount to be split using the spending management system. The calculation result is displayed on the terminal, and automatic settlement is performed using an online money transfer service.

[0142] All analyzed information, including emotional data, is used through analytical tools to develop promotional proposals for travel-related businesses and local governments. For example, if a user shows high emotional satisfaction at a particular location, that data can be used to optimize the promotional strategy.

[0143] As a concrete example, a user can take a picture of a receipt during their trip using their smartphone camera, and the device automatically analyzes it using OCR technology. The results are sent to a server, which reflects the analyzed expenses and visualizes them on the device. Furthermore, an emotion engine detects the user's enjoyment from their behavior at a restaurant and recommends options for subsequent dinners. In this way, the present invention plays a role in making the user's trip more fulfilling.

[0144] The following describes the processing flow.

[0145] Step 1:

[0146] Users save booking emails and receipt images related to their travels to their email inbox or a dedicated app.

[0147] Step 2:

[0148] The server interacts with the user's mailbox to automatically retrieve emails and image data related to their trip.

[0149] Step 3:

[0150] The server analyzes the retrieved email content using natural language processing technology to extract travel itineraries and reservation information.

[0151] Step 4:

[0152] The server uses optical character recognition technology to analyze the image data and extract expense information from the receipt image.

[0153] Step 5:

[0154] The terminal receives the analyzed data sent from the server and automatically classifies the information into categories using an information classification means.

[0155] Step 6:

[0156] Before traveling, the user sets their travel budget information on the device. The device then records this budget data.

[0157] Step 7:

[0158] The device tracks the user's spending in real time during travel and notifies the user using warning mechanisms when they are likely to exceed their budget.

[0159] Step 8:

[0160] The emotion engine activates and recognizes the user's emotional state by analyzing their voice, facial expressions, and text messages.

[0161] Step 9:

[0162] The emotion engine suggests optimal activities and plans to the user based on recognized emotions and the current travel situation.

[0163] Step 10:

[0164] After the trip ends, the server compiles all expenses within the group and uses expense management tools to fairly calculate how much each person will pay.

[0165] Step 11:

[0166] The terminal notifies the user of the calculated split amount and initiates the settlement process via the online money transfer service.

[0167] Step 12:

[0168] The server uses analyzed travel spending data, including sentiment data, to perform analysis in order to propose promotions to travel-related businesses and local governments.

[0169] This process allows users to simultaneously manage their travel expenses and enhance their emotionally-driven travel experience.

[0170] (Example 2)

[0171] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0172] Manually managing expenses during and after a trip is often time-consuming and prone to errors. Furthermore, it's difficult to enhance the experience based on emotions during the trip, highlighting the need for a system that enhances individual travel experiences. Conventional technologies lacked the means to comprehensively address these challenges.

[0173] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0174] In this invention, the server includes data extraction means for acquiring electronic communication and visual data and analyzing the information contained therein; information display means for classifying the extracted information into categories and presenting them to the user; and emotion analysis means for analyzing the user's emotions and making suggestions to improve the individual experience. This enables the automation of expense management during and after travel, as well as the improvement of the travel experience based on emotions.

[0175] "Electronic communications and visual data" refers to digital information, including emails and images, obtained from users and other sources.

[0176] "Data extraction means" refers to techniques for analyzing and extracting necessary information from electronic communication and visual data.

[0177] A "classification category" is a category used to group analyzed information based on specific criteria.

[0178] "Information display means" refers to functions and technologies for visually presenting analyzed information or classified categories to users.

[0179] An "alert system" is a technology that issues a warning when an anomaly occurs that does not meet the criteria set by the user.

[0180] A "cost management tool" is a function that allows for understanding the group's spending situation and calculating the amount each member should pay.

[0181] "Financial processing means" refers to technology for electronically settling payments based on calculated split amounts.

[0182] "Emotional analysis tools" refer to technologies that analyze a user's emotional state and provide optimal suggestions based on the results.

[0183] "Information utilization methods" refer to methods for making promotional proposals to travel agencies and local governments based on analyzed data.

[0184] "Visual recognition technology" refers to the technology used to understand the meaning of textual information and other data from images and to extract necessary information.

[0185] To implement this invention, it is first necessary to construct a system in which a server, terminal, and emotion analysis engine work together. This system is designed to improve the user's travel experience.

[0186] The server retrieves electronic communication data (e.g., emails containing travel itineraries and reservation information) and visual data (e.g., receipts and travel-related images) from the user's mailbox or dedicated app. This server incorporates natural language processing and optical character recognition technologies as data extraction methods. The server uses these technologies to analyze the acquired digital information and extract the necessary information.

[0187] The terminal receives analyzed information sent from the server, categorizes the data using an information display device, and presents it to the user. The user can use the terminal to set a budget before traveling. Based on this budget, the terminal utilizes an alert system and immediately displays a warning if spending during the trip exceeds the budget.

[0188] On the other hand, the emotion analysis engine analyzes voice and facial expression data acquired from the user in real time to understand the user's emotional state. The results of this emotion analysis are used to suggest activities and plans to improve the user's travel experience.

[0189] As a concrete example, consider a scenario where a user takes a picture of a receipt with their smartphone camera while traveling. This photographic data is immediately analyzed by the device using OCR technology, and the information is sent to a server. The server stores the analysis results in a database and visualizes them clearly on the device. Furthermore, the emotion engine can analyze the user's reaction to a restaurant they visited and suggest appropriate activities for the next step.

[0190] To support the overall functionality of this invention, users can input prompts using a generative AI model. For example, by using a prompt in the form of, "Please use OCR to analyze receipts I photographed during my trip and manage my expenses. Furthermore, please analyze my current mood and suggest activities," the system will provide the user with the most suitable suggestions.

[0191] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0192] Step 1: Data Acquisition

[0193] The server retrieves travel-related electronic and visual data from the user's mailbox or dedicated app. It uses an algorithm to filter relevant email subjects and access permissions to the user's email account as input. The output generates unanalyzed data including travel itineraries, accommodation bookings, and transportation information. The server temporarily stores this data and prepares it for analysis.

[0194] Step 2: Data Analysis

[0195] The server analyzes the acquired unanalyzed data using natural language processing and optical character recognition (OCR) technologies. The unanalyzed data obtained in step 1 is used as input. Text information is analyzed using natural language processing, and text is extracted from images using OCR. The output generates analyzed data containing travel reservation confirmation numbers, dates, and cost information.

[0196] Step 3: Information Classification

[0197] The terminal receives parsed data sent from the server and classifies the data into categories using a classification algorithm. It uses parsed data from the server as input. The terminal automatically classifies the data into categories such as "accommodation," "transportation," and "food," and generates a data structure as output to present the classified information to the user.

[0198] Step 4: Budget Management

[0199] Before traveling, the user enters their budget into the terminal. The terminal compares ongoing expenses with the set budget in real time. The inputs used are the user's set budget and the categorized expense data obtained in step 3. If the budget is exceeded, the terminal immediately displays a warning and generates a warning message as output.

[0200] Step 5: Emotion Analysis

[0201] The device acquires the user's voice and facial expression data and analyzes it in real time using an emotion analysis engine. It uses the user's sensor information and camera data as input. The emotion analysis engine identifies emotional states such as stress, joy, and excitement, and generates data that includes emotion-based improvement suggestions as output.

[0202] Step 6: Proposal Generation

[0203] The device suggests suitable travel plans and activities to the user based on emotional data obtained from the emotion analysis engine and the progress of the trip. The inputs used are the user's emotional data and travel information classified in step 3. The output generates a list of recommended plans and activities to enhance the user's experience.

[0204] Step 7: Expense settlement

[0205] The server aggregates spending data from all members during a group trip and calculates the split cost using expense management tools. It uses spending information from each group member as input. The output generates the exact amount each member should pay and online transfer instructions to support that payment. This information is sent to the terminal and displayed clearly to the user.

[0206] (Application Example 2)

[0207] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0208] Efficiently managing expenses and tracking spending during travel and outings, as well as improving the experience based on emotions, proved difficult. Furthermore, optimizing the experience through recommendations tailored to the user's emotional state and managing spending within groups presented challenges. Additionally, there was a lack of effective ways to utilize the analyzed emotional data.

[0209] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0210] In this invention, the server includes acquisition means for acquiring data and analyzing information, classification means for classifying and displaying the acquired information, and monitoring means for comparing a set budget with expenditures and issuing warnings. This enables users to efficiently manage expenses, track spending, and recommend choices based on their emotions.

[0211] "Data" refers to a collection or aggregate of information, and is what is processed by a system.

[0212] "Means of acquisition" refers to the function or method of collecting necessary data or information.

[0213] "Analysis" is the process of breaking down acquired data in order to understand its meaning and value.

[0214] A "classification method" is a method of organizing analyzed information into categories based on specific criteria.

[0215] "Display" refers to the act or device of visually presenting information to a user.

[0216] A "monitoring mechanism" is a function that continuously compares the budget with expenditures and issues a warning when an anomaly occurs.

[0217] A "management tool" is a system for appropriately organizing and controlling funds and expenditures within a group, and for performing necessary calculations.

[0218] A "settlement method" is a processing function for making payments or remittances based on calculation results.

[0219] "Emotional state" refers to data that indicates the user's psychological response and mood.

[0220] A "feedback mechanism" is a function that presents the user with the most suitable suggestions or options based on their analyzed emotional state.

[0221] "Analysis" refers to the process of examining acquired data in detail and extracting useful information from it.

[0222] "Optical character recognition technology" is a technology that automatically recognizes characters from image data and converts them into text data.

[0223] The system realizing this invention is designed to acquire data and provide the user with the optimal experience based on the analyzed information. The server acquires data sent from the user's device. This includes travel plans, purchase history, and sentiment data from voice and text. The server analyzes the acquired data using natural language processing libraries (e.g., spaCy, NLTK) and optical character recognition technology (e.g., Tesseract) to extract information. Furthermore, it utilizes sentiment recognition APIs (e.g., Microsoft® Azure® Emotion API) to grasp the user's emotional state in real time and use it to provide feedback.

[0224] The terminal receives information analyzed from the server and organizes it by category. This information is displayed intuitively through the user interface, allowing the user to check their budget management and spending status at any time.

[0225] Users can receive alerts from their device when they exceed their set budget, enabling efficient financial management. Furthermore, during group travel, the expense management system allows for the aggregation of the group's total expenses and the calculation of appropriate splitting amounts.

[0226] As a concrete example, a user sends an image taken with their smartphone camera to a server. This image is converted into text using OCR technology, and emotion recognition is used to analyze how the user felt in a particular situation. The analyzed information is evaluated by the user and used to optimize their next actions and choices.

[0227] An example of a prompt to be input to the generating AI model is, "Consider the user's current emotional state and suggest lunch options suitable for stress reduction." In this way, the present invention makes it possible to highly personalize the user's travel and outing experiences.

[0228] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0229] Step 1:

[0230] The server retrieves travel-related emails and image data from the user's device. The input is emails and images, and the output is data ready for analysis. Specifically, this involves scanning the mailbox and retrieving image data.

[0231] Step 2:

[0232] The server analyzes the acquired data using a natural language processing library (e.g., spaCy) to extract travel itineraries, reservation information, and other relevant details. The input is the data obtained in step 1, and the output is a well-organized set of information. This clarifies the user's travel plan.

[0233] Step 3:

[0234] The server uses OCR technology (e.g., Tesseract) to extract text from image data. The input is an image file, and the output is digital text. This allows paper receipts and ticket information to be treated as text data.

[0235] Step 4:

[0236] The terminal receives the analyzed information sent from the server and organizes it by category. The input is the data obtained in steps 2 and 3, and the output is the information classified by category. The information is visualized in a format suitable for the user interface.

[0237] Step 5:

[0238] The device monitors the user's set budget against actual spending and displays a warning if the budget is exceeded. Input is the user's budget information and spending data, and output is a warning message. This allows users to efficiently manage their spending.

[0239] Step 6:

[0240] The server aggregates spending data from the group and calculates the appropriate split amount. The input is each member's spending data, and the output is the calculated split amount. Users can then refer to this result to divide the payment equally.

[0241] Step 7:

[0242] The server uses an emotion recognition API to analyze the user's emotions in real time and generate feedback to improve the user experience. Input is emotion-related data from voice and text, and output is suggestions and recommended actions. Based on this, the user can make better decisions.

[0243] Step 8:

[0244] The user inputs a prompt into a generated AI model, and the system performs data calculations based on that input to provide the optimal food delivery option. The input is a prompt, and the output is a customized suggestion. A specific example of the prompt might be, "Consider the user's current emotional state and suggest a lunch option suitable for stress reduction." In this way, the data technology used enriches the user experience.

[0245] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0246] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0247] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0248] [Second Embodiment]

[0249] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0250] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0251] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0252] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0253] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0254] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0255] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0256] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0257] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0258] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0259] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0260] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0261] This invention is a system for automating expense management during and after travel, and is realized through the cooperation of a server, terminals, and users.

[0262] The server retrieves travel-related emails and image data from the user's mailbox or dedicated app. The retrieved data is then analyzed, for example, to extract information such as travel itinerary, accommodation, transportation, and expenses.

[0263] Next, the terminal automatically organizes the extracted information into categories using an information classification system and displays it on the interface for easy viewing by the user. Specifically, it is categorized into categories such as accommodation expenses, transportation expenses, and food expenses.

[0264] Users set their planned budget on the device before their trip. Based on this setting, the device monitors the user's spending in real time and issues an alert if the set budget is exceeded. This allows users to effectively manage their spending during their trip.

[0265] Furthermore, at the end of the trip, the server aggregates the expenses of the users who participated in the group trip and calculates the fair share. Based on the calculated amount, a settlement processing mechanism is used to automatically settle the payment, for example, via an online money transfer service.

[0266] Furthermore, the server uses the acquired and categorized spending data to propose promotions tailored to user preferences to travel-related businesses and local governments. This enables effective data-driven marketing.

[0267] As a concrete example, when a user books a hotel at their travel destination, the booking confirmation email is analyzed by the server, and details of the accommodation and price are automatically recorded as subordinate data. Subsequently, by taking photos of receipts from taxis and restaurants used during the trip, the device appropriately categorizes them as transportation and food expenses. After the trip, the server uses this data to distribute expenses among group members, and each person's share is settled using an online money transfer service. Through this series of processes, users are freed from the hassle of expense management, allowing them to enjoy their trip more freely.

[0268] The following describes the processing flow.

[0269] Step 1:

[0270] Users save travel-related emails and receipt images to a designated mailbox or a dedicated app.

[0271] Step 2:

[0272] The server accesses the specified mailbox or app folder to retrieve travel-related emails and image data. This data includes itinerary, fares, and booking information.

[0273] Step 3:

[0274] The server analyzes the content of the retrieved emails using natural language processing to extract important information, such as the departure date and time and the reservation amount.

[0275] Step 4:

[0276] The server uses OCR technology to extract text data from the receipt image and uses this information to verify the expenses.

[0277] Step 5:

[0278] The terminal receives the analyzed data sent from the server and automatically classifies it into categories (e.g., transportation expenses, accommodation expenses, food expenses) using an information classification system.

[0279] Step 6:

[0280] The user enters their travel budget into the device and begins budget management. This information is recorded on the device and used later for monitoring expenses.

[0281] Step 7:

[0282] The device tracks spending in real time based on user input and displays a warning to the user if the budget is exceeded.

[0283] Step 8:

[0284] After the trip ends, the server aggregates the expenditure data among the participants of the group trip and calculates each person's share using the expenditure management means.

[0285] Step 9:

[0286] The terminal displays the calculated share amount to the user and provides an option to perform the settlement process via an online money transfer service.

[0287] Step 10:

[0288] Based on the instructions from the users who have given consent, the server executes the settlement using the selected money transfer method. This operation completes all payments.

[0289] Step 11:

[0290] The server analyzes the data in order to make promotion proposals to travel-related businesses and local governments based on the analyzed expenditure data.

[0291] Through this series of steps, users can efficiently manage and settle expenses during and after the trip.

[0292] (Example 1)

[0293] Next, Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0294] Managing expenses during and after a trip is often cumbersome and time-consuming, especially in group trips where it is difficult to share costs fairly. Furthermore, in order to conduct effective marketing activities for travel-related businesses, it is necessary to collect and analyze information based on user preferences, but there are limitations to doing this manually. The purpose of the present invention is to provide an efficient system that integrates data acquisition, analysis, classification, expenditure management, settlement, and analysis in order to solve these problems.

[0295] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0296] In this invention, the server includes: information acquisition means for acquiring data using communication means and analyzing the information contained therein; information classification means for classifying the acquired information into higher-level categories and displaying them to the user; warning means for comparing the budget set by the user with expenditures and issuing a warning if the budget is exceeded; expenditure management means for managing expenditures within the community and calculating the amount to be shared; settlement processing means for processing remittances based on the calculated amount to be shared; analysis means for making proposals to stakeholders based on the analyzed expenditure data; means for extracting textual information from image data using visual recognition technology; and means for performing predictive analysis based on the analyzed data and generating a proposal model. This enables efficient expense management and marketing activities through the automated management and analysis of travel-related information.

[0297] "Communication means" is a general term for technologies or devices used to acquire and transmit electronic data.

[0298] "Information acquisition means" refers to the technology or process for analyzing and extracting necessary information from collected data.

[0299] "Information classification means" refers to a technology or algorithm for organizing and classifying acquired information into categories.

[0300] "Warning mechanisms" refer to functions that notify or alert users when specific conditions are met.

[0301] "Expenditure management tools" are techniques or processes for tracking expenditures within a community and calculating fair share of those expenditures.

[0302] "Settlement processing means" refers to the process of sending and receiving money based on the calculated share of the cost.

[0303] "Analysis means" refers to a technology or algorithm for analyzing the collected data and making proposals or predictions according to specific purposes.

[0304] "Visual recognition technology" refers to a technology for extracting and recognizing characters and other information from image data.

[0305] "Predictive analysis" is an analysis method for inferring future trends and behaviors based on past and current data.

[0306] "Proposal model" refers to a computational model for presenting optimal actions and selections based on analysis results.

[0307] This travel expense management system is an integrated expense management tool realized by the collaboration of a server, terminals, and users. Specifically, the server uses communication means to obtain travel-related data from the user's email box or dedicated application. For example, when obtaining emails, a general email server API is utilized. After data acquisition, the server uses information acquisition means to analyze the text information in the emails and the image data obtained from cloud storage.

[0308] Visual recognition technology is used for the analysis of image data. By this technology, character information is extracted from images of receipts and reservation documents, for example. Furthermore, the server uses information classification means to classify the extracted information into higher-level categories and transmit the information to the terminals.

[0309] The terminal receives the information sent from the server and displays it on the interface for the user. The terminal has a function to compare the budget set by the user with the actual expenditure based on this information, and when the budget is exceeded, it notifies the user using warning means. This means that the warning function implemented on the terminal is properly managed.

[0310] Users set a budget using an application on their device and monitor and manage expenses incurred during their trip. Upon completion of the trip, the server uses an expense management system to aggregate expenses within the group and calculate a fair share. A settlement processing system then automatically processes the payment based on the calculated share.

[0311] The analyzed spending data is further processed by analytical tools to support effective marketing activities, for example, by creating promotional proposals for travel-related businesses and relevant organizations.

[0312] For example, when a user books a hotel at their travel destination, the booking confirmation email is retrieved and analyzed by the server, and the accommodation information is displayed on the device. During the trip, simply taking a picture of the taxi receipt with the device automatically categorizes it as a transportation expense using visual recognition technology. This eliminates the need for cumbersome expense entry, allowing users to enjoy their trips more freely.

[0313] An example of a prompt for a generated AI model would be: "I would like to propose a system to help manage travel expenses. I want to develop an app that extracts necessary information from travel-related emails, categorizes expenses, and includes budget management functions. Please advise on how to implement this system."

[0314] This system allows travelers to manage and settle their expenses efficiently and easily, reducing financial stress during their trip and allowing them to enjoy their travels to the fullest.

[0315] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0316] Step 1:

[0317] The server uses communication methods to retrieve travel-related data from the user's mailbox or dedicated app. The input at this stage consists of the user's emails and image data in cloud storage. The server filters this data, identifying and collecting travel-related information. The collection process utilizes email server APIs, generating a dataset that is then passed to the information retrieval system.

[0318] Step 2:

[0319] The server uses information acquisition methods to analyze the text and image data of the acquired emails. The input consists of the email text and image files collected in step 1. The server uses natural language processing and visual recognition technologies to extract information such as travel itinerary, accommodation, and transportation, and generates structured data as output. Specifically, the Natural Language Understanding API and image recognition algorithms are executed.

[0320] Step 3:

[0321] The terminal receives structured data sent from the server and automatically organizes it into categories using an information classification system. The input for this step is the parsed information received from the server. The terminal generates a database of categories such as accommodation, transportation, and food expenses, and this is reflected in the interface as output. The information is displayed visually using a user interface such as React Native.

[0322] Step 4:

[0323] The user sets a planned budget on the device before their trip. The input is the budget amount set by the user in the application. The device monitors this setting in real time and triggers a warning function if spending exceeds the entered budget. The output is a push notification or alert triggered to the user.

[0324] Step 5:

[0325] At the end of the trip, the server uses an expense management system to aggregate the group's expenses and calculate a fair share. The input here is expense data from all participants sent from their terminals. The server runs a bill-splitting algorithm, calculates each participant's share as output, and passes it to the settlement processing system.

[0326] Step 6:

[0327] The server settles the calculated share amount using a settlement processing method via an online money transfer service. This process uses the calculated share amount as input. In actual operation, it calls a money transfer API and completes the transfer process as output.

[0328] Step 7:

[0329] The server uses analytical tools based on the analyzed data to propose promotions to stakeholders. Inputs include user preference information and spending trends. The server performs predictive analysis using an AI model, generates promotion strategies as output, and provides feedback to travel-related businesses. When using the generating AI model, these proposals are realized by appropriately creating prompt statements and inputting them into the model.

[0330] (Application Example 1)

[0331] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0332] Managing expenses during and after travel is cumbersome for many people, and group travel, in particular, presents challenges due to the complexity of organizing expenses, splitting costs, and making payments. Furthermore, individual budget management is not easy, and there is a risk of exceeding budgets, so real-time spending monitoring is required. Another challenge is to utilize the large amount of data generated during travel and provide feedback to travel-related businesses and the local economy.

[0333] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0334] This invention includes an analysis means for acquiring email and image information and analyzing the data contained therein; a display means for classifying the acquired analysis data and displaying it to the user; a notification means for comparing the budget set by the user with the expenditure and notifying the user when the budget is exceeded; an expense management means for managing expenses within the group and calculating shared expenses; a settlement means for executing remittance processing based on the calculated shared expenses; and a tracking means for integrating schedule and expense monitoring and automatically tracking expenses during travel. This makes it possible to efficiently manage expenses during travel, support the user's budget management, and automate the calculation and payment of split expenses during group travel. It can also provide effective feedback to travel-related businesses and the local economy.

[0335] "Analysis means" refers to a means that acquires email information and image information, analyzes the data contained therein, and extracts meaningful information.

[0336] A "display means" is a means that has the function of showing the acquired analysis data on the screen in a way that is easy for the user to understand.

[0337] A "notification mechanism" is a function that issues an alert when the user's set budget is exceeded, serving as a means to draw the user's attention.

[0338] An "expense management system" is a means of organizing expenses within a group and calculating shared expenses.

[0339] A "payment method" is a means of automatically processing a transfer based on the calculated shared cost and completing the payment.

[0340] A "tracking system" is a means of monitoring schedules and expenses, and recording and managing travel expenses in real time.

[0341] This invention is a system for streamlining expense management during and after travel. The server retrieves travel-related email and image information using the user's mailbox or a dedicated app. This data is analyzed using the Google Cloud Vision API and natural language processing (NLP) technology to extract information such as travel itinerary, accommodation, transportation, and expenses.

[0342] The device organizes the analyzed information into categories, classifying it into accommodation, transportation, and food expenses, and displays it on the interface. Users set a budget on the device before traveling, and the device monitors spending in real time. A feature that immediately notifies users if they exceed their budget makes it easy for them to manage their spending.

[0343] Furthermore, the server collects expenses from users participating in the group trip and calculates a fair and shared cost. Based on this calculation, it automatically settles payments using online payment services such as the Stripe API.

[0344] For example, if a user scans a restaurant receipt they took during their trip, the system can automatically categorize the restaurant expense as food expenses, helping them stay within their budget. Furthermore, it utilizes a generative AI model to create promotional suggestions tailored to the user's preferences in real time, for travel-related businesses and regions.

[0345] An example of a prompt message would be: "Set a budget for your next trip and track expenses in real time. You will be notified if you exceed your budget. After the trip, divide the group travel expenses equally and settle the accounts online."

[0346] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0347] Step 1:

[0348] The server retrieves travel-related email and image information from the user's mailbox or dedicated app. The input consists of the user's emails and image data, and the output is a collection of this data. Email information is retrieved using the POP3 or IMAP protocol, while image information is uploaded directly via the app.

[0349] Step 2:

[0350] The server processes image data using the Google Cloud Vision API and parses email data using natural language processing (NLP) techniques. The input is the email and image data obtained in step 1, and the output is extracted travel itinerary, accommodation, transportation, and cost information. Specifically, it converts receipt information from image data into text and extracts travel booking details from emails.

[0351] Step 3:

[0352] The terminal receives analysis information sent from the server and classifies it into categories such as accommodation, transportation, and food expenses. The input is the analyzed information, and the output is information organized by category. A graphical user interface (GUI) is provided to display the classification results on the user interface for easier operation.

[0353] Step 4:

[0354] The user registers their budget information, set before their trip, on their device. The input is the budget amount and its details, while the output is the budget data stored within the device. Specifically, this is achieved by the user entering the budget figures into a smartphone application.

[0355] Step 5:

[0356] The device monitors travel expenses in real time and notifies the user if an overspending is detected. Input is real-time spending data, and output is an alert for budget overruns. Notifications are sent to the smartphone as push notifications.

[0357] Step 6:

[0358] The server aggregates the expenses of all group members at the end of the trip and calculates the split amount. The input is the expense data of each individual user, and the output is the calculated split amount. Specifically, the calculation is performed using a spreadsheet-like data structure.

[0359] Step 7:

[0360] The server automatically performs settlement using an online payment service based on the calculation results. The input is the calculated split amount and each user's payment information, and the output is the completed payment result. The payment is processed using the Stripe API, and the user receives a notification.

[0361] Step 8:

[0362] The server generates reports that propose promotions to travel-related businesses and local economies based on analyzed spending data. The input is spending data and information about user preferences, and the output is a promotional report customized by a generating AI model. An example of a prompt message might be, "Set a budget for your next trip and track your expenses in real time..."

[0363] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0364] This invention is a system that automates expense management during and after travel and aims to improve the user experience based on emotions, and is realized through the cooperation of a server, terminal, emotion engine, and user.

[0365] The server retrieves travel-related emails and image data from the user's mailbox or dedicated app. This includes information such as travel itinerary, accommodation reservations, transportation methods, and expenses. This data is then analyzed using natural language processing and optical character recognition technology to extract important information.

[0366] The terminal receives analyzed data sent from the server and automatically categorizes it using an information classification system. Users set a budget before traveling, and this information is recorded on the terminal. The terminal tracks spending in real time and notifies the user using an alert system if the budget is exceeded.

[0367] The emotion engine analyzes user data, such as voice, text, and facial expressions, to recognize the user's emotions in real time. This emotion data includes items such as joy, anxiety, and stress. Based on this information, the emotion engine presents the user with the most appropriate travel plan and activities for the situation, helping to improve the experience.

[0368] If users are traveling in a group, the server aggregates spending data within the group and calculates the amount to be split using the spending management system. The calculation result is displayed on the terminal, and automatic settlement is performed using an online money transfer service.

[0369] All analyzed information, including emotional data, is used through analytical tools to develop promotional proposals for travel-related businesses and local governments. For example, if a user shows high emotional satisfaction at a particular location, that data can be used to optimize the promotional strategy.

[0370] As a concrete example, a user can take a picture of a receipt during their trip using their smartphone camera, and the device automatically analyzes it using OCR technology. The results are sent to a server, which reflects the analyzed expenses and visualizes them on the device. Furthermore, an emotion engine detects the user's enjoyment from their behavior at a restaurant and recommends options for subsequent dinners. In this way, the present invention plays a role in making the user's trip more fulfilling.

[0371] The following describes the processing flow.

[0372] Step 1:

[0373] Users save booking emails and receipt images related to their travels to their email inbox or a dedicated app.

[0374] Step 2:

[0375] The server interacts with the user's mailbox to automatically retrieve emails and image data related to their trip.

[0376] Step 3:

[0377] The server analyzes the retrieved email content using natural language processing technology to extract travel itineraries and reservation information.

[0378] Step 4:

[0379] The server uses optical character recognition technology to analyze the image data and extract expense information from the receipt image.

[0380] Step 5:

[0381] The terminal receives the analyzed data sent from the server and automatically classifies the information into categories using an information classification means.

[0382] Step 6:

[0383] Before traveling, the user sets their travel budget information on the device. The device then records this budget data.

[0384] Step 7:

[0385] The device tracks the user's spending in real time during travel and notifies the user using warning mechanisms when they are likely to exceed their budget.

[0386] Step 8:

[0387] The emotion engine activates and recognizes the user's emotional state by analyzing their voice, facial expressions, and text messages.

[0388] Step 9:

[0389] The emotion engine suggests optimal activities and plans to the user based on recognized emotions and the current travel situation.

[0390] Step 10:

[0391] After the trip ends, the server compiles all expenses within the group and uses expense management tools to fairly calculate how much each person will pay.

[0392] Step 11:

[0393] The terminal notifies the user of the calculated split amount and initiates the settlement process via the online money transfer service.

[0394] Step 12:

[0395] The server uses analyzed travel spending data, including sentiment data, to perform analysis in order to propose promotions to travel-related businesses and local governments.

[0396] This process allows users to simultaneously manage their travel expenses and enhance their emotionally-driven travel experience.

[0397] (Example 2)

[0398] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0399] Manually managing expenses during and after a trip is often time-consuming and prone to errors. Furthermore, it's difficult to enhance the experience based on emotions during the trip, highlighting the need for a system that enhances individual travel experiences. Conventional technologies lacked the means to comprehensively address these challenges.

[0400] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0401] In this invention, the server includes data extraction means for acquiring electronic communication and visual data and analyzing the information contained therein; information display means for classifying the extracted information into categories and presenting them to the user; and emotion analysis means for analyzing the user's emotions and making suggestions to improve the individual experience. This enables the automation of expense management during and after travel, as well as the improvement of the travel experience based on emotions.

[0402] "Electronic communications and visual data" refers to digital information, including emails and images, obtained from users and other sources.

[0403] "Data extraction means" refers to techniques for analyzing and extracting necessary information from electronic communication and visual data.

[0404] A "classification category" is a category used to group analyzed information based on specific criteria.

[0405] "Information display means" refers to functions and technologies for visually presenting analyzed information or classified categories to users.

[0406] An "alert system" is a technology that issues a warning when an anomaly occurs that does not meet the criteria set by the user.

[0407] A "cost management tool" is a function that allows for understanding the group's spending situation and calculating the amount each member should pay.

[0408] "Financial processing means" refers to technology for electronically settling payments based on calculated split amounts.

[0409] "Emotional analysis tools" refer to technologies that analyze a user's emotional state and provide optimal suggestions based on the results.

[0410] "Information utilization methods" refer to methods for making promotional proposals to travel agencies and local governments based on analyzed data.

[0411] "Visual recognition technology" refers to the technology used to understand the meaning of textual information and other data from images and to extract necessary information.

[0412] To implement this invention, it is first necessary to construct a system in which a server, terminal, and emotion analysis engine work together. This system is designed to improve the user's travel experience.

[0413] The server retrieves electronic communication data (e.g., emails containing travel itineraries and reservation information) and visual data (e.g., receipts and travel-related images) from the user's mailbox or dedicated app. This server incorporates natural language processing and optical character recognition technologies as data extraction methods. The server uses these technologies to analyze the acquired digital information and extract the necessary information.

[0414] The terminal receives analyzed information sent from the server, categorizes the data using an information display device, and presents it to the user. The user can use the terminal to set a budget before traveling. Based on this budget, the terminal utilizes an alert system and immediately displays a warning if spending during the trip exceeds the budget.

[0415] On the other hand, the emotion analysis engine analyzes voice and facial expression data acquired from the user in real time to understand the user's emotional state. The results of this emotion analysis are used to suggest activities and plans to improve the user's travel experience.

[0416] As a concrete example, consider a scenario where a user takes a picture of a receipt with their smartphone camera while traveling. This photographic data is immediately analyzed by the device using OCR technology, and the information is sent to a server. The server stores the analysis results in a database and visualizes them clearly on the device. Furthermore, the emotion engine can analyze the user's reaction to a restaurant they visited and suggest appropriate activities for the next step.

[0417] To support the overall functionality of this invention, users can input prompts using a generative AI model. For example, by using a prompt in the form of, "Please use OCR to analyze receipts I photographed during my trip and manage my expenses. Furthermore, please analyze my current mood and suggest activities," the system will provide the user with the most suitable suggestions.

[0418] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0419] Step 1: Data Acquisition

[0420] The server retrieves travel-related electronic and visual data from the user's mailbox or dedicated app. It uses an algorithm to filter relevant email subjects and access permissions to the user's email account as input. The output generates unanalyzed data including travel itineraries, accommodation bookings, and transportation information. The server temporarily stores this data and prepares it for analysis.

[0421] Step 2: Data Analysis

[0422] The server analyzes the acquired unanalyzed data using natural language processing and optical character recognition (OCR) technologies. The unanalyzed data obtained in step 1 is used as input. Text information is analyzed using natural language processing, and text is extracted from images using OCR. The output generates analyzed data containing travel reservation confirmation numbers, dates, and cost information.

[0423] Step 3: Information Classification

[0424] The terminal receives parsed data sent from the server and classifies the data into categories using a classification algorithm. It uses parsed data from the server as input. The terminal automatically classifies the data into categories such as "accommodation," "transportation," and "food," and generates a data structure as output to present the classified information to the user.

[0425] Step 4: Budget Management

[0426] Before traveling, the user enters their budget into the terminal. The terminal compares ongoing expenses with the set budget in real time. The inputs used are the user's set budget and the categorized expense data obtained in step 3. If the budget is exceeded, the terminal immediately displays a warning and generates a warning message as output.

[0427] Step 5: Emotion Analysis

[0428] The device acquires the user's voice and facial expression data and analyzes it in real time using an emotion analysis engine. It uses the user's sensor information and camera data as input. The emotion analysis engine identifies emotional states such as stress, joy, and excitement, and generates data that includes emotion-based improvement suggestions as output.

[0429] Step 6: Proposal Generation

[0430] The device suggests suitable travel plans and activities to the user based on emotional data obtained from the emotion analysis engine and the progress of the trip. The inputs used are the user's emotional data and travel information classified in step 3. The output generates a list of recommended plans and activities to enhance the user's experience.

[0431] Step 7: Expense settlement

[0432] The server aggregates spending data from all members during a group trip and calculates the split cost using expense management tools. It uses spending information from each group member as input. The output generates the exact amount each member should pay and online transfer instructions to support that payment. This information is sent to the terminal and displayed clearly to the user.

[0433] (Application Example 2)

[0434] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0435] Efficiently managing expenses and tracking spending during travel and outings, as well as improving the experience based on emotions, proved difficult. Furthermore, optimizing the experience through recommendations tailored to the user's emotional state and managing spending within groups presented challenges. Additionally, there was a lack of effective ways to utilize the analyzed emotional data.

[0436] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0437] In this invention, the server includes acquisition means for acquiring data and analyzing information, classification means for classifying and displaying the acquired information, and monitoring means for comparing a set budget with expenditures and issuing warnings. This enables users to efficiently manage expenses, track spending, and recommend choices based on their emotions.

[0438] "Data" refers to a collection or aggregate of information, and is what is processed by a system.

[0439] "Means of acquisition" refers to the function or method of collecting necessary data or information.

[0440] "Analysis" is the process of breaking down acquired data in order to understand its meaning and value.

[0441] A "classification method" is a method of organizing analyzed information into categories based on specific criteria.

[0442] "Display" refers to the act or device of visually presenting information to a user.

[0443] A "monitoring mechanism" is a function that continuously compares the budget with expenditures and issues a warning when an anomaly occurs.

[0444] A "management tool" is a system for appropriately organizing and controlling funds and expenditures within a group, and for performing necessary calculations.

[0445] A "settlement method" is a processing function for making payments or remittances based on calculation results.

[0446] "Emotional state" refers to data that indicates the user's psychological response and mood.

[0447] A "feedback mechanism" is a function that presents the user with the most suitable suggestions or options based on their analyzed emotional state.

[0448] "Analysis" refers to the process of examining acquired data in detail and extracting useful information from it.

[0449] "Optical character recognition technology" is a technology that automatically recognizes characters from image data and converts them into text data.

[0450] The system realizing this invention is designed to acquire data and provide the user with the optimal experience based on the analyzed information. The server acquires data sent from the user's device. This includes travel plans, purchase history, and sentiment data from voice and text. The server analyzes the acquired data using natural language processing libraries (e.g., spaCy, NLTK) and optical character recognition technology (e.g., Tesseract) to extract information. Furthermore, it utilizes sentiment recognition APIs (e.g., Microsoft Azure Emotion API) to understand the user's emotional state in real time and use this information to provide feedback.

[0451] The terminal receives information analyzed from the server and organizes it by category. This information is displayed intuitively through the user interface, allowing the user to check their budget management and spending status at any time.

[0452] Users can receive alerts from their device when they exceed their set budget, enabling efficient financial management. Furthermore, during group travel, the expense management system allows for the aggregation of the group's total expenses and the calculation of appropriate splitting amounts.

[0453] As a concrete example, a user sends an image taken with their smartphone camera to a server. This image is converted into text using OCR technology, and emotion recognition is used to analyze how the user felt in a particular situation. The analyzed information is evaluated by the user and used to optimize their next actions and choices.

[0454] An example of a prompt to be input to the generating AI model is, "Consider the user's current emotional state and suggest lunch options suitable for stress reduction." In this way, the present invention makes it possible to highly personalize the user's travel and outing experiences.

[0455] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0456] Step 1:

[0457] The server retrieves travel-related emails and image data from the user's device. The input is emails and images, and the output is data ready for analysis. Specifically, this involves scanning the mailbox and retrieving image data.

[0458] Step 2:

[0459] The server analyzes the acquired data using a natural language processing library (e.g., spaCy) to extract travel itineraries, reservation information, and other relevant details. The input is the data obtained in step 1, and the output is a well-organized set of information. This clarifies the user's travel plan.

[0460] Step 3:

[0461] The server uses OCR technology (e.g., Tesseract) to extract text from image data. The input is an image file, and the output is digital text. This allows paper receipts and ticket information to be treated as text data.

[0462] Step 4:

[0463] The terminal receives the analyzed information sent from the server and organizes it by category. The input is the data obtained in steps 2 and 3, and the output is the information classified by category. The information is visualized in a format suitable for the user interface.

[0464] Step 5:

[0465] The device monitors the user's set budget against actual spending and displays a warning if the budget is exceeded. Input is the user's budget information and spending data, and output is a warning message. This allows users to efficiently manage their spending.

[0466] Step 6:

[0467] The server aggregates spending data from the group and calculates the appropriate split amount. The input is each member's spending data, and the output is the calculated split amount. Users can then refer to this result to divide the payment equally.

[0468] Step 7:

[0469] The server uses an emotion recognition API to analyze the user's emotions in real time and generate feedback to improve the user experience. Input is emotion-related data from voice and text, and output is suggestions and recommended actions. Based on this, the user can make better decisions.

[0470] Step 8:

[0471] The user inputs a prompt into a generated AI model, and the system performs data calculations based on that input to provide the optimal food delivery option. The input is a prompt, and the output is a customized suggestion. A specific example of the prompt might be, "Consider the user's current emotional state and suggest a lunch option suitable for stress reduction." In this way, the data technology used enriches the user experience.

[0472] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0473] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0474] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0475] [Third Embodiment]

[0476] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0477] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0478] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0479] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0480] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0481] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0482] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0483] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0484] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0485] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0486] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0487] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0488] This invention is a system for automating expense management during and after travel, and is realized through the cooperation of a server, terminals, and users.

[0489] The server retrieves travel-related emails and image data from the user's mailbox or dedicated app. The retrieved data is then analyzed, for example, to extract information such as travel itinerary, accommodation, transportation, and expenses.

[0490] Next, the terminal automatically organizes the extracted information into categories using an information classification system and displays it on the interface for easy viewing by the user. Specifically, it is categorized into categories such as accommodation expenses, transportation expenses, and food expenses.

[0491] Users set their planned budget on the device before their trip. Based on this setting, the device monitors the user's spending in real time and issues an alert if the set budget is exceeded. This allows users to effectively manage their spending during their trip.

[0492] Furthermore, at the end of the trip, the server aggregates the expenses of the users who participated in the group trip and calculates the fair share. Based on the calculated amount, a settlement processing mechanism is used to automatically settle the payment, for example, via an online money transfer service.

[0493] Furthermore, the server uses the acquired and categorized spending data to propose promotions tailored to user preferences to travel-related businesses and local governments. This enables effective data-driven marketing.

[0494] As a concrete example, when a user books a hotel at their travel destination, the booking confirmation email is analyzed by the server, and details of the accommodation and price are automatically recorded as subordinate data. Subsequently, by taking photos of receipts from taxis and restaurants used during the trip, the device appropriately categorizes them as transportation and food expenses. After the trip, the server uses this data to distribute expenses among group members, and each person's share is settled using an online money transfer service. Through this series of processes, users are freed from the hassle of expense management, allowing them to enjoy their trip more freely.

[0495] The following describes the processing flow.

[0496] Step 1:

[0497] Users save travel-related emails and receipt images to a designated mailbox or a dedicated app.

[0498] Step 2:

[0499] The server accesses the specified mailbox or app folder to retrieve travel-related emails and image data. This data includes itinerary, fares, and booking information.

[0500] Step 3:

[0501] The server analyzes the content of the retrieved emails using natural language processing to extract important information, such as the departure date and time and the reservation amount.

[0502] Step 4:

[0503] The server uses OCR technology to extract text data from the receipt image and uses this information to verify the expenses.

[0504] Step 5:

[0505] The terminal receives the analyzed data sent from the server and automatically classifies it into categories (e.g., transportation expenses, accommodation expenses, food expenses) using an information classification system.

[0506] Step 6:

[0507] The user enters their travel budget into the device and begins budget management. This information is recorded on the device and used later for monitoring expenses.

[0508] Step 7:

[0509] The device tracks spending in real time based on user input and displays a warning to the user if the budget is exceeded.

[0510] Step 8:

[0511] After the trip ends, the server aggregates spending data from the group participants and uses an expense management system to calculate each person's share of the expenses.

[0512] Step 9:

[0513] The terminal displays the calculated split amount to the user and offers the option to settle the payment via an online money transfer service.

[0514] Step 10:

[0515] The server will process the payment using the selected payment method, based on instructions from the consenting user. This completes the entire payment process.

[0516] Step 11:

[0517] The server analyzes the expenditure data to make promotional proposals to travel-related businesses and local governments based on the analyzed data.

[0518] This series of steps allows users to efficiently manage and settle expenses during and after their trip.

[0519] (Example 1)

[0520] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0521] Managing travel expenses during and after a trip is often cumbersome and time-consuming, and fair cost sharing is particularly difficult in group travel. Furthermore, effective marketing activities for travel-related businesses require the collection and analysis of information based on user preferences, but there are limitations to doing this manually. To solve these problems, the present invention aims to provide an efficient system that integrates data acquisition, analysis, classification, expenditure management, settlement, and analysis.

[0522] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0523] In this invention, the server includes: information acquisition means for acquiring data using communication means and analyzing the information contained therein; information classification means for classifying the acquired information into higher-level categories and displaying them to the user; warning means for comparing the budget set by the user with expenditures and issuing a warning if the budget is exceeded; expenditure management means for managing expenditures within the community and calculating the amount to be shared; settlement processing means for processing remittances based on the calculated amount to be shared; analysis means for making proposals to stakeholders based on the analyzed expenditure data; means for extracting textual information from image data using visual recognition technology; and means for performing predictive analysis based on the analyzed data and generating a proposal model. This enables efficient expense management and marketing activities through the automated management and analysis of travel-related information.

[0524] "Communication means" is a general term for technologies or devices used to acquire and transmit electronic data.

[0525] "Information acquisition means" refers to the technology or process for analyzing and extracting necessary information from collected data.

[0526] "Information classification means" refers to a technology or algorithm for organizing and classifying acquired information into categories.

[0527] "Warning mechanisms" refer to functions that notify or alert users when specific conditions are met.

[0528] "Expenditure management tools" are techniques or processes for tracking expenditures within a community and calculating fair share of those expenditures.

[0529] "Settlement processing means" refers to the process of sending and receiving money based on the calculated share of the cost.

[0530] "Analysis tools" refer to techniques or algorithms used to analyze collected data and make suggestions or predictions tailored to specific purposes.

[0531] "Visual recognition technology" refers to technologies for extracting and recognizing text and other information from image data.

[0532] "Predictive analytics" is an analytical method used to predict future trends and behaviors based on past and present data.

[0533] A "proposal model" refers to a computational model that presents the optimal actions or choices based on the analysis results.

[0534] This travel expense management system is a comprehensive expense management tool realized through the collaboration of a server, terminals, and users. Specifically, the server uses communication methods to retrieve travel-related data from the user's email inbox or dedicated application. For example, when retrieving emails, a general mail server API is used. After retrieving the data, the server uses information retrieval methods to analyze the text information within the emails and the image data retrieved from cloud storage.

[0535] Visual recognition technology is used to analyze image data. This technology allows for the extraction of textual information from images such as receipts and reservation documents. Furthermore, the server uses information classification means to classify the extracted information into higher-level categories and transmits that information to the terminal.

[0536] The terminal receives information sent from the server and displays it to the user on its interface. Based on this information, the terminal has a function to compare the budget set by the user with actual spending, and notifies the user using a warning mechanism if the budget is exceeded. This means that the warning function implemented in the terminal is properly managed.

[0537] Users set a budget using an application on their device and monitor and manage expenses incurred during their trip. Upon completion of the trip, the server uses an expense management system to aggregate expenses within the group and calculate a fair share. A settlement processing system then automatically processes the payment based on the calculated share.

[0538] The analyzed spending data is further processed by analytical tools to support effective marketing activities, for example, by creating promotional proposals for travel-related businesses and relevant organizations.

[0539] For example, when a user books a hotel at their travel destination, the booking confirmation email is retrieved and analyzed by the server, and the accommodation information is displayed on the device. During the trip, simply taking a picture of the taxi receipt with the device automatically categorizes it as a transportation expense using visual recognition technology. This eliminates the need for cumbersome expense entry, allowing users to enjoy their trips more freely.

[0540] An example of a prompt for a generated AI model would be: "I would like to propose a system to help manage travel expenses. I want to develop an app that extracts necessary information from travel-related emails, categorizes expenses, and includes budget management functions. Please advise on how to implement this system."

[0541] This system allows travelers to manage and settle their expenses efficiently and easily, reducing financial stress during their trip and allowing them to enjoy their travels to the fullest.

[0542] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0543] Step 1:

[0544] The server uses communication methods to retrieve travel-related data from the user's mailbox or dedicated app. The input at this stage consists of the user's emails and image data in cloud storage. The server filters this data, identifying and collecting travel-related information. The collection process utilizes email server APIs, generating a dataset that is then passed to the information retrieval system.

[0545] Step 2:

[0546] The server uses information acquisition methods to analyze the text and image data of the acquired emails. The input consists of the email text and image files collected in step 1. The server uses natural language processing and visual recognition technologies to extract information such as travel itinerary, accommodation, and transportation, and generates structured data as output. Specifically, the Natural Language Understanding API and image recognition algorithms are executed.

[0547] Step 3:

[0548] The terminal receives structured data sent from the server and automatically organizes it into categories using an information classification system. The input for this step is the parsed information received from the server. The terminal generates a database of categories such as accommodation, transportation, and food expenses, and this is reflected in the interface as output. The information is displayed visually using a user interface such as React Native.

[0549] Step 4:

[0550] The user sets a planned budget on the device before their trip. The input is the budget amount set by the user in the application. The device monitors this setting in real time and triggers a warning function if spending exceeds the entered budget. The output is a push notification or alert triggered to the user.

[0551] Step 5:

[0552] At the end of the trip, the server uses an expense management system to aggregate the group's expenses and calculate a fair share. The input here is expense data from all participants sent from their terminals. The server runs a bill-splitting algorithm, calculates each participant's share as output, and passes it to the settlement processing system.

[0553] Step 6:

[0554] The server settles the calculated share amount using a settlement processing method via an online money transfer service. This process uses the calculated share amount as input. In actual operation, it calls a money transfer API and completes the transfer process as output.

[0555] Step 7:

[0556] The server uses analytical tools based on the analyzed data to propose promotions to stakeholders. Inputs include user preference information and spending trends. The server performs predictive analysis using an AI model, generates promotion strategies as output, and provides feedback to travel-related businesses. When using the generating AI model, these proposals are realized by appropriately creating prompt statements and inputting them into the model.

[0557] (Application Example 1)

[0558] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0559] Managing expenses during and after travel is cumbersome for many people, and group travel, in particular, presents challenges due to the complexity of organizing expenses, splitting costs, and making payments. Furthermore, individual budget management is not easy, and there is a risk of exceeding budgets, so real-time spending monitoring is required. Another challenge is to utilize the large amount of data generated during travel and provide feedback to travel-related businesses and the local economy.

[0560] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0561] This invention includes an analysis means for acquiring email and image information and analyzing the data contained therein; a display means for classifying the acquired analysis data and displaying it to the user; a notification means for comparing the budget set by the user with the expenditure and notifying the user when the budget is exceeded; an expense management means for managing expenses within the group and calculating shared expenses; a settlement means for executing remittance processing based on the calculated shared expenses; and a tracking means for integrating schedule and expense monitoring and automatically tracking expenses during travel. This makes it possible to efficiently manage expenses during travel, support the user's budget management, and automate the calculation and payment of split expenses during group travel. It can also provide effective feedback to travel-related businesses and the local economy.

[0562] "Analysis means" refers to a means that acquires email information and image information, analyzes the data contained therein, and extracts meaningful information.

[0563] A "display means" is a means that has the function of showing the acquired analysis data on the screen in a way that is easy for the user to understand.

[0564] A "notification mechanism" is a function that issues an alert when the user's set budget is exceeded, serving as a means to draw the user's attention.

[0565] An "expense management system" is a means of organizing expenses within a group and calculating shared expenses.

[0566] A "payment method" is a means of automatically processing a transfer based on the calculated shared cost and completing the payment.

[0567] A "tracking system" is a means of monitoring schedules and expenses, and recording and managing travel expenses in real time.

[0568] This invention is a system for streamlining expense management during and after travel. The server retrieves travel-related email and image information using the user's mailbox or a dedicated app. This data is analyzed using the Google Cloud Vision API and natural language processing (NLP) technology to extract information such as travel itinerary, accommodation, transportation, and expenses.

[0569] The device organizes the analyzed information into categories, classifying it into accommodation, transportation, and food expenses, and displays it on the interface. Users set a budget on the device before traveling, and the device monitors spending in real time. A feature that immediately notifies users if they exceed their budget makes it easy for them to manage their spending.

[0570] Furthermore, the server collects expenses from users participating in the group trip and calculates a fair and shared cost. Based on this calculation, it automatically settles payments using online payment services such as the Stripe API.

[0571] For example, if a user scans a restaurant receipt they took during their trip, the system can automatically categorize the restaurant expense as food expenses, helping them stay within their budget. Furthermore, it utilizes a generative AI model to create promotional suggestions tailored to the user's preferences in real time, for travel-related businesses and regions.

[0572] An example of a prompt message would be: "Set a budget for your next trip and track expenses in real time. You will be notified if you exceed your budget. After the trip, divide the group travel expenses equally and settle the accounts online."

[0573] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0574] Step 1:

[0575] The server retrieves travel-related email and image information from the user's mailbox or dedicated app. The input consists of the user's emails and image data, and the output is a collection of this data. Email information is retrieved using the POP3 or IMAP protocol, while image information is uploaded directly via the app.

[0576] Step 2:

[0577] The server processes image data using the Google Cloud Vision API and parses email data using natural language processing (NLP) techniques. The input is the email and image data obtained in step 1, and the output is extracted travel itinerary, accommodation, transportation, and cost information. Specifically, it converts receipt information from image data into text and extracts travel booking details from emails.

[0578] Step 3:

[0579] The terminal receives analysis information sent from the server and classifies it into categories such as accommodation, transportation, and food expenses. The input is the analyzed information, and the output is information organized by category. A graphical user interface (GUI) is provided to display the classification results on the user interface for easier operation.

[0580] Step 4:

[0581] The user registers their budget information, set before their trip, on their device. The input is the budget amount and its details, while the output is the budget data stored within the device. Specifically, this is achieved by the user entering the budget figures into a smartphone application.

[0582] Step 5:

[0583] The device monitors travel expenses in real time and notifies the user if an overspending is detected. Input is real-time spending data, and output is an alert for budget overruns. Notifications are sent to the smartphone as push notifications.

[0584] Step 6:

[0585] The server aggregates the expenses of all group members at the end of the trip and calculates the split amount. The input is the expense data of each individual user, and the output is the calculated split amount. Specifically, the calculation is performed using a spreadsheet-like data structure.

[0586] Step 7:

[0587] The server automatically performs settlement using an online payment service based on the calculation results. The input is the calculated split amount and each user's payment information, and the output is the completed payment result. The payment is processed using the Stripe API, and the user receives a notification.

[0588] Step 8:

[0589] The server generates reports that propose promotions to travel-related businesses and local economies based on analyzed spending data. The input is spending data and information about user preferences, and the output is a promotional report customized by a generating AI model. An example of a prompt message might be, "Set a budget for your next trip and track your expenses in real time..."

[0590] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0591] This invention is a system that automates expense management during and after travel and aims to improve the user experience based on emotions, and is realized through the cooperation of a server, terminal, emotion engine, and user.

[0592] The server retrieves travel-related emails and image data from the user's mailbox or dedicated app. This includes information such as travel itinerary, accommodation reservations, transportation methods, and expenses. This data is then analyzed using natural language processing and optical character recognition technology to extract important information.

[0593] The terminal receives analyzed data sent from the server and automatically categorizes it using an information classification system. Users set a budget before traveling, and this information is recorded on the terminal. The terminal tracks spending in real time and notifies the user using an alert system if the budget is exceeded.

[0594] The emotion engine analyzes user data, such as voice, text, and facial expressions, to recognize the user's emotions in real time. This emotion data includes items such as joy, anxiety, and stress. Based on this information, the emotion engine presents the user with the most appropriate travel plan and activities for the situation, helping to improve the experience.

[0595] If users are traveling in a group, the server aggregates spending data within the group and calculates the amount to be split using the spending management system. The calculation result is displayed on the terminal, and automatic settlement is performed using an online money transfer service.

[0596] All analyzed information, including emotional data, is used through analytical tools to develop promotional proposals for travel-related businesses and local governments. For example, if a user shows high emotional satisfaction at a particular location, that data can be used to optimize the promotional strategy.

[0597] As a concrete example, a user can take a picture of a receipt during their trip using their smartphone camera, and the device automatically analyzes it using OCR technology. The results are sent to a server, which reflects the analyzed expenses and visualizes them on the device. Furthermore, an emotion engine detects the user's enjoyment from their behavior at a restaurant and recommends options for subsequent dinners. In this way, the present invention plays a role in making the user's trip more fulfilling.

[0598] The following describes the processing flow.

[0599] Step 1:

[0600] Users save booking emails and receipt images related to their travels to their email inbox or a dedicated app.

[0601] Step 2:

[0602] The server interacts with the user's mailbox to automatically retrieve emails and image data related to their trip.

[0603] Step 3:

[0604] The server analyzes the retrieved email content using natural language processing technology to extract travel itineraries and reservation information.

[0605] Step 4:

[0606] The server uses optical character recognition technology to analyze the image data and extract expense information from the receipt image.

[0607] Step 5:

[0608] The terminal receives the analyzed data sent from the server and automatically classifies the information into categories using an information classification means.

[0609] Step 6:

[0610] Before traveling, the user sets their travel budget information on the device. The device then records this budget data.

[0611] Step 7:

[0612] The device tracks the user's spending in real time during travel and notifies the user using warning mechanisms when they are likely to exceed their budget.

[0613] Step 8:

[0614] The emotion engine activates and recognizes the user's emotional state by analyzing their voice, facial expressions, and text messages.

[0615] Step 9:

[0616] The emotion engine suggests optimal activities and plans to the user based on recognized emotions and the current travel situation.

[0617] Step 10:

[0618] After the trip ends, the server compiles all expenses within the group and uses expense management tools to fairly calculate how much each person will pay.

[0619] Step 11:

[0620] The terminal notifies the user of the calculated split amount and initiates the settlement process via the online money transfer service.

[0621] Step 12:

[0622] The server uses analyzed travel spending data, including sentiment data, to perform analysis in order to propose promotions to travel-related businesses and local governments.

[0623] This process allows users to simultaneously manage their travel expenses and enhance their emotionally-driven travel experience.

[0624] (Example 2)

[0625] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0626] Manually managing expenses during and after a trip is often time-consuming and prone to errors. Furthermore, it's difficult to enhance the experience based on emotions during the trip, highlighting the need for a system that enhances individual travel experiences. Conventional technologies lacked the means to comprehensively address these challenges.

[0627] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0628] In this invention, the server includes data extraction means for acquiring electronic communication and visual data and analyzing the information contained therein; information display means for classifying the extracted information into categories and presenting them to the user; and emotion analysis means for analyzing the user's emotions and making suggestions to improve the individual experience. This enables the automation of expense management during and after travel, as well as the improvement of the travel experience based on emotions.

[0629] "Electronic communications and visual data" refers to digital information, including emails and images, obtained from users and other sources.

[0630] "Data extraction means" refers to techniques for analyzing and extracting necessary information from electronic communication and visual data.

[0631] A "classification category" is a category used to group analyzed information based on specific criteria.

[0632] "Information display means" refers to functions and technologies for visually presenting analyzed information or classified categories to users.

[0633] An "alert system" is a technology that issues a warning when an anomaly occurs that does not meet the criteria set by the user.

[0634] A "cost management tool" is a function that allows for understanding the group's spending situation and calculating the amount each member should pay.

[0635] "Financial processing means" refers to technology for electronically settling payments based on calculated split amounts.

[0636] "Emotional analysis tools" refer to technologies that analyze a user's emotional state and provide optimal suggestions based on the results.

[0637] "Information utilization methods" refer to methods for making promotional proposals to travel agencies and local governments based on analyzed data.

[0638] "Visual recognition technology" refers to the technology used to understand the meaning of textual information and other data from images and to extract necessary information.

[0639] To implement this invention, it is first necessary to construct a system in which a server, terminal, and emotion analysis engine work together. This system is designed to improve the user's travel experience.

[0640] The server retrieves electronic communication data (e.g., emails containing travel itineraries and reservation information) and visual data (e.g., receipts and travel-related images) from the user's mailbox or dedicated app. This server incorporates natural language processing and optical character recognition technologies as data extraction methods. The server uses these technologies to analyze the acquired digital information and extract the necessary information.

[0641] The terminal receives analyzed information sent from the server, categorizes the data using an information display device, and presents it to the user. The user can use the terminal to set a budget before traveling. Based on this budget, the terminal utilizes an alert system and immediately displays a warning if spending during the trip exceeds the budget.

[0642] On the other hand, the emotion analysis engine analyzes voice and facial expression data acquired from the user in real time to understand the user's emotional state. The results of this emotion analysis are used to suggest activities and plans to improve the user's travel experience.

[0643] As a concrete example, consider a scenario where a user takes a picture of a receipt with their smartphone camera while traveling. This photographic data is immediately analyzed by the device using OCR technology, and the information is sent to a server. The server stores the analysis results in a database and visualizes them clearly on the device. Furthermore, the emotion engine can analyze the user's reaction to a restaurant they visited and suggest appropriate activities for the next step.

[0644] To support the overall functionality of this invention, users can input prompts using a generative AI model. For example, by using a prompt in the form of, "Please use OCR to analyze receipts I photographed during my trip and manage my expenses. Furthermore, please analyze my current mood and suggest activities," the system will provide the user with the most suitable suggestions.

[0645] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0646] Step 1: Data Acquisition

[0647] The server retrieves travel-related electronic and visual data from the user's mailbox or dedicated app. It uses an algorithm to filter relevant email subjects and access permissions to the user's email account as input. The output generates unanalyzed data including travel itineraries, accommodation bookings, and transportation information. The server temporarily stores this data and prepares it for analysis.

[0648] Step 2: Data Analysis

[0649] The server analyzes the acquired unanalyzed data using natural language processing and optical character recognition (OCR) technologies. The unanalyzed data obtained in step 1 is used as input. Text information is analyzed using natural language processing, and text is extracted from images using OCR. The output generates analyzed data containing travel reservation confirmation numbers, dates, and cost information.

[0650] Step 3: Information Classification

[0651] The terminal receives parsed data sent from the server and classifies the data into categories using a classification algorithm. It uses parsed data from the server as input. The terminal automatically classifies the data into categories such as "accommodation," "transportation," and "food," and generates a data structure as output to present the classified information to the user.

[0652] Step 4: Budget Management

[0653] Before traveling, the user enters their budget into the terminal. The terminal compares ongoing expenses with the set budget in real time. The inputs used are the user's set budget and the categorized expense data obtained in step 3. If the budget is exceeded, the terminal immediately displays a warning and generates a warning message as output.

[0654] Step 5: Emotion Analysis

[0655] The device acquires the user's voice and facial expression data and analyzes it in real time using an emotion analysis engine. It uses the user's sensor information and camera data as input. The emotion analysis engine identifies emotional states such as stress, joy, and excitement, and generates data that includes emotion-based improvement suggestions as output.

[0656] Step 6: Proposal Generation

[0657] The device suggests suitable travel plans and activities to the user based on emotional data obtained from the emotion analysis engine and the progress of the trip. The inputs used are the user's emotional data and travel information classified in step 3. The output generates a list of recommended plans and activities to enhance the user's experience.

[0658] Step 7: Expense settlement

[0659] The server aggregates spending data from all members during a group trip and calculates the split cost using expense management tools. It uses spending information from each group member as input. The output generates the exact amount each member should pay and online transfer instructions to support that payment. This information is sent to the terminal and displayed clearly to the user.

[0660] (Application Example 2)

[0661] Next, we will explain Application Example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0662] Efficiently managing expenses and tracking spending during travel and outings, as well as improving the experience based on emotions, proved difficult. Furthermore, optimizing the experience through recommendations tailored to the user's emotional state and managing spending within groups presented challenges. Additionally, there was a lack of effective ways to utilize the analyzed emotional data.

[0663] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0664] In this invention, the server includes acquisition means for acquiring data and analyzing information, classification means for classifying and displaying the acquired information, and monitoring means for comparing a set budget with expenditures and issuing warnings. This enables users to efficiently manage expenses, track spending, and recommend choices based on their emotions.

[0665] "Data" refers to a collection or aggregate of information, and is what is processed by a system.

[0666] "Means of acquisition" refers to the function or method of collecting necessary data or information.

[0667] "Analysis" is the process of breaking down acquired data in order to understand its meaning and value.

[0668] A "classification method" is a method of organizing analyzed information into categories based on specific criteria.

[0669] "Display" refers to the act or device of visually presenting information to a user.

[0670] A "monitoring mechanism" is a function that continuously compares the budget with expenditures and issues a warning when an anomaly occurs.

[0671] A "management tool" is a system for appropriately organizing and controlling funds and expenditures within a group, and for performing necessary calculations.

[0672] A "settlement method" is a processing function for making payments or remittances based on calculation results.

[0673] "Emotional state" refers to data that indicates the user's psychological response and mood.

[0674] A "feedback mechanism" is a function that presents the user with the most suitable suggestions or options based on their analyzed emotional state.

[0675] "Analysis" refers to the process of examining acquired data in detail and extracting useful information from it.

[0676] "Optical character recognition technology" is a technology that automatically recognizes characters from image data and converts them into text data.

[0677] The system realizing this invention is designed to acquire data and provide the user with the optimal experience based on the analyzed information. The server acquires data sent from the user's device. This includes travel plans, purchase history, and sentiment data from voice and text. The server analyzes the acquired data using natural language processing libraries (e.g., spaCy, NLTK) and optical character recognition technology (e.g., Tesseract) to extract information. Furthermore, it utilizes sentiment recognition APIs (e.g., Microsoft Azure Emotion API) to understand the user's emotional state in real time and use this information to provide feedback.

[0678] The terminal receives information analyzed from the server and organizes it by category. This information is displayed intuitively through the user interface, allowing the user to check their budget management and spending status at any time.

[0679] Users can receive alerts from their device when they exceed their set budget, enabling efficient financial management. Furthermore, during group travel, the expense management system allows for the aggregation of the group's total expenses and the calculation of appropriate splitting amounts.

[0680] As a concrete example, a user sends an image taken with their smartphone camera to a server. This image is converted into text using OCR technology, and emotion recognition is used to analyze how the user felt in a particular situation. The analyzed information is evaluated by the user and used to optimize their next actions and choices.

[0681] An example of a prompt to be input to the generating AI model is, "Consider the user's current emotional state and suggest lunch options suitable for stress reduction." In this way, the present invention makes it possible to highly personalize the user's travel and outing experiences.

[0682] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0683] Step 1:

[0684] The server retrieves travel-related emails and image data from the user's device. The input is emails and images, and the output is data ready for analysis. Specifically, this involves scanning the mailbox and retrieving image data.

[0685] Step 2:

[0686] The server analyzes the acquired data using a natural language processing library (e.g., spaCy) to extract travel itineraries, reservation information, and other relevant details. The input is the data obtained in step 1, and the output is a well-organized set of information. This clarifies the user's travel plan.

[0687] Step 3:

[0688] The server uses OCR technology (e.g., Tesseract) to extract text from image data. The input is an image file, and the output is digital text. This allows paper receipts and ticket information to be treated as text data.

[0689] Step 4:

[0690] The terminal receives the analyzed information sent from the server and organizes it by category. The input is the data obtained in steps 2 and 3, and the output is the information classified by category. The information is visualized in a format suitable for the user interface.

[0691] Step 5:

[0692] The device monitors the user's set budget against actual spending and displays a warning if the budget is exceeded. Input is the user's budget information and spending data, and output is a warning message. This allows users to efficiently manage their spending.

[0693] Step 6:

[0694] The server aggregates spending data from the group and calculates the appropriate split amount. The input is each member's spending data, and the output is the calculated split amount. Users can then refer to this result to divide the payment equally.

[0695] Step 7:

[0696] The server uses an emotion recognition API to analyze the user's emotions in real time and generate feedback to improve the user experience. Input is emotion-related data from voice and text, and output is suggestions and recommended actions. Based on this, the user can make better decisions.

[0697] Step 8:

[0698] The user inputs a prompt into a generated AI model, and the system performs data calculations based on that input to provide the optimal food delivery option. The input is a prompt, and the output is a customized suggestion. A specific example of the prompt might be, "Consider the user's current emotional state and suggest a lunch option suitable for stress reduction." In this way, the data technology used enriches the user experience.

[0699] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0700] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0701] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0702] [Fourth Embodiment]

[0703] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0704] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0705] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0706] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0707] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0708] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0709] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0710] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0711] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0712] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0713] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0714] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0715] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0716] This invention is a system for automating expense management during and after travel, and is realized through the cooperation of a server, terminals, and users.

[0717] The server retrieves travel-related emails and image data from the user's mailbox or dedicated app. The retrieved data is then analyzed, for example, to extract information such as travel itinerary, accommodation, transportation, and expenses.

[0718] Next, the terminal automatically organizes the extracted information into categories using an information classification system and displays it on the interface for easy viewing by the user. Specifically, it is categorized into categories such as accommodation expenses, transportation expenses, and food expenses.

[0719] Users set their planned budget on the device before their trip. Based on this setting, the device monitors the user's spending in real time and issues an alert if the set budget is exceeded. This allows users to effectively manage their spending during their trip.

[0720] Furthermore, at the end of the trip, the server aggregates the expenses of the users who participated in the group trip and calculates the fair share. Based on the calculated amount, a settlement processing mechanism is used to automatically settle the payment, for example, via an online money transfer service.

[0721] Furthermore, the server uses the acquired and categorized spending data to propose promotions tailored to user preferences to travel-related businesses and local governments. This enables effective data-driven marketing.

[0722] As a concrete example, when a user books a hotel at their travel destination, the booking confirmation email is analyzed by the server, and details of the accommodation and price are automatically recorded as subordinate data. Subsequently, by taking photos of receipts from taxis and restaurants used during the trip, the device appropriately categorizes them as transportation and food expenses. After the trip, the server uses this data to distribute expenses among group members, and each person's share is settled using an online money transfer service. Through this series of processes, users are freed from the hassle of expense management, allowing them to enjoy their trip more freely.

[0723] The following describes the processing flow.

[0724] Step 1:

[0725] Users save travel-related emails and receipt images to a designated mailbox or a dedicated app.

[0726] Step 2:

[0727] The server accesses the specified mailbox or app folder to retrieve travel-related emails and image data. This data includes itinerary, fares, and booking information.

[0728] Step 3:

[0729] The server analyzes the content of the retrieved emails using natural language processing to extract important information, such as the departure date and time and the reservation amount.

[0730] Step 4:

[0731] The server uses OCR technology to extract text data from the receipt image and uses this information to verify the expenses.

[0732] Step 5:

[0733] The terminal receives the analyzed data sent from the server and automatically classifies it into categories (e.g., transportation expenses, accommodation expenses, food expenses) using an information classification system.

[0734] Step 6:

[0735] The user enters their travel budget into the device and begins budget management. This information is recorded on the device and used later for monitoring expenses.

[0736] Step 7:

[0737] The device tracks spending in real time based on user input and displays a warning to the user if the budget is exceeded.

[0738] Step 8:

[0739] After the trip ends, the server aggregates spending data from the group participants and uses an expense management system to calculate each person's share of the expenses.

[0740] Step 9:

[0741] The terminal displays the calculated split amount to the user and offers the option to settle the payment via an online money transfer service.

[0742] Step 10:

[0743] The server will process the payment using the selected payment method, based on instructions from the consenting user. This completes the entire payment process.

[0744] Step 11:

[0745] The server analyzes the expenditure data to make promotional proposals to travel-related businesses and local governments based on the analyzed data.

[0746] This series of steps allows users to efficiently manage and settle expenses during and after their trip.

[0747] (Example 1)

[0748] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0749] Managing travel expenses during and after a trip is often cumbersome and time-consuming, and fair cost sharing is particularly difficult in group travel. Furthermore, effective marketing activities for travel-related businesses require the collection and analysis of information based on user preferences, but there are limitations to doing this manually. To solve these problems, the present invention aims to provide an efficient system that integrates data acquisition, analysis, classification, expenditure management, settlement, and analysis.

[0750] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0751] In this invention, the server includes: information acquisition means for acquiring data using communication means and analyzing the information contained therein; information classification means for classifying the acquired information into higher-level categories and displaying them to the user; warning means for comparing the budget set by the user with expenditures and issuing a warning if the budget is exceeded; expenditure management means for managing expenditures within the community and calculating the amount to be shared; settlement processing means for processing remittances based on the calculated amount to be shared; analysis means for making proposals to stakeholders based on the analyzed expenditure data; means for extracting textual information from image data using visual recognition technology; and means for performing predictive analysis based on the analyzed data and generating a proposal model. This enables efficient expense management and marketing activities through the automated management and analysis of travel-related information.

[0752] "Communication means" is a general term for technologies or devices used to acquire and transmit electronic data.

[0753] "Information acquisition means" refers to the technology or process for analyzing and extracting necessary information from collected data.

[0754] "Information classification means" refers to a technology or algorithm for organizing and classifying acquired information into categories.

[0755] "Warning mechanisms" refer to functions that notify or alert users when specific conditions are met.

[0756] "Expenditure management tools" are techniques or processes for tracking expenditures within a community and calculating fair share of those expenditures.

[0757] "Settlement processing means" refers to the process of sending and receiving money based on the calculated share of the cost.

[0758] "Analysis tools" refer to techniques or algorithms used to analyze collected data and make suggestions or predictions tailored to specific purposes.

[0759] "Visual recognition technology" refers to technologies for extracting and recognizing text and other information from image data.

[0760] "Predictive analytics" is an analytical method used to predict future trends and behaviors based on past and present data.

[0761] A "proposal model" refers to a computational model that presents the optimal actions or choices based on the analysis results.

[0762] This travel expense management system is a comprehensive expense management tool realized through the collaboration of a server, terminals, and users. Specifically, the server uses communication methods to retrieve travel-related data from the user's email inbox or dedicated application. For example, when retrieving emails, a general mail server API is used. After retrieving the data, the server uses information retrieval methods to analyze the text information within the emails and the image data retrieved from cloud storage.

[0763] Visual recognition technology is used to analyze image data. This technology allows for the extraction of textual information from images such as receipts and reservation documents. Furthermore, the server uses information classification means to classify the extracted information into higher-level categories and transmits that information to the terminal.

[0764] The terminal receives information sent from the server and displays it to the user on its interface. Based on this information, the terminal has a function to compare the budget set by the user with actual spending, and notifies the user using a warning mechanism if the budget is exceeded. This means that the warning function implemented in the terminal is properly managed.

[0765] Users set a budget using an application on their device and monitor and manage expenses incurred during their trip. Upon completion of the trip, the server uses an expense management system to aggregate expenses within the group and calculate a fair share. A settlement processing system then automatically processes the payment based on the calculated share.

[0766] The analyzed spending data is further processed by analytical tools to support effective marketing activities, for example, by creating promotional proposals for travel-related businesses and relevant organizations.

[0767] For example, when a user books a hotel at their travel destination, the booking confirmation email is retrieved and analyzed by the server, and the accommodation information is displayed on the device. During the trip, simply taking a picture of the taxi receipt with the device automatically categorizes it as a transportation expense using visual recognition technology. This eliminates the need for cumbersome expense entry, allowing users to enjoy their trips more freely.

[0768] An example of a prompt for a generated AI model would be: "I would like to propose a system to help manage travel expenses. I want to develop an app that extracts necessary information from travel-related emails, categorizes expenses, and includes budget management functions. Please advise on how to implement this system."

[0769] This system allows travelers to manage and settle their expenses efficiently and easily, reducing financial stress during their trip and allowing them to enjoy their travels to the fullest.

[0770] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0771] Step 1:

[0772] The server uses communication methods to retrieve travel-related data from the user's mailbox or dedicated app. The input at this stage consists of the user's emails and image data in cloud storage. The server filters this data, identifying and collecting travel-related information. The collection process utilizes email server APIs, generating a dataset that is then passed to the information retrieval system.

[0773] Step 2:

[0774] The server uses information acquisition methods to analyze the text and image data of the acquired emails. The input consists of the email text and image files collected in step 1. The server uses natural language processing and visual recognition technologies to extract information such as travel itinerary, accommodation, and transportation, and generates structured data as output. Specifically, the Natural Language Understanding API and image recognition algorithms are executed.

[0775] Step 3:

[0776] The terminal receives structured data sent from the server and automatically organizes it into categories using an information classification system. The input for this step is the parsed information received from the server. The terminal generates a database of categories such as accommodation, transportation, and food expenses, and this is reflected in the interface as output. The information is displayed visually using a user interface such as React Native.

[0777] Step 4:

[0778] The user sets a planned budget on the device before their trip. The input is the budget amount set by the user in the application. The device monitors this setting in real time and triggers a warning function if spending exceeds the entered budget. The output is a push notification or alert triggered to the user.

[0779] Step 5:

[0780] At the end of the trip, the server uses an expense management system to aggregate the group's expenses and calculate a fair share. The input here is expense data from all participants sent from their terminals. The server runs a bill-splitting algorithm, calculates each participant's share as output, and passes it to the settlement processing system.

[0781] Step 6:

[0782] The server settles the calculated share amount using a settlement processing method via an online money transfer service. This process uses the calculated share amount as input. In actual operation, it calls a money transfer API and completes the transfer process as output.

[0783] Step 7:

[0784] The server uses analytical tools based on the analyzed data to propose promotions to stakeholders. Inputs include user preference information and spending trends. The server performs predictive analysis using an AI model, generates promotion strategies as output, and provides feedback to travel-related businesses. When using the generating AI model, these proposals are realized by appropriately creating prompt statements and inputting them into the model.

[0785] (Application Example 1)

[0786] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0787] Managing expenses during and after travel is cumbersome for many people, and group travel, in particular, presents challenges due to the complexity of organizing expenses, splitting costs, and making payments. Furthermore, individual budget management is not easy, and there is a risk of exceeding budgets, so real-time spending monitoring is required. Another challenge is to utilize the large amount of data generated during travel and provide feedback to travel-related businesses and the local economy.

[0788] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0789] This invention includes an analysis means for acquiring email and image information and analyzing the data contained therein; a display means for classifying the acquired analysis data and displaying it to the user; a notification means for comparing the budget set by the user with the expenditure and notifying the user when the budget is exceeded; an expense management means for managing expenses within the group and calculating shared expenses; a settlement means for executing remittance processing based on the calculated shared expenses; and a tracking means for integrating schedule and expense monitoring and automatically tracking expenses during travel. This makes it possible to efficiently manage expenses during travel, support the user's budget management, and automate the calculation and payment of split expenses during group travel. It can also provide effective feedback to travel-related businesses and the local economy.

[0790] "Analysis means" refers to a means that acquires email information and image information, analyzes the data contained therein, and extracts meaningful information.

[0791] A "display means" is a means that has the function of showing the acquired analysis data on the screen in a way that is easy for the user to understand.

[0792] A "notification mechanism" is a function that issues an alert when the user's set budget is exceeded, serving as a means to draw the user's attention.

[0793] An "expense management system" is a means of organizing expenses within a group and calculating shared expenses.

[0794] A "payment method" is a means of automatically processing a transfer based on the calculated shared cost and completing the payment.

[0795] A "tracking system" is a means of monitoring schedules and expenses, and recording and managing travel expenses in real time.

[0796] This invention is a system for streamlining expense management during and after travel. The server retrieves travel-related email and image information using the user's mailbox or a dedicated app. This data is analyzed using the Google Cloud Vision API and natural language processing (NLP) technology to extract information such as travel itinerary, accommodation, transportation, and expenses.

[0797] The device organizes the analyzed information into categories, classifying it into accommodation, transportation, and food expenses, and displays it on the interface. Users set a budget on the device before traveling, and the device monitors spending in real time. A feature that immediately notifies users if they exceed their budget makes it easy for them to manage their spending.

[0798] Furthermore, the server collects expenses from users participating in the group trip and calculates a fair and shared cost. Based on this calculation, it automatically settles payments using online payment services such as the Stripe API.

[0799] For example, if a user scans a restaurant receipt they took during their trip, the system can automatically categorize the restaurant expense as food expenses, helping them stay within their budget. Furthermore, it utilizes a generative AI model to create promotional suggestions tailored to the user's preferences in real time, for travel-related businesses and regions.

[0800] An example of a prompt message would be: "Set a budget for your next trip and track expenses in real time. You will be notified if you exceed your budget. After the trip, divide the group travel expenses equally and settle the accounts online."

[0801] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0802] Step 1:

[0803] The server retrieves travel-related email and image information from the user's mailbox or dedicated app. The input consists of the user's emails and image data, and the output is a collection of this data. Email information is retrieved using the POP3 or IMAP protocol, while image information is uploaded directly via the app.

[0804] Step 2:

[0805] The server processes image data using the Google Cloud Vision API and parses email data using natural language processing (NLP) techniques. The input is the email and image data obtained in step 1, and the output is extracted travel itinerary, accommodation, transportation, and cost information. Specifically, it converts receipt information from image data into text and extracts travel booking details from emails.

[0806] Step 3:

[0807] The terminal receives analysis information sent from the server and classifies it into categories such as accommodation, transportation, and food expenses. The input is the analyzed information, and the output is information organized by category. A graphical user interface (GUI) is provided to display the classification results on the user interface for easier operation.

[0808] Step 4:

[0809] The user registers their budget information, set before their trip, on their device. The input is the budget amount and its details, while the output is the budget data stored within the device. Specifically, this is achieved by the user entering the budget figures into a smartphone application.

[0810] Step 5:

[0811] The device monitors travel expenses in real time and notifies the user if an overspending is detected. Input is real-time spending data, and output is an alert for budget overruns. Notifications are sent to the smartphone as push notifications.

[0812] Step 6:

[0813] The server aggregates the expenses of all group members at the end of the trip and calculates the split amount. The input is the expense data of each individual user, and the output is the calculated split amount. Specifically, the calculation is performed using a spreadsheet-like data structure.

[0814] Step 7:

[0815] The server automatically performs settlement using an online payment service based on the calculation results. The input is the calculated split amount and each user's payment information, and the output is the completed payment result. The payment is processed using the Stripe API, and the user receives a notification.

[0816] Step 8:

[0817] The server generates reports that propose promotions to travel-related businesses and local economies based on analyzed spending data. The input is spending data and information about user preferences, and the output is a promotional report customized by a generating AI model. An example of a prompt message might be, "Set a budget for your next trip and track your expenses in real time..."

[0818] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0819] This invention is a system that automates expense management during and after travel and aims to improve the user experience based on emotions, and is realized through the cooperation of a server, terminal, emotion engine, and user.

[0820] The server retrieves travel-related emails and image data from the user's mailbox or dedicated app. This includes information such as travel itinerary, accommodation reservations, transportation methods, and expenses. This data is then analyzed using natural language processing and optical character recognition technology to extract important information.

[0821] The terminal receives analyzed data sent from the server and automatically categorizes it using an information classification system. Users set a budget before traveling, and this information is recorded on the terminal. The terminal tracks spending in real time and notifies the user using an alert system if the budget is exceeded.

[0822] The emotion engine analyzes user data, such as voice, text, and facial expressions, to recognize the user's emotions in real time. This emotion data includes items such as joy, anxiety, and stress. Based on this information, the emotion engine presents the user with the most appropriate travel plan and activities for the situation, helping to improve the experience.

[0823] If users are traveling in a group, the server aggregates spending data within the group and calculates the amount to be split using the spending management system. The calculation result is displayed on the terminal, and automatic settlement is performed using an online money transfer service.

[0824] All analyzed information, including emotional data, is used through analytical tools to develop promotional proposals for travel-related businesses and local governments. For example, if a user shows high emotional satisfaction at a particular location, that data can be used to optimize the promotional strategy.

[0825] As a concrete example, a user can take a picture of a receipt during their trip using their smartphone camera, and the device automatically analyzes it using OCR technology. The results are sent to a server, which reflects the analyzed expenses and visualizes them on the device. Furthermore, an emotion engine detects the user's enjoyment from their behavior at a restaurant and recommends options for subsequent dinners. In this way, the present invention plays a role in making the user's trip more fulfilling.

[0826] The following describes the processing flow.

[0827] Step 1:

[0828] Users save booking emails and receipt images related to their travels to their email inbox or a dedicated app.

[0829] Step 2:

[0830] The server interacts with the user's mailbox to automatically retrieve emails and image data related to their trip.

[0831] Step 3:

[0832] The server analyzes the retrieved email content using natural language processing technology to extract travel itineraries and reservation information.

[0833] Step 4:

[0834] The server uses optical character recognition technology to analyze the image data and extract expense information from the receipt image.

[0835] Step 5:

[0836] The terminal receives the analyzed data sent from the server and automatically classifies the information into categories using an information classification means.

[0837] Step 6:

[0838] Before traveling, the user sets their travel budget information on the device. The device then records this budget data.

[0839] Step 7:

[0840] The device tracks the user's spending in real time during travel and notifies the user using warning mechanisms when they are likely to exceed their budget.

[0841] Step 8:

[0842] The emotion engine activates and recognizes the user's emotional state by analyzing their voice, facial expressions, and text messages.

[0843] Step 9:

[0844] The emotion engine suggests optimal activities and plans to the user based on recognized emotions and the current travel situation.

[0845] Step 10:

[0846] After the trip ends, the server compiles all expenses within the group and uses expense management tools to fairly calculate how much each person will pay.

[0847] Step 11:

[0848] The terminal notifies the user of the calculated split amount and initiates the settlement process via the online money transfer service.

[0849] Step 12:

[0850] The server uses analyzed travel spending data, including sentiment data, to perform analysis in order to propose promotions to travel-related businesses and local governments.

[0851] This process allows users to simultaneously manage their travel expenses and enhance their emotionally-driven travel experience.

[0852] (Example 2)

[0853] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0854] Manually managing expenses during and after a trip is often time-consuming and prone to errors. Furthermore, it's difficult to enhance the experience based on emotions during the trip, highlighting the need for a system that enhances individual travel experiences. Conventional technologies lacked the means to comprehensively address these challenges.

[0855] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0856] In this invention, the server includes data extraction means for acquiring electronic communication and visual data and analyzing the information contained therein; information display means for classifying the extracted information into categories and presenting them to the user; and emotion analysis means for analyzing the user's emotions and making suggestions to improve the individual experience. This enables the automation of expense management during and after travel, as well as the improvement of the travel experience based on emotions.

[0857] "Electronic communications and visual data" refers to digital information, including emails and images, obtained from users and other sources.

[0858] "Data extraction means" refers to techniques for analyzing and extracting necessary information from electronic communication and visual data.

[0859] A "classification category" is a category used to group analyzed information based on specific criteria.

[0860] "Information display means" refers to functions and technologies for visually presenting analyzed information or classified categories to users.

[0861] An "alert system" is a technology that issues a warning when an anomaly occurs that does not meet the criteria set by the user.

[0862] A "cost management tool" is a function that allows for understanding the group's spending situation and calculating the amount each member should pay.

[0863] "Financial processing means" refers to technology for electronically settling payments based on calculated split amounts.

[0864] "Emotional analysis tools" refer to technologies that analyze a user's emotional state and provide optimal suggestions based on the results.

[0865] "Information utilization methods" refer to methods for making promotional proposals to travel agencies and local governments based on analyzed data.

[0866] "Visual recognition technology" refers to the technology used to understand the meaning of textual information and other data from images and to extract necessary information.

[0867] To implement this invention, it is first necessary to construct a system in which a server, terminal, and emotion analysis engine work together. This system is designed to improve the user's travel experience.

[0868] The server retrieves electronic communication data (e.g., emails containing travel itineraries and reservation information) and visual data (e.g., receipts and travel-related images) from the user's mailbox or dedicated app. This server incorporates natural language processing and optical character recognition technologies as data extraction methods. The server uses these technologies to analyze the acquired digital information and extract the necessary information.

[0869] The terminal receives analyzed information sent from the server, categorizes the data using an information display device, and presents it to the user. The user can use the terminal to set a budget before traveling. Based on this budget, the terminal utilizes an alert system and immediately displays a warning if spending during the trip exceeds the budget.

[0870] On the other hand, the emotion analysis engine analyzes voice and facial expression data acquired from the user in real time to understand the user's emotional state. The results of this emotion analysis are used to suggest activities and plans to improve the user's travel experience.

[0871] As a concrete example, consider a scenario where a user takes a picture of a receipt with their smartphone camera while traveling. This photographic data is immediately analyzed by the device using OCR technology, and the information is sent to a server. The server stores the analysis results in a database and visualizes them clearly on the device. Furthermore, the emotion engine can analyze the user's reaction to a restaurant they visited and suggest appropriate activities for the next step.

[0872] To support the overall functionality of this invention, users can input prompts using a generative AI model. For example, by using a prompt in the form of, "Please use OCR to analyze receipts I photographed during my trip and manage my expenses. Furthermore, please analyze my current mood and suggest activities," the system will provide the user with the most suitable suggestions.

[0873] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0874] Step 1: Data Acquisition

[0875] The server retrieves travel-related electronic and visual data from the user's mailbox or dedicated app. It uses an algorithm to filter relevant email subjects and access permissions to the user's email account as input. The output generates unanalyzed data including travel itineraries, accommodation bookings, and transportation information. The server temporarily stores this data and prepares it for analysis.

[0876] Step 2: Data Analysis

[0877] The server analyzes the acquired unanalyzed data using natural language processing and optical character recognition (OCR) technologies. The unanalyzed data obtained in step 1 is used as input. Text information is analyzed using natural language processing, and text is extracted from images using OCR. The output generates analyzed data containing travel reservation confirmation numbers, dates, and cost information.

[0878] Step 3: Information Classification

[0879] The terminal receives parsed data sent from the server and classifies the data into categories using a classification algorithm. It uses parsed data from the server as input. The terminal automatically classifies the data into categories such as "accommodation," "transportation," and "food," and generates a data structure as output to present the classified information to the user.

[0880] Step 4: Budget Management

[0881] Before traveling, the user enters their budget into the terminal. The terminal compares ongoing expenses with the set budget in real time. The inputs used are the user's set budget and the categorized expense data obtained in step 3. If the budget is exceeded, the terminal immediately displays a warning and generates a warning message as output.

[0882] Step 5: Emotion Analysis

[0883] The device acquires the user's voice and facial expression data and analyzes it in real time using an emotion analysis engine. It uses the user's sensor information and camera data as input. The emotion analysis engine identifies emotional states such as stress, joy, and excitement, and generates data that includes emotion-based improvement suggestions as output.

[0884] Step 6: Proposal Generation

[0885] The device suggests suitable travel plans and activities to the user based on emotional data obtained from the emotion analysis engine and the progress of the trip. The inputs used are the user's emotional data and travel information classified in step 3. The output generates a list of recommended plans and activities to enhance the user's experience.

[0886] Step 7: Expense settlement

[0887] The server aggregates spending data from all members during a group trip and calculates the split cost using expense management tools. It uses spending information from each group member as input. The output generates the exact amount each member should pay and online transfer instructions to support that payment. This information is sent to the terminal and displayed clearly to the user.

[0888] (Application Example 2)

[0889] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0890] Efficiently managing expenses and tracking spending during travel and outings, as well as improving the experience based on emotions, proved difficult. Furthermore, optimizing the experience through recommendations tailored to the user's emotional state and managing spending within groups presented challenges. Additionally, there was a lack of effective ways to utilize the analyzed emotional data.

[0891] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0892] In this invention, the server includes acquisition means for acquiring data and analyzing information, classification means for classifying and displaying the acquired information, and monitoring means for comparing a set budget with expenditures and issuing warnings. This enables users to efficiently manage expenses, track spending, and recommend choices based on their emotions.

[0893] "Data" refers to a collection or aggregate of information, and is what is processed by a system.

[0894] "Means of acquisition" refers to the function or method of collecting necessary data or information.

[0895] "Analysis" is the process of breaking down acquired data in order to understand its meaning and value.

[0896] A "classification method" is a method of organizing analyzed information into categories based on specific criteria.

[0897] "Display" refers to the act or device of visually presenting information to a user.

[0898] A "monitoring mechanism" is a function that continuously compares the budget with expenditures and issues a warning when an anomaly occurs.

[0899] A "management tool" is a system for appropriately organizing and controlling funds and expenditures within a group, and for performing necessary calculations.

[0900] A "settlement method" is a processing function for making payments or remittances based on calculation results.

[0901] "Emotional state" refers to data that indicates the user's psychological response and mood.

[0902] A "feedback mechanism" is a function that presents the user with the most suitable suggestions or options based on their analyzed emotional state.

[0903] "Analysis" refers to the process of examining acquired data in detail and extracting useful information from it.

[0904] "Optical character recognition technology" is a technology that automatically recognizes characters from image data and converts them into text data.

[0905] The system realizing this invention is designed to acquire data and provide the user with the optimal experience based on the analyzed information. The server acquires data sent from the user's device. This includes travel plans, purchase history, and sentiment data from voice and text. The server analyzes the acquired data using natural language processing libraries (e.g., spaCy, NLTK) and optical character recognition technology (e.g., Tesseract) to extract information. Furthermore, it utilizes sentiment recognition APIs (e.g., Microsoft Azure Emotion API) to understand the user's emotional state in real time and use this information to provide feedback.

[0906] The terminal receives information analyzed from the server and organizes it by category. This information is displayed intuitively through the user interface, allowing the user to check their budget management and spending status at any time.

[0907] Users can receive alerts from their device when they exceed their set budget, enabling efficient financial management. Furthermore, during group travel, the expense management system allows for the aggregation of the group's total expenses and the calculation of appropriate splitting amounts.

[0908] As a concrete example, a user sends an image taken with their smartphone camera to a server. This image is converted into text using OCR technology, and emotion recognition is used to analyze how the user felt in a particular situation. The analyzed information is evaluated by the user and used to optimize their next actions and choices.

[0909] An example of a prompt to be input to the generating AI model is, "Consider the user's current emotional state and suggest lunch options suitable for stress reduction." In this way, the present invention makes it possible to highly personalize the user's travel and outing experiences.

[0910] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0911] Step 1:

[0912] The server retrieves travel-related emails and image data from the user's device. The input is emails and images, and the output is data ready for analysis. Specifically, this involves scanning the mailbox and retrieving image data.

[0913] Step 2:

[0914] The server analyzes the acquired data using a natural language processing library (e.g., spaCy) to extract travel itineraries, reservation information, and other relevant details. The input is the data obtained in step 1, and the output is a well-organized set of information. This clarifies the user's travel plan.

[0915] Step 3:

[0916] The server uses OCR technology (e.g., Tesseract) to extract text from image data. The input is an image file, and the output is digital text. This allows paper receipts and ticket information to be treated as text data.

[0917] Step 4:

[0918] The terminal receives the analyzed information sent from the server and organizes it by category. The input is the data obtained in steps 2 and 3, and the output is the information classified by category. The information is visualized in a format suitable for the user interface.

[0919] Step 5:

[0920] The device monitors the user's set budget against actual spending and displays a warning if the budget is exceeded. Input is the user's budget information and spending data, and output is a warning message. This allows users to efficiently manage their spending.

[0921] Step 6:

[0922] The server aggregates spending data from the group and calculates the appropriate split amount. The input is each member's spending data, and the output is the calculated split amount. Users can then refer to this result to divide the payment equally.

[0923] Step 7:

[0924] The server uses an emotion recognition API to analyze the user's emotions in real time and generate feedback to improve the user experience. Input is emotion-related data from voice and text, and output is suggestions and recommended actions. Based on this, the user can make better decisions.

[0925] Step 8:

[0926] The user inputs a prompt into a generated AI model, and the system performs data calculations based on that input to provide the optimal food delivery option. The input is a prompt, and the output is a customized suggestion. A specific example of the prompt might be, "Consider the user's current emotional state and suggest a lunch option suitable for stress reduction." In this way, the data technology used enriches the user experience.

[0927] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0928] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0929] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0930] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0931] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0932] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0933] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0934] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0935] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0936] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0937] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0938] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0939] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0940] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0941] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0942] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0943] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0944] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0945] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0946] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0947] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0948] The following is further disclosed regarding the embodiments described above.

[0949] (Claim 1)

[0950] Information acquisition means for acquiring emails and image data and analyzing the information contained therein,

[0951] An information classification means that categorizes acquired information and displays it to the user,

[0952] A warning system that compares the user's set budget with their expenses and issues a warning if the budget is exceeded,

[0953] A means of managing expenses within a group and calculating the amount to be split,

[0954] A settlement processing means that processes the remittance based on the calculated split amount,

[0955] A system that includes this.

[0956] (Claim 2)

[0957] The system according to claim 1, further comprising analytical means for proposing promotions to travel-related businesses and local governments based on analyzed expenditure data.

[0958] (Claim 3)

[0959] The system according to claim 1, further comprising means for extracting text from image data using optical character recognition technology.

[0960] "Example 1"

[0961] (Claim 1)

[0962] Information acquisition means that acquires data using communication means and analyzes the information contained therein,

[0963] An information classification means that classifies acquired information into higher-level categories and displays them to the user,

[0964] A warning system that compares the user's set budget with their spending and issues a warning if the budget is exceeded,

[0965] A means of managing expenditures within a community and calculating the amount of contribution,

[0966] A settlement processing means that processes the remittance based on the calculated share amount,

[0967] A system that includes analytical tools for making proposals to stakeholders based on analyzed expenditure data.

[0968] (Claim 2)

[0969] The system according to claim 1, further comprising means for extracting textual information from image data using visual recognition technology.

[0970] (Claim 3)

[0971] The system according to claim 1, further comprising means for performing predictive analysis based on analyzed data and generating a proposed model.

[0972] "Application Example 1"

[0973] (Claim 1)

[0974] An analysis means for acquiring email information and image information and analyzing the data contained therein,

[0975] A means for classifying the acquired analytical data and displaying it to the user,

[0976] A notification system that compares the user's set budget with their spending and notifies them when the budget is exceeded,

[0977] An expense management system for managing expenses within a group and calculating shared expenses,

[0978] A payment method that performs the remittance process based on the calculated shared cost,

[0979] A tracking system that integrates scheduling and expense monitoring, and automatically tracks travel expenses,

[0980] A system that includes this.

[0981] (Claim 2)

[0982] The system according to claim 1, comprising a means for making proposals to local businesses and government agencies based on analyzed expense data.

[0983] (Claim 3)

[0984] The system according to claim 1, further comprising means for extracting character information from image data using optical character recognition technology and classifying related data.

[0985] "Example 2 of combining an emotion engine"

[0986] (Claim 1)

[0987] A data extraction means for acquiring electronic communication and visual data and analyzing the information contained therein,

[0988] An information display means that categorizes the extracted information and presents it to the user,

[0989] An alert system that compares the user's set budget with their spending and issues a warning if the budget is exceeded,

[0990] A cost management tool that manages expenditures within a group and performs percentage calculations,

[0991] A financial processing method that performs electronic settlements based on calculated percentages,

[0992] An emotion analysis tool that analyzes user emotions and makes suggestions to improve individual experiences,

[0993] A system that includes this.

[0994] (Claim 2)

[0995] The system according to claim 1, further comprising means for utilizing information to propose sales promotions to travel-related businesses and local governments based on analyzed data.

[0996] (Claim 3)

[0997] The system according to claim 1, further comprising means for using visual recognition technology to extract character information from image data.

[0998] "Application example 2 when combining with an emotional engine"

[0999] (Claim 1)

[1000] A means for acquiring data and analyzing information,

[1001] A classification means for classifying and displaying acquired information,

[1002] A monitoring system that compares the set budget with actual spending and issues warnings,

[1003] A management system for managing and calculating expenditures within a group,

[1004] A settlement method that processes based on the calculated amount,

[1005] A feedback mechanism that analyzes emotional states and recommends options,

[1006] A system that includes this.

[1007] (Claim 2)

[1008] The system according to claim 1, further comprising analytical means for making proposals to businesses and government agencies based on analyzed sentiment data.

[1009] (Claim 3)

[1010] The system according to claim 1, further comprising means for using optical character recognition technology to extract text from image data. [Explanation of symbols]

[1011] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. An analysis means for acquiring email information and image information and analyzing the data contained therein, A means for classifying the acquired analytical data and displaying it to the user, A notification system that compares the user's set budget with their spending and notifies them when the budget is exceeded, An expense management system for managing expenses within a group and calculating shared expenses, A payment method that performs the remittance process based on the calculated shared cost, A tracking system that integrates scheduling and expense monitoring, and automatically tracks travel expenses, A system that includes this.

2. The system according to claim 1, comprising a means for making proposals to local businesses and government agencies based on analyzed expense data.

3. The system according to claim 1, further comprising means for extracting character information from image data using optical character recognition technology and classifying related data.

Images