system

Abstract

We provide the system. [Solution] Using data collection methods, electronic transaction information and paper-based transaction information are acquired. Using a generative model to analyze that data, we classify the transaction information. A means of automatically creating financial records, A means of evaluating household finances based on those financial records and providing financial management advice, Means of providing budget management and advice to support purchasing decisions, A system that includes simulation tools for setting long-term goals and proposing plans to achieve those goals.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern cashless societies, various transaction data complicates personal and household financial management. For this reason, it is difficult for users to create accurate household budgets without much effort and appropriately evaluate the health of household finances. In addition, the lack of tools to prevent wasteful impulse purchases and support the formulation of plans to efficiently achieve long-term financial goals is also cited as an issue.

Means for Solving the Problems

[0005] This invention provides a system that automatically creates financial records by classifying transaction information and analyzing it using a generative model, after acquiring electronic and paper-based transaction information using data collection means. This system provides household financial assessments based on financial records, offers advice on financial management, and supports purchasing decisions. Furthermore, it simplifies data entry by using speech recognition and optical character recognition technologies, enabling users to efficiently achieve their goals through long-term goal setting and plan proposals.

[0006] "Data collection means" refers to technical methods and devices for acquiring electronic transaction information and transaction information in paper format.

[0007] A "generative model" is an artificial intelligence or machine learning algorithm used to analyze acquired data and classify transaction information.

[0008] "Financial records" are records compiled by organizing income and expenses based on transaction information, and are provided to users in a format similar to a household budget book.

[0009] "Household financial assessment" is a process that analyzes a user's financial status and spending trends based on financial records to diagnose their financial health.

[0010] "Providing advice" means making specific suggestions and giving instructions to improve the user's financial health based on the results of the household financial assessment.

[0011] "Supporting purchasing decisions" is a process that provides optimal purchasing suggestions in real time, based on the user's past data and current financial status.

[0012] "Speech recognition technology" is a technology that analyzes voice input and converts it into text information, simplifying manual input.

[0013] Optical character recognition technology is a technology that digitizes printed character information, thereby streamlining data entry from paper documents.

[0014] "Long-term goal setting" is the process of clarifying important financial goals for the future and developing a plan to achieve them.

[0015] A "plan proposal" involves presenting specific steps and strategies necessary to achieve a set long-term goal. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] This invention provides a comprehensive system that enables users to efficiently manage their household finances and achieve long-term financial goals. One embodiment of this invention is described below.

[0038] Data collection

[0039] Users input electronic transaction information and paper-based information (such as receipts) into the terminal. The terminal uses voice recognition technology to convert the amount and transaction details spoken by the user into text data. Paper-based information is also digitized by scanning it using optical character recognition technology installed in the terminal. All transaction data is transmitted from the terminal to a server and stored in a central database.

[0040] Data analysis and household budget generation

[0041] The server analyzes the collected transaction data based on a generative model. This analysis categorizes each transaction (e.g., food expenses, transportation expenses, etc.). The analysis results are automatically generated as a personal financial record in the user's household budget app, which the user can view through the interface.

[0042] Household budget assessment and advice

[0043] The server uses household budget data to assess the user's financial status. For example, it analyzes the ratio of expenses to income and the breakdown of expenses by category to diagnose the health of the household finances. Based on the assessment results, it generates advice on potential savings and areas for improvement. This includes specific savings plans and suggestions for reducing unnecessary spending.

[0044] Purchasing decision support

[0045] When a user considers making a new purchase, they can enter product information into their device. The server calculates how the purchase will affect their budget based on past spending data and their current financial situation. Based on the results, the user receives real-time advice on whether to proceed with the purchase or reconsider.

[0046] Life Planning Simulator

[0047] Users can set long-term goals such as buying a home, saving for education, or preparing for retirement. Based on current financial data and projected income, the server performs simulations and provides users with savings plans and investment strategies necessary to achieve their goals. This plan can be visually viewed on the device, and is designed to help users concretely understand their own life plans.

[0048] In this way, the present invention is a system that provides total support for the user's financial management and offers the information and support necessary for a secure life in real time.

[0049] The following describes the processing flow.

[0050] Step 1:

[0051] When a user makes a cashless transaction, the terminal automatically acquires the transaction information. If voice input is used, the terminal's voice recognition engine analyzes the user's speech to identify the amount and transaction category, and collects this data. When a receipt is scanned, the terminal's optical character recognition technology converts the text information into digital data. This allows for the acquisition of cash transaction data as well.

[0052] Step 2:

[0053] The terminal sends all collected transaction data to the server at regular intervals. The server stores the received data in a database and prepares it for the next analysis process.

[0054] Step 3:

[0055] The server analyzes transaction data in the database using a generative model. This model categorizes each transaction and calculates spending trends and income / expense balances. The results are then generated as financial records provided to the user.

[0056] Step 4:

[0057] The server assesses the financial health of the household based on the generated financial records. Specific assessment points include spending ratios, the ratio of spending to income, and past trends. Based on these results, it generates specific advice for the user.

[0058] Step 5:

[0059] The advice is sent to the device and becomes available for the user to review. The displayed advice includes saving tips and specific suggestions for reducing expenses.

[0060] Step 6:

[0061] When a user enters information about a product they are considering purchasing into their terminal, the server receives that information. Based on past transaction data and budget information, the server analyzes the impact of the purchase on their financial situation and provides real-time advice on whether the purchase is a good idea.

[0062] Step 7:

[0063] When a user sets a long-term goal, they enter the target amount and deadline into their device. The server receives this information and integrates it with their current financial data. It calculates the amount of savings and investment plan needed to achieve the goal and proposes a plan to the user. This information is displayed to the user through their device.

[0064] (Example 1)

[0065] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0066] Individuals and households need systems that collect transaction information from multiple sources, automatically classify and record it, and enable accurate and efficient management. However, existing technologies have resulted in fragmented information collection, classification, recording, and advisory provision, making overall management difficult. Furthermore, they have been insufficient in supporting decision-making based on long-term plans.

[0067] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0068] In this invention, the server includes a data collection device that acquires information, a device that classifies the information using a generation algorithm that analyzes the information, and a device that automatically generates records. This enables efficient collection and analysis of information acquired from multiple information sources, allowing for real-time advice provision and long-term planning support.

[0069] "Information" refers to transaction-related data collected electronically or through paper media.

[0070] A "data acquisition device" is a device that acquires information using speech recognition technology or optical character recognition technology and stores it as electronic data.

[0071] A "generative algorithm" is an algorithm or model used to analyze acquired information and classify it into a specific category.

[0072] "Records" refer to the history and transactions of financial information automatically generated based on the analysis results.

[0073] "Advice" refers to suggestions and guidance provided to support users' financial management and decision-making based on collected and analyzed information.

[0074] A "simulation device" is a device used to formulate a plan for achieving long-term goals.

[0075] "Real-time" refers to the temporal immediacy required to perform data collection, analysis, and advice provision without delay.

[0076] This invention is a system designed to help users improve their financial management in their personal lives. Users collect daily transaction information using a terminal via voice or image input. The terminal utilizes speech recognition technology to convert the user's input into text data. Here, a "speech conversion API" is used for the speech recognition technology.

[0077] Furthermore, the terminal digitizes paper documents such as receipts that have been photographed using optical character recognition (OCR) technology. The technology used for this is called an "OCR engine." The information collected in this way is then transmitted to a server using a secure communication method.

[0078] The server collects the acquired information and analyzes it using a generative model. Specifically, it classifies the information into categories using a "machine learning algorithm" and generates records for each user. The generated records are visualized in a dedicated application on the terminal as analysis results based on the user's spending and income. Users can view this data via the terminal and check their own financial situation.

[0079] The server then evaluates the user's household finances based on the analysis results. This evaluation process generates advice to support the user's budget management and decision-making. When a user considers a purchase, the server calculates the impact of that purchase on their household finances in real time and advises whether they should proceed or reconsider.

[0080] Furthermore, the server simulates future plans based on long-term goals set by the user (for example, a housing purchase plan or savings goal). A "numerical calculation library" is used for the calculations, and the simulation results can be visually confirmed on the terminal. In this way, users can gain a concrete understanding of their own life plan and implement plans to achieve their goals.

[0081] For example, if a user wants to review their budget at the end of the month, the system analyzes past spending data and provides advice such as "Save 20% on eating out per month." In this case, a possible prompt using the generated AI model could be in the form of, "Please propose a financial strategy for a working couple in their 30s to purchase a house and save for college within five years."

[0082] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0083] Step 1:

[0084] Users input transaction information using their devices via voice or images. Specifically, users can use their smartphone's voice input function to say, "My grocery shopping this month cost 3000 yen," or take a picture of their receipt with their camera. The input voice data is converted into text using speech recognition technology, and the image data is converted into text data using optical character recognition technology. The output is text data of the transaction.

[0085] Step 2:

[0086] The terminal formats the converted text data and transfers it to the server using a secure communication method. Specifically, after a data integrity check is performed, the data is protected by encryption technology. The input in this process is the formatted text data, and the output is what is transferred to the server.

[0087] Step 3:

[0088] The server collects received text data and analyzes it using a generative AI model. First, the data is classified into different categories (e.g., food expenses, transportation expenses). The algorithm uses machine learning-based pattern recognition to classify the data into the appropriate category. The input for this step is the received text data, and the output is the categorized transaction data.

[0089] Step 4:

[0090] The server generates financial records based on the analyzed data. Specifically, expenditures are aggregated for each category, and the data is formatted as a monthly report. The output is a detailed financial record for each user, which is stored so that users can review it later.

[0091] Step 5:

[0092] The server evaluates the user's financial status based on the generated financial records. It analyzes the balance between income and expenses, and whether spending in specific categories is disproportionate to others. Based on the evaluation, it generates advice on potential savings and areas for improvement. The input is organized financial records, and the output is an advice and diagnostic report.

[0093] Step 6:

[0094] Users receive advice and reports from the server via their devices and review their contents. Users can then create new budget plans and manage their spending within the application. For example, the app might display advice such as, "By reducing this month's dining-out budget by 20%, you can save more money." The input is the report from the server, and the output is the user's understanding and action plan.

[0095] (Application Example 1)

[0096] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0097] Modern consumers primarily use electronic payments, leading to fragmented transaction records. As a result, managing household finances efficiently becomes difficult, and creating appropriate financial plans for daily spending and long-term goals is challenging. Furthermore, the difficulty in checking one's financial situation in real time when considering purchases makes unnecessary spending more likely.

[0098] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0099] In this invention, the server includes means for acquiring electronic transaction information and paper-based transaction information using data collection means, means for classifying transaction information using a generative model for analyzing the data, and means for automatically creating financial records. This enables users to manage transaction information in real time in conjunction with smart devices and receive support for efficient household budget management and purchasing decision-making.

[0100] "Data collection means" refers to technical devices or software for acquiring electronic transaction information and transaction information in paper format.

[0101] A "generative model" is a computational method or algorithm for analyzing collected data and classifying transaction information.

[0102] "Financial records" are detailed expense and income records automatically created based on the user's transaction information.

[0103] "Household financial assessment" is the process of analyzing the created financial records to determine the user's financial status.

[0104] "Advice" refers to suggestions regarding saving money and budget management provided to users based on an assessment of their household finances.

[0105] "Purchase decision support" refers to providing information to assist users in making purchasing decisions, taking into account their budget and financial situation.

[0106] "Simulation tools" are models and technologies used to present necessary plans and predictions for achieving long-term goals set by the user.

[0107] A "smart device" refers to any electronic device used to automatically import electronic transaction information and manage household finances in real time.

[0108] "Voice input" is a technology that converts spoken information from a user into text data.

[0109] Optical character recognition (OCR) is a technology used to convert information from paper documents into digital data.

[0110] This system provides users with a means to efficiently collect daily transaction information and implement financial management. The system utilizes devices such as smartphones and tablets. The server acquires electronic and paper-based transaction information from these devices through data collection methods. For voice input, the "speech_recognition" library is used, and "OpenCV" and "Tesseract OCR" are utilized for optical character recognition.

[0111] When a user enters transaction information, the server uses a generative model to analyze the data and categorize each transaction. It then automatically generates financial records and displays the household budget assessment results on the user's device. For example, by displaying expenses by category such as food, transportation, and utilities, users can visually understand their own spending habits.

[0112] Furthermore, the server assesses the user's financial status and utilizes a generative AI model to provide advice on reducing unnecessary spending. When considering new purchases, real-time advice is provided to the terminal to support purchasing decisions based on past transaction data and current financial status. In this way, users can make appropriate purchasing decisions within their budget.

[0113] In this invention, the following prompt can be used: "Recognize the following voice command and classify it by category: 'This expense is 1000 yen for food.'" Based on this prompt, the system can analyze the voice input and classify it into the appropriate category.

[0114] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0115] Step 1:

[0116] The terminal receives input from the user and acquires electronic transaction information. Input includes voice input via a smart device selected by the user and images of receipts. For voice input, the "speech_recognition" library is used to convert the voice data into text data, and for image input, "OpenCV" and "Tesseract OCR" are used for character recognition. As a result, the transaction information is output in text format.

[0117] Step 2:

[0118] The server stores transaction information received from the terminal in a database. The data stored here includes information about the costs and categories associated with each transaction. The data is stored in an organized format for later analysis.

[0119] Step 3:

[0120] The server analyzes stored transaction information and uses a generative AI model to classify the transaction information into categories. Specific transaction information entered by the user is used as input. The generative AI model learns patterns based on past data and performs category determination. This process classifies each transaction into categories such as "food expenses" or "transportation expenses," and the classification results are output.

[0121] Step 4:

[0122] Based on the category classification results generated by the server, financial records are created and displayed on the user's terminal as a household budget. This output information is visually organized in an easy-to-understand way to help users grasp their spending habits. Specifically, spending by category is shown in graphs and tables.

[0123] Step 5:

[0124] The server evaluates the user's financial status and generates advice to support purchasing decisions. The input consists of the latest financial records and historical transaction data. The generating AI model analyzes the user's spending patterns, identifying areas for wasteful spending and potential savings. As output, the terminal displays real-time financial advice tailored to the user.

[0125] Step 6:

[0126] When a user considers a new purchase, the server uses simulation tools to analyze its impact. The input includes information about the product the user is considering and its price. The server performs calculations to evaluate the feasibility of the purchase, taking into account the user's current budget and spending patterns. The output is an advisory message displayed on the terminal, indicating whether or not the purchase should be recommended.

[0127] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0128] This invention provides a system that supports financial management and purchasing decision-making that reflects the user's emotions. This system achieves support that takes into account the user's psychological state by incorporating an emotion engine into an existing household budgeting system.

[0129] Data collection

[0130] Users input transaction data into the terminal via electronic transaction information and paper documents. The terminal utilizes voice recognition and optical character recognition technologies to efficiently and accurately digitize the data. This information is transmitted to a server and centrally managed.

[0131] emotion recognition

[0132] When a user uses voice input, the device analyzes the tone and speed of their speech to recognize their emotions. User interface usage patterns (such as click frequency and selection patterns) are also analyzed by the emotion engine. This allows the user's emotional state to be identified.

[0133] Data analysis and household budget generation

[0134] The server analyzes the collected transaction data using a generative model. Each transaction is categorized, and the user's spending habits are automatically generated as a household budget. This includes cash flow and spending percentages.

[0135] Providing emotional support advice

[0136] Based on the emotions recognized by the emotion engine, the server adjusts the advice. For example, a user experiencing stress will be offered suggestions for relaxing expenses or feasible plans for saving money. This advice is sent to the device and displayed to the user in real time.

[0137] Purchasing decision support

[0138] When a user considers purchasing a new product, the system uses an emotional engine's evaluation to support their purchasing decision. For example, if the user's desire to buy is emotional, the server will offer suggestions to encourage a more rational purchasing decision.

[0139] Setting long-term goals and strengthening motivation

[0140] Users can set long-term financial goals through their device. Based on their current financial situation and emotional assessment, the server proposes a concrete plan for achieving those goals. In this process, it generates motivational messages tailored to the user's emotions to encourage positive behavior.

[0141] The system of the present invention takes into account the user's emotional state and enables more personalized financial management, thereby promoting efficient goal achievement and fund management.

[0142] The following describes the processing flow.

[0143] Step 1:

[0144] When a user conducts a transaction, the terminal collects transaction data through voice input and receipt scanning. Voice recognition technology converts the user's speech into text, identifying the amount and trading partner. Optical character recognition technology digitizes information from paper documents and converts it into a format that can be stored in a database.

[0145] Step 2:

[0146] The terminal performs additional voice analysis to evaluate the emotions behind each input. This process utilizes an emotion engine to analyze the user's voice tone and speaking patterns. The resulting emotion data is sent to the server along with the transaction data.

[0147] Step 3:

[0148] The server passes the received transaction data to a generative model, which categorizes the transaction information. The analyzed data is presented in a household budget format, visualizing the user's spending trends.

[0149] Step 4:

[0150] Based on emotional data, the server adjusts feedback regarding the user's financial situation. For example, if the user is feeling anxious, the server generates reassuring advice and budget management suggestions.

[0151] Step 5:

[0152] The generated advice is sent to the device for the user to review. By presenting content that addresses specific emotions, it encourages the user to develop an effective action plan.

[0153] Step 6:

[0154] When a user enters information about a product they are considering purchasing, the terminal sends that information to a server. The server analyzes the impact of that purchase on the user, utilizing not only past purchase history and current financial status, but also emotional data.

[0155] Step 7:

[0156] Based on the analysis results, the server recommends more rational purchasing choices while drawing attention to emotional purchasing motives. For example, it will offer suggestions to encourage calmer judgment when emotions are running high.

[0157] Step 8:

[0158] Once a user's long-term financial goals are set on the server, the server creates an achievement plan that includes emotionally sensitive motivational messages. This makes it easier to maintain motivation towards achieving those goals.

[0159] (Example 2)

[0160] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0161] Modern financial management systems primarily rely on numerical data for analysis, and do not adequately provide support that reflects the user's psychological state and emotions. This can lead to stress and emotional decisions in budget management and purchasing decisions. Furthermore, it becomes difficult to maintain motivation for setting and achieving long-term financial goals.

[0162] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0163] This invention includes a server that acquires electronic and paper-based transaction information using a data collection device, classifies the transaction information using a generative model that analyzes the information, and automatically creates financial records; a server that recognizes the user's emotional state and adjusts the advice provided based on that emotion; and a technology that determines the user's psychological state using voice analysis and user interface usage patterns. This enables more personalized financial management that takes the user's emotions into account, promoting efficient goal achievement and fund management.

[0164] A "data collection device" is a device that uses technology to acquire electronic transaction information and transaction information in paper format and convert it into a digital format.

[0165] A "generative model" is a machine learning or artificial intelligence model used to analyze and classify collected transaction information.

[0166] "Financial records" are records of economic activity that are automatically created based on transaction information classified by generative models.

[0167] "Emotional state" refers to the psychological state or feeling that a user expresses through voice input, interface usage patterns, and other means.

[0168] "Means for adjusting advice" refers to control technology that dynamically changes the content of financial management suggestions and advice according to the user's emotional state.

[0169] "Voice analysis" is a technology that analyzes voice input provided by the user to extract the speaker's intentions and emotions.

[0170] "User interface usage patterns" refer to usage characteristics such as the frequency and speed of clicks and the way users select options when operating a system.

[0171] "Psychological state" is a concept that encompasses a user's current emotions, mood, attitude, and other related factors.

[0172] This invention is a system that supports financial management and purchasing decision-making while taking into account the user's emotional state. This system integrates an emotion analysis engine with existing household management technology to achieve personalized support that reflects the user's psychological state.

[0173] Users transmit electronic transaction data and information from paper documents to a terminal via an input device. The terminal uses speech recognition technology (e.g., a general speech analysis engine) and optical character recognition technology (e.g., open-source character analysis software) to convert this data into a digital format. The digitized information is then transferred to a centrally managed server.

[0174] The server analyzes the collected transaction data using generative AI models (e.g., a modern natural language processing engine). Each transaction is automatically categorized and built into a household financial record, which includes details of expenses and monthly cash flow.

[0175] When a user uses voice input, the device analyzes the user's speech dialect and speed to recognize their emotions. It also analyzes user interface operation patterns. Based on this information, the device identifies the user's emotional state and sends this information to the server.

[0176] The server adjusts its advice based on the user's emotional state. For example, if the user is feeling stressed, it suggests spending on relaxation-related activities and develops a feasible plan for saving money. This advice is delivered to the user's device in real time.

[0177] Furthermore, when a user considers purchasing a new product, the server uses the results of sentiment analysis to support the user's emotionally-based purchasing decision. In this process, it can provide suggestions that help the user maintain caution and avoid being swayed by emotions.

[0178] As a concrete example of this system, consider the case where a user inputs, "I want to plan a family trip this summer and save money." The generative model processes this information and generates an optimal household budget management plan. An example of a prompt message in this case would be, "The user wants to increase savings for a family trip. Please generate a specific savings plan that takes the user's excitement into consideration, along with a motivating message." In this way, financial management that takes the user's emotions into account becomes possible.

[0179] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0180] Step 1:

[0181] Users input electronic transaction information or paper-based data into the terminal.

[0182] In terms of specific operations, the user uses the terminal's keyboard or scanner to input transaction information. The entered data is recorded as text or scanned images.

[0183] Step 2:

[0184] The terminal converts the input information into a digital format.

[0185] The device takes audio files or scanned images as input and uses speech recognition technology to convert the audio data into text, and optical character recognition technology to convert the image data into text. The output is text data in a unified digital format.

[0186] Step 3:

[0187] The terminal sends the converted digital data to the server.

[0188] In terms of specific operation, the terminal uses a communication module to transfer digital data to the server over the network. The output is the digital data as it is stored on the server.

[0189] Step 4:

[0190] The server analyzes the received transaction data using a generated AI model.

[0191] The server receives transaction data collected as input, categorizes it using a generative AI model, and performs analysis. The output is a financial record organized by category.

[0192] Step 5:

[0193] The server understands the user's spending habits based on financial records and automatically generates a household budget.

[0194] Specifically, the server aggregates category information from transaction data, calculates consumption trends, cash flow, and spending ratios, and creates a household budget. The output is a household budget presented in a user-friendly format.

[0195] Step 6:

[0196] The device recognizes emotions through the user's voice input and interface operations.

[0197] The device receives voice and operation pattern data as input, and uses its emotion analysis function to analyze this data and identify the user's emotional state. The output is tags and numerical evaluations related to the user's emotional state.

[0198] Step 7:

[0199] The server adjusts the advice based on the user's emotional state.

[0200] In terms of specific operation, the server uses a generative AI model to generate appropriate advice based on the user's emotional state and adjusts the economic management plan accordingly. The output is a customized piece of advice that best matches the user's emotions.

[0201] Step 8:

[0202] The server sends the adjusted advice to the terminal and provides it to the user.

[0203] The server receives pre-adjusted advice as input and sends it to the terminal, providing information to the user in real time. The output is the advice displayed on the terminal screen.

[0204] Step 9:

[0205] When a user is considering a new product, the server uses sentiment analysis data to support their purchasing decision.

[0206] In practice, the server references sentiment data and past purchase history to generate purchase suggestions and advice. The input consists of product information and the user's emotional state, while the output is specific advice to support purchasing decisions.

[0207] (Application Example 2)

[0208] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0209] Traditional financial management systems often fail to adequately consider the emotional state of users, potentially leading to inefficient purchasing decisions based on emotional impulses. Furthermore, they may not clearly present concrete plans for achieving long-term financial goals. Therefore, there is a need for methods that provide personalized advice tailored to the user's psychological state and support efficient financial management.

[0210] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0211] In this invention, the server includes means for acquiring electronic transaction information and paper-based transaction information using data collection means, means for classifying transaction information using a generative model and automatically creating financial records, and means for analyzing the user's emotional state and adjusting financial management and purchasing advice based on that state. This enables the provision of personalized advice that responds to the user's emotions, making it possible to make purchasing decisions that suppress emotional impulses and to achieve efficient financial management.

[0212] - "Data collection means" refers to means for acquiring electronic transaction information and paper-based transaction information, and for appropriately processing that information.

[0213] A "generative model" is a model used to analyze and classify acquired transaction information, and it has the function of automatically creating financial records.

[0214] "Financial records" are records generated based on a user's transaction data and contain information useful for household financial assessment and management.

[0215] "Financial management advice" is guidance provided based on the evaluation results of financial records, and offers suggestions for improving the user's financial situation.

[0216] "Emotional state" refers to the user's psychological state and is analyzed using voice recognition technology and interface usage patterns.

[0217] "Purchase advice" refers to advice that takes into account the user's mental state and supports their purchasing decision based on that state.

[0218] A "simulation tool" is a tool that has the function of proposing a plan to reach a defined long-term goal, and supports the user in achieving their goal.

[0219] This invention is a system that provides financial management and purchasing decision support while taking into account the user's emotional state. The system mainly consists of a server and terminals.

[0220] The server collects and manages user transaction data. Electronic transaction information obtained from terminals and transaction information on paper are digitized using speech recognition and optical character recognition technologies. As a result, the data is centrally transmitted to the server and integrated for management.

[0221] The server uses an emotion analysis engine to evaluate the user's emotional state. This emotion analysis is performed by analyzing data on tone and speed during voice input, as well as interface usage patterns. Emotional evaluation is a crucial process for identifying the user's psychological state.

[0222] This system also utilizes generative models to analyze user transaction information and automatically create financial records. Based on this, it analyzes the user's spending trends and generates individual household budgets. The server adjusts personalized advice based on the collected data according to the user's emotional state and provides it in real time.

[0223] For example, if a user is feeling stressed, the system might offer advice such as, "Why not choose items that help you relax?" This allows users to make more informed purchasing decisions.

[0224] An example of a prompt message would be, "Please recommend items for when the user is in a relaxed mood. The price should be under 5000 yen, and the category should be home goods." The server then uses a generative AI model to provide optimal suggestions. In this way, users can receive advice tailored to their emotional state, enabling better financial management.

[0225] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0226] Step 1:

[0227] The terminal collects transaction information from the user. Electronic transaction information is acquired in digital format, while paper-based transaction information is obtained as text data using optical character recognition technology. The entered information is centralized and transmitted to the server as digital data.

[0228] Step 2:

[0229] The server uses the received digital data to run a generative AI model and classify transaction information. Each transaction is then categorized and compiled into a financial record. The output provides detailed financial record data for each user.

[0230] Step 3:

[0231] The server performs sentiment analysis based on financial record data. It analyzes voice input data and interface usage patterns provided by the terminal to identify the emotional state. Inputs include voice patterns and click frequency, while output is the identified psychological state.

[0232] Step 4:

[0233] The server uses a generative AI model to create optimal advice for the user based on identified emotional states and financial record data. In this process, it generates prompts and adjusts the advice accordingly. The output is an advice message tailored to the user's current state.

[0234] Step 5:

[0235] The terminal displays advice messages received from the server to the user in real time. These messages include specific suggestions and ideas to help the user make better purchasing decisions.

[0236] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0237] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0238] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0239] [Second Embodiment]

[0240] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0241] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0242] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0243] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0244] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0245] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0246] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0247] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0248] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0249] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0250] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0251] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0252] This invention provides a comprehensive system that enables users to efficiently manage their household finances and achieve long-term financial goals. One embodiment of this invention is described below.

[0253] Data collection

[0254] Users input electronic transaction information and paper-based information (such as receipts) into the terminal. The terminal uses voice recognition technology to convert the amount and transaction details spoken by the user into text data. Paper-based information is also digitized by scanning it using optical character recognition technology installed in the terminal. All transaction data is transmitted from the terminal to a server and stored in a central database.

[0255] Data analysis and household budget generation

[0256] The server analyzes the collected transaction data based on a generative model. This analysis categorizes each transaction (e.g., food expenses, transportation expenses, etc.). The analysis results are automatically generated as a personal financial record in the user's household budget app, which the user can view through the interface.

[0257] Household budget assessment and advice

[0258] The server uses household budget data to assess the user's financial status. For example, it analyzes the ratio of expenses to income and the breakdown of expenses by category to diagnose the health of the household finances. Based on the assessment results, it generates advice on potential savings and areas for improvement. This includes specific savings plans and suggestions for reducing unnecessary spending.

[0259] Purchasing decision support

[0260] When a user considers making a new purchase, they can enter product information into their device. The server calculates how the purchase will affect their budget based on past spending data and their current financial situation. Based on the results, the user receives real-time advice on whether to proceed with the purchase or reconsider.

[0261] Life Planning Simulator

[0262] Users can set long-term goals such as buying a home, saving for education, or preparing for retirement. Based on current financial data and projected income, the server performs simulations and provides users with savings plans and investment strategies necessary to achieve their goals. This plan can be visually viewed on the device, and is designed to help users concretely understand their own life plans.

[0263] In this way, the present invention is a system that provides total support for the user's financial management and offers the information and support necessary for a secure life in real time.

[0264] The following describes the processing flow.

[0265] Step 1:

[0266] When a user makes a cashless transaction, the terminal automatically acquires the transaction information. If voice input is used, the terminal's voice recognition engine analyzes the user's speech to identify the amount and transaction category, and collects this data. When a receipt is scanned, the terminal's optical character recognition technology converts the text information into digital data. This allows for the acquisition of cash transaction data as well.

[0267] Step 2:

[0268] The terminal sends all collected transaction data to the server at regular intervals. The server stores the received data in a database and prepares it for the next analysis process.

[0269] Step 3:

[0270] The server analyzes transaction data in the database using a generative model. This model categorizes each transaction and calculates spending trends and income / expense balances. The results are then generated as financial records provided to the user.

[0271] Step 4:

[0272] The server assesses the financial health of the household based on the generated financial records. Specific assessment points include spending ratios, the ratio of spending to income, and past trends. Based on these results, it generates specific advice for the user.

[0273] Step 5:

[0274] The advice is sent to the device and becomes available for the user to review. The displayed advice includes saving tips and specific suggestions for reducing expenses.

[0275] Step 6:

[0276] When a user enters information about a product they are considering purchasing into their terminal, the server receives that information. Based on past transaction data and budget information, the server analyzes the impact of the purchase on their financial situation and provides real-time advice on whether the purchase is a good idea.

[0277] Step 7:

[0278] When a user sets a long-term goal, they enter the target amount and deadline into their device. The server receives this information and integrates it with their current financial data. It calculates the amount of savings and investment plan needed to achieve the goal and proposes a plan to the user. This information is displayed to the user through their device.

[0279] (Example 1)

[0280] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0281] There is a need for a system that can achieve accurate and efficient management by collecting transaction information from multiple information sources and automatically classifying and recording it in individuals and households. However, with the existing technologies, the collection, classification, recording, and advice provision of information have been fragmented, making overall management difficult. Furthermore, the support for decision-making based on long-term plans has also been insufficient.

[0282] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following respective means.

[0283] In this invention, the server includes a data collection device that acquires information, a device that classifies information using a generation algorithm that analyzes the information, and a device that automatically generates a record. As a result, the information acquired from multiple information sources can be efficiently collected and analyzed, enabling real-time advice provision and long-term plan support.

[0284] "Information" refers to transaction-related data collected through electronic or paper media.

[0285] The "data collection device" is a device that acquires information using voice recognition technology or optical character recognition technology and stores it as electronic data.

[0286] The "generation algorithm" is an algorithm or model used to analyze the acquired information and classify it into specific categories. [[ID=2l]]

[0287] "Record" refers to the history of financial information and transactions automatically generated based on the analysis results. [

[0288] "Advice" refers to proposals and guidance provided to support the user's financial management and decision-making based on the collected and analyzed information.

[0289] The "simulation device" is a device used to formulate a plan for achieving a goal based on long-term goal setting.

[0290] "Real-time" refers to the temporal immediacy required to perform data collection, analysis, and advice provision without delay.

[0291] This invention is a system designed to help users improve their financial management in their personal lives. Users collect daily transaction information using a terminal via voice or image input. The terminal utilizes speech recognition technology to convert the user's input into text data. Here, a "speech conversion API" is used for the speech recognition technology.

[0292] Furthermore, the terminal digitizes the information on paper documents such as receipts that have been photographed using optical character recognition (OCR) technology. The technology used in this process is called an "OCR engine." The information thus collected is transmitted to a server using a secure communication method.

[0293] The server collects the acquired information and analyzes it using a generative model. Specifically, it classifies the information into categories using a "machine learning algorithm" and generates records for each user. The generated records are visualized in a dedicated application on the terminal as analysis results based on the user's spending and income. Users can view this data via the terminal and check their own financial situation.

[0294] The server then evaluates the user's household finances based on the analysis results. This evaluation process generates advice to support the user's budget management and decision-making. When a user considers a purchase, the server calculates the impact of that purchase on their household finances in real time and advises whether they should proceed or reconsider.

[0295] Furthermore, the server simulates future plans based on long-term goals set by the user (for example, a housing purchase plan or savings goal). A "numerical calculation library" is used for the calculations, and the simulation results can be visually confirmed on the terminal. In this way, users can gain a concrete understanding of their own life plan and implement plans to achieve their goals.

[0296] For example, if a user wants to review their budget at the end of the month, the system analyzes past spending data and provides advice such as "Save 20% on eating out per month." In this case, a possible prompt using the generated AI model could be in the form of, "Please propose a financial strategy for a working couple in their 30s to purchase a house and save for college within five years."

[0297] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0298] Step 1:

[0299] Users input transaction information using their devices via voice or images. Specifically, users can use their smartphone's voice input function to say, "My grocery shopping this month cost 3000 yen," or take a picture of their receipt with their camera. The input voice data is converted into text using speech recognition technology, and the image data is converted into text data using optical character recognition technology. The output is text data of the transaction.

[0300] Step 2:

[0301] The terminal formats the converted text data and transfers it to the server using a secure communication method. Specifically, after a data integrity check is performed, the data is protected by encryption technology. The input in this process is the formatted text data, and the output is what is transferred to the server.

[0302] Step 3:

[0303] The server collects the received text data and performs analysis using a generated AI model. First, the data is classified into different categories (e.g., food expenses, transportation expenses). The algorithm uses pattern recognition by machine learning to classify it into appropriate categories. The input for this step is the received text data, and the output is the categorized transaction data.

[0304] Step 4:

[0305] The server generates financial records based on the analyzed data. Specifically, the total expenses for each category are calculated, and the data is formatted as a monthly report. What is output is the detailed financial record for each user, and this is stored so that the user can check it later.

[0306] Step 5:

[0307] The server evaluates the user's financial status based on the generated financial records. Here, it analyzes the balance between income and expenses and whether the expenses in a specific category are not overly skewed compared to others. Based on the evaluation results, advice regarding potential savings and areas for improvement is generated. The input is the organized financial records, and the output is a report of advice and diagnosis.

[0308] Step 6:

[0309] The user receives the advice and report provided by the server via the terminal and checks the content. The user can further perform new budget planning and expense management on the application. As a specific example here, the application displays advice such as "By reducing this month's dining-out budget by 20%, additional savings are possible." The input is the report from the server, and the output is the user's understanding and action plan.

[0310] (Application Example 1)

[0311] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0312] Modern consumers primarily use electronic payments, leading to fragmented transaction records. As a result, managing household finances efficiently becomes difficult, and creating appropriate financial plans for daily spending and long-term goals is challenging. Furthermore, the difficulty in checking one's financial situation in real time when considering purchases makes unnecessary spending more likely.

[0313] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0314] In this invention, the server includes means for acquiring electronic transaction information and paper-based transaction information using data collection means, means for classifying transaction information using a generative model for analyzing the data, and means for automatically creating financial records. This enables users to manage transaction information in real time in conjunction with smart devices and receive support for efficient household budget management and purchasing decision-making.

[0315] "Data collection means" refers to technical devices or software for acquiring electronic transaction information and transaction information in paper format.

[0316] A "generative model" is a computational method or algorithm for analyzing collected data and classifying transaction information.

[0317] "Financial records" are detailed expense and income records automatically created based on the user's transaction information.

[0318] "Household financial assessment" is the process of analyzing the created financial records to determine the user's financial status.

[0319] "Advice" refers to suggestions regarding saving money and budget management provided to users based on an assessment of their household finances.

[0320] "Purchase decision support" refers to providing information to assist users in making purchasing decisions, taking into account their budget and financial situation.

[0321] "Simulation tools" are models and technologies used to present necessary plans and predictions for achieving long-term goals set by the user.

[0322] A "smart device" refers to any electronic device used to automatically import electronic transaction information and manage household finances in real time.

[0323] "Voice input" is a technology that converts spoken information from a user into text data.

[0324] Optical character recognition (OCR) is a technology used to convert information from paper documents into digital data.

[0325] This system provides users with a means to efficiently collect daily transaction information and implement financial management. The system utilizes devices such as smartphones and tablets. The server acquires electronic and paper-based transaction information from these devices through data collection methods. For voice input, the "speech_recognition" library is used, and "OpenCV" and "Tesseract OCR" are utilized for optical character recognition.

[0326] When a user enters transaction information, the server uses a generative model to analyze the data and categorize each transaction. It then automatically generates financial records and displays the household budget assessment results on the user's device. For example, by displaying expenses by category such as food, transportation, and utilities, users can visually understand their own spending habits.

[0327] Furthermore, the server assesses the user's financial status and utilizes a generative AI model to provide advice on reducing unnecessary spending. When considering new purchases, real-time advice is provided to the terminal to support purchasing decisions based on past transaction data and current financial status. In this way, users can make appropriate purchasing decisions within their budget.

[0328] In this invention, the following prompt can be used: "Recognize the following voice command and classify it by category: 'This expense is 1000 yen for food.'" Based on this prompt, the system can analyze the voice input and classify it into the appropriate category.

[0329] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0330] Step 1:

[0331] The terminal receives input from the user and acquires electronic transaction information. Input includes voice input via a smart device selected by the user and images of receipts. For voice input, the "speech_recognition" library is used to convert the voice data into text data, and for image input, "OpenCV" and "Tesseract OCR" are used for character recognition. As a result, the transaction information is output in text format.

[0332] Step 2:

[0333] The server stores transaction information received from the terminal in a database. The data stored here includes information about the costs and categories associated with each transaction. The data is stored in an organized format for later analysis.

[0334] Step 3:

[0335] The server analyzes stored transaction information and uses a generative AI model to classify the transaction information into categories. Specific transaction information entered by the user is used as input. The generative AI model learns patterns based on past data and performs category determination. This process classifies each transaction into categories such as "food expenses" or "transportation expenses," and the classification results are output.

[0336] Step 4:

[0337] Based on the category classification results generated by the server, financial records are created and displayed on the user's terminal as a household budget. This output information is visually organized to make it easy for users to understand their spending habits. Specifically, spending by category is shown in graphs and tables.

[0338] Step 5:

[0339] The server evaluates the user's financial status and generates advice to support purchasing decisions. The input consists of the latest financial records and historical transaction data. The generating AI model analyzes the user's spending patterns, identifying areas for wasteful spending and potential savings. As output, the terminal displays real-time financial advice tailored to the user.

[0340] Step 6:

[0341] When a user considers a new purchase, the server uses simulation tools to analyze its impact. The input includes information about the product the user is considering and its price. The server performs calculations to evaluate the feasibility of the purchase, taking into account the user's current budget and spending patterns. The output is an advisory message displayed on the terminal, indicating whether or not the purchase should be recommended.

[0342] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0343] This invention provides a system that supports financial management and purchasing decision-making that reflects the user's emotions. This system achieves support that takes into account the user's psychological state by incorporating an emotion engine into an existing household financial management system.

[0344] Data collection

[0345] Users input transaction data into the terminal via electronic transaction information and paper documents. The terminal utilizes voice recognition and optical character recognition technologies to efficiently and accurately digitize the data. This information is transmitted to a server and centrally managed.

[0346] emotion recognition

[0347] When a user uses voice input, the device analyzes the tone and speed of their speech to recognize their emotions. User interface usage patterns (such as click frequency and selection patterns) are also analyzed by the emotion engine. This allows the user's emotional state to be identified.

[0348] Data analysis and household budget generation

[0349] The server analyzes the collected transaction data using a generative model. Each transaction is categorized, and the user's spending habits are automatically generated as a household budget. This includes cash flow and spending percentages.

[0350] Providing emotional support advice

[0351] Based on the emotions recognized by the emotion engine, the server adjusts the advice. For example, a user experiencing stress will be offered suggestions for relaxing expenses or feasible plans for saving money. This advice is sent to the device and displayed to the user in real time.

[0352] Purchasing decision support

[0353] When a user considers purchasing a new product, the system uses an emotional engine's evaluation to support their purchasing decision. For example, if the user's desire to buy is emotional, the server will offer suggestions to encourage a more rational purchasing decision.

[0354] Setting long-term goals and strengthening motivation

[0355] Users can set long-term financial goals through their device. Based on their current financial situation and emotional assessment, the server proposes a concrete plan for achieving those goals. In this process, it generates motivational messages tailored to the user's emotions to encourage positive behavior.

[0356] The system of the present invention takes into account the user's emotional state and enables more personalized financial management, thereby promoting efficient goal achievement and fund management.

[0357] The following describes the processing flow.

[0358] Step 1:

[0359] When a user conducts a transaction, the terminal collects transaction data through voice input and receipt scanning. Voice recognition technology converts the user's speech into text, identifying the amount and trading partner. Optical character recognition technology digitizes information from paper documents and converts it into a format that can be stored in a database.

[0360] Step 2:

[0361] The terminal performs additional voice analysis to evaluate the emotions behind each input. This process utilizes an emotion engine to analyze the user's voice tone and speaking patterns. The resulting emotion data is sent to the server along with the transaction data.

[0362] Step 3:

[0363] The server passes the received transaction data to a generative model, which categorizes the transaction information. The analyzed data is presented in a household budget format, visualizing the user's spending trends.

[0364] Step 4:

[0365] Based on emotional data, the server adjusts feedback regarding the user's financial situation. For example, if the user is feeling anxious, the server generates reassuring advice and budget management suggestions.

[0366] Step 5:

[0367] The generated advice is sent to the device for the user to review. By presenting content that addresses specific emotions, it encourages the user to develop an effective action plan.

[0368] Step 6:

[0369] When a user enters information about a product they are considering purchasing, the terminal sends that information to a server. The server analyzes the impact that purchase will have on the user, utilizing not only past purchase history and current financial status, but also emotional data.

[0370] Step 7:

[0371] Based on the analysis results, the server recommends more rational purchasing choices while drawing attention to emotional purchasing motives. For example, it will offer suggestions to encourage calmer judgment when emotions are running high.

[0372] Step 8:

[0373] Once a user's long-term financial goals are set on the server, the server creates an achievement plan that includes emotionally sensitive motivational messages. This makes it easier to maintain motivation towards achieving those goals.

[0374] (Example 2)

[0375] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0376] Modern financial management systems primarily rely on numerical data for analysis, and do not adequately provide support that reflects the user's psychological state and emotions. This can lead to stress and emotional decisions in budget management and purchasing decisions. Furthermore, it becomes difficult to maintain motivation for setting and achieving long-term financial goals.

[0377] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0378] This invention includes a server that acquires electronic and paper-based transaction information using a data collection device, classifies the transaction information using a generative model that analyzes the information, and automatically creates financial records; a server that recognizes the user's emotional state and adjusts the advice provided based on that emotion; and a technology that determines the user's psychological state using voice analysis and user interface usage patterns. This enables more personalized financial management that takes the user's emotions into account, promoting efficient goal achievement and fund management.

[0379] A "data collection device" is a device that uses technology to acquire electronic transaction information and transaction information in paper format and convert it into a digital format.

[0380] A "generative model" is a machine learning or artificial intelligence model used to analyze and classify collected transaction information.

[0381] "Financial records" are records of economic activity that are automatically created based on transaction information classified by generative models.

[0382] "Emotional state" refers to the psychological state or feeling that a user expresses through voice input, interface usage patterns, and other means.

[0383] "Means for adjusting advice" refers to control technology that dynamically changes the content of financial management suggestions and advice according to the user's emotional state.

[0384] "Voice analysis" is a technology that analyzes voice input provided by the user to extract the speaker's intentions and emotions.

[0385] "User interface usage patterns" refer to usage characteristics such as the frequency and speed of clicks and the way users select options when operating a system.

[0386] "Psychological state" is a concept that encompasses a user's current emotions, mood, attitude, and other related factors.

[0387] This invention is a system that supports financial management and purchasing decision-making while taking into account the user's emotional state. This system integrates an emotion analysis engine with existing household financial management technology to achieve personalized support that reflects the user's psychological state.

[0388] Users transmit electronic transaction data and information from paper documents to a terminal via an input device. The terminal uses speech recognition technology (e.g., a general speech analysis engine) and optical character recognition technology (e.g., open-source character analysis software) to convert this data into a digital format. The digitized information is then transferred to a centrally managed server.

[0389] The server analyzes the collected transaction data using generative AI models (e.g., a modern natural language processing engine). Each transaction is automatically categorized and built into a household financial record, which includes details of expenses and monthly cash flow.

[0390] When a user uses voice input, the device analyzes the user's speech dialect and speed to recognize their emotions. It also analyzes user interface operation patterns. Based on this information, the device identifies the user's emotional state and sends this information to the server.

[0391] The server adjusts its advice based on the user's emotional state. For example, if the user is feeling stressed, it suggests spending on relaxation-related activities and develops a feasible plan for saving money. This advice is delivered to the user's device in real time.

[0392] Furthermore, when a user considers purchasing a new product, the server uses the results of sentiment analysis to support the user's emotionally-based purchasing decision. In this process, it can provide suggestions that help the user maintain caution and avoid being swayed by emotions.

[0393] As a concrete example of this system, consider the case where a user inputs, "I want to plan a family trip this summer and save money." The generative model processes this information and generates an optimal household budget management plan. An example of a prompt message in this case would be, "The user wants to increase savings for a family trip. Please generate a specific savings plan that takes the user's excitement into consideration, along with a motivating message." In this way, financial management that takes the user's emotions into account becomes possible.

[0394] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0395] Step 1:

[0396] Users input electronic transaction information or paper-based data into the terminal.

[0397] In terms of specific operations, the user uses the terminal's keyboard or scanner to input transaction information. The entered data is recorded as text or scanned images.

[0398] Step 2:

[0399] The terminal converts the input information into a digital format.

[0400] The device takes audio files or scanned images as input and uses speech recognition technology to convert the audio data into text, and optical character recognition technology to convert the image data into text. The output is text data in a unified digital format.

[0401] Step 3:

[0402] The terminal sends the converted digital data to the server.

[0403] In terms of specific operation, the terminal uses a communication module to transfer digital data to the server over the network. The output is the digital data as it is stored on the server.

[0404] Step 4:

[0405] The server analyzes the received transaction data using a generated AI model.

[0406] The server receives transaction data collected as input, categorizes it using a generative AI model, and performs analysis. The output is a financial record organized by category.

[0407] Step 5:

[0408] The server understands the user's spending habits based on financial records and automatically generates a household budget.

[0409] Specifically, the server aggregates category information from transaction data, calculates consumption trends, cash flow, and spending ratios, and creates a household budget. The output is a household budget presented in a user-friendly format.

[0410] Step 6:

[0411] The device recognizes emotions through the user's voice input and interface operations.

[0412] The device receives voice and operation pattern data as input, and uses its emotion analysis function to analyze this data and identify the user's emotional state. The output is tags and numerical evaluations related to the user's emotional state.

[0413] Step 7:

[0414] The server adjusts the advice based on the user's emotional state.

[0415] In practice, the server uses a generative AI model to generate appropriate advice based on the user's emotional state and adjusts the economic management plan accordingly. The output is a customized piece of advice that best matches the user's emotions.

[0416] Step 8:

[0417] The server sends the adjusted advice to the terminal and provides it to the user.

[0418] The server receives a pre-adjusted advice message as input and sends it to the terminal, providing information to the user in real time. The output is the advice displayed on the terminal screen.

[0419] Step 9:

[0420] When a user is considering a new product, the server uses sentiment analysis data to support their purchasing decision.

[0421] In practice, the server references sentiment data and past purchase history to generate purchase suggestions and advice. The input consists of product information and the user's emotional state, while the output is specific advice to support purchasing decisions.

[0422] (Application Example 2)

[0423] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0424] Traditional financial management systems often fail to adequately consider the emotional state of users, potentially leading to inefficient purchasing decisions based on emotional impulses. Furthermore, they may not clearly present concrete plans for achieving long-term financial goals. Therefore, there is a need for methods that provide personalized advice tailored to the user's psychological state and support efficient financial management.

[0425] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0426] In this invention, the server includes means for acquiring electronic transaction information and paper-based transaction information using data collection means, means for classifying transaction information using a generative model and automatically creating financial records, and means for analyzing the user's emotional state and adjusting financial management and purchasing advice based on that state. This enables the provision of personalized advice that responds to the user's emotions, making it possible to make purchasing decisions that suppress emotional impulses and to achieve efficient financial management.

[0427] - "Data collection means" refers to means for acquiring electronic transaction information and paper-based transaction information, and for appropriately processing that information.

[0428] A "generative model" is a model used to analyze and classify acquired transaction information, and it has the function of automatically creating financial records.

[0429] "Financial records" are records generated based on a user's transaction data and contain information useful for household financial assessment and management.

[0430] "Financial management advice" is guidance provided based on the evaluation results of financial records, and offers suggestions for improving the user's financial situation.

[0431] "Emotional state" refers to the user's psychological state and is analyzed using voice recognition technology and interface usage patterns.

[0432] "Purchase advice" refers to advice that takes into account the user's mental state and supports their purchasing decision based on that state.

[0433] A "simulation tool" is a tool that has the function of proposing a plan to reach a defined long-term goal, and supports the user in achieving their goal.

[0434] This invention is a system that provides financial management and purchasing decision support while taking into account the user's emotional state. The system mainly consists of a server and terminals.

[0435] The server collects and manages user transaction data. Electronic transaction information obtained from terminals and transaction information on paper are digitized using speech recognition and optical character recognition technologies. As a result, the data is centrally transmitted to the server and managed in an integrated manner.

[0436] The server uses an emotion analysis engine to evaluate the user's emotional state. This emotion analysis is performed by analyzing data on tone and speed during voice input, as well as interface usage patterns. Emotional evaluation is a crucial process for identifying the user's psychological state.

[0437] This system also utilizes generative models to analyze user transaction information and automatically create financial records. Based on this, it analyzes the user's spending trends and generates individual household budgets. The server adjusts personalized advice based on the collected data according to the user's emotional state and provides it in real time.

[0438] For example, if a user is feeling stressed, the system might offer advice such as, "Why not choose items that help you relax?" This allows users to make more informed purchasing decisions.

[0439] An example of a prompt message would be, "Please recommend items for when the user is in a relaxed mood. The price should be under 5000 yen, and the category should be home goods." The server then uses a generative AI model to provide optimal suggestions. In this way, users can receive advice tailored to their emotional state, enabling better financial management.

[0440] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0441] Step 1:

[0442] The terminal collects transaction information from the user. Electronic transaction information is acquired in digital format, while paper-based transaction information is obtained as text data using optical character recognition technology. The entered information is centralized and transmitted to the server as digital data.

[0443] Step 2:

[0444] The server uses the received digital data to run a generative AI model and classify transaction information. Each transaction is categorized and compiled into a financial record. The output provides detailed financial record data for each user.

[0445] Step 3:

[0446] The server performs sentiment analysis based on financial record data. It analyzes voice input data and interface usage patterns provided by the terminal to identify the emotional state. Inputs include voice patterns and click frequency, while output is the identified psychological state.

[0447] Step 4:

[0448] The server uses a generative AI model to create optimal advice for the user based on identified emotional states and financial record data. In this process, it generates prompts and adjusts the advice accordingly. The output is an advice message tailored to the user's current state.

[0449] Step 5:

[0450] The terminal displays advice messages received from the server to the user in real time. These messages include specific suggestions and ideas to help the user make better purchasing decisions.

[0451] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0452] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0453] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0454] [Third Embodiment]

[0455] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0456] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0457] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0458] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0459] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0460] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0461] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0462] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0463] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0464] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0465] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0466] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0467] This invention provides a comprehensive system that enables users to efficiently manage their household finances and achieve long-term financial goals. One embodiment of this invention is described below.

[0468] Data collection

[0469] Users input electronic transaction information and paper-based information (such as receipts) into the terminal. The terminal uses voice recognition technology to convert the amount and transaction details spoken by the user into text data. Paper-based information is also digitized by scanning it using optical character recognition technology installed in the terminal. All transaction data is transmitted from the terminal to a server and stored in a central database.

[0470] Data analysis and household budget generation

[0471] The server analyzes the collected transaction data based on a generative model. This analysis categorizes each transaction (e.g., food expenses, transportation expenses, etc.). The analysis results are automatically generated as a personal financial record in the user's household budget app, which the user can view through the interface.

[0472] Household budget assessment and advice

[0473] The server uses household budget data to assess the user's financial status. For example, it analyzes the ratio of expenses to income and the breakdown of expenses by category to diagnose the health of the household finances. Based on the assessment results, it generates advice on potential savings and areas for improvement. This includes specific savings plans and suggestions for reducing unnecessary spending.

[0474] Purchasing decision support

[0475] When a user considers making a new purchase, they can enter product information into their device. The server calculates how the purchase will affect their budget based on past spending data and their current financial situation. Based on the results, the user receives real-time advice on whether to proceed with the purchase or reconsider.

[0476] Life Planning Simulator

[0477] Users can set long-term goals such as buying a home, saving for education, or preparing for retirement. Based on current financial data and projected income, the server performs simulations and provides users with savings plans and investment strategies necessary to achieve their goals. This plan can be visually viewed on the device, and is designed to help users concretely understand their own life plans.

[0478] In this way, the present invention is a system that provides total support for the user's financial management and offers the information and support necessary for a secure life in real time.

[0479] The following describes the processing flow.

[0480] Step 1:

[0481] When a user makes a cashless transaction, the terminal automatically acquires the transaction information. If voice input is used, the terminal's voice recognition engine analyzes the user's speech to identify the amount and transaction category, and collects this data. When a receipt is scanned, the terminal's optical character recognition technology converts the text information into digital data. This allows for the acquisition of cash transaction data as well.

[0482] Step 2:

[0483] The terminal sends all collected transaction data to the server at regular intervals. The server stores the received data in a database and prepares it for the next analysis process.

[0484] Step 3:

[0485] The server analyzes transaction data in the database using a generative model. This model categorizes each transaction and calculates spending trends and income / expense balances. The results are then generated as financial records provided to the user.

[0486] Step 4:

[0487] The server assesses the financial health of the household based on the generated financial records. Specific assessment points include spending ratios, the ratio of spending to income, and past trends. Based on these results, it generates specific advice for the user.

[0488] Step 5:

[0489] The advice is sent to the device and becomes available for the user to review. The displayed advice includes saving tips and specific suggestions for reducing expenses.

[0490] Step 6:

[0491] When a user enters information about a product they are considering purchasing into their terminal, the server receives that information. Based on past transaction data and budget information, the server analyzes the impact of the purchase on their financial situation and provides real-time advice on whether the purchase is a good idea.

[0492] Step 7:

[0493] When a user sets a long-term goal, they enter the target amount and deadline into their device. The server receives this information and integrates it with their current financial data. It calculates the amount of savings and investment plan needed to achieve the goal and proposes a plan to the user. This information is displayed to the user through their device.

[0494] (Example 1)

[0495] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0496] Individuals and households need systems that collect transaction information from multiple sources, automatically classify and record it, and enable accurate and efficient management. However, existing technologies have resulted in fragmented information collection, classification, recording, and advisory provision, making overall management difficult. Furthermore, they have been insufficient in supporting decision-making based on long-term plans.

[0497] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0498] In this invention, the server includes a data collection device that acquires information, a device that classifies the information using a generation algorithm that analyzes the information, and a device that automatically generates records. This enables efficient collection and analysis of information acquired from multiple information sources, allowing for real-time advice provision and long-term planning support.

[0499] "Information" refers to transaction-related data collected electronically or through paper media.

[0500] A "data acquisition device" is a device that acquires information using speech recognition technology or optical character recognition technology and stores it as electronic data.

[0501] A "generative algorithm" is an algorithm or model used to analyze acquired information and classify it into a specific category.

[0502] "Records" refer to the history and transactions of financial information automatically generated based on the analysis results.

[0503] "Advice" refers to suggestions and guidance provided to support users' financial management and decision-making based on collected and analyzed information.

[0504] A "simulation device" is a device used to formulate a plan for achieving long-term goals.

[0505] "Real-time" refers to the temporal immediacy required to perform data collection, analysis, and advice provision without delay.

[0506] This invention is a system designed to help users improve their financial management in their personal lives. Users collect daily transaction information using a terminal via voice or image input. The terminal utilizes speech recognition technology to convert the user's input into text data. Here, a "speech conversion API" is used for the speech recognition technology.

[0507] Furthermore, the terminal digitizes the information on paper documents such as receipts that have been photographed using optical character recognition (OCR) technology. The technology used in this process is called an "OCR engine." The information thus collected is transmitted to a server using a secure communication method.

[0508] The server collects the acquired information and analyzes it using a generative model. Specifically, it classifies the information into categories using a "machine learning algorithm" and generates records for each user. The generated records are visualized in a dedicated application on the terminal as analysis results based on the user's spending and income. Users can view this data via the terminal and check their own financial situation.

[0509] The server then evaluates the user's household finances based on the analysis results. This evaluation process generates advice to support the user's budget management and decision-making. When a user considers a purchase, the server calculates the impact of that purchase on their household finances in real time and advises whether they should proceed or reconsider.

[0510] Furthermore, the server simulates future plans based on long-term goals set by the user (for example, a housing purchase plan or savings goal). A "numerical calculation library" is used for the calculations, and the simulation results can be visually confirmed on the terminal. In this way, users can gain a concrete understanding of their own life plan and implement plans to achieve their goals.

[0511] For example, if a user wants to review their budget at the end of the month, the system analyzes past spending data and provides advice such as "Save 20% on eating out per month." In this case, a possible prompt using the generated AI model could be in the form of, "Please propose a financial strategy for a working couple in their 30s to purchase a house and save for college within five years."

[0512] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0513] Step 1:

[0514] Users input transaction information using their devices via voice or images. Specifically, users can use their smartphone's voice input function to say, "My grocery shopping this month cost 3000 yen," or take a picture of their receipt with their camera. The input voice data is converted into text using speech recognition technology, and the image data is converted into text data using optical character recognition technology. The output is text data of the transaction.

[0515] Step 2:

[0516] The terminal formats the converted text data and transfers it to the server using a secure communication method. Specifically, after a data integrity check is performed, the data is protected by encryption technology. The input in this process is the formatted text data, and the output is what is transferred to the server.

[0517] Step 3:

[0518] The server collects received text data and analyzes it using a generative AI model. First, the data is classified into different categories (e.g., food expenses, transportation expenses). The algorithm uses machine learning-based pattern recognition to classify the data into the appropriate category. The input for this step is the received text data, and the output is the categorized transaction data.

[0519] Step 4:

[0520] The server generates financial records based on the analyzed data. Specifically, expenditures are aggregated for each category, and the data is formatted into a monthly report. The output is a detailed financial record for each user, which is stored so that users can review it later.

[0521] Step 5:

[0522] The server evaluates the user's financial status based on the generated financial records. It analyzes the balance between income and expenses, and whether spending in specific categories is disproportionate to others. Based on the evaluation, it generates advice on potential savings and areas for improvement. The input is organized financial records, and the output is an advice and diagnostic report.

[0523] Step 6:

[0524] Users receive advice and reports from the server via their devices and review their contents. Users can then create new budget plans and manage their spending within the application. For example, the app might display advice such as, "By reducing this month's dining-out budget by 20%, you can save more money." The input is the report from the server, and the output is the user's understanding and action plan.

[0525] (Application Example 1)

[0526] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0527] Modern consumers primarily use electronic payments, leading to fragmented transaction records. As a result, managing household finances efficiently becomes difficult, and creating appropriate financial plans for daily spending and long-term goals is challenging. Furthermore, the difficulty in checking one's financial situation in real time when considering purchases makes unnecessary spending more likely.

[0528] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0529] In this invention, the server includes means for acquiring electronic transaction information and paper-based transaction information using data collection means, means for classifying transaction information using a generative model for analyzing the data, and means for automatically creating financial records. This enables users to manage transaction information in real time in conjunction with smart devices and receive support for efficient household budget management and purchasing decision-making.

[0530] "Data collection means" refers to technical devices or software for acquiring electronic transaction information and transaction information in paper format.

[0531] A "generative model" is a computational method or algorithm for analyzing collected data and classifying transaction information.

[0532] "Financial records" are detailed expense and income records automatically created based on the user's transaction information.

[0533] "Household financial assessment" is the process of analyzing the created financial records to determine the user's financial status.

[0534] "Advice" refers to suggestions regarding saving money and budget management provided to users based on an assessment of their household finances.

[0535] "Purchase decision support" refers to providing information to assist users in making purchasing decisions, taking into account their budget and financial situation.

[0536] "Simulation tools" are models and technologies used to present necessary plans and predictions for achieving long-term goals set by the user.

[0537] A "smart device" refers to any electronic device used to automatically import electronic transaction information and manage household finances in real time.

[0538] "Voice input" is a technology that converts spoken information from a user into text data.

[0539] Optical character recognition (OCR) is a technology used to convert information from paper documents into digital data.

[0540] This system provides users with a means to efficiently collect daily transaction information and implement financial management. The system utilizes devices such as smartphones and tablets. The server acquires electronic and paper-based transaction information from these devices through data collection methods. For voice input, the "speech_recognition" library is used, and "OpenCV" and "Tesseract OCR" are utilized for optical character recognition.

[0541] When a user enters transaction information, the server uses a generative model to analyze the data and categorize each transaction. It then automatically generates financial records and displays the household budget assessment results on the user's device. For example, by displaying expenses by category such as food, transportation, and utilities, users can visually understand their own spending habits.

[0542] Furthermore, the server assesses the user's financial status and utilizes a generative AI model to provide advice on reducing unnecessary spending. When considering new purchases, real-time advice is provided to the terminal to support purchasing decisions based on past transaction data and current financial status. In this way, users can make appropriate purchasing decisions within their budget.

[0543] In this invention, the following prompt can be used: "Recognize the following voice command and classify it by category: 'This expense is 1000 yen for food.'" Based on this prompt, the system can analyze the voice input and classify it into the appropriate category.

[0544] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0545] Step 1:

[0546] The terminal receives input from the user and acquires electronic transaction information. Input includes voice input via a smart device selected by the user and images of receipts. For voice input, the "speech_recognition" library is used to convert the voice data into text data, and for image input, "OpenCV" and "Tesseract OCR" are used for character recognition. As a result, the transaction information is output in text format.

[0547] Step 2:

[0548] The server stores transaction information received from the terminal in a database. The data stored here includes information about the costs and categories associated with each transaction. The data is stored in an organized format for later analysis.

[0549] Step 3:

[0550] The server analyzes stored transaction information and uses a generative AI model to classify the transaction information into categories. Specific transaction information entered by the user is used as input. The generative AI model learns patterns based on past data and performs category determination. This process classifies each transaction into categories such as "food expenses" or "transportation expenses," and the classification results are output.

[0551] Step 4:

[0552] Based on the category classification results generated by the server, financial records are created and displayed on the user's terminal as a household budget. This output information is visually organized to make it easy for users to understand their spending habits. Specifically, spending by category is shown in graphs and tables.

[0553] Step 5:

[0554] The server evaluates the user's financial status and generates advice to support purchasing decisions. The input consists of the latest financial records and historical transaction data. The generating AI model analyzes the user's spending patterns, identifying areas for wasteful spending and potential savings. As output, the terminal displays real-time financial advice tailored to the user.

[0555] Step 6:

[0556] When a user considers a new purchase, the server uses simulation tools to analyze its impact. The input includes information about the product the user is considering and its price. The server performs calculations to evaluate the feasibility of the purchase, taking into account the user's current budget and spending patterns. The output is an advisory message displayed on the terminal, indicating whether or not the purchase should be recommended.

[0557] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0558] This invention provides a system that supports financial management and purchasing decision-making that reflects the user's emotions. This system achieves support that takes into account the user's psychological state by incorporating an emotion engine into an existing household financial management system.

[0559] Data collection

[0560] Users input transaction data into the terminal via electronic transaction information and paper documents. The terminal utilizes voice recognition and optical character recognition technologies to efficiently and accurately digitize the data. This information is transmitted to a server and centrally managed.

[0561] emotion recognition

[0562] When a user uses voice input, the device analyzes the tone and speed of their speech to recognize their emotions. User interface usage patterns (such as click frequency and selection patterns) are also analyzed by the emotion engine. This allows the user's emotional state to be identified.

[0563] Data analysis and household budget generation

[0564] The server analyzes the collected transaction data using a generative model. Each transaction is categorized, and the user's spending habits are automatically generated as a household budget. This includes cash flow and spending percentages.

[0565] Providing emotional support advice

[0566] Based on the emotions recognized by the emotion engine, the server adjusts the advice. For example, a user experiencing stress will be offered suggestions for relaxing expenses or feasible plans for saving money. This advice is sent to the device and displayed to the user in real time.

[0567] Purchasing decision support

[0568] When a user considers purchasing a new product, the system uses an emotional engine's evaluation to support their purchasing decision. For example, if the user's desire to buy is emotional, the server will offer suggestions to encourage a more rational purchasing decision.

[0569] Setting long-term goals and strengthening motivation

[0570] Users can set long-term financial goals through their device. Based on their current financial situation and emotional assessment, the server proposes a concrete plan for achieving those goals. In this process, it generates motivational messages tailored to the user's emotions to encourage positive behavior.

[0571] The system of the present invention takes into account the user's emotional state and enables more personalized financial management, thereby promoting efficient goal achievement and fund management.

[0572] The following describes the processing flow.

[0573] Step 1:

[0574] When a user conducts a transaction, the terminal collects transaction data through voice input and receipt scanning. Voice recognition technology converts the user's speech into text, identifying the amount and trading partner. Optical character recognition technology digitizes information from paper documents and converts it into a format that can be stored in a database.

[0575] Step 2:

[0576] The terminal performs additional voice analysis to evaluate the emotions behind each input. This process utilizes an emotion engine to analyze the user's voice tone and speaking patterns. The resulting emotion data is sent to the server along with the transaction data.

[0577] Step 3:

[0578] The server passes the received transaction data to a generative model, which categorizes the transaction information. The analyzed data is presented in a household budget format, visualizing the user's spending trends.

[0579] Step 4:

[0580] Based on emotional data, the server adjusts feedback regarding the user's financial situation. For example, if the user is feeling anxious, the server generates reassuring advice and budget management suggestions.

[0581] Step 5:

[0582] The generated advice is sent to the device for the user to review. By presenting content that addresses specific emotions, it encourages the user to develop an effective action plan.

[0583] Step 6:

[0584] When a user enters information about a product they are considering purchasing, the terminal sends that information to a server. The server analyzes the impact that purchase will have on the user, utilizing not only past purchase history and current financial status, but also emotional data.

[0585] Step 7:

[0586] Based on the analysis results, the server recommends more rational purchasing choices while drawing attention to emotional purchasing motives. For example, it will offer suggestions to encourage calmer judgment when emotions are running high.

[0587] Step 8:

[0588] Once a user's long-term financial goals are set on the server, the server creates an achievement plan that includes emotionally sensitive motivational messages. This makes it easier to maintain motivation towards achieving those goals.

[0589] (Example 2)

[0590] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0591] Modern financial management systems primarily rely on numerical data for analysis, and do not adequately provide support that reflects the user's psychological state and emotions. This can lead to stress and emotional decisions in budget management and purchasing decisions. Furthermore, it becomes difficult to maintain motivation for setting and achieving long-term financial goals.

[0592] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0593] This invention includes a server that acquires electronic and paper-based transaction information using a data collection device, classifies the transaction information using a generative model that analyzes the information, and automatically creates financial records; a server that recognizes the user's emotional state and adjusts the advice provided based on that emotion; and a technology that determines the user's psychological state using voice analysis and user interface usage patterns. This enables more personalized financial management that takes the user's emotions into account, promoting efficient goal achievement and fund management.

[0594] A "data collection device" is a device that uses technology to acquire electronic transaction information and transaction information in paper format and convert it into a digital format.

[0595] A "generative model" is a machine learning or artificial intelligence model used to analyze and classify collected transaction information.

[0596] "Financial records" are records of economic activity that are automatically created based on transaction information classified by generative models.

[0597] "Emotional state" refers to the psychological state or feeling that a user expresses through voice input, interface usage patterns, and other means.

[0598] "Means for adjusting advice" refers to control technology that dynamically changes the content of financial management suggestions and advice according to the user's emotional state.

[0599] "Voice analysis" is a technology that analyzes voice input provided by the user to extract the speaker's intentions and emotions.

[0600] "User interface usage patterns" refer to usage characteristics such as the frequency and speed of clicks and the way users select options when operating a system.

[0601] "Psychological state" is a concept that encompasses a user's current emotions, mood, attitude, and other related factors.

[0602] This invention is a system that supports financial management and purchasing decision-making while taking into account the user's emotional state. This system integrates an emotion analysis engine with existing household financial management technology to achieve personalized support that reflects the user's psychological state.

[0603] Users transmit electronic transaction data and information from paper documents to a terminal via an input device. The terminal uses speech recognition technology (e.g., a general speech analysis engine) and optical character recognition technology (e.g., open-source character analysis software) to convert this data into a digital format. The digitized information is then transferred to a centrally managed server.

[0604] The server analyzes the collected transaction data using generative AI models (e.g., a modern natural language processing engine). Each transaction is automatically categorized and built into a household financial record, which includes details of expenses and monthly cash flow.

[0605] When a user uses voice input, the device analyzes the user's speech dialect and speed to recognize their emotions. It also analyzes user interface operation patterns. Based on this information, the device identifies the user's emotional state and sends this information to the server.

[0606] The server adjusts its advice based on the user's emotional state. For example, if the user is feeling stressed, it suggests spending on relaxation-related activities and develops a feasible plan for saving money. This advice is delivered to the user's device in real time.

[0607] Furthermore, when a user considers purchasing a new product, the server uses the results of sentiment analysis to support the user's emotionally-based purchasing decision. In this process, it can provide suggestions that help the user maintain caution and avoid being swayed by emotions.

[0608] As a concrete example of this system, consider the case where a user inputs, "I want to plan a family trip this summer and save money." The generative model processes this information and generates an optimal household budget management plan. An example of a prompt message in this case would be, "The user wants to increase savings for a family trip. Please generate a specific savings plan that takes the user's excitement into consideration, along with a motivating message." In this way, financial management that takes the user's emotions into account becomes possible.

[0609] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0610] Step 1:

[0611] Users input electronic transaction information or paper-based data into the terminal.

[0612] In terms of specific operations, the user uses the terminal's keyboard or scanner to input transaction information. The entered data is recorded as text or scanned images.

[0613] Step 2:

[0614] The terminal converts the input information into a digital format.

[0615] The device takes audio files or scanned images as input and uses speech recognition technology to convert the audio data into text, and optical character recognition technology to convert the image data into text. The output is text data in a unified digital format.

[0616] Step 3:

[0617] The terminal sends the converted digital data to the server.

[0618] In terms of specific operation, the terminal uses a communication module to transfer digital data to the server over the network. The output is the digital data as it is stored on the server.

[0619] Step 4:

[0620] The server analyzes the received transaction data using a generated AI model.

[0621] The server receives transaction data collected as input, categorizes it using a generative AI model, and performs analysis. The output is a financial record organized by category.

[0622] Step 5:

[0623] The server understands the user's spending habits based on financial records and automatically generates a household budget.

[0624] Specifically, the server aggregates category information from transaction data, calculates consumption trends, cash flow, and spending ratios, and creates a household budget. The output is a household budget presented in a user-friendly format.

[0625] Step 6:

[0626] The device recognizes emotions through the user's voice input and interface operations.

[0627] The device receives voice and operation pattern data as input, and uses its emotion analysis function to analyze this data and identify the user's emotional state. The output is tags and numerical evaluations related to the user's emotional state.

[0628] Step 7:

[0629] The server adjusts the advice based on the user's emotional state.

[0630] In practice, the server uses a generative AI model to generate appropriate advice based on the user's emotional state and adjusts the economic management plan accordingly. The output is a customized piece of advice that best matches the user's emotions.

[0631] Step 8:

[0632] The server sends the adjusted advice to the terminal and provides it to the user.

[0633] The server receives a pre-adjusted advice message as input and sends it to the terminal, providing information to the user in real time. The output is the advice displayed on the terminal screen.

[0634] Step 9:

[0635] When a user is considering a new product, the server uses sentiment analysis data to support their purchasing decision.

[0636] In practice, the server references sentiment data and past purchase history to generate purchase suggestions and advice. The input consists of product information and the user's emotional state, while the output is specific advice to support purchasing decisions.

[0637] (Application Example 2)

[0638] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0639] Traditional financial management systems often fail to adequately consider the emotional state of users, potentially leading to inefficient purchasing decisions based on emotional impulses. Furthermore, they may not clearly present concrete plans for achieving long-term financial goals. Therefore, there is a need for methods that provide personalized advice tailored to the user's psychological state and support efficient financial management.

[0640] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0641] In this invention, the server includes means for acquiring electronic transaction information and paper-based transaction information using data collection means, means for classifying transaction information using a generative model and automatically creating financial records, and means for analyzing the user's emotional state and adjusting financial management and purchasing advice based on that state. This enables the provision of personalized advice that responds to the user's emotions, making it possible to make purchasing decisions that suppress emotional impulses and to achieve efficient financial management.

[0642] - "Data collection means" refers to means for acquiring electronic transaction information and paper-based transaction information, and for appropriately processing that information.

[0643] A "generative model" is a model used to analyze and classify acquired transaction information, and it has the function of automatically creating financial records.

[0644] "Financial records" are records generated based on a user's transaction data and contain information useful for household financial assessment and management.

[0645] "Financial management advice" is guidance provided based on the evaluation results of financial records, and offers suggestions for improving the user's financial situation.

[0646] "Emotional state" refers to the user's psychological state and is analyzed using voice recognition technology and interface usage patterns.

[0647] "Purchase advice" refers to advice that takes into account the user's mental state and supports their purchasing decision based on that state.

[0648] A "simulation tool" is a tool that has the function of proposing a plan to reach a defined long-term goal, and supports the user in achieving their goal.

[0649] This invention is a system that provides financial management and purchasing decision support while taking into account the user's emotional state. The system mainly consists of a server and terminals.

[0650] The server collects and manages user transaction data. Electronic transaction information obtained from terminals and transaction information on paper are digitized using speech recognition and optical character recognition technologies. As a result, the data is centrally transmitted to the server and managed in an integrated manner.

[0651] The server uses an emotion analysis engine to evaluate the user's emotional state. This emotion analysis is performed by analyzing data on tone and speed during voice input, as well as interface usage patterns. Emotional evaluation is a crucial process for identifying the user's psychological state.

[0652] This system also utilizes generative models to analyze user transaction information and automatically create financial records. Based on this, it analyzes the user's spending trends and generates individual household budgets. The server adjusts personalized advice based on the collected data according to the user's emotional state and provides it in real time.

[0653] For example, if a user is feeling stressed, the system might offer advice such as, "Why not choose items that help you relax?" This allows users to make more informed purchasing decisions.

[0654] An example of a prompt message would be, "Please recommend items for when the user is in a relaxed mood. The price should be under 5000 yen, and the category should be home goods." The server then uses a generative AI model to provide optimal suggestions. In this way, users can receive advice tailored to their emotional state, enabling better financial management.

[0655] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0656] Step 1:

[0657] The terminal collects transaction information from the user. Electronic transaction information is acquired in digital format, while paper-based transaction information is obtained as text data using optical character recognition technology. The entered information is centralized and transmitted to the server as digital data.

[0658] Step 2:

[0659] The server uses the received digital data to run a generative AI model and classify transaction information. Each transaction is categorized and compiled into a financial record. The output provides detailed financial record data for each user.

[0660] Step 3:

[0661] The server performs sentiment analysis based on financial record data. It analyzes voice input data and interface usage patterns provided by the terminal to identify the emotional state. Inputs include voice patterns and click frequency, while output is the identified psychological state.

[0662] Step 4:

[0663] The server uses a generative AI model to create optimal advice for the user based on identified emotional states and financial record data. In this process, it generates prompts and adjusts the advice accordingly. The output is an advice message tailored to the user's current state.

[0664] Step 5:

[0665] The terminal displays advice messages received from the server to the user in real time. These messages include specific suggestions and ideas to help the user make better purchasing decisions.

[0666] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0667] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0668] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0669] [Fourth Embodiment]

[0670] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0671] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0672] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0673] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0674] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0675] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0676] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0677] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0678] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0679] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0680] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0681] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0682] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0683] This invention provides a comprehensive system that enables users to efficiently manage their household finances and achieve long-term financial goals. One embodiment of this invention is described below.

[0684] Data collection

[0685] Users input electronic transaction information and paper-based information (such as receipts) into the terminal. The terminal uses voice recognition technology to convert the amount and transaction details spoken by the user into text data. Paper-based information is also digitized by scanning it using optical character recognition technology installed in the terminal. All transaction data is transmitted from the terminal to a server and stored in a central database.

[0686] Data analysis and household budget generation

[0687] The server analyzes the collected transaction data based on a generative model. This analysis categorizes each transaction (e.g., food expenses, transportation expenses, etc.). The analysis results are automatically generated as a personal financial record in the user's household budget app, which the user can view through the interface.

[0688] Household budget assessment and advice

[0689] The server uses household budget data to assess the user's financial status. For example, it analyzes the ratio of expenses to income and the breakdown of expenses by category to diagnose the health of the household finances. Based on the assessment results, it generates advice on potential savings and areas for improvement. This includes specific savings plans and suggestions for reducing unnecessary spending.

[0690] Purchasing decision support

[0691] When a user considers making a new purchase, they can enter product information into their device. The server calculates how the purchase will affect their budget based on past spending data and their current financial situation. Based on the results, the user receives real-time advice on whether to proceed with the purchase or reconsider.

[0692] Life Planning Simulator

[0693] Users can set long-term goals such as buying a home, saving for education, or preparing for retirement. Based on current financial data and projected income, the server performs simulations and provides users with savings plans and investment strategies necessary to achieve their goals. This plan can be visually viewed on the device, and is designed to help users concretely understand their own life plans.

[0694] In this way, the present invention is a system that provides total support for the user's financial management and offers the information and support necessary for a secure life in real time.

[0695] The following describes the processing flow.

[0696] Step 1:

[0697] When a user makes a cashless transaction, the terminal automatically acquires the transaction information. If voice input is used, the terminal's voice recognition engine analyzes the user's speech to identify the amount and transaction category, and collects this data. When a receipt is scanned, the terminal's optical character recognition technology converts the text information into digital data. This allows for the acquisition of cash transaction data as well.

[0698] Step 2:

[0699] The terminal sends all collected transaction data to the server at regular intervals. The server stores the received data in a database and prepares it for the next analysis process.

[0700] Step 3:

[0701] The server analyzes transaction data in the database using a generative model. This model categorizes each transaction and calculates spending trends and income / expense balances. The results are then generated as financial records provided to the user.

[0702] Step 4:

[0703] The server assesses the financial health of the household based on the generated financial records. Specific assessment points include spending ratios, the ratio of spending to income, and past trends. Based on these results, it generates specific advice for the user.

[0704] Step 5:

[0705] The advice is sent to the device and becomes available for the user to review. The displayed advice includes saving tips and specific suggestions for reducing expenses.

[0706] Step 6:

[0707] When a user enters information about a product they are considering purchasing into their terminal, the server receives that information. Based on past transaction data and budget information, the server analyzes the impact of the purchase on their financial situation and provides real-time advice on whether the purchase is a good idea.

[0708] Step 7:

[0709] When a user sets a long-term goal, they enter the target amount and deadline into their device. The server receives this information and integrates it with their current financial data. It calculates the amount of savings and investment plan needed to achieve the goal and proposes a plan to the user. This information is displayed to the user through their device.

[0710] (Example 1)

[0711] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0712] Individuals and households need systems that collect transaction information from multiple sources, automatically classify and record it, and enable accurate and efficient management. However, existing technologies have resulted in fragmented information collection, classification, recording, and advisory provision, making overall management difficult. Furthermore, they have been insufficient in supporting decision-making based on long-term plans.

[0713] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0714] In this invention, the server includes a data collection device that acquires information, a device that classifies the information using a generation algorithm that analyzes the information, and a device that automatically generates records. This enables efficient collection and analysis of information acquired from multiple information sources, allowing for real-time advice provision and long-term planning support.

[0715] "Information" refers to transaction-related data collected electronically or through paper media.

[0716] A "data acquisition device" is a device that acquires information using speech recognition technology or optical character recognition technology and stores it as electronic data.

[0717] A "generative algorithm" is an algorithm or model used to analyze acquired information and classify it into a specific category.

[0718] "Records" refer to the history and transactions of financial information automatically generated based on the analysis results.

[0719] "Advice" refers to suggestions and guidance provided to support users' financial management and decision-making based on collected and analyzed information.

[0720] A "simulation device" is a device used to formulate a plan for achieving long-term goals.

[0721] "Real-time" refers to the temporal immediacy required to perform data collection, analysis, and advice provision without delay.

[0722] This invention is a system designed to help users improve their financial management in their personal lives. Users collect daily transaction information using a terminal via voice or image input. The terminal utilizes speech recognition technology to convert the user's input into text data. Here, a "speech conversion API" is used for the speech recognition technology.

[0723] Furthermore, the terminal digitizes the information on paper documents such as receipts that have been photographed using optical character recognition (OCR) technology. The technology used in this process is called an "OCR engine." The information thus collected is transmitted to a server using a secure communication method.

[0724] The server collects the acquired information and analyzes it using a generative model. Specifically, it classifies the information into categories using a "machine learning algorithm" and generates records for each user. The generated records are visualized in a dedicated application on the terminal as analysis results based on the user's spending and income. Users can view this data via the terminal and check their own financial situation.

[0725] The server then evaluates the user's household finances based on the analysis results. This evaluation process generates advice to support the user's budget management and decision-making. When a user considers a purchase, the server calculates the impact of that purchase on their household finances in real time and advises whether they should proceed or reconsider.

[0726] Furthermore, the server simulates future plans based on long-term goals set by the user (for example, a housing purchase plan or savings goal). A "numerical calculation library" is used for the calculations, and the simulation results can be visually confirmed on the terminal. In this way, users can gain a concrete understanding of their own life plan and implement plans to achieve their goals.

[0727] For example, if a user wants to review their budget at the end of the month, the system analyzes past spending data and provides advice such as "Save 20% on eating out per month." In this case, a possible prompt using the generated AI model could be in the form of, "Please propose a financial strategy for a working couple in their 30s to purchase a house and save for college within five years."

[0728] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0729] Step 1:

[0730] Users input transaction information using their devices via voice or images. Specifically, users can use their smartphone's voice input function to say, "My grocery shopping this month cost 3000 yen," or take a picture of their receipt with their camera. The input voice data is converted into text using speech recognition technology, and the image data is converted into text data using optical character recognition technology. The output is text data of the transaction.

[0731] Step 2:

[0732] The terminal formats the converted text data and transfers it to the server using a secure communication method. Specifically, after a data integrity check is performed, the data is protected by encryption technology. The input in this process is the formatted text data, and the output is what is transferred to the server.

[0733] Step 3:

[0734] The server collects received text data and analyzes it using a generative AI model. First, the data is classified into different categories (e.g., food expenses, transportation expenses). The algorithm uses machine learning-based pattern recognition to classify the data into the appropriate category. The input for this step is the received text data, and the output is the categorized transaction data.

[0735] Step 4:

[0736] The server generates financial records based on the analyzed data. Specifically, expenditures are aggregated for each category, and the data is formatted into a monthly report. The output is a detailed financial record for each user, which is stored so that users can review it later.

[0737] Step 5:

[0738] The server evaluates the user's financial status based on the generated financial records. It analyzes the balance between income and expenses, and whether spending in specific categories is disproportionate to others. Based on the evaluation, it generates advice on potential savings and areas for improvement. The input is organized financial records, and the output is an advice and diagnostic report.

[0739] Step 6:

[0740] Users receive advice and reports from the server via their devices and review their contents. Users can then create new budget plans and manage their spending within the application. For example, the app might display advice such as, "By reducing this month's dining-out budget by 20%, you can save more money." The input is the report from the server, and the output is the user's understanding and action plan.

[0741] (Application Example 1)

[0742] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0743] Modern consumers primarily use electronic payments, leading to fragmented transaction records. As a result, managing household finances efficiently becomes difficult, and creating appropriate financial plans for daily spending and long-term goals is challenging. Furthermore, the difficulty in checking one's financial situation in real time when considering purchases makes unnecessary spending more likely.

[0744] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0745] In this invention, the server includes means for acquiring electronic transaction information and paper-based transaction information using data collection means, means for classifying transaction information using a generative model for analyzing the data, and means for automatically creating financial records. This enables users to manage transaction information in real time in conjunction with smart devices and receive support for efficient household budget management and purchasing decision-making.

[0746] "Data collection means" refers to technical devices or software for acquiring electronic transaction information and transaction information in paper format.

[0747] A "generative model" is a computational method or algorithm for analyzing collected data and classifying transaction information.

[0748] "Financial records" are detailed expense and income records automatically created based on the user's transaction information.

[0749] "Household financial assessment" is the process of analyzing the created financial records to determine the user's financial status.

[0750] "Advice" refers to suggestions regarding saving money and budget management provided to users based on an assessment of their household finances.

[0751] "Purchase decision support" refers to providing information to assist users in making purchasing decisions, taking into account their budget and financial situation.

[0752] "Simulation tools" are models and technologies used to present necessary plans and predictions for achieving long-term goals set by the user.

[0753] A "smart device" refers to any electronic device used to automatically import electronic transaction information and manage household finances in real time.

[0754] "Voice input" is a technology that converts spoken information from a user into text data.

[0755] Optical character recognition (OCR) is a technology used to convert information from paper documents into digital data.

[0756] This system provides users with a means to efficiently collect daily transaction information and implement financial management. The system utilizes devices such as smartphones and tablets. The server acquires electronic and paper-based transaction information from these devices through data collection methods. For voice input, the "speech_recognition" library is used, and "OpenCV" and "Tesseract OCR" are utilized for optical character recognition.

[0757] When a user enters transaction information, the server uses a generative model to analyze the data and categorize each transaction. It then automatically generates financial records and displays the household budget assessment results on the user's device. For example, by displaying expenses by category such as food, transportation, and utilities, users can visually understand their own spending habits.

[0758] Furthermore, the server assesses the user's financial status and utilizes a generative AI model to provide advice on reducing unnecessary spending. When considering new purchases, real-time advice is provided to the terminal to support purchasing decisions based on past transaction data and current financial status. In this way, users can make appropriate purchasing decisions within their budget.

[0759] In this invention, the following prompt can be used: "Recognize the following voice command and classify it by category: 'This expense is 1000 yen for food.'" Based on this prompt, the system can analyze the voice input and classify it into the appropriate category.

[0760] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0761] Step 1:

[0762] The terminal receives input from the user and acquires electronic transaction information. Input includes voice input via a smart device selected by the user and images of receipts. For voice input, the "speech_recognition" library is used to convert the voice data into text data, and for image input, "OpenCV" and "Tesseract OCR" are used for character recognition. As a result, the transaction information is output in text format.

[0763] Step 2:

[0764] The server stores transaction information received from the terminal in a database. The data stored here includes information about the costs and categories associated with each transaction. The data is stored in an organized format for later analysis.

[0765] Step 3:

[0766] The server analyzes stored transaction information and uses a generative AI model to classify the transaction information into categories. Specific transaction information entered by the user is used as input. The generative AI model learns patterns based on past data and performs category determination. This process classifies each transaction into categories such as "food expenses" or "transportation expenses," and the classification results are output.

[0767] Step 4:

[0768] Based on the category classification results generated by the server, financial records are created and displayed on the user's terminal as a household budget. This output information is visually organized to make it easy for users to understand their spending habits. Specifically, spending by category is shown in graphs and tables.

[0769] Step 5:

[0770] The server evaluates the user's financial status and generates advice to support purchasing decisions. The input consists of the latest financial records and historical transaction data. The generating AI model analyzes the user's spending patterns, identifying areas for wasteful spending and potential savings. As output, the terminal displays real-time financial advice tailored to the user.

[0771] Step 6:

[0772] When a user considers a new purchase, the server uses simulation tools to analyze its impact. The input includes information about the product the user is considering and its price. The server performs calculations to evaluate the feasibility of the purchase, taking into account the user's current budget and spending patterns. The output is an advisory message displayed on the terminal, indicating whether or not the purchase should be recommended.

[0773] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0774] This invention provides a system that supports financial management and purchasing decision-making that reflects the user's emotions. This system achieves support that takes into account the user's psychological state by incorporating an emotion engine into an existing household financial management system.

[0775] Data collection

[0776] Users input transaction data into the terminal via electronic transaction information and paper documents. The terminal utilizes voice recognition and optical character recognition technologies to efficiently and accurately digitize the data. This information is transmitted to a server and centrally managed.

[0777] emotion recognition

[0778] When a user uses voice input, the device analyzes the tone and speed of their speech to recognize their emotions. User interface usage patterns (such as click frequency and selection patterns) are also analyzed by the emotion engine. This allows the user's emotional state to be identified.

[0779] Data analysis and household budget generation

[0780] The server analyzes the collected transaction data using a generative model. Each transaction is categorized, and the user's spending habits are automatically generated as a household budget. This includes cash flow and spending percentages.

[0781] Providing emotional support advice

[0782] Based on the emotions recognized by the emotion engine, the server adjusts the advice. For example, a user experiencing stress will be offered suggestions for relaxing expenses or feasible plans for saving money. This advice is sent to the device and displayed to the user in real time.

[0783] Purchasing decision support

[0784] When a user considers purchasing a new product, the system uses an emotional engine's evaluation to support their purchasing decision. For example, if the user's desire to buy is emotional, the server will offer suggestions to encourage a more rational purchasing decision.

[0785] Setting long-term goals and strengthening motivation

[0786] Users can set long-term financial goals through their device. Based on their current financial situation and emotional assessment, the server proposes a concrete plan for achieving those goals. In this process, it generates motivational messages tailored to the user's emotions to encourage positive behavior.

[0787] The system of the present invention takes into account the user's emotional state and enables more personalized financial management, thereby promoting efficient goal achievement and fund management.

[0788] The following describes the processing flow.

[0789] Step 1:

[0790] When a user conducts a transaction, the terminal collects transaction data through voice input and receipt scanning. Voice recognition technology converts the user's speech into text, identifying the amount and trading partner. Optical character recognition technology digitizes information from paper documents and converts it into a format that can be stored in a database.

[0791] Step 2:

[0792] The terminal performs additional voice analysis to evaluate the emotions behind each input. This process utilizes an emotion engine to analyze the user's voice tone and speaking patterns. The resulting emotion data is sent to the server along with the transaction data.

[0793] Step 3:

[0794] The server passes the received transaction data to a generative model, which categorizes the transaction information. The analyzed data is presented in a household budget format, visualizing the user's spending trends.

[0795] Step 4:

[0796] Based on emotional data, the server adjusts feedback regarding the user's financial situation. For example, if the user is feeling anxious, the server generates reassuring advice and budget management suggestions.

[0797] Step 5:

[0798] The generated advice is sent to the device for the user to review. By presenting content that addresses specific emotions, it encourages the user to develop an effective action plan.

[0799] Step 6:

[0800] When a user enters information about a product they are considering purchasing, the terminal sends that information to a server. The server analyzes the impact that purchase will have on the user, utilizing not only past purchase history and current financial status, but also emotional data.

[0801] Step 7:

[0802] Based on the analysis results, the server recommends more rational purchasing choices while drawing attention to emotional purchasing motives. For example, it will offer suggestions to encourage calmer judgment when emotions are running high.

[0803] Step 8:

[0804] Once a user's long-term financial goals are set on the server, the server creates an achievement plan that includes emotionally sensitive motivational messages. This makes it easier to maintain motivation towards achieving those goals.

[0805] (Example 2)

[0806] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0807] Modern financial management systems primarily rely on numerical data for analysis, and do not adequately provide support that reflects the user's psychological state and emotions. This can lead to stress and emotional decisions in budget management and purchasing decisions. Furthermore, it becomes difficult to maintain motivation for setting and achieving long-term financial goals.

[0808] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0809] This invention includes a server that acquires electronic and paper-based transaction information using a data collection device, classifies the transaction information using a generative model that analyzes the information, and automatically creates financial records; a server that recognizes the user's emotional state and adjusts the advice provided based on that emotion; and a technology that determines the user's psychological state using voice analysis and user interface usage patterns. This enables more personalized financial management that takes the user's emotions into account, promoting efficient goal achievement and fund management.

[0810] A "data collection device" is a device that uses technology to acquire electronic transaction information and transaction information in paper format and convert it into a digital format.

[0811] A "generative model" is a machine learning or artificial intelligence model used to analyze and classify collected transaction information.

[0812] "Financial records" are records of economic activity that are automatically created based on transaction information classified by generative models.

[0813] "Emotional state" refers to the psychological state or feeling that a user expresses through voice input, interface usage patterns, and other means.

[0814] "Means for adjusting advice" refers to control technology that dynamically changes the content of financial management suggestions and advice according to the user's emotional state.

[0815] "Voice analysis" is a technology that analyzes voice input provided by the user to extract the speaker's intentions and emotions.

[0816] "User interface usage patterns" refer to usage characteristics such as the frequency and speed of clicks and the way users select options when operating a system.

[0817] "Psychological state" is a concept that encompasses a user's current emotions, mood, attitude, and other related factors.

[0818] This invention is a system that supports financial management and purchasing decision-making while taking into account the user's emotional state. This system integrates an emotion analysis engine with existing household financial management technology to achieve personalized support that reflects the user's psychological state.

[0819] Users transmit electronic transaction data and information from paper documents to a terminal via an input device. The terminal uses speech recognition technology (e.g., a general speech analysis engine) and optical character recognition technology (e.g., open-source character analysis software) to convert this data into a digital format. The digitized information is then transferred to a centrally managed server.

[0820] The server analyzes the collected transaction data using generative AI models (e.g., a modern natural language processing engine). Each transaction is automatically categorized and built into a household financial record, which includes details of expenses and monthly cash flow.

[0821] When a user uses voice input, the device analyzes the user's speech dialect and speed to recognize their emotions. It also analyzes user interface operation patterns. Based on this information, the device identifies the user's emotional state and sends this information to the server.

[0822] The server adjusts its advice based on the user's emotional state. For example, if the user is feeling stressed, it suggests spending on relaxation-related activities and develops a feasible plan for saving money. This advice is delivered to the user's device in real time.

[0823] Furthermore, when a user considers purchasing a new product, the server uses the results of sentiment analysis to support the user's emotionally-based purchasing decision. In this process, it can provide suggestions that help the user maintain caution and avoid being swayed by emotions.

[0824] As a concrete example of this system, consider the case where a user inputs, "I want to plan a family trip this summer and save money." The generative model processes this information and generates an optimal household budget management plan. An example of a prompt message in this case would be, "The user wants to increase savings for a family trip. Please generate a specific savings plan that takes the user's excitement into consideration, along with a motivating message." In this way, financial management that takes the user's emotions into account becomes possible.

[0825] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0826] Step 1:

[0827] Users input electronic transaction information or paper-based data into the terminal.

[0828] In terms of specific operations, the user uses the terminal's keyboard or scanner to input transaction information. The entered data is recorded as text or scanned images.

[0829] Step 2:

[0830] The terminal converts the input information into a digital format.

[0831] The device takes audio files or scanned images as input and uses speech recognition technology to convert the audio data into text, and optical character recognition technology to convert the image data into text. The output is text data in a unified digital format.

[0832] Step 3:

[0833] The terminal sends the converted digital data to the server.

[0834] In terms of specific operation, the terminal uses a communication module to transfer digital data to the server over the network. The output is the digital data as it is stored on the server.

[0835] Step 4:

[0836] The server analyzes the received transaction data using a generated AI model.

[0837] The server receives transaction data collected as input, categorizes it using a generative AI model, and performs analysis. The output is a financial record organized by category.

[0838] Step 5:

[0839] The server understands the user's spending habits based on financial records and automatically generates a household budget.

[0840] Specifically, the server aggregates category information from transaction data, calculates consumption trends, cash flow, and spending ratios, and creates a household budget. The output is a household budget presented in a user-friendly format.

[0841] Step 6:

[0842] The device recognizes emotions through the user's voice input and interface operations.

[0843] The device receives voice and operation pattern data as input, and uses its emotion analysis function to analyze this data and identify the user's emotional state. The output is tags and numerical evaluations related to the user's emotional state.

[0844] Step 7:

[0845] The server adjusts the advice based on the user's emotional state.

[0846] In practice, the server uses a generative AI model to generate appropriate advice based on the user's emotional state and adjusts the economic management plan accordingly. The output is a customized piece of advice that best matches the user's emotions.

[0847] Step 8:

[0848] The server sends the adjusted advice to the terminal and provides it to the user.

[0849] The server receives a pre-adjusted advice message as input and sends it to the terminal, providing information to the user in real time. The output is the advice displayed on the terminal screen.

[0850] Step 9:

[0851] When a user is considering a new product, the server uses sentiment analysis data to support their purchasing decision.

[0852] In practice, the server references sentiment data and past purchase history to generate purchase suggestions and advice. The input consists of product information and the user's emotional state, while the output is specific advice to support purchasing decisions.

[0853] (Application Example 2)

[0854] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0855] Traditional financial management systems often fail to adequately consider the emotional state of users, potentially leading to inefficient purchasing decisions based on emotional impulses. Furthermore, they may not clearly present concrete plans for achieving long-term financial goals. Therefore, there is a need for methods that provide personalized advice tailored to the user's psychological state and support efficient financial management.

[0856] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0857] In this invention, the server includes means for acquiring electronic transaction information and paper-based transaction information using data collection means, means for classifying transaction information using a generative model and automatically creating financial records, and means for analyzing the user's emotional state and adjusting financial management and purchasing advice based on that state. This enables the provision of personalized advice that responds to the user's emotions, making it possible to make purchasing decisions that suppress emotional impulses and to achieve efficient financial management.

[0858] - "Data collection means" refers to means for acquiring electronic transaction information and paper-based transaction information, and for appropriately processing that information.

[0859] A "generative model" is a model used to analyze and classify acquired transaction information, and it has the function of automatically creating financial records.

[0860] "Financial records" are records generated based on a user's transaction data and contain information useful for household financial assessment and management.

[0861] "Financial management advice" is guidance provided based on the evaluation results of financial records, and offers suggestions for improving the user's financial situation.

[0862] "Emotional state" refers to the user's psychological state and is analyzed using voice recognition technology and interface usage patterns.

[0863] "Purchase advice" refers to advice that takes into account the user's mental state and supports their purchasing decision based on that state.

[0864] A "simulation tool" is a tool that has the function of proposing a plan to reach a defined long-term goal, and supports the user in achieving their goal.

[0865] This invention is a system that provides financial management and purchasing decision support while taking into account the user's emotional state. The system mainly consists of a server and terminals.

[0866] The server collects and manages user transaction data. Electronic transaction information obtained from terminals and transaction information on paper are digitized using speech recognition and optical character recognition technologies. As a result, the data is centrally transmitted to the server and managed in an integrated manner.

[0867] The server uses an emotion analysis engine to evaluate the user's emotional state. This emotion analysis is performed by analyzing data on tone and speed during voice input, as well as interface usage patterns. Emotional evaluation is a crucial process for identifying the user's psychological state.

[0868] This system also utilizes generative models to analyze user transaction information and automatically create financial records. Based on this, it analyzes the user's spending trends and generates individual household budgets. The server adjusts personalized advice based on the collected data according to the user's emotional state and provides it in real time.

[0869] For example, if a user is feeling stressed, the system might offer advice such as, "Why not choose items that help you relax?" This allows users to make more informed purchasing decisions.

[0870] An example of a prompt message would be, "Please recommend items for when the user is in a relaxed mood. The price should be under 5000 yen, and the category should be home goods." The server then uses a generative AI model to provide optimal suggestions. In this way, users can receive advice tailored to their emotional state, enabling better financial management.

[0871] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0872] Step 1:

[0873] The terminal collects transaction information from the user. Electronic transaction information is acquired in digital format, while paper-based transaction information is obtained as text data using optical character recognition technology. The entered information is centralized and transmitted to the server as digital data.

[0874] Step 2:

[0875] The server uses the received digital data to run a generative AI model and classify transaction information. Each transaction is categorized and compiled into a financial record. The output provides detailed financial record data for each user.

[0876] Step 3:

[0877] The server performs sentiment analysis based on financial record data. It analyzes voice input data and interface usage patterns provided by the terminal to identify the emotional state. Inputs include voice patterns and click frequency, while output is the identified psychological state.

[0878] Step 4:

[0879] The server uses a generative AI model to create optimal advice for the user based on identified emotional states and financial record data. In this process, it generates prompts and adjusts the advice accordingly. The output is an advice message tailored to the user's current state.

[0880] Step 5:

[0881] The terminal displays advice messages received from the server to the user in real time. These messages include specific suggestions and ideas to help the user make better purchasing decisions.

[0882] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0883] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0884] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0885] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0886] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0887] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0888] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0889] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0890] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0891] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0892] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0893] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0894] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0895] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0896] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0897] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0898] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0899] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0900] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0901] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0902] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0903] The following is further disclosed regarding the embodiments described above.

[0904] (Claim 1)

[0905] Using data collection methods, electronic transaction information and paper-based transaction information are acquired.

[0906] Using a generative model to analyze that data, we classify the transaction information.

[0907] A means of automatically creating financial records,

[0908] A means of evaluating household finances based on those financial records and providing financial management advice,

[0909] Means of providing budget management and advice to support purchasing decisions,

[0910] A system that includes simulation tools for setting long-term goals and proposing plans to achieve those goals.

[0911] (Claim 2)

[0912] The system according to claim 1, wherein the data acquisition means includes speech recognition technology and optical character recognition technology.

[0913] (Claim 3)

[0914] The system according to claim 1, wherein the means for supporting purchasing decisions provides real-time advice to the user based on collected past transaction data and the current financial status.

[0915] "Example 1"

[0916] (Claim 1)

[0917] A data collection device that acquires information,

[0918] A device that classifies information using a generation algorithm that analyzes that information,

[0919] A device that automatically generates records,

[0920] A device that evaluates the condition based on the records and provides advice on management,

[0921] A device that provides management and advice to support decision-making,

[0922] A system that includes a simulation device for setting long-term goals and proposing plans to achieve those goals.

[0923] (Claim 2)

[0924] The data collection device includes speech conversion technology and character recognition technology, according to claim 1.

[0925] (Claim 3)

[0926] The system according to claim 1, wherein the decision-making support device provides real-time advice to the user based on collected historical information and the current state.

[0927] "Application Example 1"

[0928] (Claim 1)

[0929] Using data collection methods, electronic transaction information and paper-based transaction information are acquired.

[0930] Using a generative model to analyze that data, we classify the transaction information.

[0931] A means of automatically creating financial records,

[0932] A means of evaluating household finances based on those financial records and providing financial management advice,

[0933] Means of providing budget management and advice to support purchasing decisions,

[0934] A simulation tool that sets long-term goals and proposes a plan to achieve those goals,

[0935] A method for automatically importing electronic transactions in conjunction with smart devices and managing household finances in real time,

[0936] A system that includes means for easily recording expenses using voice input and optical character recognition.

[0937] (Claim 2)

[0938] The system according to claim 1, wherein the data acquisition means includes voice input technology and image processing technology.

[0939] (Claim 3)

[0940] The system according to claim 1, wherein the means for supporting purchasing decisions provides the user with immediate suggestions based on collected past transaction information and current financial status.

[0941] "Example 2 of combining an emotion engine"

[0942] (Claim 1)

[0943] Using data collection devices, electronic transaction information and paper-based transaction information are acquired.

[0944] Using a generative model to analyze that information, the transaction information is classified.

[0945] A device that automatically creates financial records,

[0946] A device that assesses the household's economic situation based on its financial records and provides advice on economic management,

[0947] A device that provides budget management and advice to support purchasing decisions,

[0948] A system that includes simulation technology for setting long-term goals and proposing plans to achieve those goals,

[0949] A device that recognizes the user's emotional state and adjusts the advice provided based on that emotion,

[0950] A system that includes technology to determine a user's psychological state using voice analysis and user interface usage patterns.

[0951] (Claim 2)

[0952] The data collection device includes speech analysis technology and optical character analysis technology, according to claim 1.

[0953] (Claim 3)

[0954] The system according to claim 1, wherein the device for assisting purchasing decisions provides real-time advice to the user based on collected historical transaction data and the user's current financial status.

[0955] "Application example 2 when combining with an emotional engine"

[0956] (Claim 1)

[0957] Using data collection methods, electronic transaction information and paper-based transaction information are acquired.

[0958] Using a generative model to analyze that data, we classify the transaction information.

[0959] A means of automatically creating financial records,

[0960] A means of evaluating household finances based on those financial records and providing financial management advice,

[0961] A means of analyzing the emotional state of users and adjusting financial management and purchasing advice based on that state,

[0962] Means of providing budget management and advice to support purchasing decisions,

[0963] A system that includes simulation tools for setting long-term goals and proposing plans to achieve those goals.

[0964] (Claim 2)

[0965] The system according to claim 1, wherein the data acquisition means includes speech recognition technology and optical character recognition technology.

[0966] (Claim 3)

[0967] The system according to claim 1, wherein the means for supporting purchasing decisions provides real-time advice to the user based on collected past transaction data and current financial status, and takes into account the user's emotional state. [Explanation of Symbols]

[0968] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Using data collection methods, electronic transaction information and paper-based transaction information are acquired. Using a generative model to analyze that data, we classify the transaction information. A means of automatically creating financial records, A means of evaluating household finances based on those financial records and providing financial management advice, Means of providing budget management and advice to support purchasing decisions, A system that includes simulation tools for setting long-term goals and proposing plans to achieve those goals.

2. The system according to claim 1, wherein the data acquisition means includes speech recognition technology and optical character recognition technology.

3. The system according to claim 1, wherein the means for supporting purchasing decisions provides real-time advice to the user based on collected past transaction data and the current financial status.

Images