system

Abstract

Provide a system. 【Solution means】 Means for converting information obtained from a user by voice into character data, Means for converting image data obtained by a data acquisition device into characters, In an information processing device, means for verifying the character data as data and storing it in a storage device, Means for the information processing device to analyze past data and predict future income and expenses, Means for notifying the user of the progress in real time and sending a warning in case of an abnormality, Means including a device for visually outputting the predicted data, Means including a device for executing an operation in response to a user input, A system including the above.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance that responds to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the management of financial data, issues include input errors due to human error and a decrease in reliability due to lack of data consistency. In particular, for financial staff or project managers in small and medium-sized enterprises or large enterprises with time constraints, efficient and accurate budget management is required. However, such manual processes require a huge amount of time and labor, and as a result, there is a risk of overlooking important information. Furthermore, inaccurate prediction of future budgets and expenditures may hinder the formulation of corporate strategies.

Means for Solving the Problems

[0005] This invention provides a system that automates data input from users using voice input and scan input. First, a voice recognition means converts the user's voice instructions into text data, and an OCR technology converts image data obtained by a scanning device into text. This data is sent to a server, where the server verifies the financial data and stores it in a database. Furthermore, the server predicts future budgets and expenditures based on past data and monitors progress in real time, issuing alerts to the user in case of anomalies. It also provides individualized feedback and advice based on the user's input patterns and enables the visual display of data through a dashboard interface. This improves the efficiency of financial data management and solves new challenges.

[0006] "Voice input" refers to the process of users entering information using their voice, and the technology that allows machines to recognize that voice and convert it into text data.

[0007] A "scanning device" is a device used to convert physical documents and images into digital data, and is an input method for performing character recognition using OCR technology.

[0008] A "server" refers to a computer system that receives data sent from clients via a network and processes, stores, and analyzes it.

[0009] A "database" is a system used to efficiently manage a collection of structured data. It is accessed by a server and used for recording and retrieving information.

[0010] "Progress monitoring" refers to techniques and methods for checking the progress of processes and operations in real time and detecting anomalies or problems early.

[0011] An "alert" is a mechanism that notifies the user when certain conditions are met, with the purpose of warning about the occurrence of potential problems.

[0012] "Feedback" refers to the responses and advice that a system returns based on user actions and inputs, and is information that contributes to improving the user experience.

[0013] A "dashboard" is a visual interface that displays data, designed to allow users to see important information at a glance. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention is an automated financial data processing system that utilizes voice input and scan input, enabling users to manage their budgets more efficiently and accurately. The invention begins with the user performing voice input or scan input using a terminal including a PC, tablet, or smartphone. When the user inputs financial data such as "This month's CAPEX is 2 million yen," the terminal uses voice recognition technology to convert this voice into text data.

[0036] Similarly, paper receipts obtained by the user are digitized using the terminal's scanner, and OCR technology is used to convert the image data into text data. This data is sent to a server, which cross-references it with a financial database to verify data integrity.

[0037] The server further analyzes previously stored data to predict future budgets and expenditures. The predictions generated by the analysis are provided to the user through a visually organized dashboard, allowing the user to constantly monitor progress in real time. The server sends an alert to the user's terminal if the progress of a project or other activity exceeds a predetermined threshold.

[0038] To enable users to manage their budgets efficiently, the server generates personalized feedback. Based on previously entered data and user usage patterns, it provides strategic advice as well as suggestions for improvement and warnings. This allows users to automate many tasks that were previously done manually, enabling more accurate financial data management that supports business growth.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The user uses the device's voice input function to speak, for example, "This month's CAPEX is 2 million yen." The device activates its voice recognition technology and converts this speech into text data.

[0042] Step 2:

[0043] The user scans paper receipts and other documents using a scanner connected to the device. The device uses OCR technology to convert the scanned image data into text data.

[0044] Step 3:

[0045] The terminal sends the converted text data to the server. The server verifies the received data against the financial database to confirm its integrity. If data inconsistencies or anomalies are detected, the server logs the errors.

[0046] Step 4:

[0047] The server analyzes financial data in the database and makes forecasts for future budgets and expenditures based on historical data. The server then formats the generated forecast data for visualization and prepares it for display on a dashboard.

[0048] Step 5:

[0049] The server performs real-time monitoring and sends an alert notification to the terminal if the financial data in the database exceeds a user-defined threshold. This information is provided to the user immediately.

[0050] Step 6:

[0051] Users can view the latest financial data and forecast results through the dashboard. The server generates and displays feedback and financial advice based on the user's past behavior history and input patterns, as needed.

[0052] (Example 1)

[0053] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0054] Conventional accounting information management systems have struggled to efficiently convert voice input and scanned data into text information, and to use that information to predict future budgets and expenses. Furthermore, providing users with immediate and accurate progress updates and responding quickly to anomaly detections have been challenges.

[0055] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0056] In this invention, the server includes a device that acquires information from a user via voice and converts it into text information, a device that converts image information acquired from a recording medium into text information, and a device that verifies the text information as accounting information in a central processing unit and stores it in stored information. This enables users to manage accounting information more efficiently and quickly and to forecast future expenses.

[0057] A "user" is the entity that uses this system to perform voice input or scan input, and to manage and verify information.

[0058] "Audio" and "image information" refer to data formats obtained from the user, and include input information based on sound waves and visual information.

[0059] "Textual information" refers to data in text format that has been converted from audio or image information.

[0060] A "centralized processing unit" refers to a device that functions as a server, processing and analyzing large amounts of data, and containing computing resources for storing and managing information.

[0061] "Accounting information" refers to information related to the income and expenses of a company or individual, such as financial figures and budget data.

[0062] "Stored information" refers to a collection of historical and current data stored within a centralized processing unit.

[0063] "Predictive information" refers to data generated to estimate future income and expenses based on past and present accounting information.

[0064] An "alert" is a notification presented to the user to alert them when the system detects an anomaly.

[0065] This invention provides a system aimed at improving the efficiency of accounting information processing using voice input and image recognition. Users can input voice data via a terminal, and this voice data is converted into text format by voice recognition technology. Existing voice processing technologies such as voice recognition APIs can be used for this conversion. Specifically, when a user says, "Please report this month's operating profit," the terminal converts this voice into text information and sends that information to the server.

[0066] Furthermore, users can capture information from paper documents using a scanning device and convert the images into text information using OCR technology. This process utilizes OCR libraries and software, making it possible to accurately convert paper data such as "October 5, 2023, 1,500 yen" into text.

[0067] The data acquired by the terminal is sent to a centralized processing server, where its validity is verified. The server uses Python and AI libraries to analyze historical data, generate predictive models, and perform future accounting forecasts. These forecast results are provided to the user via visual display methods and can be viewed on the terminal screen. Visual tools are used to concisely visualize the forecast results using graphs and infographics.

[0068] For example, by inputting a prompt such as "This year's advertising budget is planned to be 10% higher than last year" into the AI ​​model, the server provides a forecast that takes the budget increase into account. This allows users to perform efficient and strategic accounting management based on detailed forecast information. Because this entire process is automated quickly and accurately, it achieves a reduction in human work and an improvement in accuracy.

[0069] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0070] Step 1:

[0071] The user makes a voice input to the terminal. The voice recognition system receives this voice data and converts it into text data. The input is in voice format, and the output is text information. For example, if the user says, "Please record this month's sales profit," the voice data is converted into text data, and the text information "Please record this month's sales profit" is generated.

[0072] Step 2:

[0073] The user scans paper documents using the terminal's scanner. The terminal uses optical character recognition (OCR) technology to convert this image data into text. The input is scanned image data, and the output is text data. Specifically, information such as "October 5, 2023, 3,000 yen" is read from a paper receipt and output as text data.

[0074] Step 3:

[0075] The terminal sends the processed text data to the server. The server compares the received data with a financial database to verify its accuracy and consistency. The input is the text information sent from the terminal, and the output is accurate data whose consistency has been verified. For example, the accuracy of dates and amounts from multiple datasets is verified.

[0076] Step 4:

[0077] The server analyzes historical accounting data and generates predictions based on new data. This analysis utilizes generative AI models and data analysis libraries. Inputs are existing and new data within the financial database, and output is forecast information for future budgets and expenditures. For example, it might predict that next month's advertising spending could increase by 8% from the forecast budget.

[0078] Step 5:

[0079] The server visualizes the forecast results as a dashboard and sends it to the terminal. Users can view the forecast results in real time and make decisions based on them. The input is forecasted financial information, and the output is information in the form of visually organized infographics and graphs. For example, visual information is generated that displays budget fluctuations with arrows and color coding.

[0080] Step 6:

[0081] The server generates and provides personalized feedback and advice to the user based on the situation. The feedback is based on past usage patterns and current data analysis. Inputs are the user's historical data and the latest analysis results, while outputs are strategic advice and points of caution. For example, it might include "suggestions for cost reduction methods to cope with increased advertising spending next month."

[0082] (Application Example 1)

[0083] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0084] In modern households, efficiently and reliably managing daily income and expenses and planning for the future is a burden for many. Manual household budgeting, in particular, is time-consuming, laborious, and often inaccurate. Furthermore, predicting future budgets and expenses is difficult, and access to appropriate feedback and advice is limited. To address these challenges, there is a need for systems that automate the efficient processing and forecasting of financial data using voice and scan input.

[0085] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0086] In this invention, the server includes means for converting information acquired from a user by voice into text data, means for converting image data obtained by a data acquisition device into text, and means for verifying the text data as data and storing it in an information processing device. This enables the automation of income and expenditure management through voice recognition within the home, detailed household budget analysis using scanned data, and appropriate feedback based on predictions.

[0087] A "user" refers to an individual or organization that uses a system to input information through voice or image data.

[0088] "Information acquired through voice" refers to various data input using the user's voice, and it is assumed that the system recognizes and processes this data.

[0089] "Text data" refers to digital data used to convert and store non-textual information such as audio and images.

[0090] A "data acquisition device" refers to a device used to scan paper receipts and documents owned by a user and acquire them as digital images.

[0091] "Image data" refers to information that represents physical form in a digital format, and includes visual information acquired through scanning.

[0092] An "information processing device" is a computer system used to verify and process text and image data, and it has the function of storing and analyzing data.

[0093] A "storage device" refers to a digital storage medium or system for saving verified data.

[0094] A "visual output device" refers to a display or projection system that shows processed or analyzed data in a way that is easy for humans to understand.

[0095] A "device that performs an action in response to user input" refers to a hardware or software system that performs a specific action or response in response to a user's request or input.

[0096] This invention provides an efficient household budget management system using voice and scan data within the home. A detailed description of its implementation follows.

[0097] Users can input information about their daily expenses and income by speaking to a voice recognition device in their home. This voice recognition utilizes common voice recognition software. The voice recognition device converts the user's voice commands into text and sends it to an information processing device. This information processing device includes a storage device where the user's financial data is stored.

[0098] In addition, users can capture physical documents such as paper receipts as digital image data by holding them over a data acquisition device in their home. These scanned images are then converted into text data using image processing software. Open-source OCR software can be used for this data conversion. The information processing device then uses the acquired text data to manage income and expenses and predict future financial conditions. For example, it can use data analysis libraries such as Python's pandas and scikit-learn to analyze past data and predict future budgets.

[0099] The predictions and analysis results generated by the information processing device are provided to the user in real time through a visual output device. This visual output is expected to utilize a home display device. Data visualization tools such as Python's Dash will be used to visualize the prediction data. If user-defined thresholds are exceeded, a warning message will be sent to prompt immediate action.

[0100] For example, when a user says to the device, "Tell me about my spending this month," voice recognition technology converts the voice into text data, and the information processing device analyzes past spending data stored in the storage device, displaying the current situation on the screen. Advice on future budgets is also provided to support the user's household financial management.

[0101] An example of a prompt for a generated AI model is: "Please suggest a way to automate budget management by recognizing household financial data from the user's voice and classifying it into the appropriate categories. Specifically, please explain in detail how the content should be analyzed and predicted when the user speaks to the robot."

[0102] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0103] Step 1:

[0104] The user provides voice input to the device. Using voice recognition technology, the voice is captured as digital audio data. The input is the user's voice commands, and the output is digital audio data.

[0105] Step 2:

[0106] The device converts acquired digital audio data into text data using speech recognition software. Specifically, it analyzes the audio data into a string and temporarily stores the conversion result. The input is digital audio data, and the output is text data.

[0107] Step 3:

[0108] The user holds the receipt over the data acquisition device to scan the image data. The terminal captures the image and converts it into text data using OCR software. The input is the image data of the receipt, and the output is the converted text data.

[0109] Step 4:

[0110] The server verifies the converted text data and stores it in the storage device as financial data. It performs processing to verify data integrity by comparing it with historical data. The input is text data obtained from speech and scanning, and the output is verified stored data.

[0111] Step 5:

[0112] The server uses accumulated data to predict future spending and budgets. It employs data analysis software to perform statistical calculations based on past trends. The input is accumulated data, and the output is predicted financial data.

[0113] Step 6:

[0114] The server transmits the analysis results to a device that displays them visually, showing the user the progress in real time. Specifically, it displays the data on a screen in the form of graphs and charts. The input is predicted financial data, and the output is visualized information.

[0115] Step 7:

[0116] The server sends an alert to the user terminal if an anomaly is detected. It generates a prompt to inform the user of the alert via the notification system. The input is the monitoring result, and the output is the warning message.

[0117] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0118] This invention combines an emotion recognition function with a system that automatically processes financial data through voice and scan input, designed to enable users to manage their budgets more effectively and flexibly. The user provides information such as "This month's CAPEX is 2 million yen" by using voice input via a terminal. The terminal converts this voice into text data using its voice recognition function, and simultaneously utilizes an emotion engine to recognize emotions from the user's speech and tone of voice.

[0119] Furthermore, when a user digitizes paper receipts or other documents using a scanner, the terminal uses OCR technology to convert them into text data, and an emotion engine infers the user's emotions based on their actions during the scanning process. This converted data is sent to a server. The server verifies the received data against a financial database to detect inconsistencies and anomalies. At this time, emotion data is also recorded and used to customize reactions.

[0120] The server analyzes past financial data based on the received data and uses generated AI to predict future budgets and expenditures. In addition, the server refers to the user's past emotional history to generate appropriate feedback and advice. For example, if the server determines that the user is anxious due to a budget shortage, it will prioritize suggesting specific solutions to alleviate those emotions.

[0121] The dashboard displays personalized information using an emotion engine, making it easier for users to understand their financial situation in real time. Furthermore, if anomalies are detected in the financial data, the server determines notification priorities based on emotion and sends alerts to the device. In this way, the system, which combines emotion recognition, reduces the mental stress of financial management, enabling more strategic and effective budget management.

[0122] The following describes the processing flow.

[0123] Step 1:

[0124] The user speaks into the device's microphone and enters the details of this month's budget by voice. The device uses speech recognition technology to convert the voice information into text data, and also uses an emotion engine to analyze the user's emotions based on the tone and speed of their voice.

[0125] Step 2:

[0126] The user digitizes receipts using a scanner connected to the device. The device activates its OCR function and extracts text data from the scanned images, while recording the user's operation speed and emotions inferred from post-processing.

[0127] Step 3:

[0128] The terminal integrates text data and sentiment data and sends it to the server. The server receives this data and verifies its validity by comparing it with an existing financial database. If inconsistencies are found, an error log is generated and corrective actions are suggested.

[0129] Step 4:

[0130] The server uses the received financial data to perform historical data analysis. It utilizes generative AI to predict future budgets and expenditures. The analysis results, along with the user's emotional history, are used to generate optimal feedback.

[0131] Step 5:

[0132] The server updates the dashboard, providing users with visually formatted data along with information adjusted by the emotion engine. This allows users to view the data in a way that takes their current emotional state into account.

[0133] Step 6:

[0134] The server runs a monitoring process, tracking fluctuations in financial and emotional data in real time. Financial data exceeding a set threshold sends alerts to the device with priority based on the user's emotional state.

[0135] Step 7:

[0136] When users receive an alert, they can take action based on feedback and emotion-based advice from the server. This allows for efficient and effective budget management while reducing stress.

[0137] (Example 2)

[0138] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0139] In modern financial management, users spend a great deal of time and effort inputting and analyzing income and expense information. Furthermore, information and warnings presented without considering the user's emotional state contribute to increased stress. This invention aims to solve these problems and enable users to manage their financial data efficiently and comfortably.

[0140] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0141] In this invention, the server includes means for analyzing emotions from the user's voice and generating emotion data, means for transmitting the converted text data and emotion data to an aggregation device, and means for the aggregation device to analyze past financial information and emotion data and predict future budgets and expenditures. This enables the user to receive personalized information and feedback according to their emotions.

[0142] A "user" refers to an individual or group that uses the system for voice input or scan input.

[0143] "Information entered by voice" refers to information that should be digitized and transmitted to the device by the user speaking.

[0144] "Text data" refers to information expressed using letters and numbers, converted from audio or images.

[0145] A "digital conversion device" refers to a device used to convert physical information, such as paper documents, into digital information.

[0146] "Image information" refers to visual information obtained from physical documents, including items that are stored and analyzed in digital format.

[0147] "Emotional analysis" refers to the process of automatically evaluating a user's emotional state based on their voice and behavior.

[0148] "Emotional data" refers to information that digitally represents the emotional state of an analyzed user.

[0149] "Data collection devices" refer to information technology devices such as computers and servers that collect and process various types of data.

[0150] "Financial information" refers to numerical data related to budgets, expenses, income, etc.

[0151] A "storage device" refers to a computer device used to store digital data for the long term.

[0152] "Forecasting future budgets and expenditures" refers to the process of estimating future financial conditions based on historical data.

[0153] "Notification priority" refers to the criteria used to determine how urgently notifications and alerts should be sent to users.

[0154] A "visualization device" refers to a device that includes displays and monitors for presenting data to users in the form of graphics or text.

[0155] This invention is a system for users to effectively manage their financial data, collecting data through voice input and scanning input, and processing it in combination with emotion recognition functionality.

[0156] The user uses a device equipped with voice input capabilities to provide information by voice, such as "This month's CAPEX is 2 million yen." The device uses commercially available voice recognition software (e.g., general voice recognition software) to convert the voice data into text data. Furthermore, the device uses an emotion analysis engine (e.g., emotion analysis software) to infer the user's emotions from their voice tone and speech content.

[0157] Furthermore, when a user scans paper documents such as receipts using a digital conversion device, the terminal uses OCR technology (e.g., general OCR software) to convert the image information into text data. In this process, too, emotion data is generated from the user's actions.

[0158] The converted text and sentiment data are sent to a server via the internet. The server receives this data, verifies it as financial information, and stores it in storage. The server further analyzes historical financial and sentiment data and uses a generative AI model (e.g., an AI analysis model) to predict future budgets and expenditures. This provides users with guidance for creating financial plans.

[0159] Furthermore, the server generates personalized feedback and advice based on emotional and financial data. This allows users to receive suggestions and warnings that take their emotional state into consideration, enabling financial management with reduced mental stress.

[0160] As a concrete example, an example of a prompt message is shown below.

[0161] "Please suggest budget improvements that reflect my feelings. I'm worried because my expenses are high this month."

[0162] This system allows users to receive emotionally sensitive information and achieve effective and comfortable budget management.

[0163] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0164] Step 1:

[0165] Users input information into the terminal using the voice input function. Specifically, users provide instructions to the terminal by voice, such as "This month's CAPEX is 2 million yen." This input is used to acquire voice data. The terminal then uses voice recognition software to convert this voice data into text data. Here, voice recognition technology is applied to analyze sound waves and convert them into text.

[0166] Step 2:

[0167] The device performs sentiment analysis using text data obtained from voice data. The sentiment analysis engine operates based on the tone, pitch, and speed of the user's voice, generating sentiment data. Here, the characteristic features of the input voice are extracted, and this data is passed through an analysis algorithm to identify the emotional state.

[0168] Step 3:

[0169] The user scans a physical receipt using a digital conversion device. This operation inputs image information. The terminal applies OCR technology to convert the scanned image information into text data. Specifically, it uses image processing technology to identify characters in the image and convert them into digital text format.

[0170] Step 4:

[0171] The device observes the user's actions during scanning operations and acquires data to infer their emotions. The emotion analysis engine evaluates the operation speed and repetitive movements to generate emotional data during the scan. Specifically, it monitors the user's behavior and calculates emotional indicators through behavioral pattern analysis.

[0172] Step 5:

[0173] The terminal sends the converted text data and generated sentiment data together to the server. Here, a security protocol is used to securely transfer the data, and the server confirms receipt of the data.

[0174] Step 6:

[0175] The server verifies the received text data as financial information and compares it with existing databases. Here, it executes database queries to confirm data consistency and accuracy and retrieves the verification results.

[0176] Step 7:

[0177] The server analyzes sentiment data and historical financial information, and uses a generative AI model to predict future budgets and expenditures. Here, the AI ​​algorithm is applied to output future predictions based on past data patterns.

[0178] Step 8:

[0179] The server sends users real-time progress updates and alerts when anomalies are detected. It prioritizes notifications based on emotional data and sends alerts accordingly. Specifically, it uses a priority notification mechanism based on the user's emotional state to send necessary information to the user's device.

[0180] (Application Example 2)

[0181] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0182] In modern society, many individuals struggle with budget management and reviewing their spending. While traditional financial management systems excel at accurate data analysis and future forecasting, they have struggled to provide support that considers the user's emotions and psychological state. Therefore, there is a growing demand for systems that can reduce stress and provide personalized advice based on emotions.

[0183] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0184] This invention includes a server comprising means for recognizing the user's emotional state based on voice or image input, a method comprising a generation algorithm for generating individual suggestions corresponding to the emotional state, and means for generating individual feedback and advice. This enables personalized financial management support based on emotions.

[0185] "Voice input" is a method of supplying information to a device using spoken language.

[0186] "Text data" refers to information in text format that has been converted from audio or image information.

[0187] An "image acquisition device" is a device that converts physical documents and images into digital data.

[0188] A "communication device" is a computer system used to send and receive data and perform certain processing.

[0189] "Financial data" refers to financial information related to income, expenses, budgets, etc.

[0190] A "storage device" is a hardware configuration for storing and retaining data and information.

[0191] "Progress notification" refers to a system informing users of the status of their operations or data processing.

[0192] "Warning methods" refer to methods used by the system to alert users when it detects a specified anomaly or problem.

[0193] "Emotional state" refers to the psychological or emotional state inferred from the user's statements and actions.

[0194] A "generative algorithm" is a computational method for creating new proposals or data based on specific input information.

[0195] An "information display device" is a display device or interface that provides digital information to users visually.

[0196] The system for carrying out the present invention consists of an integrated unit for voice input, image acquisition, and data transmission / reception. The user provides financial information using a mobile terminal through voice input and camera scanning. The terminal converts this voice into text data using Google's (registered trademark) speech recognition API. It also scans physical documents such as paper receipts using an image acquisition device and performs OCR processing using the Google Cloud Vision API. The acquired text data is then transferred to a server, which is a communication device.

[0197] The server receives this text data and verifies it by comparing it with historical financial data. The Emotion API of Microsoft® Azure® analyzes the user's utterances and psychological state during operations. The analyzed emotional state is then used to provide the user with emotion-based feedback and suggestions, utilizing a generative AI model, specifically OpenAI®'s GPT-4®. This enables more personalized financial management support that reduces psychological stress for the user.

[0198] As a concrete example, let's consider a scenario where a user enters "My dining-out expenses this month have exceeded my budget." In this case, the server detects the user's anxiety and generates and provides a suggestion for a "dining-out expense reduction plan to optimize the budget."

[0199] An example of a prompt for the generating AI model would be, "Present effective cost-cutting strategies when a user is stressed about being over budget."

[0200] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0201] Step 1:

[0202] The user provides input via voice or camera scan through their device. Voice data is converted to text data by Google's speech recognition API. Meanwhile, scanned image data is processed using OCR (optical character recognition) by the Google Cloud Vision API. In this process, the input is voice or image, and the output is text data.

[0203] Step 2:

[0204] The terminal sends the converted text data to the server. The server receives this data and verifies it as financial data. Specifically, it compares the input text data with an existing financial database to check for consistency and anomalies. The output of this step is the verification results and data ready to be sent to the server for sentiment recognition.

[0205] Step 3:

[0206] The server uses Microsoft Azure's Emotion API to recognize the user's emotional state based on their voice tone and actions during scanning. Inputs are voice data and action logs, while output is user emotional state data. Information based on emotion recognition is used to generate feedback.

[0207] Step 4:

[0208] The server uses OpenAI's GPT-4 generative AI model to generate feedback and suggestions based on the user's financial data and emotional state. Here, past financial data, current text data, and emotional state are used as input, and personalized suggestions are produced as output. The specific operation is carried out by algorithmic processing performed by the AI ​​model.

[0209] Step 5:

[0210] The server sends the generated suggestions and feedback to the terminal and displays them to the user. A dashboard providing visual information is also updated. The input to this process is the newly generated suggestions and feedback, and the output is specific guidance messages for the user.

[0211] This series of processes allows users to receive emotion-based feedback, enabling more effective financial management.

[0212] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0213] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0214] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0215] [Second Embodiment]

[0216] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0217] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0218] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0219] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0220] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0221] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0222] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0223] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0224] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0225] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0226] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0227] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0228] This invention is an automated financial data processing system that utilizes voice input and scan input, enabling users to manage their budgets more efficiently and accurately. The invention begins with the user performing voice input or scan input using a terminal including a PC, tablet, or smartphone. When the user inputs financial data such as "This month's CAPEX is 2 million yen," the terminal uses voice recognition technology to convert this voice into text data.

[0229] Similarly, paper receipts obtained by the user are digitized using the terminal's scanner, and OCR technology is used to convert the image data into text data. This data is sent to a server, which cross-references it with a financial database to verify data integrity.

[0230] The server further analyzes previously stored data to predict future budgets and expenditures. The predictions generated by the analysis are provided to the user through a visually organized dashboard, allowing the user to constantly monitor progress in real time. The server sends an alert to the user's terminal if the progress of a project or other activity exceeds a predetermined threshold.

[0231] To enable users to manage their budgets efficiently, the server generates personalized feedback. Based on previously entered data and user usage patterns, it provides strategic advice as well as suggestions for improvement and warnings. This allows users to automate many tasks that were previously done manually, enabling more accurate financial data management that supports business growth.

[0232] The following describes the processing flow.

[0233] Step 1:

[0234] The user uses the device's voice input function to speak, for example, "This month's CAPEX is 2 million yen." The device activates its voice recognition technology and converts this speech into text data.

[0235] Step 2:

[0236] The user scans paper receipts and other documents using a scanner connected to the device. The device uses OCR technology to convert the scanned image data into text data.

[0237] Step 3:

[0238] The terminal sends the converted text data to the server. The server verifies the received data against the financial database to confirm its integrity. If data inconsistencies or anomalies are detected, the server logs the errors.

[0239] Step 4:

[0240] The server analyzes financial data in the database and makes forecasts for future budgets and expenditures based on historical data. The server then formats the generated forecast data for visualization and prepares it for display on a dashboard.

[0241] Step 5:

[0242] The server performs real-time monitoring and sends an alert notification to the terminal if the financial data in the database exceeds a user-defined threshold. This information is provided to the user immediately.

[0243] Step 6:

[0244] Users can view the latest financial data and forecast results through the dashboard. The server generates and displays feedback and financial advice based on the user's past behavior history and input patterns, as needed.

[0245] (Example 1)

[0246] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0247] Conventional accounting information management systems have struggled to efficiently convert voice input and scanned data into text information, and to use that information to predict future budgets and expenses. Furthermore, providing users with immediate and accurate progress updates and responding quickly to anomaly detections have been challenges.

[0248] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0249] In this invention, the server includes a device that acquires information from a user via voice and converts it into text information, a device that converts image information acquired from a recording medium into text information, and a device that verifies the text information as accounting information in a central processing unit and stores it in stored information. This enables users to manage accounting information more efficiently and quickly and to forecast future expenses.

[0250] A "user" is the entity that uses this system to perform voice input or scan input, and to manage and verify information.

[0251] "Audio" and "image information" refer to data formats obtained from the user, and include input information based on sound waves and visual information.

[0252] "Textual information" refers to data in text format that has been converted from audio or image information.

[0253] A "centralized processing unit" refers to a device that functions as a server, processing and analyzing large amounts of data, and containing computing resources for storing and managing information.

[0254] "Accounting information" refers to information related to the income and expenses of a company or individual, such as financial figures and budget data.

[0255] "Stored information" refers to a collection of historical and current data stored within a centralized processing unit.

[0256] "Predictive information" refers to data generated to estimate future income and expenses based on past and present accounting information.

[0257] An "alert" is a notification presented to the user to alert them when the system detects an anomaly.

[0258] This invention provides a system aimed at improving the efficiency of accounting information processing using voice input and image recognition. Users can input voice data via a terminal, and this voice data is converted into text format by voice recognition technology. Existing voice processing technologies such as voice recognition APIs can be used for this conversion. Specifically, when a user says, "Please report this month's operating profit," the terminal converts this voice into text information and sends that information to the server.

[0259] Furthermore, users can capture information from paper documents using a scanning device and convert the images into text information using OCR technology. This process utilizes OCR libraries and software, making it possible to accurately convert paper data such as "October 5, 2023, 1,500 yen" into text.

[0260] The data acquired by the terminal is sent to a centralized processing server, where its validity is verified. The server uses Python and AI libraries to analyze historical data, generate predictive models, and perform future accounting forecasts. These forecast results are provided to the user via visual display methods and can be viewed on the terminal screen. Visual tools are used to concisely visualize the forecast results using graphs and infographics.

[0261] For example, by inputting a prompt such as "This year's advertising budget is planned to be 10% higher than last year" into the AI ​​model, the server provides a forecast that takes the budget increase into account. This allows users to perform efficient and strategic accounting management based on detailed forecast information. Because this entire process is automated quickly and accurately, it achieves a reduction in human work and an improvement in accuracy.

[0262] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0263] Step 1:

[0264] The user makes a voice input to the terminal. The voice recognition system receives this voice data and converts it into text data. The input is in voice format, and the output is text information. For example, if the user says, "Please record this month's sales profit," the voice data is converted into text data, and the text information "Please record this month's sales profit" is generated.

[0265] Step 2:

[0266] The user scans paper documents using the terminal's scanner. The terminal uses optical character recognition (OCR) technology to convert this image data into text. The input is scanned image data, and the output is text data. Specifically, information such as "October 5, 2023, 3,000 yen" is read from a paper receipt and output as text data.

[0267] Step 3:

[0268] The terminal sends the processed text data to the server. The server compares the received data with a financial database to verify its accuracy and consistency. The input is the text information sent from the terminal, and the output is accurate data whose consistency has been verified. For example, the accuracy of dates and amounts from multiple datasets is verified.

[0269] Step 4:

[0270] The server analyzes historical accounting data and generates predictions based on new data. This analysis utilizes generative AI models and data analysis libraries. Inputs are existing and new data within the financial database, and output is forecast information for future budgets and expenditures. For example, it might predict that next month's advertising spending could increase by 8% from the forecast budget.

[0271] Step 5:

[0272] The server visualizes the forecast results as a dashboard and sends it to the terminal. Users can view the forecast results in real time and make decisions based on them. The input is forecasted financial information, and the output is information in the form of visually organized infographics and graphs. For example, visual information is generated that displays budget fluctuations with arrows and color coding.

[0273] Step 6:

[0274] The server generates and provides personalized feedback and advice to the user based on the situation. The feedback is based on past usage patterns and current data analysis. Inputs are the user's historical data and the latest analysis results, while outputs are strategic advice and points of caution. For example, it might include "suggestions for cost reduction methods to cope with increased advertising spending next month."

[0275] (Application Example 1)

[0276] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0277] In modern households, efficiently and reliably managing daily income and expenses and planning for the future is a burden for many. Manual household budgeting, in particular, is time-consuming, laborious, and often inaccurate. Furthermore, predicting future budgets and expenses is difficult, and access to appropriate feedback and advice is limited. To address these challenges, there is a need for systems that automate the efficient processing and forecasting of financial data using voice and scan input.

[0278] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0279] In this invention, the server includes means for converting information obtained from a user by voice into character data, means for converting image data obtained by a data acquisition device into characters, and means for verifying the character data as data and storing it in a storage device in an information processing device. As a result, automation of income and expenditure management through voice recognition within a household, detailed household analysis using scan data, and appropriate feedback based on prediction become possible.

[0280] The "user" refers to a person or organization that uses the system and inputs information through voice or image data.

[0281] The "information obtained by voice" refers to various data input using the user's voice, and it is assumed that the system recognizes and processes this.

[0282] "Character data" is digital - formatted data for converting and storing non - character - form information such as voice and images.

[0283] The "data acquisition device" refers to a device for scanning paper receipts or documents owned by the user and acquiring them as digital images.

[0284] "Image data" is information that represents physical - form information in digital form and includes visual information obtained by scanning.

[0285] The "information processing device" is a computer system for verifying and processing character data and image data, and has functions for storing and analyzing data.

[0286] The "storage device" refers to a digital storage medium or system for storing verified data.

[0287] The "device for visually outputting" refers to a display or projection system for presenting processed and analyzed data in a form that is easy for humans to understand.

[0288] A "device that performs an action in response to user input" refers to a hardware or software system that performs a specific action or response in response to a user's request or input.

[0289] This invention provides an efficient household budget management system using voice and scan data within the home. A detailed description of its implementation follows.

[0290] Users can input information about their daily expenses and income by speaking to a voice recognition device in their home. This voice recognition utilizes common voice recognition software. The voice recognition device converts the user's voice commands into text and sends it to an information processing device. This information processing device includes a storage device where the user's financial data is stored.

[0291] In addition, users can capture physical documents such as paper receipts as digital image data by holding them over a data acquisition device in their home. These scanned images are then converted into text data using image processing software. Open-source OCR software can be used for this data conversion. The information processing device then uses the acquired text data to manage income and expenses and predict future financial conditions. For example, it can use data analysis libraries such as Python's pandas and scikit-learn to analyze past data and predict future budgets.

[0292] The predictions and analysis results generated by the information processing device are provided to the user in real time through a visual output device. This visual output is expected to utilize a home display device. Data visualization tools such as Python's Dash will be used to visualize the prediction data. If user-defined thresholds are exceeded, a warning message will be sent to prompt immediate action.

[0293] For example, when a user says to the device, "Tell me about my spending this month," voice recognition technology converts the voice into text data, and the information processing device analyzes past spending data stored in the storage device, displaying the current situation on the screen. Advice on future budgets is also provided to support the user's household financial management.

[0294] An example of a prompt for a generated AI model is: "Please suggest a way to automate budget management by recognizing household financial data from the user's voice and classifying it into the appropriate categories. Specifically, please explain in detail how the content should be analyzed and predicted when the user speaks to the robot."

[0295] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0296] Step 1:

[0297] The user provides voice input to the device. Using voice recognition technology, the voice is captured as digital audio data. The input is the user's voice commands, and the output is digital audio data.

[0298] Step 2:

[0299] The device converts acquired digital audio data into text data using speech recognition software. Specifically, it analyzes the audio data into a string and temporarily stores the conversion result. The input is digital audio data, and the output is text data.

[0300] Step 3:

[0301] The user holds the receipt over the data acquisition device to scan the image data. The terminal captures the image and converts it into text data using OCR software. The input is the image data of the receipt, and the output is the converted text data.

[0302] Step 4:

[0303] The server verifies the converted text data and stores it in the storage device as financial data. It performs a process to check the data consistency by comparing it with past data. The input is text data by voice and scan, and the output is the verified stored data.

[0304] Step 5:

[0305] The server uses the stored data to predict future expenses and budgets. It performs statistical calculations based on past trends using data analysis software. The input is the stored data, and the output is the predicted financial data.

[0306] Step 6:

[0307] The server sends the analysis results to a device that visually outputs them and displays the progress status to the user in real time. As a specific operation, it displays the data on the display in the form of graphs and charts. The input is the predicted financial data, and the output is the visualized information.

[0308] Step 7:

[0309] When an abnormality is detected, the server sends a warning to the user terminal. It generates a prompt to notify the user of the alert via the notification system. The input is the monitoring result, and the output is the warning message.

[0310] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform specific processing using the user's emotion.

[0311] This invention combines an emotion recognition function with a system that automatically processes financial data through voice and scan input, designed to enable users to manage their budgets more effectively and flexibly. The user provides information such as "This month's CAPEX is 2 million yen" by using voice input via a terminal. The terminal converts this voice into text data using its voice recognition function, and simultaneously utilizes an emotion engine to recognize emotions from the user's speech and tone of voice.

[0312] Furthermore, when a user digitizes paper receipts or other documents using a scanner, the terminal uses OCR technology to convert them into text data, and an emotion engine infers the user's emotions based on their actions during the scanning process. This converted data is sent to a server. The server verifies the received data against a financial database to detect inconsistencies and anomalies. At this time, emotion data is also recorded and used to customize reactions.

[0313] The server analyzes past financial data based on the received data and uses generated AI to predict future budgets and expenditures. In addition, the server refers to the user's past emotional history to generate appropriate feedback and advice. For example, if the server determines that the user is anxious due to a budget shortage, it will prioritize suggesting specific solutions to alleviate those emotions.

[0314] The dashboard displays personalized information using an emotion engine, making it easier for users to understand their financial situation in real time. Furthermore, if anomalies are detected in the financial data, the server determines notification priorities based on emotion and sends alerts to the device. In this way, the system, which combines emotion recognition, reduces the mental stress of financial management, enabling more strategic and effective budget management.

[0315] The following describes the processing flow.

[0316] Step 1:

[0317] The user speaks into the device's microphone and enters the details of this month's budget by voice. The device uses speech recognition technology to convert the voice information into text data, and also uses an emotion engine to analyze the user's emotions based on the tone and speed of their voice.

[0318] Step 2:

[0319] The user digitizes receipts using a scanner connected to the device. The device activates its OCR function and extracts text data from the scanned images, while recording the user's operation speed and emotions inferred from post-processing.

[0320] Step 3:

[0321] The terminal integrates text data and sentiment data and sends it to the server. The server receives this data and verifies its validity by comparing it with an existing financial database. If inconsistencies are found, an error log is generated and corrective actions are suggested.

[0322] Step 4:

[0323] The server uses the received financial data to perform historical data analysis. It utilizes generative AI to predict future budgets and expenditures. The analysis results, along with the user's emotional history, are used to generate optimal feedback.

[0324] Step 5:

[0325] The server updates the dashboard, providing users with visually formatted data along with information adjusted by the emotion engine. This allows users to view the data in a way that takes their current emotional state into account.

[0326] Step 6:

[0327] The server runs a monitoring process, tracking fluctuations in financial and emotional data in real time. Financial data exceeding a set threshold sends alerts to the device with priority based on the user's emotional state.

[0328] Step 7:

[0329] When users receive an alert, they can take action based on feedback and emotion-based advice from the server. This allows for efficient and effective budget management while reducing stress.

[0330] (Example 2)

[0331] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0332] In modern financial management, users spend a great deal of time and effort inputting and analyzing income and expense information. Furthermore, information and warnings presented without considering the user's emotional state contribute to increased stress. This invention aims to solve these problems and enable users to manage their financial data efficiently and comfortably.

[0333] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0334] In this invention, the server includes means for analyzing emotions from the user's voice and generating emotion data, means for transmitting the converted text data and emotion data to an aggregation device, and means for the aggregation device to analyze past financial information and emotion data and predict future budgets and expenditures. This enables the user to receive personalized information and feedback according to their emotions.

[0335] A "user" refers to an individual or group that uses the system for voice input or scan input.

[0336] "Information entered by voice" refers to information that should be digitized and transmitted to the device by the user speaking.

[0337] "Text data" refers to information expressed using letters and numbers, converted from audio or images.

[0338] A "digital conversion device" refers to a device used to convert physical information, such as paper documents, into digital information.

[0339] "Image information" refers to visual information obtained from physical documents, including items that are stored and analyzed in digital format.

[0340] "Emotional analysis" refers to the process of automatically evaluating a user's emotional state based on their voice and behavior.

[0341] "Emotional data" refers to information that digitally represents the emotional state of an analyzed user.

[0342] "Data collection devices" refer to information technology devices such as computers and servers that collect and process various types of data.

[0343] "Financial information" refers to numerical data related to budgets, expenses, income, etc.

[0344] A "storage device" refers to a computer device used to store digital data for the long term.

[0345] "Forecasting future budgets and expenditures" refers to the process of estimating future financial conditions based on historical data.

[0346] "Notification priority" refers to the criteria used to determine how urgently notifications and alerts should be sent to users.

[0347] A "visualization device" refers to a device that includes displays and monitors for presenting data to users in the form of graphics or text.

[0348] This invention is a system for users to effectively manage their financial data, collecting data through voice input and scanning input, and processing it in combination with emotion recognition functionality.

[0349] The user uses a device equipped with voice input capabilities to provide information by voice, such as "This month's CAPEX is 2 million yen." The device uses commercially available voice recognition software (e.g., general voice recognition software) to convert the voice data into text data. Furthermore, the device uses an emotion analysis engine (e.g., emotion analysis software) to infer the user's emotions from their voice tone and speech content.

[0350] Furthermore, when a user scans paper documents such as receipts using a digital conversion device, the terminal uses OCR technology (e.g., general OCR software) to convert the image information into text data. In this process, too, emotion data is generated from the user's actions.

[0351] The converted text and sentiment data are sent to a server via the internet. The server receives this data, verifies it as financial information, and stores it in storage. The server further analyzes historical financial and sentiment data and uses a generative AI model (e.g., an AI analysis model) to predict future budgets and expenditures. This provides users with guidance for creating financial plans.

[0352] Furthermore, the server generates personalized feedback and advice based on emotional and financial data. This allows users to receive suggestions and warnings that take their emotional state into consideration, enabling financial management with reduced mental stress.

[0353] As a concrete example, an example of a prompt message is shown below.

[0354] "Please suggest budget improvements that reflect my feelings. I'm worried because my expenses are high this month."

[0355] This system allows users to receive emotionally sensitive information and achieve effective and comfortable budget management.

[0356] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0357] Step 1:

[0358] Users input information into the terminal using the voice input function. Specifically, users provide instructions to the terminal by voice, such as "This month's CAPEX is 2 million yen." This input is used to acquire voice data. The terminal then uses voice recognition software to convert this voice data into text data. Here, voice recognition technology is applied to analyze sound waves and convert them into text.

[0359] Step 2:

[0360] The device performs sentiment analysis using text data obtained from voice data. The sentiment analysis engine operates based on the tone, pitch, and speed of the user's voice, generating sentiment data. Here, the characteristic features of the input voice are extracted, and this data is passed through an analysis algorithm to identify the emotional state.

[0361] Step 3:

[0362] The user scans a physical receipt using a digital conversion device. This operation inputs image information. The terminal applies OCR technology to convert the scanned image information into text data. Specifically, it uses image processing technology to identify characters in the image and convert them into digital text format.

[0363] Step 4:

[0364] The device observes the user's actions during scanning operations and acquires data to infer their emotions. The emotion analysis engine evaluates the operation speed and repetitive movements to generate emotional data during the scan. Specifically, it monitors the user's behavior and calculates emotional indicators through behavioral pattern analysis.

[0365] Step 5:

[0366] The terminal sends the converted text data and generated sentiment data together to the server. Here, a security protocol is used to securely transfer the data, and the server confirms receipt of the data.

[0367] Step 6:

[0368] The server verifies the received text data as financial information and compares it with existing databases. Here, it executes database queries to confirm data consistency and accuracy and retrieves the verification results.

[0369] Step 7:

[0370] The server analyzes sentiment data and historical financial information, and uses a generative AI model to predict future budgets and expenditures. Here, the AI ​​algorithm is applied to output future predictions based on past data patterns.

[0371] Step 8:

[0372] The server sends users real-time progress updates and alerts when anomalies are detected. It prioritizes notifications based on emotional data and sends alerts accordingly. Specifically, it uses a priority notification mechanism based on the user's emotional state to send necessary information to the user's device.

[0373] (Application Example 2)

[0374] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0375] In modern society, many individuals struggle with budget management and reviewing their spending. While traditional financial management systems excel at accurate data analysis and future forecasting, they have struggled to provide support that considers the user's emotions and psychological state. Therefore, there is a growing demand for systems that can reduce stress and provide personalized advice based on emotions.

[0376] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0377] This invention includes a server comprising means for recognizing the user's emotional state based on voice or image input, a method comprising a generation algorithm for generating individual suggestions corresponding to the emotional state, and means for generating individual feedback and advice. This enables personalized financial management support based on emotions.

[0378] "Voice input" is a method of supplying information to a device using spoken language.

[0379] "Text data" refers to information in text format that has been converted from audio or image information.

[0380] An "image acquisition device" is a device that converts physical documents and images into digital data.

[0381] A "communication device" is a computer system used to send and receive data and perform certain processing.

[0382] "Financial data" refers to financial information related to income, expenses, budgets, etc.

[0383] A "storage device" is a hardware configuration for storing and retaining data and information.

[0384] "Progress notification" refers to a system informing users of the status of their operations or data processing.

[0385] "Warning methods" refer to methods used by the system to alert users when it detects a specified anomaly or problem.

[0386] "Emotional state" refers to the psychological or emotional state inferred from the user's statements and actions.

[0387] A "generative algorithm" is a computational method for creating new proposals or data based on specific input information.

[0388] An "information display device" is a display device or interface that provides digital information to users visually.

[0389] The system for implementing the present invention consists of an integrated unit comprising voice input, image acquisition, and data transmission / reception. The user provides financial information using a mobile terminal via voice input or camera scanning. The terminal converts this voice into text data using Google's speech recognition API. It also scans physical documents such as paper receipts using an image acquisition device and performs OCR processing using the Google Cloud Vision API. The resulting text data is then transferred to a server, which is a communication device.

[0390] The server receives this text data and verifies it against historical financial data. Microsoft Azure's Emotion API analyzes the user's utterances and psychological state during their actions. The analyzed emotional state is then used with a generative AI model, specifically OpenAI's GPT-4, to provide the user with emotion-based feedback and suggestions. This results in more personalized financial management support that reduces psychological stress for the user.

[0391] As a concrete example, let's consider a scenario where a user enters "My dining-out expenses this month have exceeded my budget." In this case, the server detects the user's anxiety and generates and provides a suggestion for a "dining-out expense reduction plan to optimize the budget."

[0392] An example of a prompt for the generating AI model would be, "Present effective cost-cutting strategies when a user is stressed about being over budget."

[0393] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0394] Step 1:

[0395] The user provides input via voice or camera scan through their device. Voice data is converted to text data by Google's speech recognition API. Meanwhile, scanned image data is processed using OCR (optical character recognition) by the Google Cloud Vision API. In this process, the input is voice or image, and the output is text data.

[0396] Step 2:

[0397] The terminal sends the converted text data to the server. The server receives this data and verifies it as financial data. Specifically, it compares the input text data with an existing financial database to check for consistency and anomalies. The output of this step is the verification results and data ready to be sent to the server for sentiment recognition.

[0398] Step 3:

[0399] The server uses Microsoft Azure's Emotion API to recognize the user's emotional state based on their voice tone and actions during scanning. Inputs are voice data and action logs, while output is user emotional state data. Information based on emotion recognition is used to generate feedback.

[0400] Step 4:

[0401] The server uses OpenAI's GPT-4 generative AI model to generate feedback and suggestions based on the user's financial data and emotional state. Here, past financial data, current text data, and emotional state are used as input, and personalized suggestions are produced as output. The specific operation is carried out by algorithmic processing performed by the AI ​​model.

[0402] Step 5:

[0403] The server sends the generated suggestions and feedback to the terminal and displays them to the user. A dashboard providing visual information is also updated. The input to this process is the newly generated suggestions and feedback, and the output is specific guidance messages for the user.

[0404] This series of processes allows users to receive emotion-based feedback, enabling more effective financial management.

[0405] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0406] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0407] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0408] [Third Embodiment]

[0409] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0410] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0411] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0412] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0413] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0414] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0415] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0416] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0417] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0418] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0419] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0420] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0421] This invention is an automated financial data processing system that utilizes voice input and scan input, enabling users to manage their budgets more efficiently and accurately. The invention begins with the user performing voice input or scan input using a terminal including a PC, tablet, or smartphone. When the user inputs financial data such as "This month's CAPEX is 2 million yen," the terminal uses voice recognition technology to convert this voice into text data.

[0422] Similarly, paper receipts obtained by the user are digitized using the terminal's scanner, and OCR technology is used to convert the image data into text data. This data is sent to a server, which cross-references it with a financial database to verify data integrity.

[0423] The server further analyzes previously stored data to predict future budgets and expenditures. The predictions generated by the analysis are provided to the user through a visually organized dashboard, allowing the user to constantly monitor progress in real time. The server sends an alert to the user's terminal if the progress of a project or other activity exceeds a predetermined threshold.

[0424] To enable users to manage their budgets efficiently, the server generates personalized feedback. Based on previously entered data and user usage patterns, it provides strategic advice as well as suggestions for improvement and warnings. This allows users to automate many tasks that were previously done manually, enabling more accurate financial data management that supports business growth.

[0425] The following describes the processing flow.

[0426] Step 1:

[0427] The user uses the device's voice input function to speak, for example, "This month's CAPEX is 2 million yen." The device activates its voice recognition technology and converts this speech into text data.

[0428] Step 2:

[0429] The user scans paper receipts and other documents using a scanner connected to the device. The device uses OCR technology to convert the scanned image data into text data.

[0430] Step 3:

[0431] The terminal sends the converted text data to the server. The server verifies the received data against the financial database to confirm its integrity. If data inconsistencies or anomalies are detected, the server logs the errors.

[0432] Step 4:

[0433] The server analyzes financial data in the database and makes forecasts for future budgets and expenditures based on historical data. The server then formats the generated forecast data for visualization and prepares it for display on a dashboard.

[0434] Step 5:

[0435] The server performs real-time monitoring and sends an alert notification to the terminal if the financial data in the database exceeds a user-defined threshold. This information is provided to the user immediately.

[0436] Step 6:

[0437] Users can view the latest financial data and forecast results through the dashboard. The server generates and displays feedback and financial advice based on the user's past behavior history and input patterns, as needed.

[0438] (Example 1)

[0439] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0440] Conventional accounting information management systems have struggled to efficiently convert voice input and scanned data into text information, and to use that information to predict future budgets and expenses. Furthermore, providing users with immediate and accurate progress updates and responding quickly to anomaly detections have been challenges.

[0441] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0442] In this invention, the server includes a device that acquires information from a user via voice and converts it into text information, a device that converts image information acquired from a recording medium into text information, and a device that verifies the text information as accounting information in a central processing unit and stores it in stored information. This enables users to manage accounting information more efficiently and quickly and to forecast future expenses.

[0443] A "user" is the entity that uses this system to perform voice input or scan input, and to manage and verify information.

[0444] "Audio" and "image information" refer to data formats obtained from the user, and include input information based on sound waves and visual information.

[0445] "Textual information" refers to data in text format that has been converted from audio or image information.

[0446] A "centralized processing unit" refers to a device that functions as a server, processing and analyzing large amounts of data, and containing computing resources for storing and managing information.

[0447] "Accounting information" refers to information related to the income and expenses of a company or individual, such as financial figures and budget data.

[0448] "Stored information" refers to a collection of historical and current data stored within a centralized processing unit.

[0449] "Predictive information" refers to data generated to estimate future income and expenses based on past and present accounting information.

[0450] An "alert" is a notification presented to the user to alert them when the system detects an anomaly.

[0451] This invention provides a system aimed at improving the efficiency of accounting information processing using voice input and image recognition. Users can input voice data via a terminal, and this voice data is converted into text format by voice recognition technology. Existing voice processing technologies such as voice recognition APIs can be used for this conversion. Specifically, when a user says, "Please report this month's operating profit," the terminal converts this voice into text information and sends that information to the server.

[0452] Furthermore, users can capture information from paper documents using a scanning device and convert the images into text information using OCR technology. This process utilizes OCR libraries and software, making it possible to accurately convert paper data such as "October 5, 2023, 1,500 yen" into text.

[0453] The data acquired by the terminal is sent to a centralized processing server, where its validity is verified. The server uses Python and AI libraries to analyze historical data, generate predictive models, and perform future accounting forecasts. These forecast results are provided to the user via visual display methods and can be viewed on the terminal screen. Visual tools are used to concisely visualize the forecast results using graphs and infographics.

[0454] For example, by inputting a prompt such as "This year's advertising budget is planned to be 10% higher than last year" into the AI ​​model, the server provides a forecast that takes the budget increase into account. This allows users to perform efficient and strategic accounting management based on detailed forecast information. Because this entire process is automated quickly and accurately, it achieves a reduction in human work and an improvement in accuracy.

[0455] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0456] Step 1:

[0457] The user makes a voice input to the terminal. The voice recognition system receives this voice data and converts it into text data. The input is in voice format, and the output is text information. For example, if the user says, "Please record this month's sales profit," the voice data is converted into text data, and the text information "Please record this month's sales profit" is generated.

[0458] Step 2:

[0459] The user scans paper documents using the terminal's scanner. The terminal uses optical character recognition (OCR) technology to convert this image data into text. The input is scanned image data, and the output is text data. Specifically, information such as "October 5, 2023, 3,000 yen" is read from a paper receipt and output as text data.

[0460] Step 3:

[0461] The terminal sends the processed text data to the server. The server compares the received data with a financial database to verify its accuracy and consistency. The input is the text information sent from the terminal, and the output is accurate data whose consistency has been verified. For example, the accuracy of dates and amounts from multiple datasets is verified.

[0462] Step 4:

[0463] The server analyzes historical accounting data and generates predictions based on new data. This analysis utilizes generative AI models and data analysis libraries. Inputs are existing and new data within the financial database, and output is forecast information for future budgets and expenditures. For example, it might predict that next month's advertising spending could increase by 8% from the forecast budget.

[0464] Step 5:

[0465] The server visualizes the forecast results as a dashboard and sends it to the terminal. Users can view the forecast results in real time and make decisions based on them. The input is forecasted financial information, and the output is information in the form of visually organized infographics and graphs. For example, visual information is generated that displays budget fluctuations with arrows and color coding.

[0466] Step 6:

[0467] The server generates and provides personalized feedback and advice to the user based on the situation. The feedback is based on past usage patterns and current data analysis. Inputs are the user's historical data and the latest analysis results, while outputs are strategic advice and points of caution. For example, it might include "suggestions for cost reduction methods to cope with increased advertising spending next month."

[0468] (Application Example 1)

[0469] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0470] In modern households, efficiently and reliably managing daily income and expenses and planning for the future is a burden for many. Manual household budgeting, in particular, is time-consuming, laborious, and often inaccurate. Furthermore, predicting future budgets and expenses is difficult, and access to appropriate feedback and advice is limited. To address these challenges, there is a need for systems that automate the efficient processing and forecasting of financial data using voice and scan input.

[0471] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0472] In this invention, the server includes means for converting information acquired from a user by voice into text data, means for converting image data obtained by a data acquisition device into text, and means for verifying the text data as data and storing it in an information processing device. This enables the automation of income and expenditure management through voice recognition within the home, detailed household budget analysis using scanned data, and appropriate feedback based on predictions.

[0473] A "user" refers to an individual or organization that uses a system to input information through voice or image data.

[0474] "Information acquired through voice" refers to various data input using the user's voice, and it is assumed that the system recognizes and processes this data.

[0475] "Text data" refers to digital data used to convert and store non-textual information such as audio and images.

[0476] A "data acquisition device" refers to a device used to scan paper receipts and documents owned by a user and acquire them as digital images.

[0477] "Image data" refers to information that represents physical form in a digital format, and includes visual information acquired through scanning.

[0478] An "information processing device" is a computer system used to verify and process text and image data, and it has the function of storing and analyzing data.

[0479] A "storage device" refers to a digital storage medium or system for saving verified data.

[0480] A "visual output device" refers to a display or projection system that shows processed or analyzed data in a way that is easy for humans to understand.

[0481] A "device that performs an action in response to user input" refers to a hardware or software system that performs a specific action or response in response to a user's request or input.

[0482] This invention provides an efficient household budget management system using voice and scan data within the home. A detailed description of its implementation follows.

[0483] Users can input information about their daily expenses and income by speaking to a voice recognition device in their home. This voice recognition utilizes common voice recognition software. The voice recognition device converts the user's voice commands into text and sends it to an information processing device. This information processing device includes a storage device where the user's financial data is stored.

[0484] In addition, users can capture physical documents such as paper receipts as digital image data by holding them over a data acquisition device in their home. These scanned images are then converted into text data using image processing software. Open-source OCR software can be used for this data conversion. The information processing device then uses the acquired text data to manage income and expenses and predict future financial conditions. For example, it can use data analysis libraries such as Python's pandas and scikit-learn to analyze past data and predict future budgets.

[0485] The predictions and analysis results generated by the information processing device are provided to the user in real time through a visual output device. This visual output is expected to utilize a home display device. Data visualization tools such as Python's Dash will be used to visualize the prediction data. If user-defined thresholds are exceeded, a warning message will be sent to prompt immediate action.

[0486] For example, when a user says to the device, "Tell me about my spending this month," voice recognition technology converts the voice into text data, and the information processing device analyzes past spending data stored in the storage device, displaying the current situation on the screen. Advice on future budgets is also provided to support the user's household financial management.

[0487] An example of a prompt for a generated AI model is: "Please suggest a way to automate budget management by recognizing household financial data from the user's voice and classifying it into the appropriate categories. Specifically, please explain in detail how the content should be analyzed and predicted when the user speaks to the robot."

[0488] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0489] Step 1:

[0490] The user provides voice input to the device. Using voice recognition technology, the voice is captured as digital audio data. The input is the user's voice commands, and the output is digital audio data.

[0491] Step 2:

[0492] The device converts acquired digital audio data into text data using speech recognition software. Specifically, it analyzes the audio data into a string and temporarily stores the conversion result. The input is digital audio data, and the output is text data.

[0493] Step 3:

[0494] The user holds the receipt over the data acquisition device to scan the image data. The terminal captures the image and converts it into text data using OCR software. The input is the image data of the receipt, and the output is the converted text data.

[0495] Step 4:

[0496] The server verifies the converted text data and stores it in the storage device as financial data. It performs processing to verify data integrity by comparing it with historical data. The input is text data obtained from speech and scanning, and the output is verified stored data.

[0497] Step 5:

[0498] The server uses accumulated data to predict future spending and budgets. It employs data analysis software to perform statistical calculations based on past trends. The input is accumulated data, and the output is predicted financial data.

[0499] Step 6:

[0500] The server transmits the analysis results to a device that displays them visually, showing the user the progress in real time. Specifically, it displays the data on a screen in the form of graphs and charts. The input is predicted financial data, and the output is visualized information.

[0501] Step 7:

[0502] The server sends an alert to the user terminal if an anomaly is detected. It generates a prompt to inform the user of the alert via the notification system. The input is the monitoring result, and the output is the warning message.

[0503] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0504] This invention combines an emotion recognition function with a system that automatically processes financial data through voice and scan input, designed to enable users to manage their budgets more effectively and flexibly. The user provides information such as "This month's CAPEX is 2 million yen" by using voice input via a terminal. The terminal converts this voice into text data using its voice recognition function, and simultaneously utilizes an emotion engine to recognize emotions from the user's speech and tone of voice.

[0505] Furthermore, when a user digitizes paper receipts or other documents using a scanner, the terminal uses OCR technology to convert them into text data, and an emotion engine infers the user's emotions based on their actions during the scanning process. This converted data is sent to a server. The server verifies the received data against a financial database to detect inconsistencies and anomalies. At this time, emotion data is also recorded and used to customize reactions.

[0506] The server analyzes past financial data based on the received data and uses generated AI to predict future budgets and expenditures. In addition, the server refers to the user's past emotional history to generate appropriate feedback and advice. For example, if the server determines that the user is anxious due to a budget shortage, it will prioritize suggesting specific solutions to alleviate those emotions.

[0507] The dashboard displays personalized information using an emotion engine, making it easier for users to understand their financial situation in real time. Furthermore, if anomalies are detected in the financial data, the server determines notification priorities based on emotion and sends alerts to the device. In this way, the system, which combines emotion recognition, reduces the mental stress of financial management, enabling more strategic and effective budget management.

[0508] The following describes the processing flow.

[0509] Step 1:

[0510] The user speaks into the device's microphone and enters the details of this month's budget by voice. The device uses speech recognition technology to convert the voice information into text data, and also uses an emotion engine to analyze the user's emotions based on the tone and speed of their voice.

[0511] Step 2:

[0512] The user digitizes receipts using a scanner connected to the device. The device activates its OCR function and extracts text data from the scanned images, while recording the user's operation speed and emotions inferred from post-processing.

[0513] Step 3:

[0514] The terminal integrates text data and sentiment data and sends it to the server. The server receives this data and verifies its validity by comparing it with an existing financial database. If inconsistencies are found, an error log is generated and corrective actions are suggested.

[0515] Step 4:

[0516] The server uses the received financial data to perform historical data analysis. It utilizes generative AI to predict future budgets and expenditures. The analysis results, along with the user's emotional history, are used to generate optimal feedback.

[0517] Step 5:

[0518] The server updates the dashboard, providing users with visually formatted data along with information adjusted by the emotion engine. This allows users to view the data in a way that takes their current emotional state into account.

[0519] Step 6:

[0520] The server runs a monitoring process, tracking fluctuations in financial and emotional data in real time. Financial data exceeding a set threshold sends alerts to the device with priority based on the user's emotional state.

[0521] Step 7:

[0522] When users receive an alert, they can take action based on feedback and emotion-based advice from the server. This allows for efficient and effective budget management while reducing stress.

[0523] (Example 2)

[0524] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0525] In modern financial management, users spend a great deal of time and effort inputting and analyzing income and expense information. Furthermore, information and warnings presented without considering the user's emotional state contribute to increased stress. This invention aims to solve these problems and enable users to manage their financial data efficiently and comfortably.

[0526] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0527] In this invention, the server includes means for analyzing emotions from the user's voice and generating emotion data, means for transmitting the converted text data and emotion data to an aggregation device, and means for the aggregation device to analyze past financial information and emotion data and predict future budgets and expenditures. This enables the user to receive personalized information and feedback according to their emotions.

[0528] A "user" refers to an individual or group that uses the system for voice input or scan input.

[0529] "Information entered by voice" refers to information that should be digitized and transmitted to the device by the user speaking.

[0530] "Text data" refers to information expressed using letters and numbers, converted from audio or images.

[0531] A "digital conversion device" refers to a device used to convert physical information, such as paper documents, into digital information.

[0532] "Image information" refers to visual information obtained from physical documents, including items that are stored and analyzed in digital format.

[0533] "Emotional analysis" refers to the process of automatically evaluating a user's emotional state based on their voice and behavior.

[0534] "Emotional data" refers to information that digitally represents the emotional state of an analyzed user.

[0535] "Data collection devices" refer to information technology devices such as computers and servers that collect and process various types of data.

[0536] "Financial information" refers to numerical data related to budgets, expenses, income, etc.

[0537] A "storage device" refers to a computer device used to store digital data for the long term.

[0538] "Forecasting future budgets and expenditures" refers to the process of estimating future financial conditions based on historical data.

[0539] "Notification priority" refers to the criteria used to determine how urgently notifications and alerts should be sent to users.

[0540] A "visualization device" refers to a device that includes displays and monitors for presenting data to users in the form of graphics or text.

[0541] This invention is a system for users to effectively manage their financial data, collecting data through voice input and scanning input, and processing it in combination with emotion recognition functionality.

[0542] The user uses a device equipped with voice input capabilities to provide information by voice, such as "This month's CAPEX is 2 million yen." The device uses commercially available voice recognition software (e.g., general voice recognition software) to convert the voice data into text data. Furthermore, the device uses an emotion analysis engine (e.g., emotion analysis software) to infer the user's emotions from their voice tone and speech content.

[0543] Furthermore, when a user scans paper documents such as receipts using a digital conversion device, the terminal uses OCR technology (e.g., general OCR software) to convert the image information into text data. In this process, too, emotion data is generated from the user's actions.

[0544] The converted text and sentiment data are sent to a server via the internet. The server receives this data, verifies it as financial information, and stores it in storage. The server further analyzes historical financial and sentiment data and uses a generative AI model (e.g., an AI analysis model) to predict future budgets and expenditures. This provides users with guidance for creating financial plans.

[0545] Furthermore, the server generates personalized feedback and advice based on emotional and financial data. This allows users to receive suggestions and warnings that take their emotional state into consideration, enabling financial management with reduced mental stress.

[0546] As a concrete example, an example of a prompt message is shown below.

[0547] "Please suggest budget improvements that reflect my feelings. I'm worried because my expenses are high this month."

[0548] This system allows users to receive emotionally sensitive information and achieve effective and comfortable budget management.

[0549] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0550] Step 1:

[0551] Users input information into the terminal using the voice input function. Specifically, users provide instructions to the terminal by voice, such as "This month's CAPEX is 2 million yen." This input is used to acquire voice data. The terminal then uses voice recognition software to convert this voice data into text data. Here, voice recognition technology is applied to analyze sound waves and convert them into text.

[0552] Step 2:

[0553] The device performs sentiment analysis using text data obtained from voice data. The sentiment analysis engine operates based on the tone, pitch, and speed of the user's voice, generating sentiment data. Here, the characteristic features of the input voice are extracted, and this data is passed through an analysis algorithm to identify the emotional state.

[0554] Step 3:

[0555] The user scans a physical receipt using a digital conversion device. This operation inputs image information. The terminal applies OCR technology to convert the scanned image information into text data. Specifically, it uses image processing technology to identify characters in the image and convert them into digital text format.

[0556] Step 4:

[0557] The device observes the user's actions during scanning operations and acquires data to infer their emotions. The emotion analysis engine evaluates the operation speed and repetitive movements to generate emotional data during the scan. Specifically, it monitors the user's behavior and calculates emotional indicators through behavioral pattern analysis.

[0558] Step 5:

[0559] The terminal sends the converted text data and generated sentiment data together to the server. Here, a security protocol is used to securely transfer the data, and the server confirms receipt of the data.

[0560] Step 6:

[0561] The server verifies the received text data as financial information and compares it with existing databases. Here, it executes database queries to confirm data consistency and accuracy and retrieves the verification results.

[0562] Step 7:

[0563] The server analyzes sentiment data and historical financial information, and uses a generative AI model to predict future budgets and expenditures. Here, the AI ​​algorithm is applied to output future predictions based on past data patterns.

[0564] Step 8:

[0565] The server sends users real-time progress updates and alerts when anomalies are detected. It prioritizes notifications based on emotional data and sends alerts accordingly. Specifically, it uses a priority notification mechanism based on the user's emotional state to send necessary information to the user's device.

[0566] (Application Example 2)

[0567] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0568] In modern society, many individuals struggle with budget management and reviewing their spending. While traditional financial management systems excel at accurate data analysis and future forecasting, they have struggled to provide support that considers the user's emotions and psychological state. Therefore, there is a growing demand for systems that can reduce stress and provide personalized advice based on emotions.

[0569] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0570] This invention includes a server comprising means for recognizing the user's emotional state based on voice or image input, a method comprising a generation algorithm for generating individual suggestions corresponding to the emotional state, and means for generating individual feedback and advice. This enables personalized financial management support based on emotions.

[0571] "Voice input" is a method of supplying information to a device using spoken language.

[0572] "Text data" refers to information in text format that has been converted from audio or image information.

[0573] An "image acquisition device" is a device that converts physical documents and images into digital data.

[0574] A "communication device" is a computer system used to send and receive data and perform certain processing.

[0575] "Financial data" refers to financial information related to income, expenses, budgets, etc.

[0576] A "storage device" is a hardware configuration for storing and retaining data and information.

[0577] "Progress notification" refers to a system informing users of the status of their operations or data processing.

[0578] "Warning methods" refer to methods used by the system to alert users when it detects a specified anomaly or problem.

[0579] "Emotional state" refers to the psychological or emotional state inferred from the user's statements and actions.

[0580] A "generative algorithm" is a computational method for creating new proposals or data based on specific input information.

[0581] An "information display device" is a display device or interface that provides digital information to users visually.

[0582] The system for implementing the present invention consists of an integrated unit comprising voice input, image acquisition, and data transmission / reception. The user provides financial information using a mobile terminal via voice input or camera scanning. The terminal converts this voice into text data using Google's speech recognition API. It also scans physical documents such as paper receipts using an image acquisition device and performs OCR processing using the Google Cloud Vision API. The resulting text data is then transferred to a server, which is a communication device.

[0583] The server receives this text data and verifies it against historical financial data. Microsoft Azure's Emotion API analyzes the user's utterances and psychological state during their actions. The analyzed emotional state is then used with a generative AI model, specifically OpenAI's GPT-4, to provide the user with emotion-based feedback and suggestions. This results in more personalized financial management support that reduces psychological stress for the user.

[0584] As a concrete example, let's consider a scenario where a user enters "My dining-out expenses this month have exceeded my budget." In this case, the server detects the user's anxiety and generates and provides a suggestion for a "dining-out expense reduction plan to optimize the budget."

[0585] An example of a prompt for the generating AI model would be, "Present effective cost-cutting strategies when a user is stressed about being over budget."

[0586] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0587] Step 1:

[0588] The user provides input via voice or camera scan through their device. Voice data is converted to text data by Google's speech recognition API. Meanwhile, scanned image data is processed using OCR (optical character recognition) by the Google Cloud Vision API. In this process, the input is voice or image, and the output is text data.

[0589] Step 2:

[0590] The terminal sends the converted text data to the server. The server receives this data and verifies it as financial data. Specifically, it compares the input text data with an existing financial database to check for consistency and anomalies. The output of this step is the verification results and data ready to be sent to the server for sentiment recognition.

[0591] Step 3:

[0592] The server uses Microsoft Azure's Emotion API to recognize the user's emotional state based on their voice tone and actions during scanning. Inputs are voice data and action logs, while output is user emotional state data. Information based on emotion recognition is used to generate feedback.

[0593] Step 4:

[0594] The server uses OpenAI's GPT-4 generative AI model to generate feedback and suggestions based on the user's financial data and emotional state. Here, past financial data, current text data, and emotional state are used as input, and personalized suggestions are produced as output. The specific operation is carried out by algorithmic processing performed by the AI ​​model.

[0595] Step 5:

[0596] The server sends the generated suggestions and feedback to the terminal and displays them to the user. A dashboard providing visual information is also updated. The input to this process is the newly generated suggestions and feedback, and the output is specific guidance messages for the user.

[0597] This series of processes allows users to receive emotion-based feedback, enabling more effective financial management.

[0598] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0599] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0600] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0601] [Fourth Embodiment]

[0602] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0603] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0604] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0605] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0606] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0607] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0608] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0609] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0610] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0611] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0612] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0613] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0614] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0615] This invention is an automated financial data processing system that utilizes voice input and scan input, enabling users to manage their budgets more efficiently and accurately. The invention begins with the user performing voice input or scan input using a terminal including a PC, tablet, or smartphone. When the user inputs financial data such as "This month's CAPEX is 2 million yen," the terminal uses voice recognition technology to convert this voice into text data.

[0616] Similarly, paper receipts obtained by the user are digitized using the terminal's scanner, and OCR technology is used to convert the image data into text data. This data is sent to a server, which cross-references it with a financial database to verify data integrity.

[0617] The server further analyzes previously stored data to predict future budgets and expenditures. The predictions generated by the analysis are provided to the user through a visually organized dashboard, allowing the user to constantly monitor progress in real time. The server sends an alert to the user's terminal if the progress of a project or other activity exceeds a predetermined threshold.

[0618] To enable users to manage their budgets efficiently, the server generates personalized feedback. Based on previously entered data and user usage patterns, it provides strategic advice as well as suggestions for improvement and warnings. This allows users to automate many tasks that were previously done manually, enabling more accurate financial data management that supports business growth.

[0619] The following describes the processing flow.

[0620] Step 1:

[0621] The user uses the device's voice input function to speak, for example, "This month's CAPEX is 2 million yen." The device activates its voice recognition technology and converts this speech into text data.

[0622] Step 2:

[0623] The user scans paper receipts and other documents using a scanner connected to the device. The device uses OCR technology to convert the scanned image data into text data.

[0624] Step 3:

[0625] The terminal sends the converted text data to the server. The server verifies the received data against the financial database to confirm its integrity. If data inconsistencies or anomalies are detected, the server logs the errors.

[0626] Step 4:

[0627] The server analyzes financial data in the database and makes forecasts for future budgets and expenditures based on historical data. The server then formats the generated forecast data for visualization and prepares it for display on a dashboard.

[0628] Step 5:

[0629] The server performs real-time monitoring and sends an alert notification to the terminal if the financial data in the database exceeds a user-defined threshold. This information is provided to the user immediately.

[0630] Step 6:

[0631] Users can view the latest financial data and forecast results through the dashboard. The server generates and displays feedback and financial advice based on the user's past behavior history and input patterns, as needed.

[0632] (Example 1)

[0633] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0634] Conventional accounting information management systems have struggled to efficiently convert voice input and scanned data into text information, and to use that information to predict future budgets and expenses. Furthermore, providing users with immediate and accurate progress updates and responding quickly to anomaly detections have been challenges.

[0635] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0636] In this invention, the server includes a device that acquires information from a user via voice and converts it into text information, a device that converts image information acquired from a recording medium into text information, and a device that verifies the text information as accounting information in a central processing unit and stores it in stored information. This enables users to manage accounting information more efficiently and quickly and to forecast future expenses.

[0637] A "user" is the entity that uses this system to perform voice input or scan input, and to manage and verify information.

[0638] "Audio" and "image information" refer to data formats obtained from the user, and include input information based on sound waves and visual information.

[0639] "Textual information" refers to data in text format that has been converted from audio or image information.

[0640] A "centralized processing unit" refers to a device that functions as a server, processing and analyzing large amounts of data, and containing computing resources for storing and managing information.

[0641] "Accounting information" refers to information related to the income and expenses of a company or individual, such as financial figures and budget data.

[0642] "Stored information" refers to a collection of historical and current data stored within a centralized processing unit.

[0643] "Predictive information" refers to data generated to estimate future income and expenses based on past and present accounting information.

[0644] An "alert" is a notification presented to the user to alert them when the system detects an anomaly.

[0645] This invention provides a system aimed at improving the efficiency of accounting information processing using voice input and image recognition. Users can input voice data via a terminal, and this voice data is converted into text format by voice recognition technology. Existing voice processing technologies such as voice recognition APIs can be used for this conversion. Specifically, when a user says, "Please report this month's operating profit," the terminal converts this voice into text information and sends that information to the server.

[0646] Furthermore, users can capture information from paper documents using a scanning device and convert the images into text information using OCR technology. This process utilizes OCR libraries and software, making it possible to accurately convert paper data such as "October 5, 2023, 1,500 yen" into text.

[0647] The data acquired by the terminal is sent to a centralized processing server, where its validity is verified. The server uses Python and AI libraries to analyze historical data, generate predictive models, and perform future accounting forecasts. These forecast results are provided to the user via visual display methods and can be viewed on the terminal screen. Visual tools are used to concisely visualize the forecast results using graphs and infographics.

[0648] For example, by inputting a prompt such as "This year's advertising budget is planned to be 10% higher than last year" into the AI ​​model, the server provides a forecast that takes the budget increase into account. This allows users to perform efficient and strategic accounting management based on detailed forecast information. Because this entire process is automated quickly and accurately, it achieves a reduction in human work and an improvement in accuracy.

[0649] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0650] Step 1:

[0651] The user makes a voice input to the terminal. The voice recognition system receives this voice data and converts it into text data. The input is in voice format, and the output is text information. For example, if the user says, "Please record this month's sales profit," the voice data is converted into text data, and the text information "Please record this month's sales profit" is generated.

[0652] Step 2:

[0653] The user scans paper documents using the terminal's scanner. The terminal uses optical character recognition (OCR) technology to convert this image data into text. The input is scanned image data, and the output is text data. Specifically, information such as "October 5, 2023, 3,000 yen" is read from a paper receipt and output as text data.

[0654] Step 3:

[0655] The terminal sends the processed text data to the server. The server compares the received data with a financial database to verify its accuracy and consistency. The input is the text information sent from the terminal, and the output is accurate data whose consistency has been verified. For example, the accuracy of dates and amounts from multiple datasets is verified.

[0656] Step 4:

[0657] The server analyzes historical accounting data and generates predictions based on new data. This analysis utilizes generative AI models and data analysis libraries. Inputs are existing and new data within the financial database, and output is forecast information for future budgets and expenditures. For example, it might predict that next month's advertising spending could increase by 8% from the forecast budget.

[0658] Step 5:

[0659] The server visualizes the forecast results as a dashboard and sends it to the terminal. Users can view the forecast results in real time and make decisions based on them. The input is forecasted financial information, and the output is information in the form of visually organized infographics and graphs. For example, visual information is generated that displays budget fluctuations with arrows and color coding.

[0660] Step 6:

[0661] The server generates and provides personalized feedback and advice to the user based on the situation. The feedback is based on past usage patterns and current data analysis. Inputs are the user's historical data and the latest analysis results, while outputs are strategic advice and points of caution. For example, it might include "suggestions for cost reduction methods to cope with increased advertising spending next month."

[0662] (Application Example 1)

[0663] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0664] In modern households, efficiently and reliably managing daily income and expenses and planning for the future is a burden for many. Manual household budgeting, in particular, is time-consuming, laborious, and often inaccurate. Furthermore, predicting future budgets and expenses is difficult, and access to appropriate feedback and advice is limited. To address these challenges, there is a need for systems that automate the efficient processing and forecasting of financial data using voice and scan input.

[0665] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0666] In this invention, the server includes means for converting information acquired from a user by voice into text data, means for converting image data obtained by a data acquisition device into text, and means for verifying the text data as data and storing it in an information processing device. This enables the automation of income and expenditure management through voice recognition within the home, detailed household budget analysis using scanned data, and appropriate feedback based on predictions.

[0667] A "user" refers to an individual or organization that uses a system to input information through voice or image data.

[0668] "Information acquired through voice" refers to various data input using the user's voice, and it is assumed that the system recognizes and processes this data.

[0669] "Text data" refers to digital data used to convert and store non-textual information such as audio and images.

[0670] A "data acquisition device" refers to a device used to scan paper receipts and documents owned by a user and acquire them as digital images.

[0671] "Image data" refers to information that represents physical form in a digital format, and includes visual information acquired through scanning.

[0672] An "information processing device" is a computer system used to verify and process text and image data, and it has the function of storing and analyzing data.

[0673] A "storage device" refers to a digital storage medium or system for saving verified data.

[0674] A "visual output device" refers to a display or projection system that shows processed or analyzed data in a way that is easy for humans to understand.

[0675] A "device that performs an action in response to user input" refers to a hardware or software system that performs a specific action or response in response to a user's request or input.

[0676] This invention provides an efficient household budget management system using voice and scan data within the home. A detailed description of its implementation follows.

[0677] Users can input information about their daily expenses and income by speaking to a voice recognition device in their home. This voice recognition utilizes common voice recognition software. The voice recognition device converts the user's voice commands into text and sends it to an information processing device. This information processing device includes a storage device where the user's financial data is stored.

[0678] In addition, users can capture physical documents such as paper receipts as digital image data by holding them over a data acquisition device in their home. These scanned images are then converted into text data using image processing software. Open-source OCR software can be used for this data conversion. The information processing device then uses the acquired text data to manage income and expenses and predict future financial conditions. For example, it can use data analysis libraries such as Python's pandas and scikit-learn to analyze past data and predict future budgets.

[0679] The predictions and analysis results generated by the information processing device are provided to the user in real time through a visual output device. This visual output is expected to utilize a home display device. Data visualization tools such as Python's Dash will be used to visualize the prediction data. If user-defined thresholds are exceeded, a warning message will be sent to prompt immediate action.

[0680] For example, when a user says to the device, "Tell me about my spending this month," voice recognition technology converts the voice into text data, and the information processing device analyzes past spending data stored in the storage device, displaying the current situation on the screen. Advice on future budgets is also provided to support the user's household financial management.

[0681] An example of a prompt for a generated AI model is: "Please suggest a way to automate budget management by recognizing household financial data from the user's voice and classifying it into the appropriate categories. Specifically, please explain in detail how the content should be analyzed and predicted when the user speaks to the robot."

[0682] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0683] Step 1:

[0684] The user provides voice input to the device. Using voice recognition technology, the voice is captured as digital audio data. The input is the user's voice commands, and the output is digital audio data.

[0685] Step 2:

[0686] The device converts acquired digital audio data into text data using speech recognition software. Specifically, it analyzes the audio data into a string and temporarily stores the conversion result. The input is digital audio data, and the output is text data.

[0687] Step 3:

[0688] The user holds the receipt over the data acquisition device to scan the image data. The terminal captures the image and converts it into text data using OCR software. The input is the image data of the receipt, and the output is the converted text data.

[0689] Step 4:

[0690] The server verifies the converted text data and stores it in the storage device as financial data. It performs processing to verify data integrity by comparing it with historical data. The input is text data obtained from speech and scanning, and the output is verified stored data.

[0691] Step 5:

[0692] The server uses accumulated data to predict future spending and budgets. It employs data analysis software to perform statistical calculations based on past trends. The input is accumulated data, and the output is predicted financial data.

[0693] Step 6:

[0694] The server transmits the analysis results to a device that displays them visually, showing the user the progress in real time. Specifically, it displays the data on a screen in the form of graphs and charts. The input is predicted financial data, and the output is visualized information.

[0695] Step 7:

[0696] The server sends an alert to the user terminal if an anomaly is detected. It generates a prompt to inform the user of the alert via the notification system. The input is the monitoring result, and the output is the warning message.

[0697] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0698] This invention combines an emotion recognition function with a system that automatically processes financial data through voice and scan input, designed to enable users to manage their budgets more effectively and flexibly. The user provides information such as "This month's CAPEX is 2 million yen" by using voice input via a terminal. The terminal converts this voice into text data using its voice recognition function, and simultaneously utilizes an emotion engine to recognize emotions from the user's speech and tone of voice.

[0699] Furthermore, when a user digitizes paper receipts or other documents using a scanner, the terminal uses OCR technology to convert them into text data, and an emotion engine infers the user's emotions based on their actions during the scanning process. This converted data is sent to a server. The server verifies the received data against a financial database to detect inconsistencies and anomalies. At this time, emotion data is also recorded and used to customize reactions.

[0700] The server analyzes past financial data based on the received data and uses generated AI to predict future budgets and expenditures. In addition, the server refers to the user's past emotional history to generate appropriate feedback and advice. For example, if the server determines that the user is anxious due to a budget shortage, it will prioritize suggesting specific solutions to alleviate those emotions.

[0701] The dashboard displays personalized information using an emotion engine, making it easier for users to understand their financial situation in real time. Furthermore, if anomalies are detected in the financial data, the server determines notification priorities based on emotion and sends alerts to the device. In this way, the system, which combines emotion recognition, reduces the mental stress of financial management, enabling more strategic and effective budget management.

[0702] The following describes the processing flow.

[0703] Step 1:

[0704] The user speaks into the device's microphone and enters the details of this month's budget by voice. The device uses speech recognition technology to convert the voice information into text data, and also uses an emotion engine to analyze the user's emotions based on the tone and speed of their voice.

[0705] Step 2:

[0706] The user digitizes receipts using a scanner connected to the device. The device activates its OCR function and extracts text data from the scanned images, while recording the user's operation speed and emotions inferred from post-processing.

[0707] Step 3:

[0708] The terminal integrates text data and sentiment data and sends it to the server. The server receives this data and verifies its validity by comparing it with an existing financial database. If inconsistencies are found, an error log is generated and corrective actions are suggested.

[0709] Step 4:

[0710] The server uses the received financial data to perform historical data analysis. It utilizes generative AI to predict future budgets and expenditures. The analysis results, along with the user's emotional history, are used to generate optimal feedback.

[0711] Step 5:

[0712] The server updates the dashboard, providing users with visually formatted data along with information adjusted by the emotion engine. This allows users to view the data in a way that takes their current emotional state into account.

[0713] Step 6:

[0714] The server runs a monitoring process, tracking fluctuations in financial and emotional data in real time. Financial data exceeding a set threshold sends alerts to the device with priority based on the user's emotional state.

[0715] Step 7:

[0716] When users receive an alert, they can take action based on feedback and emotion-based advice from the server. This allows for efficient and effective budget management while reducing stress.

[0717] (Example 2)

[0718] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0719] In modern financial management, users spend a great deal of time and effort inputting and analyzing income and expense information. Furthermore, information and warnings presented without considering the user's emotional state contribute to increased stress. This invention aims to solve these problems and enable users to manage their financial data efficiently and comfortably.

[0720] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0721] In this invention, the server includes means for analyzing emotions from the user's voice and generating emotion data, means for transmitting the converted text data and emotion data to an aggregation device, and means for the aggregation device to analyze past financial information and emotion data and predict future budgets and expenditures. This enables the user to receive personalized information and feedback according to their emotions.

[0722] A "user" refers to an individual or group that uses the system for voice input or scan input.

[0723] "Information entered by voice" refers to information that should be digitized and transmitted to the device by the user speaking.

[0724] "Text data" refers to information expressed using letters and numbers, converted from audio or images.

[0725] A "digital conversion device" refers to a device used to convert physical information, such as paper documents, into digital information.

[0726] "Image information" refers to visual information obtained from physical documents, including items that are stored and analyzed in digital format.

[0727] "Emotional analysis" refers to the process of automatically evaluating a user's emotional state based on their voice and behavior.

[0728] "Emotional data" refers to information that digitally represents the emotional state of an analyzed user.

[0729] "Data collection devices" refer to information technology devices such as computers and servers that collect and process various types of data.

[0730] "Financial information" refers to numerical data related to budgets, expenses, income, etc.

[0731] A "storage device" refers to a computer device used to store digital data for the long term.

[0732] "Forecasting future budgets and expenditures" refers to the process of estimating future financial conditions based on historical data.

[0733] "Notification priority" refers to the criteria used to determine how urgently notifications and alerts should be sent to users.

[0734] A "visualization device" refers to a device that includes displays and monitors for presenting data to users in the form of graphics or text.

[0735] This invention is a system for users to effectively manage their financial data, collecting data through voice input and scanning input, and processing it in combination with emotion recognition functionality.

[0736] The user uses a device equipped with voice input capabilities to provide information by voice, such as "This month's CAPEX is 2 million yen." The device uses commercially available voice recognition software (e.g., general voice recognition software) to convert the voice data into text data. Furthermore, the device uses an emotion analysis engine (e.g., emotion analysis software) to infer the user's emotions from their voice tone and speech content.

[0737] Furthermore, when a user scans paper documents such as receipts using a digital conversion device, the terminal uses OCR technology (e.g., general OCR software) to convert the image information into text data. In this process, too, emotion data is generated from the user's actions.

[0738] The converted text and sentiment data are sent to a server via the internet. The server receives this data, verifies it as financial information, and stores it in storage. The server further analyzes historical financial and sentiment data and uses a generative AI model (e.g., an AI analysis model) to predict future budgets and expenditures. This provides users with guidance for creating financial plans.

[0739] Furthermore, the server generates personalized feedback and advice based on emotional and financial data. This allows users to receive suggestions and warnings that take their emotional state into consideration, enabling financial management with reduced mental stress.

[0740] As a concrete example, an example of a prompt message is shown below.

[0741] "Please suggest budget improvements that reflect my feelings. I'm worried because my expenses are high this month."

[0742] This system allows users to receive emotionally sensitive information and achieve effective and comfortable budget management.

[0743] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0744] Step 1:

[0745] Users input information into the terminal using the voice input function. Specifically, users provide instructions to the terminal by voice, such as "This month's CAPEX is 2 million yen." This input is used to acquire voice data. The terminal then uses voice recognition software to convert this voice data into text data. Here, voice recognition technology is applied to analyze sound waves and convert them into text.

[0746] Step 2:

[0747] The device performs sentiment analysis using text data obtained from voice data. The sentiment analysis engine operates based on the tone, pitch, and speed of the user's voice, generating sentiment data. Here, the characteristic features of the input voice are extracted, and this data is passed through an analysis algorithm to identify the emotional state.

[0748] Step 3:

[0749] The user scans a physical receipt using a digital conversion device. This operation inputs image information. The terminal applies OCR technology to convert the scanned image information into text data. Specifically, it uses image processing technology to identify characters in the image and convert them into digital text format.

[0750] Step 4:

[0751] The device observes the user's actions during scanning operations and acquires data to infer their emotions. The emotion analysis engine evaluates the operation speed and repetitive movements to generate emotional data during the scan. Specifically, it monitors the user's behavior and calculates emotional indicators through behavioral pattern analysis.

[0752] Step 5:

[0753] The terminal sends the converted text data and generated sentiment data together to the server. Here, a security protocol is used to securely transfer the data, and the server confirms receipt of the data.

[0754] Step 6:

[0755] The server verifies the received text data as financial information and compares it with existing databases. Here, it executes database queries to confirm data consistency and accuracy and retrieves the verification results.

[0756] Step 7:

[0757] The server analyzes sentiment data and historical financial information, and uses a generative AI model to predict future budgets and expenditures. Here, the AI ​​algorithm is applied to output future predictions based on past data patterns.

[0758] Step 8:

[0759] The server sends users real-time progress updates and alerts when anomalies are detected. It prioritizes notifications based on emotional data and sends alerts accordingly. Specifically, it uses a priority notification mechanism based on the user's emotional state to send necessary information to the user's device.

[0760] (Application Example 2)

[0761] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0762] In modern society, many individuals struggle with budget management and reviewing their spending. While traditional financial management systems excel at accurate data analysis and future forecasting, they have struggled to provide support that considers the user's emotions and psychological state. Therefore, there is a growing demand for systems that can reduce stress and provide personalized advice based on emotions.

[0763] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0764] This invention includes a server comprising means for recognizing the user's emotional state based on voice or image input, a method comprising a generation algorithm for generating individual suggestions corresponding to the emotional state, and means for generating individual feedback and advice. This enables personalized financial management support based on emotions.

[0765] "Voice input" is a method of supplying information to a device using spoken language.

[0766] "Text data" refers to information in text format that has been converted from audio or image information.

[0767] An "image acquisition device" is a device that converts physical documents and images into digital data.

[0768] A "communication device" is a computer system used to send and receive data and perform certain processing.

[0769] "Financial data" refers to financial information related to income, expenses, budgets, etc.

[0770] A "storage device" is a hardware configuration for storing and retaining data and information.

[0771] "Progress notification" refers to a system informing users of the status of their operations or data processing.

[0772] "Warning methods" refer to methods used by the system to alert users when it detects a specified anomaly or problem.

[0773] "Emotional state" refers to the psychological or emotional state inferred from the user's statements and actions.

[0774] A "generative algorithm" is a computational method for creating new proposals or data based on specific input information.

[0775] An "information display device" is a display device or interface that provides digital information to users visually.

[0776] The system for implementing the present invention consists of an integrated unit comprising voice input, image acquisition, and data transmission / reception. The user provides financial information using a mobile terminal via voice input or camera scanning. The terminal converts this voice into text data using Google's speech recognition API. It also scans physical documents such as paper receipts using an image acquisition device and performs OCR processing using the Google Cloud Vision API. The resulting text data is then transferred to a server, which is a communication device.

[0777] The server receives this text data and verifies it against historical financial data. Microsoft Azure's Emotion API analyzes the user's utterances and psychological state during their actions. The analyzed emotional state is then used with a generative AI model, specifically OpenAI's GPT-4, to provide the user with emotion-based feedback and suggestions. This results in more personalized financial management support that reduces psychological stress for the user.

[0778] As a concrete example, let's consider a scenario where a user enters "My dining-out expenses this month have exceeded my budget." In this case, the server detects the user's anxiety and generates and provides a suggestion for a "dining-out expense reduction plan to optimize the budget."

[0779] An example of a prompt for the generating AI model would be, "Present effective cost-cutting strategies when a user is stressed about being over budget."

[0780] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0781] Step 1:

[0782] The user provides input via voice or camera scan through their device. Voice data is converted to text data by Google's speech recognition API. Meanwhile, scanned image data is processed using OCR (optical character recognition) by the Google Cloud Vision API. In this process, the input is voice or image, and the output is text data.

[0783] Step 2:

[0784] The terminal sends the converted text data to the server. The server receives this data and verifies it as financial data. Specifically, it compares the input text data with an existing financial database to check for consistency and anomalies. The output of this step is the verification results and data ready to be sent to the server for sentiment recognition.

[0785] Step 3:

[0786] The server uses Microsoft Azure's Emotion API to recognize the user's emotional state based on their voice tone and actions during scanning. Inputs are voice data and action logs, while output is user emotional state data. Information based on emotion recognition is used to generate feedback.

[0787] Step 4:

[0788] The server uses OpenAI's GPT-4 generative AI model to generate feedback and suggestions based on the user's financial data and emotional state. Here, past financial data, current text data, and emotional state are used as input, and personalized suggestions are produced as output. The specific operation is carried out by algorithmic processing performed by the AI ​​model.

[0789] Step 5:

[0790] The server sends the generated suggestions and feedback to the terminal and displays them to the user. A dashboard providing visual information is also updated. The input to this process is the newly generated suggestions and feedback, and the output is specific guidance messages for the user.

[0791] This series of processes allows users to receive emotion-based feedback, enabling more effective financial management.

[0792] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0793] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0794] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0795] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0796] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0797] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0798] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0799] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0800] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0801] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0802] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0803] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0804] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0805] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0806] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0807] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0808] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0809] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0810] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0811] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0812] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0813] The following is further disclosed regarding the embodiments described above.

[0814] (Claim 1)

[0815] A means of converting information input by the user via voice into text data,

[0816] A means for converting image data obtained by a scanning device into text,

[0817] The server provides a means for verifying the text data as financial data and storing it in a database.

[0818] The aforementioned server has means for analyzing past financial data and predicting future budgets and expenditures,

[0819] A means of notifying users of progress in real time and issuing alerts in case of anomalies,

[0820] A system that includes this.

[0821] (Claim 2)

[0822] The system according to claim 1, further comprising means for generating individualized feedback and advice based on the user's past input patterns and financial data.

[0823] (Claim 3)

[0824] The system according to claim 1, further comprising an interface with a dashboard that visually displays the predictive data and processing results generated by the server.

[0825] "Example 1"

[0826] (Claim 1)

[0827] A device that acquires information from a user via voice and converts it into text information,

[0828] A device that converts image information acquired from a recording medium into text information,

[0829] In a centralized processing system, there is a device that verifies the aforementioned textual information as accounting information and stores it in the stored information,

[0830] The aforementioned centralized processing unit is a device that analyzes past accounting information and predicts future budgets and expenses,

[0831] A device that immediately notifies the user of the progress and issues an alarm in case of an abnormality,

[0832] A system that includes this.

[0833] (Claim 2)

[0834] The system according to claim 1, further comprising a device for generating individual opinions and advice based on the user's history input patterns and accounting information.

[0835] (Claim 3)

[0836] The system according to claim 1, further comprising an operation screen equipped with an information board that visually displays the prediction information and processing results generated by the centralized processing device.

[0837] "Application Example 1"

[0838] (Claim 1)

[0839] A means of converting information obtained from the user via voice into text data,

[0840] A means for converting image data obtained by a data acquisition device into characters,

[0841] An information processing device includes means for verifying the character data as data and storing it in a storage device,

[0842] The aforementioned information processing device includes means for analyzing past data and predicting future income and expenses,

[0843] A means of notifying users of progress in real time and issuing warnings in case of abnormalities,

[0844] Means including a device for visually outputting the predicted data,

[0845] Means including a device that performs an action in response to user input,

[0846] A system that includes this.

[0847] (Claim 2)

[0848] The system according to claim 1, further comprising means for generating individual responses or advice based on the user's past input patterns and data.

[0849] (Claim 3)

[0850] The system according to claim 1, wherein the predictive data and processing results generated by the information processing device are output via an interface equipped with a display means.

[0851] "Example 2 of combining an emotion engine"

[0852] (Claim 1)

[0853] A means of converting information entered by the user via voice into text data,

[0854] A means for converting image information obtained by a digital conversion device into character data,

[0855] A means of analyzing emotions from a user's voice and generating emotion data,

[0856] Means for transmitting converted text data and emotion data to an accumulating device,

[0857] In the integration device, means for verifying the character data as financial information and storing it in a storage device,

[0858] The aforementioned data collection device includes means for analyzing past financial information and sentiment data to predict future budgets and expenditures,

[0859] A means of notifying users of progress in real time, and in the event of an anomaly, determining the notification priority based on emotion and issuing an alert,

[0860] A system that includes this.

[0861] (Claim 2)

[0862] The system according to claim 1, further comprising means for generating individualized feedback and advice based on the user's past input patterns, financial information, and sentiment data.

[0863] (Claim 3)

[0864] The system according to claim 1, further comprising a visualization device that provides personalized information based on sentiment data for visually displaying the predictive information and processing results generated by the aforementioned aggregation device.

[0865] "Application example 2 when combining with an emotional engine"

[0866] (Claim 1)

[0867] A method for converting information input via voice into text data,

[0868] A method for converting image information obtained from an image acquisition device into text,

[0869] A method for verifying the character data as financial data and storing it in a storage device in a communication device,

[0870] The aforementioned communication device analyzes past financial data and estimates future budgets and expenditures,

[0871] A method to notify users of the progress in real time and issue warnings in case of abnormalities,

[0872] A method for recognizing a user's emotional state based on voice or image input,

[0873] A method comprising a generation algorithm that generates individual suggestions according to emotional state,

[0874] A system that includes this.

[0875] (Claim 2)

[0876] The system according to claim 1, further comprising means for generating individualized feedback and advice based on the user's past input patterns, financial data, and emotional state.

[0877] (Claim 3)

[0878] The system according to claim 1, further comprising an information display device that visually displays predictive data, processing results, and feedback based on emotional state generated by the communication device. [Explanation of symbols]

[0879] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of converting information obtained from the user via voice into text data, A means for converting image data obtained by a data acquisition device into characters, An information processing device includes means for verifying the character data as data and storing it in a storage device, The aforementioned information processing device includes means for analyzing past data and predicting future income and expenses, A means of notifying users of progress in real time and issuing warnings in case of abnormalities, Means including a device for visually outputting the predicted data, Means including a device that performs an action in response to user input, A system that includes this.

2. The system according to claim 1, further comprising means for generating individual responses and advice based on the user's past input patterns and data.

3. The system according to claim 1, wherein the predictive data and processing results generated by the information processing device are output via an interface equipped with a display means.

Images