System

Abstract

A system is provided.SOLUTION: Transmitting, by a user using a portable communication terminal, image data related to a transaction via a message application; receiving the transmitted image data and extracting text information in the image using an optical character recognition technology; A system comprising: means for automatically performing journalizing and recording for each type of transaction; means for converting data into a format for final tax return based on the recorded journalizing data; means for understanding and correcting an instruction by voice or text from a user using a natural language processing technique; and means for analyzing the recorded data and generating advice on tax and asset management.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, the number of self-employed individuals and individuals with side jobs has increased, but managing income and expenses and filing tax returns takes a great deal of time and effort. Furthermore, neglecting these tasks can lead to tax problems. Traditional methods require manually managing receipts and invoices individually, journalizing and entering data, and often requiring expensive consultations with a tax accountant. These issues pose a significant burden, especially for individuals with low financial literacy. Therefore, there is a need for a system that allows for easy and efficient income and expense management and tax return filing. [Means for solving the problem]

[0005] The present invention provides a means for a user to use a mobile communication terminal to send image data related to a transaction via a messaging application, and a means for receiving the sent image data and extracting text information from the image using optical character recognition technology. The system also includes a means for analyzing the extracted text information, automatically journalizing and recording transactions by type, and converting the recorded journal data into a format for tax returns. Furthermore, the system also includes a means for understanding and correcting voice or text instructions from the user using natural language processing technology, and a means for analyzing the recorded data and generating advice on taxes and asset management, thereby improving the efficiency of income and expenditure management and facilitating tax return filing.

[0006] A "portable communication terminal" refers to a device such as a smartphone or tablet that is portable, connects to the Internet, and has communication capabilities.

[0007] "Image data" refers to data that contains visual information, such as photographs taken with a digital camera or smartphone camera, or scanned documents.

[0008] "Messaging application" refers to a software application that enables a user to send and receive text messages, images, audio, etc.

[0009] "Optical character recognition technology" refers to technology that reads printed or handwritten characters from digital images and converts them into text data.

[0010] "Text information" refers to character string information such as letters, numbers, and symbols extracted from image data.

[0011] "Journal entries" refer to accounting procedures in which the content of a transaction is classified and recorded under appropriate account headings according to certain criteria.

[0012] "Format for tax returns" refers to the format and data format required for tax returns, created based on the tax laws of each country.

[0013] "Natural language processing technology" refers to the technology that allows computers to understand and analyze human language.

[0014] "Tax and asset management advice" refers to advice that suggests optimal methods for tax savings and asset formation based on income and expenditure data. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention provides a system that allows users to efficiently manage income and expenditures and file tax returns by sending image data related to transactions via a message application using a portable communication terminal. This system operates in conjunction with the following programs.

[0037] 1. User sends image data

[0038] Users use their smartphones to take photos of receipts or invoices, which are then sent to the system via a messaging application (e.g., LINE).

[0039] The user logs in with LINE and sends a photo of a receipt or invoice to the chat.

[0040] 2. Receiving and analyzing image data

[0041] The server receives the image data sent by the user using the LINE API, and then extracts the text information from the image using optical character recognition technology (AI-OCR).

[0042] The server receives the image data, runs the AI-OCR engine, and extracts the text.

[0043] For example, text information such as "October 1, 2023," "10,000 yen," and "transportation expenses" is extracted from a receipt.

[0044] 3. Automatic journalization of extracted text

[0045] The extracted text information is analyzed by the server and automatically journalized by type of income and expenditure, with transactions classified into appropriate account items according to pre-set journalization rules.

[0046] Based on the analysis results, the server automatically records the expenses in categories such as "transportation expenses."

[0047] For example, "10,000 yen" is classified as "transportation expenses."

[0048] 4. Generate tax return data

[0049] The automatically posted data is aggregated on a server and converted into monthly and annual income and expenditure data in a format suitable for tax returns. This format is provided in accordance with the tax laws of each country.

[0050] The server aggregates the journal data by period and converts it into tax return format.

[0051] The data for tax returns will be sent to the user via LINE.

[0052] 5. Natural language instructions and corrections

[0053] Users can use natural language on LINE to check and edit their income and expenditure data. For example, if they say, "I want to edit my travel expenses for October," the server will activate its natural language processing engine, understand the user's instructions, and make the appropriate edits.

[0054] The user sends instructions in natural language, which the server analyzes using an NLP engine.

[0055] The server reviews the data and makes any necessary corrections.

[0056] 6. Collaboration with tax accountant AI

[0057] The server uses the collected income and expenditure data to generate optimal tax advice using a tax accountant AI algorithm, allowing users to receive appropriate advice on tax savings and asset management.

[0058] The server sends the data to the tax accountant AI, which generates optimal advice.

[0059] The advice will be sent to the user via LINE.

[0060] 7. Data Storage and Financial Services

[0061] The server encrypts and securely stores all collected data, and provides financial product recommendations and investment advice as needed, thus supporting users' financial growth.

[0062] The server encrypts and stores the data and suggests financial products.

[0063] Users receive financial advice and product details via LINE.

[0064] Specific examples

[0065] For example, let's say that self-employed business owner Sato wants to organize his income and expenses at the end of the month. Sato takes a photo of this month's receipts with his smartphone and sends it to the system via LINE. The server receives the image data and uses AI-OCR to extract the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server automatically accounts for this data and records it as entertainment expenses. Then, at the end of the month, data for tax returns is automatically generated and notified to Sato via LINE. When Sato sends a message via LINE saying, "I would like to review my entertainment expenses," the server understands the instruction and reconfirms and corrects it. Furthermore, a tax accountant AI analyzes Sato's data and provides tax-saving advice. Ultimately, Sato can manage his income and expenses and file his tax returns simply and efficiently through this system.

[0066] The processing flow will be explained below.

[0067] Step 1:

[0068] The user takes a photo of a receipt or invoice using a mobile communication device (smartphone). The user opens the LINE app and sends the photo to the LINE chat. This sends the image data to the system.

[0069] Step 2:

[0070] The server uses the LINE API to receive the image data sent by the user. The received image data is saved in its original format and passed to the next processing step.

[0071] Step 3:

[0072] The server activates optical character recognition (AI-OCR) technology to extract text information from the received image data. Specifically, the image processing engine identifies letters and numbers in the image and converts them into text. For example, information such as "October 1, 2023," "10,000 yen," and "transportation expenses" is extracted.

[0073] Step 4:

[0074] The extracted text information is analyzed by the server. The server identifies the type of transaction based on the text information and automatically journalizes it. Specifically, it classifies the transaction into the appropriate account category based on information such as the "date," "amount," and "item name." For example, "10,000 yen" is journalized as "transportation expenses."

[0075] Step 5:

[0076] The server aggregates the extracted and journalized data and generates monthly or annual income and expenditure reports, which are then converted into tax return formats and prepared for easy user access later.

[0077] Step 6:

[0078] The server notifies the user of the generated tax return data via LINE chat. The user can then check the data in the LINE app and request corrections if necessary.

[0079] Step 7:

[0080] The user sends a natural language instruction on LINE, such as "I would like to revise the transportation expenses for October." This instruction is sent to the server and analyzed by the natural language processing engine.

[0081] Step 8:

[0082] Based on the analysis results, the server understands the user's instructions and corrects the relevant journal data. For example, it rechecks "October travel expenses" and makes any necessary corrections.

[0083] Step 9:

[0084] The server provides all collected income and expenditure data to the AI ​​tax accountant, who then analyzes the data and provides optimal tax advice on tax-saving strategies and asset formation.

[0085] Step 10:

[0086] The server notifies the user of the generated advice via LINE chat, and the user can receive the advice in the LINE app and take the necessary action.

[0087] Step 11:

[0088] The server encrypts and securely stores all financial data, and the server uses this data to provide more advanced financial services and investment advice.

[0089] Step 12:

[0090] Users receive financial advice and product recommendations on LINE, which helps them make more effective financial decisions.

[0091] Example 1

[0092] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0093] Currently, managing income and expenses and filing tax returns takes a lot of time and effort. In particular, the process of manually recording receipts and invoices, journalizing them, and preparing data for tax returns is cumbersome and prone to errors. It is also difficult to receive real-time corrections and tax advice using natural language. This results in users expending a huge amount of effort and hinders efficient income and expense management.

[0094] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0095] In this invention, the server includes means for a user to use a wireless communication terminal to send image information related to a transaction via communication software, means for receiving the sent image information and extracting text information from the image using optical character recognition technology, means for analyzing the extracted text information and automatically journalizing and recording the transaction by type, means for converting the recorded journal data into a reporting format, means for understanding and correcting verbal or text instructions from the user using natural language processing technology, and means for analyzing the recorded data and generating advice on tax and asset management. This allows users to significantly reduce their workload and enable efficient and accurate income and expenditure management and tax return filing.

[0096] A "wireless communication terminal" is an electronic device that allows communication while moving, and includes smartphones, tablets, laptops, and the like.

[0097] "Communication software" refers to a program for sending and receiving data, and includes message applications, chat applications, and the like.

[0098] "Image information" means visual data captured electronically, including photographs and scanned data such as receipts and invoices.

[0099] "Optical character recognition technology" is a technology that analyzes characters in an image and converts them into text data, and is known as OCR (Optical Character Recognition).

[0100] "Character information" is data that is identified as characters, and includes text files and character string information.

[0101] The "means for automatically making and recording entries" is a system for analyzing extracted text information and classifying and recording it according to pre-set criteria.

[0102] "Format for tax return" means a standardized data format used for tax returns, including formats conforming to the laws and regulations of each country.

[0103] "Natural language processing technology" is a technology that understands and analyzes human language, and is a system for interpreting instructions entered in language or text.

[0104] The "means for generating tax and asset management advice" is a system that analyzes recorded data and makes recommendations on optimal tax strategies and asset management.

[0105] This invention is a system that allows users to efficiently manage income and expenditures and file tax returns using a wireless communication terminal.It works by sending image information related to transactions via communication software, and a server processes and analyzes that information.

[0106] System Overview

[0107] The system includes the following major components:

[0108] 1. User's wireless communication device (smartphone, tablet, laptop, etc.)

[0109] 2. Communication software (messaging and chat applications)

[0110] 3. Server

[0111] 4. Optical Character Recognition Technology (OCR engine, e.g. Tesseract OCR)

[0112] 5. Natural Language Processing Technology (NLP engine, e.g. GPT-3)

[0113] 6. Database

[0114] System Operation

[0115] Sending and receiving image information

[0116] A user uses a wireless communication terminal to take images of transaction-related documents such as receipts and invoices, which are then sent to a server via communication software.

[0117] The server receives the image data sent by the user using the API of the communication software, and the received image data is stored in the server's storage.

[0118] Image information analysis and classification

[0119] The server uses optical character recognition technology (OCR engine) to extract text information from the received image data. For example, from a receipt, text information such as "October 1, 2023," "10,000 yen," and "transportation expenses" is extracted.

[0120] The extracted text information is analyzed within the server, automatically categorized by transaction type, and recorded in a database based on pre-set rules.

[0121] Generate tax return data

[0122] The server aggregates the recorded journal data and converts it into a reporting format. This format is provided in a format that complies with the laws and regulations of each country. The generated reporting data is notified to the user via communication software.

[0123] Natural language instructions and corrections

[0124] Users can use natural language on the communication software to check and modify income and expenditure data. For example, if a user issues a command such as "I want to modify my travel expenses for October," the server will use natural language processing technology to analyze the user's command and make the appropriate modifications.

[0125] Providing tax advice

[0126] The server analyzes the recorded data and generates optimal advice on tax and asset management using a tax accountant AI algorithm, which is then communicated to the user via communication software.

[0127] Data Storage and Financial Services

[0128] The server encrypts and securely stores all data, and also provides financial product suggestions and investment advice based on the user's income and expenditure data, thereby supporting the user's financial growth.

[0129] Specific examples

[0130] For example, say a user wants to organize this month's income and expenses. The user takes a photo of a receipt with their smartphone and sends it to the system via communication software. The server analyzes the received image information using an OCR engine and extracts the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The extracted data is automatically accounted for as "entertainment expenses." At the end of the month, the server generates data for tax returns and notifies the user. Furthermore, if the user instructs in natural language that they would like to "review entertainment expenses," the server understands the instruction and makes the appropriate corrections. Through the above process, users can efficiently and accurately manage their income and expenses and file their tax returns.

[0131] Prompt Sentence Examples

[0132] 1. "I have sent you a scanned image of the receipt. Please let me know the extracted text data."

[0133] 2. "Please automatically post this month's travel expenses and display the related data."

[0134] 3. "Please generate the income and expenditure data for October in the tax return format and notify me."

[0135] 4. "How do I correct my income and expenditure data using natural language?"

[0136] 5. "Provide tax-saving advice."

[0137] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0138] Step 1: Send image information

[0139] The user uses the wireless communication terminal to capture image information related to the transaction.

[0140] Input: Captured image information (photos of receipts and invoices)

[0141] Operation: The user starts the communication software (messaging application) and logs in. Next, the user attaches the captured image information to the chat screen and presses the send button.

[0142] Output: Image information sent to the server via communication software

[0143] Step 2: Receiving image information

[0144] The server receives the image information sent by the user using the API of the communication software.

[0145] Input: Image information sent by the user

[0146] Operation: The server calls the API of the communication software and saves the received image information in storage.

[0147] Output: Image information saved in storage

[0148] Step 3: Analyze image information

[0149] The server uses optical character recognition technology (OCR engine) to extract text information from the received image information.

[0150] Input: Image information saved in storage

[0151] Operation: The server starts the OCR engine, analyzes the image data, and extracts text information. For example, from an image of a receipt, it obtains the text information "October 1, 2023," "10,000 yen," and "Transportation expenses."

[0152] Output: Extracted text information

[0153] Step 4: Journalize textual information

[0154] The server analyzes the extracted text information and automatically records and journals the transaction by type.

[0155] Input: Extracted text information

[0156] Operation: The server analyzes the text information and classifies it into the appropriate account based on the journal entry rules. For example, the information "October 1, 2023," "10,000 yen," and "Transportation expenses" is journalized as transportation expenses.

[0157] Output: Journalized data is recorded in the database

[0158] Step 5: Generate data for declaration

[0159] The server aggregates the recorded journal data and converts it into a format for reporting.

[0160] Input: Journal entry data recorded in the database

[0161] How it works: The server aggregates accounting data for a specific period and converts it into a reporting format that complies with the laws and regulations of each country. For example, it totals travel expenses, business expenses, and other expenses for one month.

[0162] Output: Data converted into a declaration format

[0163] Step 6: Natural Language Processing Remediation

[0164] Users can check and modify income and expenditure data in natural language through the communication software.

[0165] Input: Natural language instructions from the user (e.g., "I would like to correct the travel expenses for October.")

[0166] How it works: The server uses natural language processing (NLP) technology to parse the instructions and modify the specific data. For example, it extracts the entry for "October travel expenses" and modifies it to the specified value.

[0167] Output: Database updated with modified data

[0168] Step 7: Providing tax advice

[0169] The server analyzes the recorded data and generates optimal advice on tax and asset management.

[0170] Input: All income and expenditure data recorded in the database

[0171] How it works: The server uses the tax accountant AI algorithm to analyze the data and generate optimal advice, such as proposals for tax savings and asset management.

[0172] Output: The generated tax advice is communicated to the user via the communication software.

[0173] Step 8: Data Storage and Financial Services

[0174] The server encrypts and securely stores all data and provides financial product recommendations and investment advice as needed.

[0175] Input: All transaction data, journal data, user income and expenditure data

[0176] How it works: The server encrypts your transaction data and stores it in a secure database. It also analyzes your income and expenditure data and generates financial product and investment recommendations tailored to you.

[0177] Output: Encrypted and securely stored data, as well as financial product suggestions and investment advice notified to the user.

[0178] (Application example 1)

[0179] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0180] Previously, users had to manually collect image data related to transactions, sort them manually, and prepare tax returns, which required a great deal of effort and was prone to errors. Furthermore, there was a lack of efficient ways to manage income and expenditures or receive tax advice, placing a heavy burden on users. Furthermore, there was no system in place for managing daily expenses in real time and receiving appropriate financial advice. To solve these issues, a new method was needed.

[0181] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0182] In this invention, the server includes: means for a user to use a mobile communication device to send image data related to transactions via a messaging application; means for receiving the sent image data and extracting text information from the image using optical character recognition technology; means for analyzing the extracted text information and automatically journalizing and recording transactions by type; means for converting the recorded journal data into a format for tax returns; means for understanding and correcting voice or text instructions from the user using natural language processing technology; means for analyzing the recorded data and generating advice on tax and asset management; means for a user to send image data via an application on the mobile communication device and compile data related to financial transactions; means for providing up-to-date financial information based on the compiled data; and means for using a generative AI model to generate prompts related to income and expenditure management according to the user's requests. This allows users to efficiently manage transaction-related data and receive appropriate tax advice. It also allows users to manage their daily expenses in real time and receive optimal financial advice.

[0183] A "portable communication terminal" refers to an electronic device that is portable and capable of internet communication and data transmission and reception.

[0184] "Image data" refers to visual information obtained by optical means and expressed in digital form.

[0185] "Messaging application" refers to software that enables users to send and receive messages, including text, images, and audio.

[0186] "Optical character recognition technology" refers to the technology that extracts character information from image data and converts it into text data.

[0187] "Text information" refers to information expressed as a string of characters.

[0188] "Means for automatically making and recording journal entries" refers to the process of classifying text information extracted by the system by transaction type and saving the results in a database or the like.

[0189] A "tax return format" refers to income and expenditure data organized in the format required for submission to tax authorities.

[0190] "Natural language processing technology" refers to the technology that enables computers to understand and analyze text and speech written in natural language.

[0191] "Financial transaction related data" refers to economic information related to payments and transactions.

[0192] "Up-to-date financial information" means the most recent economic data, including recent transactions and expenditures.

[0193] A "generative AI model" refers to a model of artificial intelligence that uses machine learning algorithms to generate appropriate outputs for a specific task.

[0194] "Prompt" refers to the text content generated in response to a user instruction or question.

[0195] This invention is a system that improves the efficiency of income and expenditure management and tax return filing by allowing users to send image data related to transactions via a message application using a portable communication terminal. This system operates with the following configuration and program.

[0196] 1. User sends image data

[0197] A user takes a photo of a receipt or invoice using their smartphone. This photo is sent to the system via a messaging application (e.g., a general messaging application). The user logs in to the application and sends the photo of the receipt or invoice to the chat.

[0198] 2. Receiving and analyzing image data

[0199] The server receives image data sent by the user using a messaging API. It then uses AI-OCR (optical character recognition) technology to extract text information from the image. For example, it can use the Google Cloud Vision API. The server analyzes the image data and obtains information such as the date, time, amount, and subject.

[0200] 3. Automatic journalization of extracted text

[0201] The extracted text information is analyzed by the server and automatically journalized by transaction type. During this process, transactions are classified into appropriate account items according to pre-set journalization rules. For example, they may be classified into categories such as "transportation expenses" and "entertainment expenses."

[0202] 4. Generate tax return data

[0203] The automatically posted data is aggregated on the server and converted into a format for tax returns as monthly or annual income and expenditure data. This format is provided in a format that complies with the tax laws of each country. The data for tax returns is then notified to the user.

[0204] 5. Natural language instructions and corrections

[0205] Users can use natural language on the messaging application to check and modify their income and expenditure data. For example, if they issue a command such as "I want to modify my travel expenses for October," the server will activate its natural language processing engine, understand the user's command, and make the appropriate modifications. It uses OpenAI GPT-4 and other technologies to analyze the command and make the necessary modifications.

[0206] 6. Generating tax advice

[0207] The server generates tax and asset management advice based on the collected income and expenditure data, allowing users to receive appropriate advice on tax savings and asset management. Using a generative AI model, advice is provided that is tailored to the user's specific situation.

[0208] 7. Providing real-time financial information

[0209] A user sends image data via an application on a portable communication terminal, and data related to financial transactions is compiled. The system provides the latest financial information based on the compiled data. For example, in response to a request such as "What is the total amount of expenses this month?", the system displays the latest financial information in real time.

[0210] Specific examples

[0211] For example, say a self-employed person wants to organize their income and expenses at the end of the month. The user takes a photo of this month's receipts with their smartphone and sends it to the system from the application. The server receives the image data and uses AI-OCR to extract the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server automatically journalizes this data and records it as entertainment expenses. Then, at the end of the month, data for tax returns is automatically generated and notified to the user.

[0212] When a user sends a message saying, "I'd like to review my entertainment expenses," the server understands the instruction and reconfirms and corrects it. Furthermore, a generative AI model analyzes the user's data and provides tax-saving advice. Ultimately, the system allows users to easily and efficiently manage their income and expenses and file tax returns.

[0213] Prompt Sentence Examples

[0214] "I would like to revise my travel expenses for October."

[0215] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0216] Step 1:

[0217] A user uses a portable communication terminal to take a photo of a receipt or invoice related to a transaction and transmits the image data to a server via a message application. The input is the image data taken by the user, and the output is the image data transmitted to the server via the message application.

[0218] Step 2:

[0219] The server receives image data sent by the user using a messaging API. The input is the image data sent by the user, and the output is the image data received by the server.

[0220] Step 3:

[0221] The server applies AI-OCR (optical character recognition) technology to the received image data and extracts the text information within the image. The input is the received image data, and the output is text information. Specifically, the image is analyzed using Google Cloud Vision API, etc., and text information such as the date, time, amount, and transaction items is extracted.

[0222] Step 4:

[0223] The server analyzes the extracted text information and automatically journalizes and records each type of transaction. The input is the extracted text information, and the output is automatically journalized data. Specifically, the data is classified into categories such as "transportation expenses" and "entertainment expenses" and recorded in the appropriate account.

[0224] Step 5:

[0225] The server aggregates the recorded journal data and converts it into a format for tax returns as monthly or annual income and expenditure data. The input is automatically journalized data, and the output is data converted into a format for tax returns.

[0226] Step 6:

[0227] The user uses a messaging application to input correction instructions in natural language. For example, they might send a message saying, "I would like to correct the travel expenses for October." The input is a natural language instruction from the user, and the output is the target data to be corrected based on that instruction.

[0228] Step 7:

[0229] The server uses natural language processing technology to analyze user instructions and make appropriate data corrections. The input is the user's natural language instructions, and the output is the corrected income and expenditure data. Specifically, OpenAI GPT-4 and other technologies are used to interpret the user's intent and update the relevant parts of the database.

[0230] Step 8:

[0231] The server generates advice on tax and asset management based on the collected income and expenditure data. The input is the recorded income and expenditure data, and the output is the generated advice. Using a generative AI model, it provides specific advice tailored to the user's situation.

[0232] Step 9:

[0233] The server aggregates data related to image data transmissions and financial transactions performed by users via applications on mobile communication terminals in real time and provides the latest financial information. The input is the transmitted image data and transaction data, and the output is the updated latest financial information. The data is processed in real time, and the most recent financial information is displayed when the user requests it.

[0234] Examples of concrete examples and prompts

[0235] For example, say a self-employed person wants to organize their income and expenses at the end of the month. The user takes a photo of this month's receipts with their smartphone and sends it to the system from the application. The server receives the image data and uses AI-OCR to extract the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server automatically journalizes this data and records it as entertainment expenses. Then, at the end of the month, data for tax returns is automatically generated and notified to the user.

[0236] Prompt Sentence Examples

[0237] "I would like to revise my travel expenses for October."

[0238] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0239] The present invention combines an emotion engine with a system that allows users to efficiently manage income and expenditures and file tax returns by sending image data related to transactions via a message application using a portable communication terminal. This system operates in conjunction with the following programs.

[0240] 1. User sends image data

[0241] Users use their smartphones to take photos of receipts or invoices, which are then sent to the system via a messaging application (e.g., LINE).

[0242] The user logs in with LINE and sends a photo of a receipt or invoice to the chat.

[0243] 2. Receiving and analyzing image data

[0244] The server receives the image data sent by the user using the LINE API, and then extracts the text information from the image using optical character recognition technology (AI-OCR).

[0245] The server receives the image data, runs the AI-OCR engine, and extracts the text.

[0246] For example, text information such as "October 1, 2023," "10,000 yen," and "transportation expenses" is extracted from a receipt.

[0247] 3. Automatic journalization of extracted text

[0248] The extracted text information is analyzed by the server and automatically journalized by type of income and expenditure, with transactions classified into appropriate account items according to pre-set journalization rules.

[0249] Based on the analysis results, the server automatically records the expenses in categories such as "transportation expenses."

[0250] For example, "10,000 yen" is classified as "transportation expenses."

[0251] 4. Generate tax return data

[0252] The automatically posted data is aggregated on the server and reflected in monthly and annual income and expenditure reports. This data is converted into a tax return format and prepared for easy access by users later.

[0253] The server aggregates the journal data by period and converts it into tax return format.

[0254] The data for tax returns will be sent to the user via LINE.

[0255] 5. Natural language instructions and corrections

[0256] Users can use natural language on LINE to check and edit their income and expenditure data. For example, if they say, "I want to edit my travel expenses for October," the server will activate its natural language processing engine, understand the user's instructions, and make the appropriate edits.

[0257] The user sends instructions in natural language, which the server analyzes using an NLP engine.

[0258] The server reviews the data and makes any necessary corrections.

[0259] 6. Collaboration with tax accountant AI

[0260] The server uses the collected income and expenditure data to generate optimal tax advice using a tax accountant AI algorithm, allowing users to receive appropriate advice on tax savings and asset management.

[0261] The server sends the data to the tax accountant AI, which generates optimal advice.

[0262] The advice will be sent to the user via LINE.

[0263] 7. Data Storage and Financial Services

[0264] The server encrypts and securely stores all collected data, and provides financial product recommendations and investment advice as needed, thus supporting users' financial growth.

[0265] The server encrypts and stores the data and suggests financial products.

[0266] Users receive financial advice and product details via LINE.

[0267] 8. Introducing and utilizing an emotion engine

[0268] The server incorporates a new emotion engine that analyzes emotions based on the user's voice or text data and optimizes response content. This engine recognizes the user's emotions in real time and can respond appropriately.

[0269] The user sends a voice message such as "Today is stressful and difficult."

[0270] The server receives the voice data and uses an emotion engine to analyze the user's emotional state.

[0271] The server provides appropriate feedback via LINE based on the user's emotions (e.g., "You seem stressed. Let me know if there's anything I can do to help you").

[0272] Specific examples

[0273] For example, let's say that self-employed Tanaka wants to organize his income and expenses at the end of the month. Tanaka takes a photo of this month's receipts with his smartphone and sends it to the system via LINE. The server receives the image data and uses AI-OCR to extract the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server automatically accounts for this data and records it as entertainment expenses. Then, at the end of the month, data for tax returns is automatically generated and notified to Tanaka via LINE. When Tanaka sends a message via LINE saying, "I would like to review my entertainment expenses," the server understands the instruction and reconfirms and corrects it. Furthermore, the tax accountant AI analyzes Tanaka's data and provides tax-saving advice.

[0274] One day, Tanaka sends a voice message to LINE saying, "I'm feeling very stressed today," and the server's emotion engine analyzes Tanaka's emotions and provides appropriate feedback, such as, "It seems you're feeling stressed. Let me know if there's anything I can help you with." This allows Tanaka to not only efficiently manage his income and expenses and file his tax return through the system, but also receive emotional support.

[0275] The processing flow will be explained below.

[0276] Step 1:

[0277] The user takes a photo of a receipt or invoice using a mobile communication device (smartphone). The user opens the LINE app and sends the photo to the LINE chat. This sends the image data to the system.

[0278] Step 2:

[0279] The server uses the LINE API to receive the image data sent by the user. The received image data is saved in its original format and passed to the next processing step.

[0280] Step 3:

[0281] The server activates optical character recognition (AI-OCR) technology to extract text information from the received image data. Specifically, the image processing engine identifies letters and numbers in the image and converts them into text. For example, information such as "October 1, 2023," "10,000 yen," and "transportation expenses" is extracted.

[0282] Step 4:

[0283] The extracted text information is analyzed by the server. The server identifies the type of transaction based on the text information and automatically journalizes it. Specifically, it classifies the transaction into the appropriate account category based on information such as the "date," "amount," and "item name." For example, "10,000 yen" is journalized as "transportation expenses."

[0284] Step 5:

[0285] The server aggregates the extracted and journalized data and generates monthly or annual income and expenditure reports, which are then converted into tax return formats and prepared for easy user access later.

[0286] Step 6:

[0287] The server notifies the user of the generated tax return data via LINE chat. The user can then check the data in the LINE app and request corrections if necessary.

[0288] Step 7:

[0289] The user sends a natural language instruction on LINE, such as "I would like to revise the transportation expenses for October." This instruction is sent to the server and analyzed by the natural language processing engine.

[0290] Step 8:

[0291] Based on the analysis results, the server understands the user's instructions and corrects the relevant journal data. For example, it rechecks "October travel expenses" and makes any necessary corrections.

[0292] Step 9:

[0293] The server provides all collected income and expenditure data to the AI ​​tax accountant, who then analyzes the data and provides optimal tax advice on tax-saving strategies and asset formation.

[0294] Step 10:

[0295] The server notifies the user of the generated advice via LINE chat, and the user can receive the advice in the LINE app and take the necessary action.

[0296] Step 11:

[0297] The server uses an emotion engine to analyze the user's voice or text data in real time. For example, if a user says, "I'm feeling stressed today," the server recognizes the voice data and analyzes the user's emotional state.

[0298] Step 12:

[0299] Based on the analysis results of the emotion engine, the server provides appropriate feedback according to the user's emotions. For example, it may send a message via LINE saying, "You seem to be feeling stressed. Please let us know if there is anything we can do to help."

[0300] Step 13:

[0301] The server encrypts and securely stores all financial data, and the server uses this data to provide more advanced financial services and investment advice.

[0302] Step 14:

[0303] Users receive financial advice and product recommendations on LINE, which helps them make more effective financial decisions.

[0304] Specific examples

[0305] For example, let's say that self-employed Tanaka wants to organize his income and expenses at the end of the month. Tanaka takes a photo of this month's receipts with his smartphone and sends it to the system via LINE. The server receives the image data and uses AI-OCR to extract the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server automatically accounts for this data and records it as entertainment expenses. Then, at the end of the month, data for tax returns is automatically generated and notified to Tanaka via LINE. When Tanaka sends a message via LINE saying, "I would like to review my entertainment expenses," the server understands the instruction and reconfirms and corrects it. Furthermore, the tax accountant AI analyzes Tanaka's data and provides tax-saving advice.

[0306] One day, Tanaka sends a voice message to LINE saying, "I'm feeling very stressed today," and the server's emotion engine analyzes Tanaka's emotions and provides appropriate feedback, such as, "It seems you're feeling stressed. Let me know if there's anything I can help you with." This allows Tanaka to not only efficiently manage his income and expenses and file his tax return through the system, but also receive emotional support.

[0307] Example 2

[0308] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0309] Traditional income and expenditure management and tax return systems required time and effort for manual data entry, resulting in inefficiency. They also often failed to accurately record collected data or provide adequate tax advice. Furthermore, they lacked support that took into account the user's emotional and stress levels, resulting in a lack of improvement in the overall user experience.

[0310] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for a user to use a portable communication terminal to send image data related to a transaction via a message application; means for receiving the sent image data and extracting text information from the image using optical character recognition technology; means for analyzing the extracted text information and automatically journalizing and recording the transaction type; means for converting the recorded journal data into a format for tax returns; means for understanding and correcting voice or text instructions from the user using natural language processing technology; means for analyzing the recorded data and generating advice on taxes and asset management; and means for recognizing emotions based on the user's voice or text data using emotion analysis technology and optimizing the response. This not only improves the efficiency of income and expenditure management and tax return filing, but also enables improved accuracy of tax advice and emotional support for users.

[0311] "User" refers to any individual or legal entity that uses the System.

[0312] A "portable communication terminal" refers to a terminal device such as a smartphone or tablet that is portable and capable of wireless communication.

[0313] "Transaction" refers to the act of exchanging money or goods in economic activities.

[0314] "Image data" refers to visual information represented in digital form.

[0315] "Message application" refers to an application that sends and receives text messages and image data over the Internet.

[0316] "Optical character recognition technology" refers to technology that extracts characters from an image as digital data.

[0317] "Text information" refers to data expressed as a string of characters.

[0318] "Journal entry" refers to the accounting process of classifying and recording transactions into account items.

[0319] "Server" refers to a device or system that provides computer resources over a network.

[0320] "Tax return format" refers to the format of the income and expenditure report to be submitted to the tax office.

[0321] "Natural language processing technology" refers to technology for understanding human language and responding appropriately.

[0322] "Voice or text instructions" refers to operation instructions given to the system by the user using words or letters.

[0323] "Tax advice" refers to the act or content of providing appropriate advice regarding tax.

[0324] "Asset management" refers to the efficient management and administration of personal or corporate assets.

[0325] "Emotion analysis technology" refers to the technology of analyzing human emotions and psychological states from voice and text.

[0326] "Optimizing response content" refers to optimizing the responses and support provided by the system according to the user's needs and condition.

[0327] This invention is a system that allows users to efficiently manage income and expenditures and file tax returns using a mobile communication device. This system mainly combines a messaging application, optical character recognition technology (AI-OCR), natural language processing technology (NLP), tax accountant AI algorithms, and sentiment analysis technology. Specific examples of the use of various hardware and software are shown below.

[0328] A user uses a mobile communication device (e.g., a smartphone or tablet) to take a photo of a receipt or invoice related to a transaction. At that time, the user uses a messaging application (e.g., LINE) to send the image data to the system. The user logs in to LINE and sends a photo of the receipt or invoice as a chat message.

[0329] The server receives image data sent by the user using LINE's API. The server then uses optical character recognition technology (e.g., Google Cloud Vision API) to extract text information from the image. This AI-OCR engine identifies characters in the image and extracts them as numerical or text data. For example, from a receipt, the following information is extracted: "October 1, 2023," "10,000 yen," and "Transportation expenses."

[0330] The extracted text information is analyzed by the server and automatically journalized according to pre-set journalization rules. Based on the analysis results, the server classifies the data into appropriate account items for each type of income and expenditure. For example, "10,000 yen" is journalized as "transportation expenses."

[0331] The server converts the collected data into a format for tax returns. This data is reflected in monthly and annual income and expenditure reports and prepared as tax return data. The server then aggregates this data and converts it into a tax return format (e.g., XML for tax software). Users are notified via LINE for easy access to their tax return data.

[0332] Users can also send voice or text instructions on LINE using natural language processing technology. For example, if you send an instruction such as "I would like to revise my travel expenses for October," the server will use an NLP engine (e.g., OpenAI GPT-3) to analyze the message, understand the user's instruction, and make the appropriate revisions.

[0333] Based on the collected data, the server uses a tax accountant AI algorithm (e.g., Watson Tax Advisor) to generate optimal tax advice. Users can receive advice on tax savings and asset management via LINE. The server also encrypts and securely stores revenue data and provides users with financial products and investment advice.

[0334] Emotion analysis technology (e.g., IBM Watson Tone Analyzer) recognizes emotions from a user's voice or text data and provides feedback in real time. For example, if a user sends a voice message saying, "I'm feeling very stressed today," the server analyzes the emotion and replies, "It sounds like you're feeling stressed. Let me know if there's anything I can do to help."

[0335] Specific examples

[0336] A self-employed individual wants to organize their income and expenses at the end of the month. They take a photo of this month's receipts with their smartphone and send it to the system via LINE. The server receives the image data and uses AI-OCR to extract the following information: "October 15, 2023," "5,000 yen," and "Entertainment Expenses." The server automatically accounts for this data and records it as entertainment expenses. At the end of the month, tax return data is automatically generated and notified via LINE. When the self-employed individual sends a message via LINE saying, "I'd like to review my entertainment expenses," the server understands the instruction and reconfirms and corrects it. Furthermore, a tax accountant AI analyzes the data and provides tax-saving advice. One day, the self-employed individual sends a voice message via LINE saying, "I'm feeling stressed today and it's bothering me," and the server's emotion engine analyzes the emotion and provides appropriate feedback, such as, "It sounds like you're stressed. Let me know if there's anything I can do to help."

[0337] Prompt Sentence Examples

[0338] "Please send a photo of your receipt via LINE to record your income and expenses."

[0339] "I would like to revise my travel expenses for October 2023."

[0340] "Can you give me some tax saving advice?"

[0341] "I'm having a lot of stress today"

[0342] In this way, the system helps users manage their finances and file their tax returns efficiently, and also provides emotional support.

[0343] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0344] Step 1: Sending image data

[0345] The user takes a photo of an image related to the transaction (such as a receipt or invoice) using a mobile communication terminal (e.g., a smartphone) and sends it to the system via a messaging application (e.g., LINE).

[0346] Input: Image data of receipts and invoices

[0347] Output: Image data sent to LINE

[0348] Specific operation: The user launches the camera app and takes a picture of a receipt or invoice. The user selects the image in the LINE app and sends it as a chat message.

[0349] Step 2: Receiving image data

[0350] The server uses the LINE API to receive the image data sent by the user.

[0351] Input: Image data sent via the LINE API

[0352] Output: Image data stored on the server

[0353] Specific operation: The server periodically checks the LINE API to receive new image data, which is then stored in the database.

[0354] Step 3: Extracting text information

[0355] The server uses optical character recognition technology (AI-OCR) to extract text information from the received image data.

[0356] Input: Saved image data

[0357] Output: Extracted text information

[0358] How it works: The server launches an AI-OCR engine such as Google Cloud Vision API, identifies the characters in the image, and generates text data. For example, the following information is extracted: "October 1, 2023," "10,000 yen," and "transportation expenses."

[0359] Step 4: Parsing and journalizing extracted text

[0360] The server analyzes the extracted text information and automatically records and journals the transactions by type.

[0361] Input: Extracted text information

[0362] Output: Journal entry data

[0363] Specific operation: The server analyzes the text information based on pre-defined rules and classifies it into categories such as "transportation expenses" and "entertainment expenses." For example, "10,000 yen" is journalized as "transportation expenses." The journalized data is then saved in a database.

[0364] Step 5: Generate tax return data

[0365] The server converts the data into a format for tax returns based on the accounting data.

[0366] Input: Journal data

[0367] Output: Data in tax return format

[0368] Specific operation: The server aggregates monthly and annual income and expenditure reports and converts them into a format for tax returns (e.g., XML for tax software). The user is notified via LINE that "tax return data is ready."

[0369] Step 6: Correcting data using natural language

[0370] Users can check and edit income and expenditure data using natural language on LINE.

[0371] Input: Natural language instructions from the user

[0372] Output: Corrected balance data

[0373] Specific operation: The user sends a message on LINE saying, "I would like to correct my travel expenses for October." The server uses an NLP engine (e.g., OpenAI GPT-3) to analyze the message and understand the instructions. The server then reviews the target data and makes any necessary corrections.

[0374] Step 7: Advice generation by tax accountant AI

[0375] The server sends the collected data to the tax accountant AI, which generates optimal tax advice.

[0376] Input: Collected income and expenditure data

[0377] Output: Tax advice

[0378] Specific operation: The server sends the collected data to a tax accountant AI (e.g., Watson Tax Advisor). The tax accountant AI analyzes the data and generates appropriate tax advice. The advice is then sent to the user via LINE.

[0379] Step 8: Optimize your response with sentiment analysis

[0380] The server uses emotion analysis technology to recognize the user's emotions and optimize the response.

[0381] Input: Voice or text data from the user

[0382] Output: Emotion-based feedback

[0383] Specific operation: The user sends a voice message stating, "I'm feeling very stressed today." The server receives the voice data and analyzes the user's emotions using an emotion analysis engine (e.g., IBM Watson Tone Analyzer). Appropriate feedback based on the user's emotions is then provided via LINE (e.g., "It sounds like you're feeling stressed. Please let me know if there's anything I can do to help you").

[0384] This processing flow allows the system to streamline users' income and expenditure management and tax return filing, and also provides emotional support.

[0385] (Application example 2)

[0386] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0387] Managing transaction-related income and expenditures and filing tax returns are time-consuming and labor-intensive tasks, especially for small businesses and sole proprietors. Traditional manual data entry and accounting processes are prone to errors, and users without specialized knowledge of tax or asset management find it difficult to obtain appropriate support. Furthermore, few systems take into account the user's emotions and mental state, which leads to a poor user experience. Therefore, the challenge is to provide a system that streamlines income and expenditure management and tax return filing and provides emotionally sensitive feedback.

[0388] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to use a portable communication terminal to send image data related to a transaction via a message application; means for receiving the sent image data and extracting text information from the image using optical character recognition technology; means for analyzing the extracted text information and automatically journalizing and recording the transaction type; means for converting the recorded journal data into a format for tax returns; means for understanding and correcting voice or text instructions from the user using natural language processing technology; means for analyzing the recorded data and generating advice on taxes and asset management; and means for analyzing the user's emotions and providing feedback according to the user's emotions using an emotion engine that optimizes response content. This improves the efficiency of income and expenditure management and tax return filing and also enables support that takes into account the user's emotions and mental state.

[0389] A "portable communication terminal" is a device that is portable and has communication capabilities, and mainly refers to smartphones and tablets.

[0390] "Transaction-related image data" refers to image data, such as receipts and invoices, that contains information about financial transactions.

[0391] A "messaging application" is application software that allows users to send and receive messages, such as text and images, via the Internet, and includes LINE and WhatsApp.

[0392] "Optical character recognition technology" is a technology that extracts character information from image data, and is also known as OCR (Optical Character Recognition).

[0393] "Text information" refers to information such as letters and numbers, and includes character data extracted from an image.

[0394] "Journal entries" are the process of classifying transactions into specific account items and recording them in the ledger.

[0395] "Tax format" means the conversion of collected financial data into a specific format for use in tax returns.

[0396] "Natural language processing technology" is a technology that allows computers to understand, analyze, and respond to human language, and is also known as NLP (Natural Language Processing).

[0397] An "emotion engine" is a technology that analyzes user data such as voice and text and estimates their emotional state.

[0398] "Feedback" refers to the response or advice that a system gives to a user in response to their input.

[0399] "Tax and asset management advice" means providing expertise based on the user's financial data and offering advice on tax optimization and asset management.

[0400] This invention is a system that combines an emotion engine with a system that allows users to use a portable communication terminal to send image data related to transactions via a message application, thereby streamlining income and expenditure management and tax return filing.

[0401] The server receives image data related to transactions sent by users via messaging applications such as LINE and WhatsApp. It extracts text information from the received image data using optical character recognition (OCR). The extracted text information is stored in the server's database, analyzed by an automatic accounting program, and automatically recorded and accounted for by transaction type.

[0402] Furthermore, the recorded journal data is automatically converted into a format for tax returns and provided to users in a format that can be used for tax returns. This also implements a means of generating advice on taxes and asset management, and the tax accountant AI provides optimal advice based on the data.

[0403] Users can send voice or text instructions to the server through an interface that uses natural language processing technology. For example, they can send a message to a friend saying, "I want to revise my travel expenses for October." The server analyzes this instruction, reviews the relevant data, and makes any necessary adjustments. Furthermore, if a user sends a message expressing their emotions, the server's emotion engine analyzes it and provides feedback appropriate to the user's emotional state.

[0404] The hardware used includes a server and a mobile communication device, and the software includes the LINE Messaging API, an OCR engine (such as Tesseract OCR), MySQL, a natural language processing engine (such as spaCy), and a sentiment analysis model (such as the Transformers sentiment analysis pipeline).

[0405] For example, a user can take a photo of a receipt with their smartphone and send it to the system via a messaging application. The image data is transferred to the server, where OCR extracts the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server records this data as entertainment expenses and automatically generates data for tax returns. If the user then issues a natural language instruction such as "I'd like to correct the entertainment expenses for October," the system will correct the relevant data. Furthermore, if the user voice-transmits "I'm feeling very stressed today," the emotion engine will analyze the emotion and provide feedback such as "It sounds like you're stressed. Please let me know if there's anything I can do to help."

[0406] An example prompt is:

[0407] "Go to My Page and check your credit card transaction details from last month. Then, send an image of the receipt to LINE and correct the relevant transaction. Also, if you would like to receive advice on asset management, please let the system know."

[0408] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0409] Step 1:

[0410] A user uses a mobile communication terminal to send image data related to a transaction (e.g., a photo of a receipt or invoice) to the system via a message application. The input of this step is the image data of the receipt or invoice, and the output is that it reaches the server via the message application. Specifically, the user takes a photo of the receipt using the camera function of their smartphone and sends the image via the chat function of the message application.

[0411] Step 2:

[0412] The server receives image data sent by the user using the API of the messaging application. The input of this step is the image data sent through the messaging application, and the output is that it is saved on the server. Specifically, the server uses the API of the messaging application to detect new messages, receive the image file, and save it in the database.

[0413] Step 3:

[0414] The server extracts text information from the received image data using optical character recognition (OCR). The input for this step is the image data, and the output is the text information extracted from the image. Specifically, the server uses an OCR engine (e.g., Tesseract OCR) to extract the characters in the image in text format.

[0415] Step 4:

[0416] The server analyzes the extracted text information and automatically journalizes and records it for each type of transaction. The input to this step is the text information extracted by OCR, and the output is the analyzed transaction information being journalized and recorded in a database. Specifically, the server uses an analysis program to analyze the text information, classify it into the appropriate account category, and save it in the database.

[0417] Step 5:

[0418] The server converts the recorded journal data into a format for tax returns. The input for this step is the journal data, and the output is the data converted into a format for tax returns. Specifically, the server aggregates the journal data and converts it into a tax return format that complies with the tax laws of each country.

[0419] Step 6:

[0420] The user sends voice or text instructions to the server using natural language processing technology. The input for this step is the user's voice or text instruction, and the output is that the instruction is understood and executed. Specifically, the user enters "I would like to revise the travel expenses for October," and the instruction is sent to the server.

[0421] Step 7:

[0422] The server analyzes the user's instructions using a natural language processing engine and makes any necessary corrections. The input to this step is the natural language instruction from the user, and the output is the execution of data correction processing. Specifically, the server uses a natural language processing engine (e.g., spaCy) to analyze the user's instructions and correct the target data.

[0423] Step 8:

[0424] The server analyzes the recorded data and generates advice on tax and asset management. The input for this step is the recorded income and expenditure data, and the output is advice on tax and asset management. Specifically, the server uses the tax accountant AI algorithm to analyze the data and generate optimal advice.

[0425] Step 9:

[0426] The server analyzes the user's emotions and provides feedback based on the user's emotions using an emotion engine that optimizes the response content. The input for this step is the user's emotional expression in voice or text, and the output is a feedback message based on the emotion analysis results. Specifically, the server uses an emotion engine (e.g., a Transformers emotion analysis model) to analyze the user's emotional state and generate appropriate feedback.

[0427] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0428] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0429] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0430] [Second embodiment]

[0431] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0432] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0433] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0434] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0435] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0436] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0437] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0438] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0439] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0440] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0441] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0442] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0443] The present invention provides a system that allows users to efficiently manage income and expenditures and file tax returns by sending image data related to transactions via a message application using a portable communication terminal. This system operates in conjunction with the following programs.

[0444] 1. User sends image data

[0445] Users use their smartphones to take photos of receipts or invoices, which are then sent to the system via a messaging application (e.g., LINE).

[0446] The user logs in with LINE and sends a photo of a receipt or invoice to the chat.

[0447] 2. Receiving and analyzing image data

[0448] The server receives the image data sent by the user using the LINE API, and then extracts the text information from the image using optical character recognition technology (AI-OCR).

[0449] The server receives the image data, runs the AI-OCR engine, and extracts the text.

[0450] For example, text information such as "October 1, 2023," "10,000 yen," and "transportation expenses" is extracted from a receipt.

[0451] 3. Automatic journalization of extracted text

[0452] The extracted text information is analyzed by the server and automatically journalized by type of income and expenditure, with transactions classified into appropriate account items according to pre-set journalization rules.

[0453] Based on the analysis results, the server automatically records the expenses in categories such as "transportation expenses."

[0454] For example, "10,000 yen" is classified as "transportation expenses."

[0455] 4. Generate tax return data

[0456] The automatically posted data is aggregated on a server and converted into monthly and annual income and expenditure data in a format suitable for tax returns. This format is provided in accordance with the tax laws of each country.

[0457] The server aggregates the journal data by period and converts it into tax return format.

[0458] The data for tax returns will be sent to the user via LINE.

[0459] 5. Natural language instructions and corrections

[0460] Users can use natural language on LINE to check and edit their income and expenditure data. For example, if they say, "I want to edit my travel expenses for October," the server will activate its natural language processing engine, understand the user's instructions, and make the appropriate edits.

[0461] The user sends instructions in natural language, which the server analyzes using an NLP engine.

[0462] The server reviews the data and makes any necessary corrections.

[0463] 6. Collaboration with tax accountant AI

[0464] The server uses the collected income and expenditure data to generate optimal tax advice using a tax accountant AI algorithm, allowing users to receive appropriate advice on tax savings and asset management.

[0465] The server sends the data to the tax accountant AI, which generates optimal advice.

[0466] The advice will be sent to the user via LINE.

[0467] 7. Data Storage and Financial Services

[0468] The server encrypts and securely stores all collected data, and provides financial product recommendations and investment advice as needed, thus supporting users' financial growth.

[0469] The server encrypts and stores the data and suggests financial products.

[0470] Users receive financial advice and product details via LINE.

[0471] Specific examples

[0472] For example, let's say that self-employed business owner Sato wants to organize his income and expenses at the end of the month. Sato takes a photo of this month's receipts with his smartphone and sends it to the system via LINE. The server receives the image data and uses AI-OCR to extract the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server automatically accounts for this data and records it as entertainment expenses. Then, at the end of the month, data for tax returns is automatically generated and notified to Sato via LINE. When Sato sends a message via LINE saying, "I would like to review my entertainment expenses," the server understands the instruction and reconfirms and corrects it. Furthermore, a tax accountant AI analyzes Sato's data and provides tax-saving advice. Ultimately, Sato can manage his income and expenses and file his tax returns simply and efficiently through this system.

[0473] The processing flow will be explained below.

[0474] Step 1:

[0475] The user takes a photo of a receipt or invoice using a mobile communication device (smartphone). The user opens the LINE app and sends the photo to the LINE chat. This sends the image data to the system.

[0476] Step 2:

[0477] The server uses the LINE API to receive the image data sent by the user. The received image data is saved in its original format and passed to the next processing step.

[0478] Step 3:

[0479] The server activates optical character recognition (AI-OCR) technology to extract text information from the received image data. Specifically, the image processing engine identifies letters and numbers in the image and converts them into text. For example, information such as "October 1, 2023," "10,000 yen," and "transportation expenses" is extracted.

[0480] Step 4:

[0481] The extracted text information is analyzed by the server. The server identifies the type of transaction based on the text information and automatically journalizes it. Specifically, it classifies the transaction into the appropriate account category based on information such as the "date," "amount," and "item name." For example, "10,000 yen" is journalized as "transportation expenses."

[0482] Step 5:

[0483] The server aggregates the extracted and journalized data and generates monthly or annual income and expenditure reports, which are then converted into tax return formats and prepared for easy user access later.

[0484] Step 6:

[0485] The server notifies the user of the generated tax return data via LINE chat. The user can then check the data in the LINE app and request corrections if necessary.

[0486] Step 7:

[0487] The user sends a natural language instruction on LINE, such as "I would like to revise the transportation expenses for October." This instruction is sent to the server and analyzed by the natural language processing engine.

[0488] Step 8:

[0489] Based on the analysis results, the server understands the user's instructions and corrects the relevant journal data. For example, it rechecks "October travel expenses" and makes any necessary corrections.

[0490] Step 9:

[0491] The server provides all collected income and expenditure data to the AI ​​tax accountant, who then analyzes the data and provides optimal tax advice on tax-saving strategies and asset formation.

[0492] Step 10:

[0493] The server notifies the user of the generated advice via LINE chat, and the user can receive the advice in the LINE app and take the necessary action.

[0494] Step 11:

[0495] The server encrypts and securely stores all financial data, and the server uses this data to provide more advanced financial services and investment advice.

[0496] Step 12:

[0497] Users receive financial advice and product recommendations on LINE, which helps them make more effective financial decisions.

[0498] Example 1

[0499] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0500] Currently, managing income and expenses and filing tax returns takes a lot of time and effort. In particular, the process of manually recording receipts and invoices, journalizing them, and preparing data for tax returns is cumbersome and prone to errors. It is also difficult to receive real-time corrections and tax advice using natural language. This results in users expending a huge amount of effort and hinders efficient income and expense management.

[0501] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0502] In this invention, the server includes means for a user to use a wireless communication terminal to send image information related to a transaction via communication software, means for receiving the sent image information and extracting text information from the image using optical character recognition technology, means for analyzing the extracted text information and automatically journalizing and recording the transaction by type, means for converting the recorded journal data into a reporting format, means for understanding and correcting verbal or text instructions from the user using natural language processing technology, and means for analyzing the recorded data and generating advice on tax and asset management. This allows users to significantly reduce their workload and enable efficient and accurate income and expenditure management and tax return filing.

[0503] A "wireless communication terminal" is an electronic device that allows communication while moving, and includes smartphones, tablets, laptops, and the like.

[0504] "Communication software" refers to a program for sending and receiving data, and includes message applications, chat applications, and the like.

[0505] "Image information" means visual data captured electronically, including photographs and scanned data such as receipts and invoices.

[0506] "Optical character recognition technology" is a technology that analyzes characters in an image and converts them into text data, and is known as OCR (Optical Character Recognition).

[0507] "Character information" is data that is identified as characters, and includes text files and character string information.

[0508] The "means for automatically making and recording entries" is a system for analyzing extracted text information and classifying and recording it according to pre-set criteria.

[0509] "Format for tax return" means a standardized data format used for tax returns, including formats conforming to the laws and regulations of each country.

[0510] "Natural language processing technology" is a technology that understands and analyzes human language, and is a system for interpreting instructions entered in language or text.

[0511] The "means for generating tax and asset management advice" is a system that analyzes recorded data and makes recommendations on optimal tax strategies and asset management.

[0512] This invention is a system that allows users to efficiently manage income and expenditures and file tax returns using a wireless communication terminal.It works by sending image information related to transactions via communication software, and a server processes and analyzes that information.

[0513] System Overview

[0514] The system includes the following major components:

[0515] 1. User's wireless communication device (smartphone, tablet, laptop, etc.)

[0516] 2. Communication software (messaging and chat applications)

[0517] 3. Server

[0518] 4. Optical Character Recognition Technology (OCR engine, e.g. Tesseract OCR)

[0519] 5. Natural Language Processing Technology (NLP engine, e.g. GPT-3)

[0520] 6. Database

[0521] System Operation

[0522] Sending and receiving image information

[0523] A user uses a wireless communication terminal to take images of transaction-related documents such as receipts and invoices, which are then sent to a server via communication software.

[0524] The server receives the image data sent by the user using the API of the communication software, and the received image data is stored in the server's storage.

[0525] Image information analysis and classification

[0526] The server uses optical character recognition technology (OCR engine) to extract text information from the received image data. For example, from a receipt, text information such as "October 1, 2023," "10,000 yen," and "transportation expenses" is extracted.

[0527] The extracted text information is analyzed within the server, automatically categorized by transaction type, and recorded in a database based on pre-set rules.

[0528] Generate tax return data

[0529] The server aggregates the recorded journal data and converts it into a reporting format. This format is provided in a format that complies with the laws and regulations of each country. The generated reporting data is notified to the user via communication software.

[0530] Natural language instructions and corrections

[0531] Users can use natural language on the communication software to check and modify income and expenditure data. For example, if a user issues a command such as "I want to modify my travel expenses for October," the server will use natural language processing technology to analyze the user's command and make the appropriate modifications.

[0532] Providing tax advice

[0533] The server analyzes the recorded data and generates optimal advice on tax and asset management using a tax accountant AI algorithm, which is then communicated to the user via communication software.

[0534] Data Storage and Financial Services

[0535] The server encrypts and securely stores all data, and also provides financial product suggestions and investment advice based on the user's income and expenditure data, thereby supporting the user's financial growth.

[0536] Specific examples

[0537] For example, say a user wants to organize this month's income and expenses. The user takes a photo of a receipt with their smartphone and sends it to the system via communication software. The server analyzes the received image information using an OCR engine and extracts the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The extracted data is automatically accounted for as "entertainment expenses." At the end of the month, the server generates data for tax returns and notifies the user. Furthermore, if the user instructs in natural language that they would like to "review entertainment expenses," the server understands the instruction and makes the appropriate corrections. Through the above process, users can efficiently and accurately manage their income and expenses and file their tax returns.

[0538] Prompt Sentence Examples

[0539] 1. "I have sent you a scanned image of the receipt. Please let me know the extracted text data."

[0540] 2. "Please automatically post this month's travel expenses and display the related data."

[0541] 3. "Please generate the income and expenditure data for October in the tax return format and notify me."

[0542] 4. "How do I correct my income and expenditure data using natural language?"

[0543] 5. "Provide tax-saving advice."

[0544] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0545] Step 1: Send image information

[0546] The user uses the wireless communication terminal to capture image information related to the transaction.

[0547] Input: Captured image information (photos of receipts and invoices)

[0548] Operation: The user starts the communication software (messaging application) and logs in. Next, the user attaches the captured image information to the chat screen and presses the send button.

[0549] Output: Image information sent to the server via communication software

[0550] Step 2: Receiving image information

[0551] The server receives the image information sent by the user using the API of the communication software.

[0552] Input: Image information sent by the user

[0553] Operation: The server calls the API of the communication software and saves the received image information in storage.

[0554] Output: Image information saved in storage

[0555] Step 3: Analyze image information

[0556] The server uses optical character recognition technology (OCR engine) to extract text information from the received image information.

[0557] Input: Image information saved in storage

[0558] Operation: The server starts the OCR engine, analyzes the image data, and extracts text information. For example, from an image of a receipt, it obtains the text information "October 1, 2023," "10,000 yen," and "Transportation expenses."

[0559] Output: Extracted text information

[0560] Step 4: Journalize textual information

[0561] The server analyzes the extracted text information and automatically records and journals the transaction by type.

[0562] Input: Extracted text information

[0563] Operation: The server analyzes the text information and classifies it into the appropriate account based on the journal entry rules. For example, the information "October 1, 2023," "10,000 yen," and "Transportation expenses" is journalized as transportation expenses.

[0564] Output: Journalized data is recorded in the database

[0565] Step 5: Generate data for declaration

[0566] The server aggregates the recorded journal data and converts it into a format for reporting.

[0567] Input: Journal entry data recorded in the database

[0568] How it works: The server aggregates accounting data for a specific period and converts it into a reporting format that complies with the laws and regulations of each country. For example, it totals travel expenses, business expenses, and other expenses for one month.

[0569] Output: Data converted into a declaration format

[0570] Step 6: Natural Language Processing Remediation

[0571] Users can check and modify income and expenditure data in natural language through the communication software.

[0572] Input: Natural language instructions from the user (e.g., "I would like to correct the travel expenses for October.")

[0573] How it works: The server uses natural language processing (NLP) technology to parse the instructions and modify the specific data. For example, it extracts the entry for "October travel expenses" and modifies it to the specified value.

[0574] Output: Database updated with modified data

[0575] Step 7: Providing tax advice

[0576] The server analyzes the recorded data and generates optimal advice on tax and asset management.

[0577] Input: All income and expenditure data recorded in the database

[0578] How it works: The server uses the tax accountant AI algorithm to analyze the data and generate optimal advice, such as proposals for tax savings and asset management.

[0579] Output: The generated tax advice is communicated to the user via the communication software.

[0580] Step 8: Data Storage and Financial Services

[0581] The server encrypts and securely stores all data and provides financial product recommendations and investment advice as needed.

[0582] Input: All transaction data, journal data, user income and expenditure data

[0583] How it works: The server encrypts your transaction data and stores it in a secure database. It also analyzes your income and expenditure data and generates financial product and investment recommendations tailored to you.

[0584] Output: Encrypted and securely stored data, as well as financial product suggestions and investment advice notified to the user.

[0585] (Application example 1)

[0586] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0587] Previously, users had to manually collect image data related to transactions, sort them manually, and prepare tax returns, which required a great deal of effort and was prone to errors. Furthermore, there was a lack of efficient ways to manage income and expenditures or receive tax advice, placing a heavy burden on users. Furthermore, there was no system in place for managing daily expenses in real time and receiving appropriate financial advice. To solve these issues, a new method was needed.

[0588] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0589] In this invention, the server includes: means for a user to use a mobile communication device to send image data related to transactions via a messaging application; means for receiving the sent image data and extracting text information from the image using optical character recognition technology; means for analyzing the extracted text information and automatically journalizing and recording transactions by type; means for converting the recorded journal data into a format for tax returns; means for understanding and correcting voice or text instructions from the user using natural language processing technology; means for analyzing the recorded data and generating advice on tax and asset management; means for a user to send image data via an application on the mobile communication device and compile data related to financial transactions; means for providing up-to-date financial information based on the compiled data; and means for using a generative AI model to generate prompts related to income and expenditure management according to the user's requests. This allows users to efficiently manage transaction-related data and receive appropriate tax advice. It also allows users to manage their daily expenses in real time and receive optimal financial advice.

[0590] A "portable communication terminal" refers to an electronic device that is portable and capable of internet communication and data transmission and reception.

[0591] "Image data" refers to visual information obtained by optical means and expressed in digital form.

[0592] "Messaging application" refers to software that enables users to send and receive messages, including text, images, and audio.

[0593] "Optical character recognition technology" refers to the technology that extracts character information from image data and converts it into text data.

[0594] "Text information" refers to information expressed as a string of characters.

[0595] "Means for automatically making and recording journal entries" refers to the process of classifying text information extracted by the system by transaction type and saving the results in a database or the like.

[0596] A "tax return format" refers to income and expenditure data organized in the format required for submission to tax authorities.

[0597] "Natural language processing technology" refers to the technology that enables computers to understand and analyze text and speech written in natural language.

[0598] "Financial transaction related data" refers to economic information related to payments and transactions.

[0599] "Up-to-date financial information" means the most recent economic data, including recent transactions and expenditures.

[0600] A "generative AI model" refers to a model of artificial intelligence that uses machine learning algorithms to generate appropriate outputs for a specific task.

[0601] "Prompt" refers to the text content generated in response to a user instruction or question.

[0602] This invention is a system that improves the efficiency of income and expenditure management and tax return filing by allowing users to send image data related to transactions via a message application using a portable communication terminal. This system operates with the following configuration and program.

[0603] 1. User sends image data

[0604] A user takes a photo of a receipt or invoice using their smartphone. This photo is sent to the system via a messaging application (e.g., a general messaging application). The user logs in to the application and sends the photo of the receipt or invoice to the chat.

[0605] 2. Receiving and analyzing image data

[0606] The server receives image data sent by the user using a messaging API. It then uses AI-OCR (optical character recognition) technology to extract text information from the image. For example, it can use the Google Cloud Vision API. The server analyzes the image data and obtains information such as the date, time, amount, and subject.

[0607] 3. Automatic journalization of extracted text

[0608] The extracted text information is analyzed by the server and automatically journalized by transaction type. During this process, transactions are classified into appropriate account items according to pre-set journalization rules. For example, they may be classified into categories such as "transportation expenses" and "entertainment expenses."

[0609] 4. Generate tax return data

[0610] The automatically posted data is aggregated on the server and converted into a format for tax returns as monthly or annual income and expenditure data. This format is provided in a format that complies with the tax laws of each country. The data for tax returns is then notified to the user.

[0611] 5. Natural language instructions and corrections

[0612] Users can use natural language on the messaging application to check and modify their income and expenditure data. For example, if they issue a command such as "I want to modify my travel expenses for October," the server will activate its natural language processing engine, understand the user's command, and make the appropriate modifications. It uses OpenAI GPT-4 and other technologies to analyze the command and make the necessary modifications.

[0613] 6. Generating tax advice

[0614] The server generates tax and asset management advice based on the collected income and expenditure data, allowing users to receive appropriate advice on tax savings and asset management. Using a generative AI model, advice is provided that is tailored to the user's specific situation.

[0615] 7. Providing real-time financial information

[0616] A user sends image data via an application on a portable communication terminal, and data related to financial transactions is compiled. The system provides the latest financial information based on the compiled data. For example, in response to a request such as "What is the total amount of expenses this month?", the system displays the latest financial information in real time.

[0617] Specific examples

[0618] For example, say a self-employed person wants to organize their income and expenses at the end of the month. The user takes a photo of this month's receipts with their smartphone and sends it to the system from the application. The server receives the image data and uses AI-OCR to extract the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server automatically journalizes this data and records it as entertainment expenses. Then, at the end of the month, data for tax returns is automatically generated and notified to the user.

[0619] When a user sends a message saying, "I'd like to review my entertainment expenses," the server understands the instruction and reconfirms and corrects it. Furthermore, a generative AI model analyzes the user's data and provides tax-saving advice. Ultimately, the system allows users to easily and efficiently manage their income and expenses and file tax returns.

[0620] Prompt Sentence Examples

[0621] "I would like to revise my travel expenses for October."

[0622] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0623] Step 1:

[0624] A user uses a portable communication terminal to take a photo of a receipt or invoice related to a transaction and transmits the image data to a server via a message application. The input is the image data taken by the user, and the output is the image data transmitted to the server via the message application.

[0625] Step 2:

[0626] The server receives image data sent by the user using a messaging API. The input is the image data sent by the user, and the output is the image data received by the server.

[0627] Step 3:

[0628] The server applies AI-OCR (optical character recognition) technology to the received image data and extracts the text information within the image. The input is the received image data, and the output is text information. Specifically, the image is analyzed using Google Cloud Vision API, etc., and text information such as the date, time, amount, and transaction items is extracted.

[0629] Step 4:

[0630] The server analyzes the extracted text information and automatically journalizes and records each type of transaction. The input is the extracted text information, and the output is automatically journalized data. Specifically, the data is classified into categories such as "transportation expenses" and "entertainment expenses" and recorded in the appropriate account.

[0631] Step 5:

[0632] The server aggregates the recorded journal data and converts it into a format for tax returns as monthly or annual income and expenditure data. The input is automatically journalized data, and the output is data converted into a format for tax returns.

[0633] Step 6:

[0634] The user uses a messaging application to input correction instructions in natural language. For example, they might send a message saying, "I would like to correct the travel expenses for October." The input is a natural language instruction from the user, and the output is the target data to be corrected based on that instruction.

[0635] Step 7:

[0636] The server uses natural language processing technology to analyze user instructions and make appropriate data corrections. The input is the user's natural language instructions, and the output is the corrected income and expenditure data. Specifically, OpenAI GPT-4 and other technologies are used to interpret the user's intent and update the relevant parts of the database.

[0637] Step 8:

[0638] The server generates advice on tax and asset management based on the collected income and expenditure data. The input is the recorded income and expenditure data, and the output is the generated advice. Using a generative AI model, it provides specific advice tailored to the user's situation.

[0639] Step 9:

[0640] The server aggregates data related to image data transmissions and financial transactions performed by users via applications on mobile communication terminals in real time and provides the latest financial information. The input is the transmitted image data and transaction data, and the output is the updated latest financial information. The data is processed in real time, and the most recent financial information is displayed when the user requests it.

[0641] Examples of concrete examples and prompts

[0642] For example, say a self-employed person wants to organize their income and expenses at the end of the month. The user takes a photo of this month's receipts with their smartphone and sends it to the system from the application. The server receives the image data and uses AI-OCR to extract the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server automatically journalizes this data and records it as entertainment expenses. Then, at the end of the month, data for tax returns is automatically generated and notified to the user.

[0643] Prompt Sentence Examples

[0644] "I would like to revise my travel expenses for October."

[0645] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0646] The present invention combines an emotion engine with a system that allows users to efficiently manage income and expenditures and file tax returns by sending image data related to transactions via a message application using a portable communication terminal. This system operates in conjunction with the following programs.

[0647] 1. User sends image data

[0648] Users use their smartphones to take photos of receipts or invoices, which are then sent to the system via a messaging application (e.g., LINE).

[0649] The user logs in with LINE and sends a photo of a receipt or invoice to the chat.

[0650] 2. Receiving and analyzing image data

[0651] The server receives the image data sent by the user using the LINE API, and then extracts the text information from the image using optical character recognition technology (AI-OCR).

[0652] The server receives the image data, runs the AI-OCR engine, and extracts the text.

[0653] For example, text information such as "October 1, 2023," "10,000 yen," and "transportation expenses" is extracted from a receipt.

[0654] 3. Automatic journalization of extracted text

[0655] The extracted text information is analyzed by the server and automatically journalized by type of income and expenditure, with transactions classified into appropriate account items according to pre-set journalization rules.

[0656] Based on the analysis results, the server automatically records the expenses in categories such as "transportation expenses."

[0657] For example, "10,000 yen" is classified as "transportation expenses."

[0658] 4. Generate tax return data

[0659] The automatically posted data is aggregated on the server and reflected in monthly and annual income and expenditure reports. This data is converted into a tax return format and prepared for easy access by users later.

[0660] The server aggregates the journal data by period and converts it into tax return format.

[0661] The data for tax returns will be sent to the user via LINE.

[0662] 5. Natural language instructions and corrections

[0663] Users can use natural language on LINE to check and edit their income and expenditure data. For example, if they say, "I want to edit my travel expenses for October," the server will activate its natural language processing engine, understand the user's instructions, and make the appropriate edits.

[0664] The user sends instructions in natural language, which the server analyzes using an NLP engine.

[0665] The server reviews the data and makes any necessary corrections.

[0666] 6. Collaboration with tax accountant AI

[0667] The server uses the collected income and expenditure data to generate optimal tax advice using a tax accountant AI algorithm, allowing users to receive appropriate advice on tax savings and asset management.

[0668] The server sends the data to the tax accountant AI, which generates optimal advice.

[0669] The advice will be sent to the user via LINE.

[0670] 7. Data Storage and Financial Services

[0671] The server encrypts and securely stores all collected data, and provides financial product recommendations and investment advice as needed, thus supporting users' financial growth.

[0672] The server encrypts and stores the data and suggests financial products.

[0673] Users receive financial advice and product details via LINE.

[0674] 8. Introducing and utilizing an emotion engine

[0675] The server incorporates a new emotion engine that analyzes emotions based on the user's voice or text data and optimizes response content. This engine recognizes the user's emotions in real time and can respond appropriately.

[0676] The user sends a voice message such as "Today is stressful and difficult."

[0677] The server receives the voice data and uses an emotion engine to analyze the user's emotional state.

[0678] The server provides appropriate feedback via LINE based on the user's emotions (e.g., "You seem stressed. Let me know if there's anything I can do to help you").

[0679] Specific examples

[0680] For example, let's say that self-employed Tanaka wants to organize his income and expenses at the end of the month. Tanaka takes a photo of this month's receipts with his smartphone and sends it to the system via LINE. The server receives the image data and uses AI-OCR to extract the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server automatically accounts for this data and records it as entertainment expenses. Then, at the end of the month, data for tax returns is automatically generated and notified to Tanaka via LINE. When Tanaka sends a message via LINE saying, "I would like to review my entertainment expenses," the server understands the instruction and reconfirms and corrects it. Furthermore, the tax accountant AI analyzes Tanaka's data and provides tax-saving advice.

[0681] One day, Tanaka sends a voice message to LINE saying, "I'm feeling very stressed today," and the server's emotion engine analyzes Tanaka's emotions and provides appropriate feedback, such as, "It seems you're feeling stressed. Let me know if there's anything I can help you with." This allows Tanaka to not only efficiently manage his income and expenses and file his tax return through the system, but also receive emotional support.

[0682] The processing flow will be explained below.

[0683] Step 1:

[0684] The user takes a photo of a receipt or invoice using a mobile communication device (smartphone). The user opens the LINE app and sends the photo to the LINE chat. This sends the image data to the system.

[0685] Step 2:

[0686] The server uses the LINE API to receive the image data sent by the user. The received image data is saved in its original format and passed to the next processing step.

[0687] Step 3:

[0688] The server activates optical character recognition (AI-OCR) technology to extract text information from the received image data. Specifically, the image processing engine identifies letters and numbers in the image and converts them into text. For example, information such as "October 1, 2023," "10,000 yen," and "transportation expenses" is extracted.

[0689] Step 4:

[0690] The extracted text information is analyzed by the server. The server identifies the type of transaction based on the text information and automatically journalizes it. Specifically, it classifies the transaction into the appropriate account category based on information such as the "date," "amount," and "item name." For example, "10,000 yen" is journalized as "transportation expenses."

[0691] Step 5:

[0692] The server aggregates the extracted and journalized data and generates monthly or annual income and expenditure reports, which are then converted into tax return formats and prepared for easy user access later.

[0693] Step 6:

[0694] The server notifies the user of the generated tax return data via LINE chat. The user can then check the data in the LINE app and request corrections if necessary.

[0695] Step 7:

[0696] The user sends a natural language instruction on LINE, such as "I would like to revise the transportation expenses for October." This instruction is sent to the server and analyzed by the natural language processing engine.

[0697] Step 8:

[0698] Based on the analysis results, the server understands the user's instructions and corrects the relevant journal data. For example, it rechecks "October travel expenses" and makes any necessary corrections.

[0699] Step 9:

[0700] The server provides all collected income and expenditure data to the AI ​​tax accountant, who then analyzes the data and provides optimal tax advice on tax-saving strategies and asset formation.

[0701] Step 10:

[0702] The server notifies the user of the generated advice via LINE chat, and the user can receive the advice in the LINE app and take the necessary action.

[0703] Step 11:

[0704] The server uses an emotion engine to analyze the user's voice or text data in real time. For example, if a user says, "I'm feeling stressed today," the server recognizes the voice data and analyzes the user's emotional state.

[0705] Step 12:

[0706] Based on the analysis results of the emotion engine, the server provides appropriate feedback according to the user's emotions. For example, it may send a message via LINE saying, "You seem to be feeling stressed. Please let us know if there is anything we can do to help."

[0707] Step 13:

[0708] The server encrypts and securely stores all financial data, and the server uses this data to provide more advanced financial services and investment advice.

[0709] Step 14:

[0710] Users receive financial advice and product recommendations on LINE, which helps them make more effective financial decisions.

[0711] Specific examples

[0712] For example, let's say that self-employed Tanaka wants to organize his income and expenses at the end of the month. Tanaka takes a photo of this month's receipts with his smartphone and sends it to the system via LINE. The server receives the image data and uses AI-OCR to extract the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server automatically accounts for this data and records it as entertainment expenses. Then, at the end of the month, data for tax returns is automatically generated and notified to Tanaka via LINE. When Tanaka sends a message via LINE saying, "I would like to review my entertainment expenses," the server understands the instruction and reconfirms and corrects it. Furthermore, the tax accountant AI analyzes Tanaka's data and provides tax-saving advice.

[0713] One day, Tanaka sends a voice message to LINE saying, "I'm feeling very stressed today," and the server's emotion engine analyzes Tanaka's emotions and provides appropriate feedback, such as, "It seems you're feeling stressed. Let me know if there's anything I can help you with." This allows Tanaka to not only efficiently manage his income and expenses and file his tax return through the system, but also receive emotional support.

[0714] Example 2

[0715] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0716] Traditional income and expenditure management and tax return systems required time and effort for manual data entry, resulting in inefficiency. They also often failed to accurately record collected data or provide adequate tax advice. Furthermore, they lacked support that took into account the user's emotional and stress levels, resulting in a lack of improvement in the overall user experience.

[0717] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for a user to use a portable communication terminal to send image data related to a transaction via a message application; means for receiving the sent image data and extracting text information from the image using optical character recognition technology; means for analyzing the extracted text information and automatically journalizing and recording the transaction type; means for converting the recorded journal data into a format for tax returns; means for understanding and correcting voice or text instructions from the user using natural language processing technology; means for analyzing the recorded data and generating advice on taxes and asset management; and means for recognizing emotions based on the user's voice or text data using emotion analysis technology and optimizing the response. This not only improves the efficiency of income and expenditure management and tax return filing, but also enables improved accuracy of tax advice and emotional support for users.

[0718] "User" refers to any individual or legal entity that uses the System.

[0719] A "portable communication terminal" refers to a terminal device such as a smartphone or tablet that is portable and capable of wireless communication.

[0720] "Transaction" refers to the act of exchanging money or goods in economic activities.

[0721] "Image data" refers to visual information represented in digital form.

[0722] "Message application" refers to an application that sends and receives text messages and image data over the Internet.

[0723] "Optical character recognition technology" refers to technology that extracts characters from an image as digital data.

[0724] "Text information" refers to data expressed as a string of characters.

[0725] "Journal entry" refers to the accounting process of classifying and recording transactions into account items.

[0726] "Server" refers to a device or system that provides computer resources over a network.

[0727] "Tax return format" refers to the format of the income and expenditure report to be submitted to the tax office.

[0728] "Natural language processing technology" refers to technology for understanding human language and responding appropriately.

[0729] "Voice or text instructions" refers to operation instructions given to the system by the user using words or letters.

[0730] "Tax advice" refers to the act or content of providing appropriate advice regarding tax.

[0731] "Asset management" refers to the efficient management and administration of personal or corporate assets.

[0732] "Emotion analysis technology" refers to the technology of analyzing human emotions and psychological states from voice and text.

[0733] "Optimizing response content" refers to optimizing the responses and support provided by the system according to the user's needs and condition.

[0734] This invention is a system that allows users to efficiently manage income and expenditures and file tax returns using a mobile communication device. This system mainly combines a messaging application, optical character recognition technology (AI-OCR), natural language processing technology (NLP), tax accountant AI algorithms, and sentiment analysis technology. Specific examples of the use of various hardware and software are shown below.

[0735] A user uses a mobile communication device (e.g., a smartphone or tablet) to take a photo of a receipt or invoice related to a transaction. At that time, the user uses a messaging application (e.g., LINE) to send the image data to the system. The user logs in to LINE and sends a photo of the receipt or invoice as a chat message.

[0736] The server receives image data sent by the user using LINE's API. The server then uses optical character recognition technology (e.g., Google Cloud Vision API) to extract text information from the image. This AI-OCR engine identifies characters in the image and extracts them as numerical or text data. For example, from a receipt, the following information is extracted: "October 1, 2023," "10,000 yen," and "Transportation expenses."

[0737] The extracted text information is analyzed by the server and automatically journalized according to pre-set journalization rules. Based on the analysis results, the server classifies the data into appropriate account items for each type of income and expenditure. For example, "10,000 yen" is journalized as "transportation expenses."

[0738] The server converts the collected data into a format for tax returns. This data is reflected in monthly and annual income and expenditure reports and prepared as tax return data. The server then aggregates this data and converts it into a tax return format (e.g., XML for tax software). Users are notified via LINE for easy access to their tax return data.

[0739] Users can also send voice or text instructions on LINE using natural language processing technology. For example, if you send an instruction such as "I would like to revise my travel expenses for October," the server will use an NLP engine (e.g., OpenAI GPT-3) to analyze the message, understand the user's instruction, and make the appropriate revisions.

[0740] Based on the collected data, the server uses a tax accountant AI algorithm (e.g., Watson Tax Advisor) to generate optimal tax advice. Users can receive advice on tax savings and asset management via LINE. The server also encrypts and securely stores revenue data and provides users with financial products and investment advice.

[0741] Emotion analysis technology (e.g., IBM Watson Tone Analyzer) recognizes emotions from a user's voice or text data and provides feedback in real time. For example, if a user sends a voice message saying, "I'm feeling very stressed today," the server analyzes the emotion and replies, "It sounds like you're feeling stressed. Let me know if there's anything I can do to help."

[0742] Specific examples

[0743] A self-employed individual wants to organize their income and expenses at the end of the month. They take a photo of this month's receipts with their smartphone and send it to the system via LINE. The server receives the image data and uses AI-OCR to extract the following information: "October 15, 2023," "5,000 yen," and "Entertainment Expenses." The server automatically accounts for this data and records it as entertainment expenses. At the end of the month, tax return data is automatically generated and notified via LINE. When the self-employed individual sends a message via LINE saying, "I'd like to review my entertainment expenses," the server understands the instruction and reconfirms and corrects it. Furthermore, a tax accountant AI analyzes the data and provides tax-saving advice. One day, the self-employed individual sends a voice message via LINE saying, "I'm feeling stressed today and it's bothering me," and the server's emotion engine analyzes the emotion and provides appropriate feedback, such as, "It sounds like you're stressed. Let me know if there's anything I can do to help."

[0744] Prompt Sentence Examples

[0745] "Please send a photo of your receipt via LINE to record your income and expenses."

[0746] "I would like to revise my travel expenses for October 2023."

[0747] "Can you give me some tax saving advice?"

[0748] "I'm having a lot of stress today"

[0749] In this way, the system helps users manage their finances and file their tax returns efficiently, and also provides emotional support.

[0750] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0751] Step 1: Sending image data

[0752] The user takes a photo of an image related to the transaction (such as a receipt or invoice) using a mobile communication terminal (e.g., a smartphone) and sends it to the system via a messaging application (e.g., LINE).

[0753] Input: Image data of receipts and invoices

[0754] Output: Image data sent to LINE

[0755] Specific operation: The user launches the camera app and takes a picture of a receipt or invoice. The user selects the image in the LINE app and sends it as a chat message.

[0756] Step 2: Receiving image data

[0757] The server uses the LINE API to receive the image data sent by the user.

[0758] Input: Image data sent via the LINE API

[0759] Output: Image data stored on the server

[0760] Specific operation: The server periodically checks the LINE API to receive new image data, which is then stored in the database.

[0761] Step 3: Extracting text information

[0762] The server uses optical character recognition technology (AI-OCR) to extract text information from the received image data.

[0763] Input: Saved image data

[0764] Output: Extracted text information

[0765] How it works: The server launches an AI-OCR engine such as Google Cloud Vision API, identifies the characters in the image, and generates text data. For example, the following information is extracted: "October 1, 2023," "10,000 yen," and "transportation expenses."

[0766] Step 4: Parsing and journalizing extracted text

[0767] The server analyzes the extracted text information and automatically records and journals the transactions by type.

[0768] Input: Extracted text information

[0769] Output: Journal entry data

[0770] Specific operation: The server analyzes the text information based on pre-defined rules and classifies it into categories such as "transportation expenses" and "entertainment expenses." For example, "10,000 yen" is journalized as "transportation expenses." The journalized data is then saved in a database.

[0771] Step 5: Generate tax return data

[0772] The server converts the data into a format for tax returns based on the accounting data.

[0773] Input: Journal data

[0774] Output: Data in tax return format

[0775] Specific operation: The server aggregates monthly and annual income and expenditure reports and converts them into a format for tax returns (e.g., XML for tax software). The user is notified via LINE that "tax return data is ready."

[0776] Step 6: Correcting data using natural language

[0777] Users can check and edit income and expenditure data using natural language on LINE.

[0778] Input: Natural language instructions from the user

[0779] Output: Corrected balance data

[0780] Specific operation: The user sends a message on LINE saying, "I would like to correct my travel expenses for October." The server uses an NLP engine (e.g., OpenAI GPT-3) to analyze the message and understand the instructions. The server then reviews the target data and makes any necessary corrections.

[0781] Step 7: Advice generation by tax accountant AI

[0782] The server sends the collected data to the tax accountant AI, which generates optimal tax advice.

[0783] Input: Collected income and expenditure data

[0784] Output: Tax advice

[0785] Specific operation: The server sends the collected data to a tax accountant AI (e.g., Watson Tax Advisor). The tax accountant AI analyzes the data and generates appropriate tax advice. The advice is then sent to the user via LINE.

[0786] Step 8: Optimize your response with sentiment analysis

[0787] The server uses emotion analysis technology to recognize the user's emotions and optimize the response.

[0788] Input: Voice or text data from the user

[0789] Output: Emotion-based feedback

[0790] Specific operation: The user sends a voice message stating, "I'm feeling very stressed today." The server receives the voice data and analyzes the user's emotions using an emotion analysis engine (e.g., IBM Watson Tone Analyzer). Appropriate feedback based on the user's emotions is then provided via LINE (e.g., "It sounds like you're feeling stressed. Please let me know if there's anything I can do to help you").

[0791] This processing flow allows the system to streamline users' income and expenditure management and tax return filing, and also provides emotional support.

[0792] (Application example 2)

[0793] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0794] Managing transaction-related income and expenditures and filing tax returns are time-consuming and labor-intensive tasks, especially for small businesses and sole proprietors. Traditional manual data entry and accounting processes are prone to errors, and users without specialized knowledge of tax or asset management find it difficult to obtain appropriate support. Furthermore, few systems take into account the user's emotions and mental state, which leads to a poor user experience. Therefore, the challenge is to provide a system that streamlines income and expenditure management and tax return filing and provides emotionally sensitive feedback.

[0795] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to use a portable communication terminal to send image data related to a transaction via a message application; means for receiving the sent image data and extracting text information from the image using optical character recognition technology; means for analyzing the extracted text information and automatically journalizing and recording the transaction type; means for converting the recorded journal data into a format for tax returns; means for understanding and correcting voice or text instructions from the user using natural language processing technology; means for analyzing the recorded data and generating advice on taxes and asset management; and means for analyzing the user's emotions and providing feedback according to the user's emotions using an emotion engine that optimizes response content. This improves the efficiency of income and expenditure management and tax return filing and also enables support that takes into account the user's emotions and mental state.

[0796] A "portable communication terminal" is a device that is portable and has communication capabilities, and mainly refers to smartphones and tablets.

[0797] "Transaction-related image data" refers to image data, such as receipts and invoices, that contains information about financial transactions.

[0798] A "messaging application" is application software that allows users to send and receive messages, such as text and images, via the Internet, and includes LINE and WhatsApp.

[0799] "Optical character recognition technology" is a technology that extracts character information from image data, and is also known as OCR (Optical Character Recognition).

[0800] "Text information" refers to information such as letters and numbers, and includes character data extracted from an image.

[0801] "Journal entries" are the process of classifying transactions into specific account items and recording them in the ledger.

[0802] "Tax format" means the conversion of collected financial data into a specific format for use in tax returns.

[0803] "Natural language processing technology" is a technology that allows computers to understand, analyze, and respond to human language, and is also known as NLP (Natural Language Processing).

[0804] An "emotion engine" is a technology that analyzes user data such as voice and text and estimates their emotional state.

[0805] "Feedback" refers to the response or advice that a system gives to a user in response to their input.

[0806] "Tax and asset management advice" means providing expertise based on the user's financial data and offering advice on tax optimization and asset management.

[0807] This invention is a system that combines an emotion engine with a system that allows users to use a portable communication terminal to send image data related to transactions via a message application, thereby streamlining income and expenditure management and tax return filing.

[0808] The server receives image data related to transactions sent by users via messaging applications such as LINE and WhatsApp. It extracts text information from the received image data using optical character recognition (OCR). The extracted text information is stored in the server's database, analyzed by an automatic accounting program, and automatically recorded and accounted for by transaction type.

[0809] Furthermore, the recorded journal data is automatically converted into a format for tax returns and provided to users in a format that can be used for tax returns. This also implements a means of generating advice on taxes and asset management, and the tax accountant AI provides optimal advice based on the data.

[0810] Users can send voice or text instructions to the server through an interface that uses natural language processing technology. For example, they can send a message to a friend saying, "I want to revise my travel expenses for October." The server analyzes this instruction, reviews the relevant data, and makes any necessary adjustments. Furthermore, if a user sends a message expressing their emotions, the server's emotion engine analyzes it and provides feedback appropriate to the user's emotional state.

[0811] The hardware used includes a server and a mobile communication device, and the software includes the LINE Messaging API, an OCR engine (such as Tesseract OCR), MySQL, a natural language processing engine (such as spaCy), and a sentiment analysis model (such as the Transformers sentiment analysis pipeline).

[0812] For example, a user can take a photo of a receipt with their smartphone and send it to the system via a messaging application. The image data is transferred to the server, where OCR extracts the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server records this data as entertainment expenses and automatically generates data for tax returns. If the user then issues a natural language instruction such as "I'd like to correct the entertainment expenses for October," the system will correct the relevant data. Furthermore, if the user voice-transmits "I'm feeling very stressed today," the emotion engine will analyze the emotion and provide feedback such as "It sounds like you're stressed. Please let me know if there's anything I can do to help."

[0813] An example prompt is:

[0814] "Go to My Page and check your credit card transaction details from last month. Then, send an image of the receipt to LINE and correct the relevant transaction. Also, if you would like to receive advice on asset management, please let the system know."

[0815] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0816] Step 1:

[0817] A user uses a mobile communication terminal to send image data related to a transaction (e.g., a photo of a receipt or invoice) to the system via a message application. The input of this step is the image data of the receipt or invoice, and the output is that it reaches the server via the message application. Specifically, the user takes a photo of the receipt using the camera function of their smartphone and sends the image via the chat function of the message application.

[0818] Step 2:

[0819] The server receives image data sent by the user using the API of the messaging application. The input of this step is the image data sent through the messaging application, and the output is that it is saved on the server. Specifically, the server uses the API of the messaging application to detect new messages, receive the image file, and save it in the database.

[0820] Step 3:

[0821] The server extracts text information from the received image data using optical character recognition (OCR). The input for this step is the image data, and the output is the text information extracted from the image. Specifically, the server uses an OCR engine (e.g., Tesseract OCR) to extract the characters in the image in text format.

[0822] Step 4:

[0823] The server analyzes the extracted text information and automatically journalizes and records it for each type of transaction. The input to this step is the text information extracted by OCR, and the output is the analyzed transaction information being journalized and recorded in a database. Specifically, the server uses an analysis program to analyze the text information, classify it into the appropriate account category, and save it in the database.

[0824] Step 5:

[0825] The server converts the recorded journal data into a format for tax returns. The input for this step is the journal data, and the output is the data converted into a format for tax returns. Specifically, the server aggregates the journal data and converts it into a tax return format that complies with the tax laws of each country.

[0826] Step 6:

[0827] The user sends voice or text instructions to the server using natural language processing technology. The input for this step is the user's voice or text instruction, and the output is that the instruction is understood and executed. Specifically, the user enters "I would like to revise the travel expenses for October," and the instruction is sent to the server.

[0828] Step 7:

[0829] The server analyzes the user's instructions using a natural language processing engine and makes any necessary corrections. The input to this step is the natural language instruction from the user, and the output is the execution of data correction processing. Specifically, the server uses a natural language processing engine (e.g., spaCy) to analyze the user's instructions and correct the target data.

[0830] Step 8:

[0831] The server analyzes the recorded data and generates advice on tax and asset management. The input for this step is the recorded income and expenditure data, and the output is advice on tax and asset management. Specifically, the server uses the tax accountant AI algorithm to analyze the data and generate optimal advice.

[0832] Step 9:

[0833] The server analyzes the user's emotions and provides feedback based on the user's emotions using an emotion engine that optimizes the response content. The input for this step is the user's emotional expression in voice or text, and the output is a feedback message based on the emotion analysis results. Specifically, the server uses an emotion engine (e.g., a Transformers emotion analysis model) to analyze the user's emotional state and generate appropriate feedback.

[0834] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0835] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0836] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0837] [Third embodiment]

[0838] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0839] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0840] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0841] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0842] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0843] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0844] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0845] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0846] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0847] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0848] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0849] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0850] The present invention provides a system that allows users to efficiently manage income and expenditures and file tax returns by sending image data related to transactions via a message application using a portable communication terminal. This system operates in conjunction with the following programs.

[0851] 1. User sends image data

[0852] Users use their smartphones to take photos of receipts or invoices, which are then sent to the system via a messaging application (e.g., LINE).

[0853] The user logs in with LINE and sends a photo of a receipt or invoice to the chat.

[0854] 2. Receiving and analyzing image data

[0855] The server receives the image data sent by the user using the LINE API, and then extracts the text information from the image using optical character recognition technology (AI-OCR).

[0856] The server receives the image data, runs the AI-OCR engine, and extracts the text.

[0857] For example, text information such as "October 1, 2023," "10,000 yen," and "transportation expenses" is extracted from a receipt.

[0858] 3. Automatic journalization of extracted text

[0859] The extracted text information is analyzed by the server and automatically journalized by type of income and expenditure, with transactions classified into appropriate account items according to pre-set journalization rules.

[0860] Based on the analysis results, the server automatically records the expenses in categories such as "transportation expenses."

[0861] For example, "10,000 yen" is classified as "transportation expenses."

[0862] 4. Generate tax return data

[0863] The automatically posted data is aggregated on a server and converted into monthly and annual income and expenditure data in a format suitable for tax returns. This format is provided in accordance with the tax laws of each country.

[0864] The server aggregates the journal data by period and converts it into tax return format.

[0865] The data for tax returns will be sent to the user via LINE.

[0866] 5. Natural language instructions and corrections

[0867] Users can use natural language on LINE to check and edit their income and expenditure data. For example, if they say, "I want to edit my travel expenses for October," the server will activate its natural language processing engine, understand the user's instructions, and make the appropriate edits.

[0868] The user sends instructions in natural language, which the server analyzes using an NLP engine.

[0869] The server reviews the data and makes any necessary corrections.

[0870] 6. Collaboration with tax accountant AI

[0871] The server uses the collected income and expenditure data to generate optimal tax advice using a tax accountant AI algorithm, allowing users to receive appropriate advice on tax savings and asset management.

[0872] The server sends the data to the tax accountant AI, which generates optimal advice.

[0873] The advice will be sent to the user via LINE.

[0874] 7. Data Storage and Financial Services

[0875] The server encrypts and securely stores all collected data, and provides financial product recommendations and investment advice as needed, thus supporting users' financial growth.

[0876] The server encrypts and stores the data and suggests financial products.

[0877] Users receive financial advice and product details via LINE.

[0878] Specific examples

[0879] For example, let's say that self-employed business owner Sato wants to organize his income and expenses at the end of the month. Sato takes a photo of this month's receipts with his smartphone and sends it to the system via LINE. The server receives the image data and uses AI-OCR to extract the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server automatically accounts for this data and records it as entertainment expenses. Then, at the end of the month, data for tax returns is automatically generated and notified to Sato via LINE. When Sato sends a message via LINE saying, "I would like to review my entertainment expenses," the server understands the instruction and reconfirms and corrects it. Furthermore, a tax accountant AI analyzes Sato's data and provides tax-saving advice. Ultimately, Sato can manage his income and expenses and file his tax returns simply and efficiently through this system.

[0880] The processing flow will be explained below.

[0881] Step 1:

[0882] The user takes a photo of a receipt or invoice using a mobile communication device (smartphone). The user opens the LINE app and sends the photo to the LINE chat. This sends the image data to the system.

[0883] Step 2:

[0884] The server uses the LINE API to receive the image data sent by the user. The received image data is saved in its original format and passed to the next processing step.

[0885] Step 3:

[0886] The server activates optical character recognition (AI-OCR) technology to extract text information from the received image data. Specifically, the image processing engine identifies letters and numbers in the image and converts them into text. For example, information such as "October 1, 2023," "10,000 yen," and "transportation expenses" is extracted.

[0887] Step 4:

[0888] The extracted text information is analyzed by the server. The server identifies the type of transaction based on the text information and automatically journalizes it. Specifically, it classifies the transaction into the appropriate account category based on information such as the "date," "amount," and "item name." For example, "10,000 yen" is journalized as "transportation expenses."

[0889] Step 5:

[0890] The server aggregates the extracted and journalized data and generates monthly or annual income and expenditure reports, which are then converted into tax return formats and prepared for easy user access later.

[0891] Step 6:

[0892] The server notifies the user of the generated tax return data via LINE chat. The user can then check the data in the LINE app and request corrections if necessary.

[0893] Step 7:

[0894] The user sends a natural language instruction on LINE, such as "I would like to revise the transportation expenses for October." This instruction is sent to the server and analyzed by the natural language processing engine.

[0895] Step 8:

[0896] Based on the analysis results, the server understands the user's instructions and corrects the relevant journal data. For example, it rechecks "October travel expenses" and makes any necessary corrections.

[0897] Step 9:

[0898] The server provides all collected income and expenditure data to the AI ​​tax accountant, who then analyzes the data and provides optimal tax advice on tax-saving strategies and asset formation.

[0899] Step 10:

[0900] The server notifies the user of the generated advice via LINE chat, and the user can receive the advice in the LINE app and take the necessary action.

[0901] Step 11:

[0902] The server encrypts and securely stores all financial data, and the server uses this data to provide more advanced financial services and investment advice.

[0903] Step 12:

[0904] Users receive financial advice and product recommendations on LINE, which helps them make more effective financial decisions.

[0905] Example 1

[0906] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0907] Currently, managing income and expenses and filing tax returns takes a lot of time and effort. In particular, the process of manually recording receipts and invoices, journalizing them, and preparing data for tax returns is cumbersome and prone to errors. It is also difficult to receive real-time corrections and tax advice using natural language. This results in users expending a huge amount of effort and hinders efficient income and expense management.

[0908] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0909] In this invention, the server includes means for a user to use a wireless communication terminal to send image information related to a transaction via communication software, means for receiving the sent image information and extracting text information from the image using optical character recognition technology, means for analyzing the extracted text information and automatically journalizing and recording the transaction by type, means for converting the recorded journal data into a reporting format, means for understanding and correcting verbal or text instructions from the user using natural language processing technology, and means for analyzing the recorded data and generating advice on tax and asset management. This allows users to significantly reduce their workload and enable efficient and accurate income and expenditure management and tax return filing.

[0910] A "wireless communication terminal" is an electronic device that allows communication while moving, and includes smartphones, tablets, laptops, and the like.

[0911] "Communication software" refers to a program for sending and receiving data, and includes message applications, chat applications, and the like.

[0912] "Image information" means visual data captured electronically, including photographs and scanned data such as receipts and invoices.

[0913] "Optical character recognition technology" is a technology that analyzes characters in an image and converts them into text data, and is known as OCR (Optical Character Recognition).

[0914] "Character information" is data that is identified as characters, and includes text files and character string information.

[0915] The "means for automatically making and recording entries" is a system for analyzing extracted text information and classifying and recording it according to pre-set criteria.

[0916] "Format for tax return" means a standardized data format used for tax returns, including formats conforming to the laws and regulations of each country.

[0917] "Natural language processing technology" is a technology that understands and analyzes human language, and is a system for interpreting instructions entered in language or text.

[0918] The "means for generating tax and asset management advice" is a system that analyzes recorded data and makes recommendations on optimal tax strategies and asset management.

[0919] This invention is a system that allows users to efficiently manage income and expenditures and file tax returns using a wireless communication terminal.It works by sending image information related to transactions via communication software, and a server processes and analyzes that information.

[0920] System Overview

[0921] The system includes the following major components:

[0922] 1. User's wireless communication device (smartphone, tablet, laptop, etc.)

[0923] 2. Communication software (messaging and chat applications)

[0924] 3. Server

[0925] 4. Optical Character Recognition Technology (OCR engine, e.g. Tesseract OCR)

[0926] 5. Natural Language Processing Technology (NLP engine, e.g. GPT-3)

[0927] 6. Database

[0928] System Operation

[0929] Sending and receiving image information

[0930] A user uses a wireless communication terminal to take images of transaction-related documents such as receipts and invoices, which are then sent to a server via communication software.

[0931] The server receives the image data sent by the user using the API of the communication software, and the received image data is stored in the server's storage.

[0932] Image information analysis and classification

[0933] The server uses optical character recognition technology (OCR engine) to extract text information from the received image data. For example, from a receipt, text information such as "October 1, 2023," "10,000 yen," and "transportation expenses" is extracted.

[0934] The extracted text information is analyzed within the server, automatically categorized by transaction type, and recorded in a database based on pre-set rules.

[0935] Generate tax return data

[0936] The server aggregates the recorded journal data and converts it into a reporting format. This format is provided in a format that complies with the laws and regulations of each country. The generated reporting data is notified to the user via communication software.

[0937] Natural language instructions and corrections

[0938] Users can use natural language on the communication software to check and modify income and expenditure data. For example, if a user issues a command such as "I want to modify my travel expenses for October," the server will use natural language processing technology to analyze the user's command and make the appropriate modifications.

[0939] Providing tax advice

[0940] The server analyzes the recorded data and generates optimal advice on tax and asset management using a tax accountant AI algorithm, which is then communicated to the user via communication software.

[0941] Data Storage and Financial Services

[0942] The server encrypts and securely stores all data, and also provides financial product suggestions and investment advice based on the user's income and expenditure data, thereby supporting the user's financial growth.

[0943] Specific examples

[0944] For example, say a user wants to organize this month's income and expenses. The user takes a photo of a receipt with their smartphone and sends it to the system via communication software. The server analyzes the received image information using an OCR engine and extracts the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The extracted data is automatically accounted for as "entertainment expenses." At the end of the month, the server generates data for tax returns and notifies the user. Furthermore, if the user instructs in natural language that they would like to "review entertainment expenses," the server understands the instruction and makes the appropriate corrections. Through the above process, users can efficiently and accurately manage their income and expenses and file their tax returns.

[0945] Prompt Sentence Examples

[0946] 1. "I have sent you a scanned image of the receipt. Please let me know the extracted text data."

[0947] 2. "Please automatically post this month's travel expenses and display the related data."

[0948] 3. "Please generate the income and expenditure data for October in the tax return format and notify me."

[0949] 4. "How do I correct my income and expenditure data using natural language?"

[0950] 5. "Provide tax-saving advice."

[0951] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0952] Step 1: Send image information

[0953] The user uses the wireless communication terminal to capture image information related to the transaction.

[0954] Input: Captured image information (photos of receipts and invoices)

[0955] Operation: The user starts the communication software (messaging application) and logs in. Next, the user attaches the captured image information to the chat screen and presses the send button.

[0956] Output: Image information sent to the server via communication software

[0957] Step 2: Receiving image information

[0958] The server receives the image information sent by the user using the API of the communication software.

[0959] Input: Image information sent by the user

[0960] Operation: The server calls the API of the communication software and saves the received image information in storage.

[0961] Output: Image information saved in storage

[0962] Step 3: Analyze image information

[0963] The server uses optical character recognition technology (OCR engine) to extract text information from the received image information.

[0964] Input: Image information saved in storage

[0965] Operation: The server starts the OCR engine, analyzes the image data, and extracts text information. For example, from an image of a receipt, it obtains the text information "October 1, 2023," "10,000 yen," and "Transportation expenses."

[0966] Output: Extracted text information

[0967] Step 4: Journalize textual information

[0968] The server analyzes the extracted text information and automatically records and journals the transaction by type.

[0969] Input: Extracted text information

[0970] Operation: The server analyzes the text information and classifies it into the appropriate account based on the journal entry rules. For example, the information "October 1, 2023," "10,000 yen," and "Transportation expenses" is journalized as transportation expenses.

[0971] Output: Journalized data is recorded in the database

[0972] Step 5: Generate data for declaration

[0973] The server aggregates the recorded journal data and converts it into a format for reporting.

[0974] Input: Journal entry data recorded in the database

[0975] How it works: The server aggregates accounting data for a specific period and converts it into a reporting format that complies with the laws and regulations of each country. For example, it totals travel expenses, business expenses, and other expenses for one month.

[0976] Output: Data converted into a declaration format

[0977] Step 6: Natural Language Processing Remediation

[0978] Users can check and modify income and expenditure data in natural language through the communication software.

[0979] Input: Natural language instructions from the user (e.g., "I would like to correct the travel expenses for October.")

[0980] How it works: The server uses natural language processing (NLP) technology to parse the instructions and modify the specific data. For example, it extracts the entry for "October travel expenses" and modifies it to the specified value.

[0981] Output: Database updated with modified data

[0982] Step 7: Providing tax advice

[0983] The server analyzes the recorded data and generates optimal advice on tax and asset management.

[0984] Input: All income and expenditure data recorded in the database

[0985] How it works: The server uses the tax accountant AI algorithm to analyze the data and generate optimal advice, such as proposals for tax savings and asset management.

[0986] Output: The generated tax advice is communicated to the user via the communication software.

[0987] Step 8: Data Storage and Financial Services

[0988] The server encrypts and securely stores all data and provides financial product recommendations and investment advice as needed.

[0989] Input: All transaction data, journal data, user income and expenditure data

[0990] How it works: The server encrypts your transaction data and stores it in a secure database. It also analyzes your income and expenditure data and generates financial product and investment recommendations tailored to you.

[0991] Output: Encrypted and securely stored data, as well as financial product suggestions and investment advice notified to the user.

[0992] (Application example 1)

[0993] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0994] Previously, users had to manually collect image data related to transactions, sort them manually, and prepare tax returns, which required a great deal of effort and was prone to errors. Furthermore, there was a lack of efficient ways to manage income and expenditures or receive tax advice, placing a heavy burden on users. Furthermore, there was no system in place for managing daily expenses in real time and receiving appropriate financial advice. To solve these issues, a new method was needed.

[0995] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0996] In this invention, the server includes: means for a user to use a mobile communication device to send image data related to transactions via a messaging application; means for receiving the sent image data and extracting text information from the image using optical character recognition technology; means for analyzing the extracted text information and automatically journalizing and recording transactions by type; means for converting the recorded journal data into a format for tax returns; means for understanding and correcting voice or text instructions from the user using natural language processing technology; means for analyzing the recorded data and generating advice on tax and asset management; means for a user to send image data via an application on the mobile communication device and compile data related to financial transactions; means for providing up-to-date financial information based on the compiled data; and means for using a generative AI model to generate prompts related to income and expenditure management according to the user's requests. This allows users to efficiently manage transaction-related data and receive appropriate tax advice. It also allows users to manage their daily expenses in real time and receive optimal financial advice.

[0997] A "portable communication terminal" refers to an electronic device that is portable and capable of internet communication and data transmission and reception.

[0998] "Image data" refers to visual information obtained by optical means and expressed in digital form.

[0999] "Messaging application" refers to software that enables users to send and receive messages, including text, images, and audio.

[1000] "Optical character recognition technology" refers to the technology that extracts character information from image data and converts it into text data.

[1001] "Text information" refers to information expressed as a string of characters.

[1002] "Means for automatically making and recording journal entries" refers to the process of classifying text information extracted by the system by transaction type and saving the results in a database or the like.

[1003] A "tax return format" refers to income and expenditure data organized in the format required for submission to tax authorities.

[1004] "Natural language processing technology" refers to the technology that enables computers to understand and analyze text and speech written in natural language.

[1005] "Financial transaction related data" refers to economic information related to payments and transactions.

[1006] "Up-to-date financial information" means the most recent economic data, including recent transactions and expenditures.

[1007] A "generative AI model" refers to a model of artificial intelligence that uses machine learning algorithms to generate appropriate outputs for a specific task.

[1008] "Prompt" refers to the text content generated in response to a user instruction or question.

[1009] This invention is a system that improves the efficiency of income and expenditure management and tax return filing by allowing users to send image data related to transactions via a message application using a portable communication terminal. This system operates with the following configuration and program.

[1010] 1. User sends image data

[1011] A user takes a photo of a receipt or invoice using their smartphone. This photo is sent to the system via a messaging application (e.g., a general messaging application). The user logs in to the application and sends the photo of the receipt or invoice to the chat.

[1012] 2. Receiving and analyzing image data

[1013] The server receives image data sent by the user using a messaging API. It then uses AI-OCR (optical character recognition) technology to extract text information from the image. For example, it can use the Google Cloud Vision API. The server analyzes the image data and obtains information such as the date, time, amount, and subject.

[1014] 3. Automatic journalization of extracted text

[1015] The extracted text information is analyzed by the server and automatically journalized by transaction type. During this process, transactions are classified into appropriate account items according to pre-set journalization rules. For example, they may be classified into categories such as "transportation expenses" and "entertainment expenses."

[1016] 4. Generate tax return data

[1017] The automatically posted data is aggregated on the server and converted into a format for tax returns as monthly or annual income and expenditure data. This format is provided in a format that complies with the tax laws of each country. The data for tax returns is then notified to the user.

[1018] 5. Natural language instructions and corrections

[1019] Users can use natural language on the messaging application to check and modify their income and expenditure data. For example, if they issue a command such as "I want to modify my travel expenses for October," the server will activate its natural language processing engine, understand the user's command, and make the appropriate modifications. It uses OpenAI GPT-4 and other technologies to analyze the command and make the necessary modifications.

[1020] 6. Generating tax advice

[1021] The server generates tax and asset management advice based on the collected income and expenditure data, allowing users to receive appropriate advice on tax savings and asset management. Using a generative AI model, advice is provided that is tailored to the user's specific situation.

[1022] 7. Providing real-time financial information

[1023] A user sends image data via an application on a portable communication terminal, and data related to financial transactions is compiled. The system provides the latest financial information based on the compiled data. For example, in response to a request such as "What is the total amount of expenses this month?", the system displays the latest financial information in real time.

[1024] Specific examples

[1025] For example, say a self-employed person wants to organize their income and expenses at the end of the month. The user takes a photo of this month's receipts with their smartphone and sends it to the system from the application. The server receives the image data and uses AI-OCR to extract the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server automatically journalizes this data and records it as entertainment expenses. Then, at the end of the month, data for tax returns is automatically generated and notified to the user.

[1026] When a user sends a message saying, "I'd like to review my entertainment expenses," the server understands the instruction and reconfirms and corrects it. Furthermore, a generative AI model analyzes the user's data and provides tax-saving advice. Ultimately, the system allows users to easily and efficiently manage their income and expenses and file tax returns.

[1027] Prompt Sentence Examples

[1028] "I would like to revise my travel expenses for October."

[1029] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1030] Step 1:

[1031] A user uses a portable communication terminal to take a photo of a receipt or invoice related to a transaction and transmits the image data to a server via a message application. The input is the image data taken by the user, and the output is the image data transmitted to the server via the message application.

[1032] Step 2:

[1033] The server receives image data sent by the user using a messaging API. The input is the image data sent by the user, and the output is the image data received by the server.

[1034] Step 3:

[1035] The server applies AI-OCR (optical character recognition) technology to the received image data and extracts the text information within the image. The input is the received image data, and the output is text information. Specifically, the image is analyzed using Google Cloud Vision API, etc., and text information such as the date, time, amount, and transaction items is extracted.

[1036] Step 4:

[1037] The server analyzes the extracted text information and automatically journalizes and records each type of transaction. The input is the extracted text information, and the output is automatically journalized data. Specifically, the data is classified into categories such as "transportation expenses" and "entertainment expenses" and recorded in the appropriate account.

[1038] Step 5:

[1039] The server aggregates the recorded journal data and converts it into a format for tax returns as monthly or annual income and expenditure data. The input is automatically journalized data, and the output is data converted into a format for tax returns.

[1040] Step 6:

[1041] The user uses a messaging application to input correction instructions in natural language. For example, they might send a message saying, "I would like to correct the travel expenses for October." The input is a natural language instruction from the user, and the output is the target data to be corrected based on that instruction.

[1042] Step 7:

[1043] The server uses natural language processing technology to analyze user instructions and make appropriate data corrections. The input is the user's natural language instructions, and the output is the corrected income and expenditure data. Specifically, OpenAI GPT-4 and other technologies are used to interpret the user's intent and update the relevant parts of the database.

[1044] Step 8:

[1045] The server generates advice on tax and asset management based on the collected income and expenditure data. The input is the recorded income and expenditure data, and the output is the generated advice. Using a generative AI model, it provides specific advice tailored to the user's situation.

[1046] Step 9:

[1047] The server aggregates data related to image data transmissions and financial transactions performed by users via applications on mobile communication terminals in real time and provides the latest financial information. The input is the transmitted image data and transaction data, and the output is the updated latest financial information. The data is processed in real time, and the most recent financial information is displayed when the user requests it.

[1048] Examples of concrete examples and prompts

[1049] For example, say a self-employed person wants to organize their income and expenses at the end of the month. The user takes a photo of this month's receipts with their smartphone and sends it to the system from the application. The server receives the image data and uses AI-OCR to extract the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server automatically journalizes this data and records it as entertainment expenses. Then, at the end of the month, data for tax returns is automatically generated and notified to the user.

[1050] Prompt Sentence Examples

[1051] "I would like to revise my travel expenses for October."

[1052] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1053] The present invention combines an emotion engine with a system that allows users to efficiently manage income and expenditures and file tax returns by sending image data related to transactions via a message application using a portable communication terminal. This system operates in conjunction with the following programs.

[1054] 1. User sends image data

[1055] Users use their smartphones to take photos of receipts or invoices, which are then sent to the system via a messaging application (e.g., LINE).

[1056] The user logs in with LINE and sends a photo of a receipt or invoice to the chat.

[1057] 2. Receiving and analyzing image data

[1058] The server receives the image data sent by the user using the LINE API, and then extracts the text information from the image using optical character recognition technology (AI-OCR).

[1059] The server receives the image data, runs the AI-OCR engine, and extracts the text.

[1060] For example, text information such as "October 1, 2023," "10,000 yen," and "transportation expenses" is extracted from a receipt.

[1061] 3. Automatic journalization of extracted text

[1062] The extracted text information is analyzed by the server and automatically journalized by type of income and expenditure, with transactions classified into appropriate account items according to pre-set journalization rules.

[1063] Based on the analysis results, the server automatically records the expenses in categories such as "transportation expenses."

[1064] For example, "10,000 yen" is classified as "transportation expenses."

[1065] 4. Generate tax return data

[1066] The automatically posted data is aggregated on the server and reflected in monthly and annual income and expenditure reports. This data is converted into a tax return format and prepared for easy access by users later.

[1067] The server aggregates the journal data by period and converts it into tax return format.

[1068] The data for tax returns will be sent to the user via LINE.

[1069] 5. Natural language instructions and corrections

[1070] Users can use natural language on LINE to check and edit their income and expenditure data. For example, if they say, "I want to edit my travel expenses for October," the server will activate its natural language processing engine, understand the user's instructions, and make the appropriate edits.

[1071] The user sends instructions in natural language, which the server analyzes using an NLP engine.

[1072] The server reviews the data and makes any necessary corrections.

[1073] 6. Collaboration with tax accountant AI

[1074] The server uses the collected income and expenditure data to generate optimal tax advice using a tax accountant AI algorithm, allowing users to receive appropriate advice on tax savings and asset management.

[1075] The server sends the data to the tax accountant AI, which generates optimal advice.

[1076] The advice will be sent to the user via LINE.

[1077] 7. Data Storage and Financial Services

[1078] The server encrypts and securely stores all collected data, and provides financial product recommendations and investment advice as needed, thus supporting users' financial growth.

[1079] The server encrypts and stores the data and suggests financial products.

[1080] Users receive financial advice and product details via LINE.

[1081] 8. Introducing and utilizing an emotion engine

[1082] The server incorporates a new emotion engine that analyzes emotions based on the user's voice or text data and optimizes response content. This engine recognizes the user's emotions in real time and can respond appropriately.

[1083] The user sends a voice message such as "Today is stressful and difficult."

[1084] The server receives the voice data and uses an emotion engine to analyze the user's emotional state.

[1085] The server provides appropriate feedback via LINE based on the user's emotions (e.g., "You seem stressed. Let me know if there's anything I can do to help you").

[1086] Specific examples

[1087] For example, let's say that self-employed Tanaka wants to organize his income and expenses at the end of the month. Tanaka takes a photo of this month's receipts with his smartphone and sends it to the system via LINE. The server receives the image data and uses AI-OCR to extract the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server automatically accounts for this data and records it as entertainment expenses. Then, at the end of the month, data for tax returns is automatically generated and notified to Tanaka via LINE. When Tanaka sends a message via LINE saying, "I would like to review my entertainment expenses," the server understands the instruction and reconfirms and corrects it. Furthermore, the tax accountant AI analyzes Tanaka's data and provides tax-saving advice.

[1088] One day, Tanaka sends a voice message to LINE saying, "I'm feeling very stressed today," and the server's emotion engine analyzes Tanaka's emotions and provides appropriate feedback, such as, "It seems you're feeling stressed. Let me know if there's anything I can help you with." This allows Tanaka to not only efficiently manage his income and expenses and file his tax return through the system, but also receive emotional support.

[1089] The processing flow will be explained below.

[1090] Step 1:

[1091] The user takes a photo of a receipt or invoice using a mobile communication device (smartphone). The user opens the LINE app and sends the photo to the LINE chat. This sends the image data to the system.

[1092] Step 2:

[1093] The server uses the LINE API to receive the image data sent by the user. The received image data is saved in its original format and passed to the next processing step.

[1094] Step 3:

[1095] The server activates optical character recognition (AI-OCR) technology to extract text information from the received image data. Specifically, the image processing engine identifies letters and numbers in the image and converts them into text. For example, information such as "October 1, 2023," "10,000 yen," and "transportation expenses" is extracted.

[1096] Step 4:

[1097] The extracted text information is analyzed by the server. The server identifies the type of transaction based on the text information and automatically journalizes it. Specifically, it classifies the transaction into the appropriate account category based on information such as the "date," "amount," and "item name." For example, "10,000 yen" is journalized as "transportation expenses."

[1098] Step 5:

[1099] The server aggregates the extracted and journalized data and generates monthly or annual income and expenditure reports, which are then converted into tax return formats and prepared for easy user access later.

[1100] Step 6:

[1101] The server notifies the user of the generated tax return data via LINE chat. The user can then check the data in the LINE app and request corrections if necessary.

[1102] Step 7:

[1103] The user sends a natural language instruction on LINE, such as "I would like to revise the transportation expenses for October." This instruction is sent to the server and analyzed by the natural language processing engine.

[1104] Step 8:

[1105] Based on the analysis results, the server understands the user's instructions and corrects the relevant journal data. For example, it rechecks "October travel expenses" and makes any necessary corrections.

[1106] Step 9:

[1107] The server provides all collected income and expenditure data to the AI ​​tax accountant, who then analyzes the data and provides optimal tax advice on tax-saving strategies and asset formation.

[1108] Step 10:

[1109] The server notifies the user of the generated advice via LINE chat, and the user can receive the advice in the LINE app and take the necessary action.

[1110] Step 11:

[1111] The server uses an emotion engine to analyze the user's voice or text data in real time. For example, if a user says, "I'm feeling stressed today," the server recognizes the voice data and analyzes the user's emotional state.

[1112] Step 12:

[1113] Based on the analysis results of the emotion engine, the server provides appropriate feedback according to the user's emotions. For example, it may send a message via LINE saying, "You seem to be feeling stressed. Please let us know if there is anything we can do to help."

[1114] Step 13:

[1115] The server encrypts and securely stores all financial data, and the server uses this data to provide more advanced financial services and investment advice.

[1116] Step 14:

[1117] Users receive financial advice and product recommendations on LINE, which helps them make more effective financial decisions.

[1118] Specific examples

[1119] For example, let's say that self-employed Tanaka wants to organize his income and expenses at the end of the month. Tanaka takes a photo of this month's receipts with his smartphone and sends it to the system via LINE. The server receives the image data and uses AI-OCR to extract the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server automatically accounts for this data and records it as entertainment expenses. Then, at the end of the month, data for tax returns is automatically generated and notified to Tanaka via LINE. When Tanaka sends a message via LINE saying, "I would like to review my entertainment expenses," the server understands the instruction and reconfirms and corrects it. Furthermore, the tax accountant AI analyzes Tanaka's data and provides tax-saving advice.

[1120] One day, Tanaka sends a voice message to LINE saying, "I'm feeling very stressed today," and the server's emotion engine analyzes Tanaka's emotions and provides appropriate feedback, such as, "It seems you're feeling stressed. Let me know if there's anything I can help you with." This allows Tanaka to not only efficiently manage his income and expenses and file his tax return through the system, but also receive emotional support.

[1121] Example 2

[1122] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1123] Traditional income and expenditure management and tax return systems required time and effort for manual data entry, resulting in inefficiency. They also often failed to accurately record collected data or provide adequate tax advice. Furthermore, they lacked support that took into account the user's emotional and stress levels, resulting in a lack of improvement in the overall user experience.

[1124] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for a user to use a portable communication terminal to send image data related to a transaction via a message application; means for receiving the sent image data and extracting text information from the image using optical character recognition technology; means for analyzing the extracted text information and automatically journalizing and recording the transaction type; means for converting the recorded journal data into a format for tax returns; means for understanding and correcting voice or text instructions from the user using natural language processing technology; means for analyzing the recorded data and generating advice on taxes and asset management; and means for recognizing emotions based on the user's voice or text data using emotion analysis technology and optimizing the response. This not only improves the efficiency of income and expenditure management and tax return filing, but also enables improved accuracy of tax advice and emotional support for users.

[1125] "User" refers to any individual or legal entity that uses the System.

[1126] A "portable communication terminal" refers to a terminal device such as a smartphone or tablet that is portable and capable of wireless communication.

[1127] "Transaction" refers to the act of exchanging money or goods in economic activities.

[1128] "Image data" refers to visual information represented in digital form.

[1129] "Message application" refers to an application that sends and receives text messages and image data over the Internet.

[1130] "Optical character recognition technology" refers to technology that extracts characters from an image as digital data.

[1131] "Text information" refers to data expressed as a string of characters.

[1132] "Journal entry" refers to the accounting process of classifying and recording transactions into account items.

[1133] "Server" refers to a device or system that provides computer resources over a network.

[1134] "Tax return format" refers to the format of the income and expenditure report to be submitted to the tax office.

[1135] "Natural language processing technology" refers to technology for understanding human language and responding appropriately.

[1136] "Voice or text instructions" refers to operation instructions given to the system by the user using words or letters.

[1137] "Tax advice" refers to the act or content of providing appropriate advice regarding tax.

[1138] "Asset management" refers to the efficient management and administration of personal or corporate assets.

[1139] "Emotion analysis technology" refers to the technology of analyzing human emotions and psychological states from voice and text.

[1140] "Optimizing response content" refers to optimizing the responses and support provided by the system according to the user's needs and condition.

[1141] This invention is a system that allows users to efficiently manage income and expenditures and file tax returns using a mobile communication device. This system mainly combines a messaging application, optical character recognition technology (AI-OCR), natural language processing technology (NLP), tax accountant AI algorithms, and sentiment analysis technology. Specific examples of the use of various hardware and software are shown below.

[1142] A user uses a mobile communication device (e.g., a smartphone or tablet) to take a photo of a receipt or invoice related to a transaction. At that time, the user uses a messaging application (e.g., LINE) to send the image data to the system. The user logs in to LINE and sends a photo of the receipt or invoice as a chat message.

[1143] The server receives image data sent by the user using LINE's API. The server then uses optical character recognition technology (e.g., Google Cloud Vision API) to extract text information from the image. This AI-OCR engine identifies characters in the image and extracts them as numerical or text data. For example, from a receipt, the following information is extracted: "October 1, 2023," "10,000 yen," and "Transportation expenses."

[1144] The extracted text information is analyzed by the server and automatically journalized according to pre-set journalization rules. Based on the analysis results, the server classifies the data into appropriate account items for each type of income and expenditure. For example, "10,000 yen" is journalized as "transportation expenses."

[1145] The server converts the collected data into a format for tax returns. This data is reflected in monthly and annual income and expenditure reports and prepared as tax return data. The server then aggregates this data and converts it into a tax return format (e.g., XML for tax software). Users are notified via LINE for easy access to their tax return data.

[1146] Users can also send voice or text instructions on LINE using natural language processing technology. For example, if you send an instruction such as "I would like to revise my travel expenses for October," the server will use an NLP engine (e.g., OpenAI GPT-3) to analyze the message, understand the user's instruction, and make the appropriate revisions.

[1147] Based on the collected data, the server uses a tax accountant AI algorithm (e.g., Watson Tax Advisor) to generate optimal tax advice. Users can receive advice on tax savings and asset management via LINE. The server also encrypts and securely stores revenue data and provides users with financial products and investment advice.

[1148] Emotion analysis technology (e.g., IBM Watson Tone Analyzer) recognizes emotions from a user's voice or text data and provides feedback in real time. For example, if a user sends a voice message saying, "I'm feeling very stressed today," the server analyzes the emotion and replies, "It sounds like you're feeling stressed. Let me know if there's anything I can do to help."

[1149] Specific examples

[1150] A self-employed individual wants to organize their income and expenses at the end of the month. They take a photo of this month's receipts with their smartphone and send it to the system via LINE. The server receives the image data and uses AI-OCR to extract the following information: "October 15, 2023," "5,000 yen," and "Entertainment Expenses." The server automatically accounts for this data and records it as entertainment expenses. At the end of the month, tax return data is automatically generated and notified via LINE. When the self-employed individual sends a message via LINE saying, "I'd like to review my entertainment expenses," the server understands the instruction and reconfirms and corrects it. Furthermore, a tax accountant AI analyzes the data and provides tax-saving advice. One day, the self-employed individual sends a voice message via LINE saying, "I'm feeling stressed today and it's bothering me," and the server's emotion engine analyzes the emotion and provides appropriate feedback, such as, "It sounds like you're stressed. Let me know if there's anything I can do to help."

[1151] Prompt Sentence Examples

[1152] "Please send a photo of your receipt via LINE to record your income and expenses."

[1153] "I would like to revise my travel expenses for October 2023."

[1154] "Can you give me some tax saving advice?"

[1155] "I'm having a lot of stress today"

[1156] In this way, the system helps users manage their finances and file their tax returns efficiently, and also provides emotional support.

[1157] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1158] Step 1: Sending image data

[1159] The user takes a photo of an image related to the transaction (such as a receipt or invoice) using a mobile communication terminal (e.g., a smartphone) and sends it to the system via a messaging application (e.g., LINE).

[1160] Input: Image data of receipts and invoices

[1161] Output: Image data sent to LINE

[1162] Specific operation: The user launches the camera app and takes a picture of a receipt or invoice. The user selects the image in the LINE app and sends it as a chat message.

[1163] Step 2: Receiving image data

[1164] The server uses the LINE API to receive the image data sent by the user.

[1165] Input: Image data sent via the LINE API

[1166] Output: Image data stored on the server

[1167] Specific operation: The server periodically checks the LINE API to receive new image data, which is then stored in the database.

[1168] Step 3: Extracting text information

[1169] The server uses optical character recognition technology (AI-OCR) to extract text information from the received image data.

[1170] Input: Saved image data

[1171] Output: Extracted text information

[1172] How it works: The server launches an AI-OCR engine such as Google Cloud Vision API, identifies the characters in the image, and generates text data. For example, the following information is extracted: "October 1, 2023," "10,000 yen," and "transportation expenses."

[1173] Step 4: Parsing and journalizing extracted text

[1174] The server analyzes the extracted text information and automatically records and journals the transactions by type.

[1175] Input: Extracted text information

[1176] Output: Journal entry data

[1177] Specific operation: The server analyzes the text information based on pre-defined rules and classifies it into categories such as "transportation expenses" and "entertainment expenses." For example, "10,000 yen" is journalized as "transportation expenses." The journalized data is then saved in a database.

[1178] Step 5: Generate tax return data

[1179] The server converts the data into a format for tax returns based on the accounting data.

[1180] Input: Journal data

[1181] Output: Data in tax return format

[1182] Specific operation: The server aggregates monthly and annual income and expenditure reports and converts them into a format for tax returns (e.g., XML for tax software). The user is notified via LINE that "tax return data is ready."

[1183] Step 6: Correcting data using natural language

[1184] Users can check and edit income and expenditure data using natural language on LINE.

[1185] Input: Natural language instructions from the user

[1186] Output: Corrected balance data

[1187] Specific operation: The user sends a message on LINE saying, "I would like to correct my travel expenses for October." The server uses an NLP engine (e.g., OpenAI GPT-3) to analyze the message and understand the instructions. The server then reviews the target data and makes any necessary corrections.

[1188] Step 7: Advice generation by tax accountant AI

[1189] The server sends the collected data to the tax accountant AI, which generates optimal tax advice.

[1190] Input: Collected income and expenditure data

[1191] Output: Tax advice

[1192] Specific operation: The server sends the collected data to a tax accountant AI (e.g., Watson Tax Advisor). The tax accountant AI analyzes the data and generates appropriate tax advice. The advice is then sent to the user via LINE.

[1193] Step 8: Optimize your response with sentiment analysis

[1194] The server uses emotion analysis technology to recognize the user's emotions and optimize the response.

[1195] Input: Voice or text data from the user

[1196] Output: Emotion-based feedback

[1197] Specific operation: The user sends a voice message stating, "I'm feeling very stressed today." The server receives the voice data and analyzes the user's emotions using an emotion analysis engine (e.g., IBM Watson Tone Analyzer). Appropriate feedback based on the user's emotions is then provided via LINE (e.g., "It sounds like you're feeling stressed. Please let me know if there's anything I can do to help you").

[1198] This processing flow allows the system to streamline users' income and expenditure management and tax return filing, and also provides emotional support.

[1199] (Application example 2)

[1200] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1201] Managing transaction-related income and expenditures and filing tax returns are time-consuming and labor-intensive tasks, especially for small businesses and sole proprietors. Traditional manual data entry and accounting processes are prone to errors, and users without specialized knowledge of tax or asset management find it difficult to obtain appropriate support. Furthermore, few systems take into account the user's emotions and mental state, which leads to a poor user experience. Therefore, the challenge is to provide a system that streamlines income and expenditure management and tax return filing and provides emotionally sensitive feedback.

[1202] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to use a portable communication terminal to send image data related to a transaction via a message application; means for receiving the sent image data and extracting text information from the image using optical character recognition technology; means for analyzing the extracted text information and automatically journalizing and recording the transaction type; means for converting the recorded journal data into a format for tax returns; means for understanding and correcting voice or text instructions from the user using natural language processing technology; means for analyzing the recorded data and generating advice on taxes and asset management; and means for analyzing the user's emotions and providing feedback according to the user's emotions using an emotion engine that optimizes response content. This improves the efficiency of income and expenditure management and tax return filing and also enables support that takes into account the user's emotions and mental state.

[1203] A "portable communication terminal" is a device that is portable and has communication capabilities, and mainly refers to smartphones and tablets.

[1204] "Transaction-related image data" refers to image data, such as receipts and invoices, that contains information about financial transactions.

[1205] A "messaging application" is application software that allows users to send and receive messages, such as text and images, via the Internet, and includes LINE and WhatsApp.

[1206] "Optical character recognition technology" is a technology that extracts character information from image data, and is also known as OCR (Optical Character Recognition).

[1207] "Text information" refers to information such as letters and numbers, and includes character data extracted from an image.

[1208] "Journal entries" are the process of classifying transactions into specific account items and recording them in the ledger.

[1209] "Tax format" means the conversion of collected financial data into a specific format for use in tax returns.

[1210] "Natural language processing technology" is a technology that allows computers to understand, analyze, and respond to human language, and is also known as NLP (Natural Language Processing).

[1211] An "emotion engine" is a technology that analyzes user data such as voice and text and estimates their emotional state.

[1212] "Feedback" refers to the response or advice that a system gives to a user in response to their input.

[1213] "Tax and asset management advice" means providing expertise based on the user's financial data and offering advice on tax optimization and asset management.

[1214] This invention is a system that combines an emotion engine with a system that allows users to use a portable communication terminal to send image data related to transactions via a message application, thereby streamlining income and expenditure management and tax return filing.

[1215] The server receives image data related to transactions sent by users via messaging applications such as LINE and WhatsApp. It extracts text information from the received image data using optical character recognition (OCR). The extracted text information is stored in the server's database, analyzed by an automatic accounting program, and automatically recorded and accounted for by transaction type.

[1216] Furthermore, the recorded journal data is automatically converted into a format for tax returns and provided to users in a format that can be used for tax returns. This also implements a means of generating advice on taxes and asset management, and the tax accountant AI provides optimal advice based on the data.

[1217] Users can send voice or text instructions to the server through an interface that uses natural language processing technology. For example, they can send a message to a friend saying, "I want to revise my travel expenses for October." The server analyzes this instruction, reviews the relevant data, and makes any necessary adjustments. Furthermore, if a user sends a message expressing their emotions, the server's emotion engine analyzes it and provides feedback appropriate to the user's emotional state.

[1218] The hardware used includes a server and a mobile communication device, and the software includes the LINE Messaging API, an OCR engine (such as Tesseract OCR), MySQL, a natural language processing engine (such as spaCy), and a sentiment analysis model (such as the Transformers sentiment analysis pipeline).

[1219] For example, a user can take a photo of a receipt with their smartphone and send it to the system via a messaging application. The image data is transferred to the server, where OCR extracts the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server records this data as entertainment expenses and automatically generates data for tax returns. If the user then issues a natural language instruction such as "I'd like to correct the entertainment expenses for October," the system will correct the relevant data. Furthermore, if the user voice-transmits "I'm feeling very stressed today," the emotion engine will analyze the emotion and provide feedback such as "It sounds like you're stressed. Please let me know if there's anything I can do to help."

[1220] An example prompt is:

[1221] "Go to My Page and check your credit card transaction details from last month. Then, send an image of the receipt to LINE and correct the relevant transaction. Also, if you would like to receive advice on asset management, please let the system know."

[1222] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1223] Step 1:

[1224] A user uses a mobile communication terminal to send image data related to a transaction (e.g., a photo of a receipt or invoice) to the system via a message application. The input of this step is the image data of the receipt or invoice, and the output is that it reaches the server via the message application. Specifically, the user takes a photo of the receipt using the camera function of their smartphone and sends the image via the chat function of the message application.

[1225] Step 2:

[1226] The server receives image data sent by the user using the API of the messaging application. The input of this step is the image data sent through the messaging application, and the output is that it is saved on the server. Specifically, the server uses the API of the messaging application to detect new messages, receive the image file, and save it in the database.

[1227] Step 3:

[1228] The server extracts text information from the received image data using optical character recognition (OCR). The input for this step is the image data, and the output is the text information extracted from the image. Specifically, the server uses an OCR engine (e.g., Tesseract OCR) to extract the characters in the image in text format.

[1229] Step 4:

[1230] The server analyzes the extracted text information and automatically journalizes and records it for each type of transaction. The input to this step is the text information extracted by OCR, and the output is the analyzed transaction information being journalized and recorded in a database. Specifically, the server uses an analysis program to analyze the text information, classify it into the appropriate account category, and save it in the database.

[1231] Step 5:

[1232] The server converts the recorded journal data into a format for tax returns. The input for this step is the journal data, and the output is the data converted into a format for tax returns. Specifically, the server aggregates the journal data and converts it into a tax return format that complies with the tax laws of each country.

[1233] Step 6:

[1234] The user sends voice or text instructions to the server using natural language processing technology. The input for this step is the user's voice or text instruction, and the output is that the instruction is understood and executed. Specifically, the user enters "I would like to revise the travel expenses for October," and the instruction is sent to the server.

[1235] Step 7:

[1236] The server analyzes the user's instructions using a natural language processing engine and makes any necessary corrections. The input to this step is the natural language instruction from the user, and the output is the execution of data correction processing. Specifically, the server uses a natural language processing engine (e.g., spaCy) to analyze the user's instructions and correct the target data.

[1237] Step 8:

[1238] The server analyzes the recorded data and generates advice on tax and asset management. The input for this step is the recorded income and expenditure data, and the output is advice on tax and asset management. Specifically, the server uses the tax accountant AI algorithm to analyze the data and generate optimal advice.

[1239] Step 9:

[1240] The server analyzes the user's emotions and provides feedback based on the user's emotions using an emotion engine that optimizes the response content. The input for this step is the user's emotional expression in voice or text, and the output is a feedback message based on the emotion analysis results. Specifically, the server uses an emotion engine (e.g., a Transformers emotion analysis model) to analyze the user's emotional state and generate appropriate feedback.

[1241] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1242] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1243] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1244] [Fourth embodiment]

[1245] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1246] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1247] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1248] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1249] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1250] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1251] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1252] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1253] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1254] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1255] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1256] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1257] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1258] The present invention provides a system that allows users to efficiently manage income and expenditures and file tax returns by sending image data related to transactions via a message application using a portable communication terminal. This system operates in conjunction with the following programs.

[1259] 1. User sends image data

[1260] Users use their smartphones to take photos of receipts or invoices, which are then sent to the system via a messaging application (e.g., LINE).

[1261] The user logs in with LINE and sends a photo of a receipt or invoice to the chat.

[1262] 2. Receiving and analyzing image data

[1263] The server receives the image data sent by the user using the LINE API, and then extracts the text information from the image using optical character recognition technology (AI-OCR).

[1264] The server receives the image data, runs the AI-OCR engine, and extracts the text.

[1265] For example, text information such as "October 1, 2023," "10,000 yen," and "transportation expenses" is extracted from a receipt.

[1266] 3. Automatic journalization of extracted text

[1267] The extracted text information is analyzed by the server and automatically journalized by type of income and expenditure, with transactions classified into appropriate account items according to pre-set journalization rules.

[1268] Based on the analysis results, the server automatically records the expenses in categories such as "transportation expenses."

[1269] For example, "10,000 yen" is classified as "transportation expenses."

[1270] 4. Generate tax return data

[1271] The automatically posted data is aggregated on a server and converted into monthly and annual income and expenditure data in a format suitable for tax returns. This format is provided in accordance with the tax laws of each country.

[1272] The server aggregates the journal data by period and converts it into tax return format.

[1273] The data for tax returns will be sent to the user via LINE.

[1274] 5. Natural language instructions and corrections

[1275] Users can use natural language on LINE to check and edit their income and expenditure data. For example, if they say, "I want to edit my travel expenses for October," the server will activate its natural language processing engine, understand the user's instructions, and make the appropriate edits.

[1276] The user sends instructions in natural language, which the server analyzes using an NLP engine.

[1277] The server reviews the data and makes any necessary corrections.

[1278] 6. Collaboration with tax accountant AI

[1279] The server uses the collected income and expenditure data to generate optimal tax advice using a tax accountant AI algorithm, allowing users to receive appropriate advice on tax savings and asset management.

[1280] The server sends the data to the tax accountant AI, which generates optimal advice.

[1281] The advice will be sent to the user via LINE.

[1282] 7. Data Storage and Financial Services

[1283] The server encrypts and securely stores all collected data, and provides financial product recommendations and investment advice as needed, thus supporting users' financial growth.

[1284] The server encrypts and stores the data and suggests financial products.

[1285] Users receive financial advice and product details via LINE.

[1286] Specific examples

[1287] For example, let's say that self-employed business owner Sato wants to organize his income and expenses at the end of the month. Sato takes a photo of this month's receipts with his smartphone and sends it to the system via LINE. The server receives the image data and uses AI-OCR to extract the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server automatically accounts for this data and records it as entertainment expenses. Then, at the end of the month, data for tax returns is automatically generated and notified to Sato via LINE. When Sato sends a message via LINE saying, "I would like to review my entertainment expenses," the server understands the instruction and reconfirms and corrects it. Furthermore, a tax accountant AI analyzes Sato's data and provides tax-saving advice. Ultimately, Sato can manage his income and expenses and file his tax returns simply and efficiently through this system.

[1288] The processing flow will be explained below.

[1289] Step 1:

[1290] The user takes a photo of a receipt or invoice using a mobile communication device (smartphone). The user opens the LINE app and sends the photo to the LINE chat. This sends the image data to the system.

[1291] Step 2:

[1292] The server uses the LINE API to receive the image data sent by the user. The received image data is saved in its original format and passed to the next processing step.

[1293] Step 3:

[1294] The server activates optical character recognition (AI-OCR) technology to extract text information from the received image data. Specifically, the image processing engine identifies letters and numbers in the image and converts them into text. For example, information such as "October 1, 2023," "10,000 yen," and "transportation expenses" is extracted.

[1295] Step 4:

[1296] The extracted text information is analyzed by the server. The server identifies the type of transaction based on the text information and automatically journalizes it. Specifically, it classifies the transaction into the appropriate account category based on information such as the "date," "amount," and "item name." For example, "10,000 yen" is journalized as "transportation expenses."

[1297] Step 5:

[1298] The server aggregates the extracted and journalized data and generates monthly or annual income and expenditure reports, which are then converted into tax return formats and prepared for easy user access later.

[1299] Step 6:

[1300] The server notifies the user of the generated tax return data via LINE chat. The user can then check the data in the LINE app and request corrections if necessary.

[1301] Step 7:

[1302] The user sends a natural language instruction on LINE, such as "I would like to revise the transportation expenses for October." This instruction is sent to the server and analyzed by the natural language processing engine.

[1303] Step 8:

[1304] Based on the analysis results, the server understands the user's instructions and corrects the relevant journal data. For example, it rechecks "October travel expenses" and makes any necessary corrections.

[1305] Step 9:

[1306] The server provides all collected income and expenditure data to the AI ​​tax accountant, who then analyzes the data and provides optimal tax advice on tax-saving strategies and asset formation.

[1307] Step 10:

[1308] The server notifies the user of the generated advice via LINE chat, and the user can receive the advice in the LINE app and take the necessary action.

[1309] Step 11:

[1310] The server encrypts and securely stores all financial data, and the server uses this data to provide more advanced financial services and investment advice.

[1311] Step 12:

[1312] Users receive financial advice and product recommendations on LINE, which helps them make more effective financial decisions.

[1313] Example 1

[1314] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1315] Currently, managing income and expenses and filing tax returns takes a lot of time and effort. In particular, the process of manually recording receipts and invoices, journalizing them, and preparing data for tax returns is cumbersome and prone to errors. It is also difficult to receive real-time corrections and tax advice using natural language. This results in users expending a huge amount of effort and hinders efficient income and expense management.

[1316] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1317] In this invention, the server includes means for a user to use a wireless communication terminal to send image information related to a transaction via communication software, means for receiving the sent image information and extracting text information from the image using optical character recognition technology, means for analyzing the extracted text information and automatically journalizing and recording the transaction by type, means for converting the recorded journal data into a reporting format, means for understanding and correcting verbal or text instructions from the user using natural language processing technology, and means for analyzing the recorded data and generating advice on tax and asset management. This allows users to significantly reduce their workload and enable efficient and accurate income and expenditure management and tax return filing.

[1318] A "wireless communication terminal" is an electronic device that allows communication while moving, and includes smartphones, tablets, laptops, and the like.

[1319] "Communication software" refers to a program for sending and receiving data, and includes message applications, chat applications, and the like.

[1320] "Image information" means visual data captured electronically, including photographs and scanned data such as receipts and invoices.

[1321] "Optical character recognition technology" is a technology that analyzes characters in an image and converts them into text data, and is known as OCR (Optical Character Recognition).

[1322] "Character information" is data that is identified as characters, and includes text files and character string information.

[1323] The "means for automatically making and recording entries" is a system for analyzing extracted text information and classifying and recording it according to pre-set criteria.

[1324] "Format for tax return" means a standardized data format used for tax returns, including formats conforming to the laws and regulations of each country.

[1325] "Natural language processing technology" is a technology that understands and analyzes human language, and is a system for interpreting instructions entered in language or text.

[1326] The "means for generating tax and asset management advice" is a system that analyzes recorded data and makes recommendations on optimal tax strategies and asset management.

[1327] This invention is a system that allows users to efficiently manage income and expenditures and file tax returns using a wireless communication terminal.It works by sending image information related to transactions via communication software, and a server processes and analyzes that information.

[1328] System Overview

[1329] The system includes the following major components:

[1330] 1. User's wireless communication device (smartphone, tablet, laptop, etc.)

[1331] 2. Communication software (messaging and chat applications)

[1332] 3. Server

[1333] 4. Optical Character Recognition Technology (OCR engine, e.g. Tesseract OCR)

[1334] 5. Natural Language Processing Technology (NLP engine, e.g. GPT-3)

[1335] 6. Database

[1336] System Operation

[1337] Sending and receiving image information

[1338] A user uses a wireless communication terminal to take images of transaction-related documents such as receipts and invoices, which are then sent to a server via communication software.

[1339] The server receives the image data sent by the user using the API of the communication software, and the received image data is stored in the server's storage.

[1340] Image information analysis and classification

[1341] The server uses optical character recognition technology (OCR engine) to extract text information from the received image data. For example, from a receipt, text information such as "October 1, 2023," "10,000 yen," and "transportation expenses" is extracted.

[1342] The extracted text information is analyzed within the server, automatically categorized by transaction type, and recorded in a database based on pre-set rules.

[1343] Generate tax return data

[1344] The server aggregates the recorded journal data and converts it into a reporting format. This format is provided in a format that complies with the laws and regulations of each country. The generated reporting data is notified to the user via communication software.

[1345] Natural language instructions and corrections

[1346] Users can use natural language on the communication software to check and modify income and expenditure data. For example, if a user issues a command such as "I want to modify my travel expenses for October," the server will use natural language processing technology to analyze the user's command and make the appropriate modifications.

[1347] Providing tax advice

[1348] The server analyzes the recorded data and generates optimal advice on tax and asset management using a tax accountant AI algorithm, which is then communicated to the user via communication software.

[1349] Data Storage and Financial Services

[1350] The server encrypts and securely stores all data, and also provides financial product suggestions and investment advice based on the user's income and expenditure data, thereby supporting the user's financial growth.

[1351] Specific examples

[1352] For example, say a user wants to organize this month's income and expenses. The user takes a photo of a receipt with their smartphone and sends it to the system via communication software. The server analyzes the received image information using an OCR engine and extracts the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The extracted data is automatically accounted for as "entertainment expenses." At the end of the month, the server generates data for tax returns and notifies the user. Furthermore, if the user instructs in natural language that they would like to "review entertainment expenses," the server understands the instruction and makes the appropriate corrections. Through the above process, users can efficiently and accurately manage their income and expenses and file their tax returns.

[1353] Prompt Sentence Examples

[1354] 1. "I have sent you a scanned image of the receipt. Please let me know the extracted text data."

[1355] 2. "Please automatically post this month's travel expenses and display the related data."

[1356] 3. "Please generate the income and expenditure data for October in the tax return format and notify me."

[1357] 4. "How do I correct my income and expenditure data using natural language?"

[1358] 5. "Provide tax-saving advice."

[1359] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1360] Step 1: Send image information

[1361] The user uses the wireless communication terminal to capture image information related to the transaction.

[1362] Input: Captured image information (photos of receipts and invoices)

[1363] Operation: The user starts the communication software (messaging application) and logs in. Next, the user attaches the captured image information to the chat screen and presses the send button.

[1364] Output: Image information sent to the server via communication software

[1365] Step 2: Receiving image information

[1366] The server receives the image information sent by the user using the API of the communication software.

[1367] Input: Image information sent by the user

[1368] Operation: The server calls the API of the communication software and saves the received image information in storage.

[1369] Output: Image information saved in storage

[1370] Step 3: Analyze image information

[1371] The server uses optical character recognition technology (OCR engine) to extract text information from the received image information.

[1372] Input: Image information saved in storage

[1373] Operation: The server starts the OCR engine, analyzes the image data, and extracts text information. For example, from an image of a receipt, it obtains the text information "October 1, 2023," "10,000 yen," and "Transportation expenses."

[1374] Output: Extracted text information

[1375] Step 4: Journalize textual information

[1376] The server analyzes the extracted text information and automatically records and journals the transaction by type.

[1377] Input: Extracted text information

[1378] Operation: The server analyzes the text information and classifies it into the appropriate account based on the journal entry rules. For example, the information "October 1, 2023," "10,000 yen," and "Transportation expenses" is journalized as transportation expenses.

[1379] Output: Journalized data is recorded in the database

[1380] Step 5: Generate data for declaration

[1381] The server aggregates the recorded journal data and converts it into a format for reporting.

[1382] Input: Journal entry data recorded in the database

[1383] How it works: The server aggregates accounting data for a specific period and converts it into a reporting format that complies with the laws and regulations of each country. For example, it totals travel expenses, business expenses, and other expenses for one month.

[1384] Output: Data converted into a declaration format

[1385] Step 6: Natural Language Processing Remediation

[1386] Users can check and modify income and expenditure data in natural language through the communication software.

[1387] Input: Natural language instructions from the user (e.g., "I would like to correct the travel expenses for October.")

[1388] How it works: The server uses natural language processing (NLP) technology to parse the instructions and modify the specific data. For example, it extracts the entry for "October travel expenses" and modifies it to the specified value.

[1389] Output: Database updated with modified data

[1390] Step 7: Providing tax advice

[1391] The server analyzes the recorded data and generates optimal advice on tax and asset management.

[1392] Input: All income and expenditure data recorded in the database

[1393] How it works: The server uses the tax accountant AI algorithm to analyze the data and generate optimal advice, such as proposals for tax savings and asset management.

[1394] Output: The generated tax advice is communicated to the user via the communication software.

[1395] Step 8: Data Storage and Financial Services

[1396] The server encrypts and securely stores all data and provides financial product recommendations and investment advice as needed.

[1397] Input: All transaction data, journal data, user income and expenditure data

[1398] How it works: The server encrypts your transaction data and stores it in a secure database. It also analyzes your income and expenditure data and generates financial product and investment recommendations tailored to you.

[1399] Output: Encrypted and securely stored data, as well as financial product suggestions and investment advice notified to the user.

[1400] (Application example 1)

[1401] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1402] Previously, users had to manually collect image data related to transactions, sort them manually, and prepare tax returns, which required a great deal of effort and was prone to errors. Furthermore, there was a lack of efficient ways to manage income and expenditures or receive tax advice, placing a heavy burden on users. Furthermore, there was no system in place for managing daily expenses in real time and receiving appropriate financial advice. To solve these issues, a new method was needed.

[1403] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1404] In this invention, the server includes: means for a user to use a mobile communication device to send image data related to transactions via a messaging application; means for receiving the sent image data and extracting text information from the image using optical character recognition technology; means for analyzing the extracted text information and automatically journalizing and recording transactions by type; means for converting the recorded journal data into a format for tax returns; means for understanding and correcting voice or text instructions from the user using natural language processing technology; means for analyzing the recorded data and generating advice on tax and asset management; means for a user to send image data via an application on the mobile communication device and compile data related to financial transactions; means for providing up-to-date financial information based on the compiled data; and means for using a generative AI model to generate prompts related to income and expenditure management according to the user's requests. This allows users to efficiently manage transaction-related data and receive appropriate tax advice. It also allows users to manage their daily expenses in real time and receive optimal financial advice.

[1405] A "portable communication terminal" refers to an electronic device that is portable and capable of internet communication and data transmission and reception.

[1406] "Image data" refers to visual information obtained by optical means and expressed in digital form.

[1407] "Messaging application" refers to software that enables users to send and receive messages, including text, images, and audio.

[1408] "Optical character recognition technology" refers to the technology that extracts character information from image data and converts it into text data.

[1409] "Text information" refers to information expressed as a string of characters.

[1410] "Means for automatically making and recording journal entries" refers to the process of classifying text information extracted by the system by transaction type and saving the results in a database or the like.

[1411] A "tax return format" refers to income and expenditure data organized in the format required for submission to tax authorities.

[1412] "Natural language processing technology" refers to the technology that enables computers to understand and analyze text and speech written in natural language.

[1413] "Financial transaction related data" refers to economic information related to payments and transactions.

[1414] "Up-to-date financial information" means the most recent economic data, including recent transactions and expenditures.

[1415] A "generative AI model" refers to a model of artificial intelligence that uses machine learning algorithms to generate appropriate outputs for a specific task.

[1416] "Prompt" refers to the text content generated in response to a user instruction or question.

[1417] This invention is a system that improves the efficiency of income and expenditure management and tax return filing by allowing users to send image data related to transactions via a message application using a portable communication terminal. This system operates with the following configuration and program.

[1418] 1. User sends image data

[1419] A user takes a photo of a receipt or invoice using their smartphone. This photo is sent to the system via a messaging application (e.g., a general messaging application). The user logs in to the application and sends the photo of the receipt or invoice to the chat.

[1420] 2. Receiving and analyzing image data

[1421] The server receives image data sent by the user using a messaging API. It then uses AI-OCR (optical character recognition) technology to extract text information from the image. For example, it can use the Google Cloud Vision API. The server analyzes the image data and obtains information such as the date, time, amount, and subject.

[1422] 3. Automatic journalization of extracted text

[1423] The extracted text information is analyzed by the server and automatically journalized by transaction type. During this process, transactions are classified into appropriate account items according to pre-set journalization rules. For example, they may be classified into categories such as "transportation expenses" and "entertainment expenses."

[1424] 4. Generate tax return data

[1425] The automatically posted data is aggregated on the server and converted into a format for tax returns as monthly or annual income and expenditure data. This format is provided in a format that complies with the tax laws of each country. The data for tax returns is then notified to the user.

[1426] 5. Natural language instructions and corrections

[1427] Users can use natural language on the messaging application to check and modify their income and expenditure data. For example, if they issue a command such as "I want to modify my travel expenses for October," the server will activate its natural language processing engine, understand the user's command, and make the appropriate modifications. It uses OpenAI GPT-4 and other technologies to analyze the command and make the necessary modifications.

[1428] 6. Generating tax advice

[1429] The server generates tax and asset management advice based on the collected income and expenditure data, allowing users to receive appropriate advice on tax savings and asset management. Using a generative AI model, advice is provided that is tailored to the user's specific situation.

[1430] 7. Providing real-time financial information

[1431] A user sends image data via an application on a portable communication terminal, and data related to financial transactions is compiled. The system provides the latest financial information based on the compiled data. For example, in response to a request such as "What is the total amount of expenses this month?", the system displays the latest financial information in real time.

[1432] Specific examples

[1433] For example, say a self-employed person wants to organize their income and expenses at the end of the month. The user takes a photo of this month's receipts with their smartphone and sends it to the system from the application. The server receives the image data and uses AI-OCR to extract the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server automatically journalizes this data and records it as entertainment expenses. Then, at the end of the month, data for tax returns is automatically generated and notified to the user.

[1434] When a user sends a message saying, "I'd like to review my entertainment expenses," the server understands the instruction and reconfirms and corrects it. Furthermore, a generative AI model analyzes the user's data and provides tax-saving advice. Ultimately, the system allows users to easily and efficiently manage their income and expenses and file tax returns.

[1435] Prompt Sentence Examples

[1436] "I would like to revise my travel expenses for October."

[1437] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1438] Step 1:

[1439] A user uses a portable communication terminal to take a photo of a receipt or invoice related to a transaction and transmits the image data to a server via a message application. The input is the image data taken by the user, and the output is the image data transmitted to the server via the message application.

[1440] Step 2:

[1441] The server receives image data sent by the user using a messaging API. The input is the image data sent by the user, and the output is the image data received by the server.

[1442] Step 3:

[1443] The server applies AI-OCR (optical character recognition) technology to the received image data and extracts the text information within the image. The input is the received image data, and the output is text information. Specifically, the image is analyzed using Google Cloud Vision API, etc., and text information such as the date, time, amount, and transaction items is extracted.

[1444] Step 4:

[1445] The server analyzes the extracted text information and automatically journalizes and records each type of transaction. The input is the extracted text information, and the output is automatically journalized data. Specifically, the data is classified into categories such as "transportation expenses" and "entertainment expenses" and recorded in the appropriate account.

[1446] Step 5:

[1447] The server aggregates the recorded journal data and converts it into a format for tax returns as monthly or annual income and expenditure data. The input is automatically journalized data, and the output is data converted into a format for tax returns.

[1448] Step 6:

[1449] The user uses a messaging application to input correction instructions in natural language. For example, they might send a message saying, "I would like to correct the travel expenses for October." The input is a natural language instruction from the user, and the output is the target data to be corrected based on that instruction.

[1450] Step 7:

[1451] The server uses natural language processing technology to analyze user instructions and make appropriate data corrections. The input is the user's natural language instructions, and the output is the corrected income and expenditure data. Specifically, OpenAI GPT-4 and other technologies are used to interpret the user's intent and update the relevant parts of the database.

[1452] Step 8:

[1453] The server generates advice on tax and asset management based on the collected income and expenditure data. The input is the recorded income and expenditure data, and the output is the generated advice. Using a generative AI model, it provides specific advice tailored to the user's situation.

[1454] Step 9:

[1455] The server aggregates data related to image data transmissions and financial transactions performed by users via applications on mobile communication terminals in real time and provides the latest financial information. The input is the transmitted image data and transaction data, and the output is the updated latest financial information. The data is processed in real time, and the most recent financial information is displayed when the user requests it.

[1456] Examples of concrete examples and prompts

[1457] For example, say a self-employed person wants to organize their income and expenses at the end of the month. The user takes a photo of this month's receipts with their smartphone and sends it to the system from the application. The server receives the image data and uses AI-OCR to extract the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server automatically journalizes this data and records it as entertainment expenses. Then, at the end of the month, data for tax returns is automatically generated and notified to the user.

[1458] Prompt Sentence Examples

[1459] "I would like to revise my travel expenses for October."

[1460] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1461] The present invention combines an emotion engine with a system that allows users to efficiently manage income and expenditures and file tax returns by sending image data related to transactions via a message application using a portable communication terminal. This system operates in conjunction with the following programs.

[1462] 1. User sends image data

[1463] Users use their smartphones to take photos of receipts or invoices, which are then sent to the system via a messaging application (e.g., LINE).

[1464] The user logs in with LINE and sends a photo of a receipt or invoice to the chat.

[1465] 2. Receiving and analyzing image data

[1466] The server receives the image data sent by the user using the LINE API, and then extracts the text information from the image using optical character recognition technology (AI-OCR).

[1467] The server receives the image data, runs the AI-OCR engine, and extracts the text.

[1468] For example, text information such as "October 1, 2023," "10,000 yen," and "transportation expenses" is extracted from a receipt.

[1469] 3. Automatic journalization of extracted text

[1470] The extracted text information is analyzed by the server and automatically journalized by type of income and expenditure, with transactions classified into appropriate account items according to pre-set journalization rules.

[1471] Based on the analysis results, the server automatically records the expenses in categories such as "transportation expenses."

[1472] For example, "10,000 yen" is classified as "transportation expenses."

[1473] 4. Generate tax return data

[1474] The automatically posted data is aggregated on the server and reflected in monthly and annual income and expenditure reports. This data is converted into a tax return format and prepared for easy access by users later.

[1475] The server aggregates the journal data by period and converts it into tax return format.

[1476] The data for tax returns will be sent to the user via LINE.

[1477] 5. Natural language instructions and corrections

[1478] Users can use natural language on LINE to check and edit their income and expenditure data. For example, if they say, "I want to edit my travel expenses for October," the server will activate its natural language processing engine, understand the user's instructions, and make the appropriate edits.

[1479] The user sends instructions in natural language, which the server analyzes using an NLP engine.

[1480] The server reviews the data and makes any necessary corrections.

[1481] 6. Collaboration with tax accountant AI

[1482] The server uses the collected income and expenditure data to generate optimal tax advice using a tax accountant AI algorithm, allowing users to receive appropriate advice on tax savings and asset management.

[1483] The server sends the data to the tax accountant AI, which generates optimal advice.

[1484] The advice will be sent to the user via LINE.

[1485] 7. Data Storage and Financial Services

[1486] The server encrypts and securely stores all collected data, and provides financial product recommendations and investment advice as needed, thus supporting users' financial growth.

[1487] The server encrypts and stores the data and suggests financial products.

[1488] Users receive financial advice and product details via LINE.

[1489] 8. Introducing and utilizing an emotion engine

[1490] The server incorporates a new emotion engine that analyzes emotions based on the user's voice or text data and optimizes response content. This engine recognizes the user's emotions in real time and can respond appropriately.

[1491] The user sends a voice message such as "Today is stressful and difficult."

[1492] The server receives the voice data and uses an emotion engine to analyze the user's emotional state.

[1493] The server provides appropriate feedback via LINE based on the user's emotions (e.g., "You seem stressed. Let me know if there's anything I can do to help you").

[1494] Specific examples

[1495] For example, let's say that self-employed Tanaka wants to organize his income and expenses at the end of the month. Tanaka takes a photo of this month's receipts with his smartphone and sends it to the system via LINE. The server receives the image data and uses AI-OCR to extract the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server automatically accounts for this data and records it as entertainment expenses. Then, at the end of the month, data for tax returns is automatically generated and notified to Tanaka via LINE. When Tanaka sends a message via LINE saying, "I would like to review my entertainment expenses," the server understands the instruction and reconfirms and corrects it. Furthermore, the tax accountant AI analyzes Tanaka's data and provides tax-saving advice.

[1496] One day, Tanaka sends a voice message to LINE saying, "I'm feeling very stressed today," and the server's emotion engine analyzes Tanaka's emotions and provides appropriate feedback, such as, "It seems you're feeling stressed. Let me know if there's anything I can help you with." This allows Tanaka to not only efficiently manage his income and expenses and file his tax return through the system, but also receive emotional support.

[1497] The processing flow will be explained below.

[1498] Step 1:

[1499] The user takes a photo of a receipt or invoice using a mobile communication device (smartphone). The user opens the LINE app and sends the photo to the LINE chat. This sends the image data to the system.

[1500] Step 2:

[1501] The server uses the LINE API to receive the image data sent by the user. The received image data is saved in its original format and passed to the next processing step.

[1502] Step 3:

[1503] The server activates optical character recognition (AI-OCR) technology to extract text information from the received image data. Specifically, the image processing engine identifies letters and numbers in the image and converts them into text. For example, information such as "October 1, 2023," "10,000 yen," and "transportation expenses" is extracted.

[1504] Step 4:

[1505] The extracted text information is analyzed by the server. The server identifies the type of transaction based on the text information and automatically journalizes it. Specifically, it classifies the transaction into the appropriate account category based on information such as the "date," "amount," and "item name." For example, "10,000 yen" is journalized as "transportation expenses."

[1506] Step 5:

[1507] The server aggregates the extracted and journalized data and generates monthly or annual income and expenditure reports, which are then converted into tax return formats and prepared for easy user access later.

[1508] Step 6:

[1509] The server notifies the user of the generated tax return data via LINE chat. The user can then check the data in the LINE app and request corrections if necessary.

[1510] Step 7:

[1511] The user sends a natural language instruction on LINE, such as "I would like to revise the transportation expenses for October." This instruction is sent to the server and analyzed by the natural language processing engine.

[1512] Step 8:

[1513] Based on the analysis results, the server understands the user's instructions and corrects the relevant journal data. For example, it rechecks "October travel expenses" and makes any necessary corrections.

[1514] Step 9:

[1515] The server provides all collected income and expenditure data to the AI ​​tax accountant, who then analyzes the data and provides optimal tax advice on tax-saving strategies and asset formation.

[1516] Step 10:

[1517] The server notifies the user of the generated advice via LINE chat, and the user can receive the advice in the LINE app and take the necessary action.

[1518] Step 11:

[1519] The server uses an emotion engine to analyze the user's voice or text data in real time. For example, if a user says, "I'm feeling stressed today," the server recognizes the voice data and analyzes the user's emotional state.

[1520] Step 12:

[1521] Based on the analysis results of the emotion engine, the server provides appropriate feedback according to the user's emotions. For example, it may send a message via LINE saying, "You seem to be feeling stressed. Please let us know if there is anything we can do to help."

[1522] Step 13:

[1523] The server encrypts and securely stores all financial data, and the server uses this data to provide more advanced financial services and investment advice.

[1524] Step 14:

[1525] Users receive financial advice and product recommendations on LINE, which helps them make more effective financial decisions.

[1526] Specific examples

[1527] For example, let's say that self-employed Tanaka wants to organize his income and expenses at the end of the month. Tanaka takes a photo of this month's receipts with his smartphone and sends it to the system via LINE. The server receives the image data and uses AI-OCR to extract the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server automatically accounts for this data and records it as entertainment expenses. Then, at the end of the month, data for tax returns is automatically generated and notified to Tanaka via LINE. When Tanaka sends a message via LINE saying, "I would like to review my entertainment expenses," the server understands the instruction and reconfirms and corrects it. Furthermore, the tax accountant AI analyzes Tanaka's data and provides tax-saving advice.

[1528] One day, Tanaka sends a voice message to LINE saying, "I'm feeling very stressed today," and the server's emotion engine analyzes Tanaka's emotions and provides appropriate feedback, such as, "It seems you're feeling stressed. Let me know if there's anything I can help you with." This allows Tanaka to not only efficiently manage his income and expenses and file his tax return through the system, but also receive emotional support.

[1529] Example 2

[1530] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1531] Traditional income and expenditure management and tax return systems required time and effort for manual data entry, resulting in inefficiency. They also often failed to accurately record collected data or provide adequate tax advice. Furthermore, they lacked support that took into account the user's emotional and stress levels, resulting in a lack of improvement in the overall user experience.

[1532] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for a user to use a portable communication terminal to send image data related to a transaction via a message application; means for receiving the sent image data and extracting text information from the image using optical character recognition technology; means for analyzing the extracted text information and automatically journalizing and recording the transaction type; means for converting the recorded journal data into a format for tax returns; means for understanding and correcting voice or text instructions from the user using natural language processing technology; means for analyzing the recorded data and generating advice on taxes and asset management; and means for recognizing emotions based on the user's voice or text data using emotion analysis technology and optimizing the response. This not only improves the efficiency of income and expenditure management and tax return filing, but also enables improved accuracy of tax advice and emotional support for users.

[1533] "User" refers to any individual or legal entity that uses the System.

[1534] A "portable communication terminal" refers to a terminal device such as a smartphone or tablet that is portable and capable of wireless communication.

[1535] "Transaction" refers to the act of exchanging money or goods in economic activities.

[1536] "Image data" refers to visual information represented in digital form.

[1537] "Message application" refers to an application that sends and receives text messages and image data over the Internet.

[1538] "Optical character recognition technology" refers to technology that extracts characters from an image as digital data.

[1539] "Text information" refers to data expressed as a string of characters.

[1540] "Journal entry" refers to the accounting process of classifying and recording transactions into account items.

[1541] "Server" refers to a device or system that provides computer resources over a network.

[1542] "Tax return format" refers to the format of the income and expenditure report to be submitted to the tax office.

[1543] "Natural language processing technology" refers to technology for understanding human language and responding appropriately.

[1544] "Voice or text instructions" refers to operation instructions given to the system by the user using words or letters.

[1545] "Tax advice" refers to the act or content of providing appropriate advice regarding tax.

[1546] "Asset management" refers to the efficient management and administration of personal or corporate assets.

[1547] "Emotion analysis technology" refers to the technology of analyzing human emotions and psychological states from voice and text.

[1548] "Optimizing response content" refers to optimizing the responses and support provided by the system according to the user's needs and condition.

[1549] This invention is a system that allows users to efficiently manage income and expenditures and file tax returns using a mobile communication device. This system mainly combines a messaging application, optical character recognition technology (AI-OCR), natural language processing technology (NLP), tax accountant AI algorithms, and sentiment analysis technology. Specific examples of the use of various hardware and software are shown below.

[1550] A user uses a mobile communication device (e.g., a smartphone or tablet) to take a photo of a receipt or invoice related to a transaction. At that time, the user uses a messaging application (e.g., LINE) to send the image data to the system. The user logs in to LINE and sends a photo of the receipt or invoice as a chat message.

[1551] The server receives image data sent by the user using LINE's API. The server then uses optical character recognition technology (e.g., Google Cloud Vision API) to extract text information from the image. This AI-OCR engine identifies characters in the image and extracts them as numerical or text data. For example, from a receipt, the following information is extracted: "October 1, 2023," "10,000 yen," and "Transportation expenses."

[1552] The extracted text information is analyzed by the server and automatically journalized according to pre-set journalization rules. Based on the analysis results, the server classifies the data into appropriate account items for each type of income and expenditure. For example, "10,000 yen" is journalized as "transportation expenses."

[1553] The server converts the collected data into a format for tax returns. This data is reflected in monthly and annual income and expenditure reports and prepared as tax return data. The server then aggregates this data and converts it into a tax return format (e.g., XML for tax software). Users are notified via LINE for easy access to their tax return data.

[1554] Users can also send voice or text instructions on LINE using natural language processing technology. For example, if you send an instruction such as "I would like to revise my travel expenses for October," the server will use an NLP engine (e.g., OpenAI GPT-3) to analyze the message, understand the user's instruction, and make the appropriate revisions.

[1555] Based on the collected data, the server uses a tax accountant AI algorithm (e.g., Watson Tax Advisor) to generate optimal tax advice. Users can receive advice on tax savings and asset management via LINE. The server also encrypts and securely stores revenue data and provides users with financial products and investment advice.

[1556] Emotion analysis technology (e.g., IBM Watson Tone Analyzer) recognizes emotions from a user's voice or text data and provides feedback in real time. For example, if a user sends a voice message saying, "I'm feeling very stressed today," the server analyzes the emotion and replies, "It sounds like you're feeling stressed. Let me know if there's anything I can do to help."

[1557] Specific examples

[1558] A self-employed individual wants to organize their income and expenses at the end of the month. They take a photo of this month's receipts with their smartphone and send it to the system via LINE. The server receives the image data and uses AI-OCR to extract the following information: "October 15, 2023," "5,000 yen," and "Entertainment Expenses." The server automatically accounts for this data and records it as entertainment expenses. At the end of the month, tax return data is automatically generated and notified via LINE. When the self-employed individual sends a message via LINE saying, "I'd like to review my entertainment expenses," the server understands the instruction and reconfirms and corrects it. Furthermore, a tax accountant AI analyzes the data and provides tax-saving advice. One day, the self-employed individual sends a voice message via LINE saying, "I'm feeling stressed today and it's bothering me," and the server's emotion engine analyzes the emotion and provides appropriate feedback, such as, "It sounds like you're stressed. Let me know if there's anything I can do to help."

[1559] Prompt Sentence Examples

[1560] "Please send a photo of your receipt via LINE to record your income and expenses."

[1561] "I would like to revise my travel expenses for October 2023."

[1562] "Can you give me some tax saving advice?"

[1563] "I'm having a lot of stress today"

[1564] In this way, the system helps users manage their finances and file their tax returns efficiently, and also provides emotional support.

[1565] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1566] Step 1: Sending image data

[1567] The user takes a photo of an image related to the transaction (such as a receipt or invoice) using a mobile communication terminal (e.g., a smartphone) and sends it to the system via a messaging application (e.g., LINE).

[1568] Input: Image data of receipts and invoices

[1569] Output: Image data sent to LINE

[1570] Specific operation: The user launches the camera app and takes a picture of a receipt or invoice. The user selects the image in the LINE app and sends it as a chat message.

[1571] Step 2: Receiving image data

[1572] The server uses the LINE API to receive the image data sent by the user.

[1573] Input: Image data sent via the LINE API

[1574] Output: Image data stored on the server

[1575] Specific operation: The server periodically checks the LINE API to receive new image data, which is then stored in the database.

[1576] Step 3: Extracting text information

[1577] The server uses optical character recognition technology (AI-OCR) to extract text information from the received image data.

[1578] Input: Saved image data

[1579] Output: Extracted text information

[1580] How it works: The server launches an AI-OCR engine such as Google Cloud Vision API, identifies the characters in the image, and generates text data. For example, the following information is extracted: "October 1, 2023," "10,000 yen," and "transportation expenses."

[1581] Step 4: Parsing and journalizing extracted text

[1582] The server analyzes the extracted text information and automatically records and journals the transactions by type.

[1583] Input: Extracted text information

[1584] Output: Journal entry data

[1585] Specific operation: The server analyzes the text information based on pre-defined rules and classifies it into categories such as "transportation expenses" and "entertainment expenses." For example, "10,000 yen" is journalized as "transportation expenses." The journalized data is then saved in a database.

[1586] Step 5: Generate tax return data

[1587] The server converts the data into a format for tax returns based on the accounting data.

[1588] Input: Journal data

[1589] Output: Data in tax return format

[1590] Specific operation: The server aggregates monthly and annual income and expenditure reports and converts them into a format for tax returns (e.g., XML for tax software). The user is notified via LINE that "tax return data is ready."

[1591] Step 6: Correcting data using natural language

[1592] Users can check and edit income and expenditure data using natural language on LINE.

[1593] Input: Natural language instructions from the user

[1594] Output: Corrected balance data

[1595] Specific operation: The user sends a message on LINE saying, "I would like to correct my travel expenses for October." The server uses an NLP engine (e.g., OpenAI GPT-3) to analyze the message and understand the instructions. The server then reviews the target data and makes any necessary corrections.

[1596] Step 7: Advice generation by tax accountant AI

[1597] The server sends the collected data to the tax accountant AI, which generates optimal tax advice.

[1598] Input: Collected income and expenditure data

[1599] Output: Tax advice

[1600] Specific operation: The server sends the collected data to a tax accountant AI (e.g., Watson Tax Advisor). The tax accountant AI analyzes the data and generates appropriate tax advice. The advice is then sent to the user via LINE.

[1601] Step 8: Optimize your response with sentiment analysis

[1602] The server uses emotion analysis technology to recognize the user's emotions and optimize the response.

[1603] Input: Voice or text data from the user

[1604] Output: Emotion-based feedback

[1605] Specific operation: The user sends a voice message stating, "I'm feeling very stressed today." The server receives the voice data and analyzes the user's emotions using an emotion analysis engine (e.g., IBM Watson Tone Analyzer). Appropriate feedback based on the user's emotions is then provided via LINE (e.g., "It sounds like you're feeling stressed. Please let me know if there's anything I can do to help you").

[1606] This processing flow allows the system to streamline users' income and expenditure management and tax return filing, and also provides emotional support.

[1607] (Application example 2)

[1608] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1609] Managing transaction-related income and expenditures and filing tax returns are time-consuming and labor-intensive tasks, especially for small businesses and sole proprietors. Traditional manual data entry and accounting processes are prone to errors, and users without specialized knowledge of tax or asset management find it difficult to obtain appropriate support. Furthermore, few systems take into account the user's emotions and mental state, which leads to a poor user experience. Therefore, the challenge is to provide a system that streamlines income and expenditure management and tax return filing and provides emotionally sensitive feedback.

[1610] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to use a portable communication terminal to send image data related to a transaction via a message application; means for receiving the sent image data and extracting text information from the image using optical character recognition technology; means for analyzing the extracted text information and automatically journalizing and recording the transaction type; means for converting the recorded journal data into a format for tax returns; means for understanding and correcting voice or text instructions from the user using natural language processing technology; means for analyzing the recorded data and generating advice on taxes and asset management; and means for analyzing the user's emotions and providing feedback according to the user's emotions using an emotion engine that optimizes response content. This improves the efficiency of income and expenditure management and tax return filing and also enables support that takes into account the user's emotions and mental state.

[1611] A "portable communication terminal" is a device that is portable and has communication capabilities, and mainly refers to smartphones and tablets.

[1612] "Transaction-related image data" refers to image data, such as receipts and invoices, that contains information about financial transactions.

[1613] A "messaging application" is application software that allows users to send and receive messages, such as text and images, via the Internet, and includes LINE and WhatsApp.

[1614] "Optical character recognition technology" is a technology that extracts character information from image data, and is also known as OCR (Optical Character Recognition).

[1615] "Text information" refers to information such as letters and numbers, and includes character data extracted from an image.

[1616] "Journal entries" are the process of classifying transactions into specific account items and recording them in the ledger.

[1617] "Tax format" means the conversion of collected financial data into a specific format for use in tax returns.

[1618] "Natural language processing technology" is a technology that allows computers to understand, analyze, and respond to human language, and is also known as NLP (Natural Language Processing).

[1619] An "emotion engine" is a technology that analyzes user data such as voice and text and estimates their emotional state.

[1620] "Feedback" refers to the response or advice that a system gives to a user in response to their input.

[1621] "Tax and asset management advice" means providing expertise based on the user's financial data and offering advice on tax optimization and asset management.

[1622] This invention is a system that combines an emotion engine with a system that allows users to use a portable communication terminal to send image data related to transactions via a message application, thereby streamlining income and expenditure management and tax return filing.

[1623] The server receives image data related to transactions sent by users via messaging applications such as LINE and WhatsApp. It extracts text information from the received image data using optical character recognition (OCR). The extracted text information is stored in the server's database, analyzed by an automatic accounting program, and automatically recorded and accounted for by transaction type.

[1624] Furthermore, the recorded journal data is automatically converted into a format for tax returns and provided to users in a format that can be used for tax returns. This also implements a means of generating advice on taxes and asset management, and the tax accountant AI provides optimal advice based on the data.

[1625] Users can send voice or text instructions to the server through an interface that uses natural language processing technology. For example, they can send a message to a friend saying, "I want to revise my travel expenses for October." The server analyzes this instruction, reviews the relevant data, and makes any necessary adjustments. Furthermore, if a user sends a message expressing their emotions, the server's emotion engine analyzes it and provides feedback appropriate to the user's emotional state.

[1626] The hardware used includes a server and a mobile communication device, and the software includes the LINE Messaging API, an OCR engine (such as Tesseract OCR), MySQL, a natural language processing engine (such as spaCy), and a sentiment analysis model (such as the Transformers sentiment analysis pipeline).

[1627] For example, a user can take a photo of a receipt with their smartphone and send it to the system via a messaging application. The image data is transferred to the server, where OCR extracts the information "October 15, 2023," "5,000 yen," and "entertainment expenses." The server records this data as entertainment expenses and automatically generates data for tax returns. If the user then issues a natural language instruction such as "I'd like to correct the entertainment expenses for October," the system will correct the relevant data. Furthermore, if the user voice-transmits "I'm feeling very stressed today," the emotion engine will analyze the emotion and provide feedback such as "It sounds like you're stressed. Please let me know if there's anything I can do to help."

[1628] An example prompt is:

[1629] "Go to My Page and check your credit card transaction details from last month. Then, send an image of the receipt to LINE and correct the relevant transaction. Also, if you would like to receive advice on asset management, please let the system know."

[1630] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1631] Step 1:

[1632] A user uses a mobile communication terminal to send image data related to a transaction (e.g., a photo of a receipt or invoice) to the system via a message application. The input of this step is the image data of the receipt or invoice, and the output is that it reaches the server via the message application. Specifically, the user takes a photo of the receipt using the camera function of their smartphone and sends the image via the chat function of the message application.

[1633] Step 2:

[1634] The server receives image data sent by the user using the API of the messaging application. The input of this step is the image data sent through the messaging application, and the output is that it is saved on the server. Specifically, the server uses the API of the messaging application to detect new messages, receive the image file, and save it in the database.

[1635] Step 3:

[1636] The server extracts text information from the received image data using optical character recognition (OCR). The input for this step is the image data, and the output is the text information extracted from the image. Specifically, the server uses an OCR engine (e.g., Tesseract OCR) to extract the characters in the image in text format.

[1637] Step 4:

[1638] The server analyzes the extracted text information and automatically journalizes and records it for each type of transaction. The input to this step is the text information extracted by OCR, and the output is the analyzed transaction information being journalized and recorded in a database. Specifically, the server uses an analysis program to analyze the text information, classify it into the appropriate account category, and save it in the database.

[1639] Step 5:

[1640] The server converts the recorded journal data into a format for tax returns. The input for this step is the journal data, and the output is the data converted into a format for tax returns. Specifically, the server aggregates the journal data and converts it into a tax return format that complies with the tax laws of each country.

[1641] Step 6:

[1642] The user sends voice or text instructions to the server using natural language processing technology. The input for this step is the user's voice or text instruction, and the output is that the instruction is understood and executed. Specifically, the user enters "I would like to revise the travel expenses for October," and the instruction is sent to the server.

[1643] Step 7:

[1644] The server analyzes the user's instructions using a natural language processing engine and makes any necessary corrections. The input to this step is the natural language instruction from the user, and the output is the execution of data correction processing. Specifically, the server uses a natural language processing engine (e.g., spaCy) to analyze the user's instructions and correct the target data.

[1645] Step 8:

[1646] The server analyzes the recorded data and generates advice on tax and asset management. The input for this step is the recorded income and expenditure data, and the output is advice on tax and asset management. Specifically, the server uses the tax accountant AI algorithm to analyze the data and generate optimal advice.

[1647] Step 9:

[1648] The server analyzes the user's emotions and provides feedback based on the user's emotions using an emotion engine that optimizes the response content. The input for this step is the user's emotional expression in voice or text, and the output is a feedback message based on the emotion analysis results. Specifically, the server uses an emotion engine (e.g., a Transformers emotion analysis model) to analyze the user's emotional state and generate appropriate feedback.

[1649] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1650] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data ...

Claims

1. A means for a user to send image data related to a transaction via a message application using a portable communication terminal; means for receiving the transmitted image data and extracting text information within the image using optical character recognition technology; A means to analyze the extracted text information and automatically record and journalize each type of transaction. A means for converting the recorded journal data into a format for tax returns; means for understanding and correcting voice or text instructions from a user using natural language processing technology; a means of analysing the recorded data and generating tax and wealth management advice; A system including:

2. The system of claim 1 , wherein the transmitted image data is a receipt or an invoice.

3. The system according to claim 1 , wherein the format for the tax return conforms to the tax laws of each country.

Images