System

Abstract

A system is provided.SOLUTION: The system of claim 1, wherein the AI engine further comprises: means for generating an Tax forms based on the AI engine; and means for submitting the Tax forms to an on-line system, wherein the on-line system comprises: means for generating an invoice based on the analysis engine; and means for submitting the invoice to the on-line system.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Filing tax returns is a time-consuming and labor-intensive process for sole proprietors. In particular, the task of organizing receipts and payment data and properly allocating expenses to each category is cumbersome and requires specialized knowledge for many people, creating a significant burden. Furthermore, the process from preparing to submitting tax returns is time-consuming and can disrupt business operations. There is a need for a solution to these problems and reduce the burden on sole proprietors. [Means for solving the problem]

[0005] The present invention provides a means for users to upload receipt images and payment data from their terminals to a server, and includes a means for the server to format the received data and send it to an AI engine. The AI ​​engine analyzes the data using image recognition and natural language processing technologies and allocates expenses. The server also provides a means for generating tax return documents based on the allocated data and submitting the generated tax return documents to an online system. This automates the cumbersome task of allocating expenses, document preparation, and submission procedures, resulting in a system that significantly reduces the burden on sole proprietors.

[0006] A "server" is a computer system that provides services to other computers (clients) on a network.

[0007] A "terminal" is a device such as a computer or smartphone that is operated by a user.

[0008] "User" refers to an individual or corporation that uses this system to upload receipt images and payment data.

[0009] A "receipt" is a document that proves receipt of money in a transaction.

[0010] "Payment Data" means data containing payment information made through electronic means.

[0011] An "AI engine" is a software component that uses artificial intelligence technology to analyze and classify data.

[0012] "Image recognition technology" is a technology in which a computer analyzes the contents of an image and identifies the object.

[0013] "Natural language processing technology" is a technology that allows computers to understand and process human language.

[0014] An "expense item" is a category for classifying expenses and expenditures.

[0015] A "tax return" is a document submitted to tax authorities that contains details of income and expenses.

[0016] "Online System" means a web-based service system available via the Internet. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This invention is a system that greatly simplifies and streamlines the preparation of tax return documents. This system significantly reduces the burden on sole proprietors by automating the complicated process of allocating expenses, generating tax return documents, and even submitting them online.

[0039] Program processing overview

[0040] The program of this system has the following functions:

[0041] 1. The user uploads an image of the receipt and payment data.

[0042] Users use their smartphones or PCs to upload images of receipts and payment data to the server, using a dedicated application or web browser.

[0043] 2. The server receives the data, formats it, and sends it to the AI ​​engine

[0044] The server receives image data and payment data sent by the user, converts it into an appropriate format, and sends the formatted data to the AI ​​engine.

[0045] 3. The AI ​​engine analyzes the data and allocates expenses

[0046] The AI ​​engine uses image recognition and natural language processing technologies to extract necessary information (such as date, amount, and expense type) from receipt images and payment data. Based on the extracted information, it automatically allocates each expense to the appropriate expense category.

[0047] 4. The server generates tax return documents based on the sorted data.

[0048] The server stores the expense information sent from the AI ​​engine in a database and automatically generates tax return documents based on that information, calculating the totals of income, expenditures, and expenses and accurately filling out the information required for the return.

[0049] 5. The server submits the generated tax return to the online system.

[0050] The server then processes the generated tax return documents for submission to an online system (such as e-Tax), eliminating the need for users to download and mail the documents.

[0051] Specific examples

[0052] 1. The user takes a photo of the receipt with their smartphone and uploads it to the server using a dedicated app.

[0053] For example, a user takes a taxi and receives a receipt for the fare of 5,000 yen. The user takes a photo of the receipt with the smartphone camera and uploads it to the server via a dedicated app.

[0054] 2. The server formats the received image data and sends it to the AI ​​engine.

[0055] When the server receives the receipt image, it removes noise from the image and formats it so that text information can be easily extracted. The formatted image data is then sent to the AI ​​engine.

[0056] 3. The AI ​​engine extracts the necessary information from the receipt image and determines the expense item

[0057] The AI ​​engine uses image recognition technology to extract text information such as "October 1, 2023," "5,000 yen," and "transportation expenses." Natural language processing technology then categorizes this information as transportation expenses.

[0058] 4. The server automatically generates tax return documents

[0059] The server retrieves the information about the 5,000 yen allocated as transportation expenses from the AI ​​engine and automatically enters it into the appropriate field on the tax return. This calculates the total of income, expenditures, and expenses, and generates an accurate tax return.

[0060] 5. The server submits the generated tax return to the online system.

[0061] The server logs into the e-Tax system and submits the generated tax return. The user is notified when the submission is complete.

[0062] This system allows users to carry out the complicated tax return procedures almost without thinking about them, enabling smooth data management and submission of tax return documents, allowing users to focus more on running their business.

[0063] The processing flow will be explained below.

[0064] Step 1:

[0065] The user opens the dedicated application using a smartphone or PC. After taking a picture of the receipt or importing the payment data, the user clicks the "Upload" button in the application. This operation causes the device to upload the receipt image and payment data to the server.

[0066] Step 2:

[0067] The server receives the image data and payment data sent by the user, and performs preprocessing (e.g., noise removal, resolution adjustment) on the received data to maintain the image resolution at an appropriate level.

[0068] Step 3:

[0069] The server sends the preprocessed image data to the AI ​​engine, where it is converted into an appropriate format and formatted for efficient analysis.

[0070] Step 4:

[0071] The AI ​​engine uses image recognition technology to extract text information from receipt images, such as date, amount, and category (transportation expenses, entertainment expenses, etc.) using a unique algorithm.

[0072] Step 5:

[0073] The AI ​​engine uses natural language processing technology to analyze the extracted text information and categorize it into the appropriate expense category. Specifically, it categorizes keywords such as "taxi" and "fare" as transportation expenses.

[0074] Step 6:

[0075] The server stores the classification information sent from the AI ​​engine in a database, which accumulates the allocated expense information and makes it available for subsequent processing.

[0076] Step 7:

[0077] The server automatically generates tax return documents based on the expense classification information retrieved from the database. Specifically, it inputs appropriate data into the income, expenditure, and expense information fields according to the tax return template, and calculates the total amount.

[0078] Step 8:

[0079] The server submits the generated tax return documents to the e-Tax system. At this time, the server uses the authentication information for logging in to the e-Tax system to carry out the appropriate upload procedure.

[0080] Step 9:

[0081] The server receives the results of the tax return submission and confirms that the submission was successful. It then sends a notification of the completion of the submission to the user.

[0082] This detailed processing step allows users to automate the entire tax return process by simply taking and uploading an image of their receipt.

[0083] Example 1

[0084] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0085] The complicated and time-consuming process of preparing tax returns by hand for sole proprietors has led to problems of human error and reduced work efficiency. Furthermore, it is a significant burden to individually categorize receipts, prepare tax return forms, and then submit them online.

[0086] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0087] In this invention, the server includes a means for a user to send business data from an information input device to a processing device, a means for converting and formatting the received data into an appropriate format by the processing device, and a means for sending the formatted data by the processing device to an analysis device. This automates the preparation of documents required for filing tax returns and the allocation of data to expense items, enabling online submission of tax returns.

[0088] An "information input device" is a device that allows a user to input business data and transmit it to a server, and includes smartphones, personal computers, and the like.

[0089] A "processor" is a server or computer system that receives data sent by a user and converts and formats it into an appropriate format.

[0090] "Business data" refers to various transaction information and receipt image data that are generated by users and sent to the server.

[0091] An "analysis device" is an AI engine or dedicated hardware that analyzes the formatted data sent from the processing device and classifies expenses.

[0092] "Image processing technology" refers to technology for extracting and processing necessary information from digital images, and includes noise removal and text recognition.

[0093] "Automatic analysis technology" is a technology that allows an AI engine to analyze data and classify information using natural language processing technology.

[0094] "Report" refers to tax returns and other business reports generated based on the allocated data.

[0095] "Online services" refers to various services provided via the Internet, including tax return submission systems such as e-Tax.

[0096] This invention is a system for streamlining the tax return process, where users submit business data and the data is analyzed, classified, and generated using a server and an AI engine. The system aims to automate many manual tasks and reduce the burden on users.

[0097] System configuration

[0098] The system consists of the following main components:

[0099] 1. Information input device

[0100] A device that allows users to input business data and send it to a server. Specifically, it includes smartphones (iOS or Android) and personal computers (Windows or Mac). Dedicated applications and web browsers (Chrome or Firefox) are used.

[0101] 2. Processing equipment

[0102] It is a server or computer system that receives data sent by users and converts and formats it into an appropriate format. It uses a Linux-based server and an image processing library (OpenCV). It also includes a database (MySQL or PostgreSQL).

[0103] 3. Analysis device

[0104] This is an AI engine and dedicated hardware that analyzes the formatted data sent from the processing device and classifies expenses. The AI ​​engine uses deep learning frameworks such as TensorFlow and PyTorch and an OCR library (Tesseract), and performs analysis using natural language processing libraries (spaCy and NLTK).

[0105] 4. Online Services

[0106] These are various services provided via the Internet, including tax return submission systems such as e-Tax.

[0107] Specific examples

[0108] Next, a concrete example of the operation of this system will be given.

[0109] 1. The user takes a photo of the receipt with their smartphone and uploads it to the server using a dedicated app.

[0110] Suppose a user takes a taxi and receives a receipt for the fare of 5,000 yen. The user takes a photo of this receipt with the smartphone camera and uploads the image to the server using a dedicated app. An example of a prompt sentence is, "Please take a photo of the restaurant receipt and upload it using the dedicated app. Please make sure that the amount and date are clearly visible."

[0111] 2. The server formats the received image data and sends it to the AI ​​engine.

[0112] The server uses OpenCV to remove noise from the images it receives and formats them so that text information can be easily extracted. The formatted image data is then sent to the AI ​​engine.

[0113] 3. The AI ​​engine extracts the necessary information from the receipt image and determines the expense item

[0114] The AI ​​engine uses Tesseract to extract information such as "October 1, 2023," "5,000 yen," and "transportation expenses," and then uses natural language processing technology (spaCy) to classify this as transportation expenses.

[0115] 4. The server automatically generates tax return documents

[0116] The server uses JasperReports to generate the necessary tax return documents based on the classification information from the AI ​​engine, and calculates the totals of income, expenses, and costs to accurately reflect them on the tax return documents.

[0117] 5. The server submits the generated tax return to the online system.

[0118] The server logs into the e-Tax system through the API and submits the generated tax return, which notifies the user and eliminates the need to download and mail the documents.

[0119] This system allows users to simplify the cumbersome manual process of filing tax returns, improving work efficiency.

[0120] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0121] Step 1:

[0122] The user uses an information input device (smartphone or PC) to send business data (receipt images and payment data) to the server. Specifically, the user uses a dedicated application or web browser to take a photo of the taxi receipt with the smartphone camera and upload the image. The input is image data taken by the user, which is then sent to the server.

[0123] Step 2:

[0124] The server converts and formats the received image data into an appropriate format. Specifically, the server uses an image processing library (OpenCV) to remove noise and format the data so that text information can be easily extracted. This processing improves the accuracy of reading the image data. The input is the raw image data sent by the user, and the output is formatted image data.

[0125] Step 3:

[0126] The server sends the formatted image data to the analysis device (AI engine). The server uses an API for data transmission to pass the formatted data to the AI ​​engine. The input is the formatted image data, and the output is the data sent to the analysis device.

[0127] Step 4:

[0128] The analysis device (AI engine) uses image processing technology (OCR) and automatic analysis technology to analyze the data and determine the expense items. Specifically, the AI ​​engine uses the OCR library (Tesseract) to extract text information such as "October 1, 2023" and "5,000 yen," and classifies it as "transportation expenses" using natural language processing technology (spaCy). The input is formatted image data, and the output is text information with classified expense items.

[0129] Step 5:

[0130] The server generates a report based on the expense classification data received from the analysis device. Specifically, the server uses JasperReports to calculate the totals of income, expenditures, and expenses, and automatically generates tax return documents. The input is text information with expense classifications, and the output is the automatically generated tax return documents.

[0131] Step 6:

[0132] The server submits the generated tax return to an online service (such as e-Tax). The server logs in to the e-Tax system via API and uploads the tax return. The input is the automatically generated tax return, and the output is a notification to the online service that the return has been submitted. This process notifies the user that the return has been submitted.

[0133] (Application example 1)

[0134] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0135] Currently, filing tax returns is a very complicated and time-consuming task for store owners and sole proprietors. In particular, processes such as organizing receipts, allocating expenses, and creating tax return documents and submitting them online require specialized knowledge and require a great deal of time and effort. This prevents them from focusing on running their business, resulting in a decline in work efficiency. The present invention aims to significantly reduce the burden of filing tax returns and provide a method for doing so more efficiently.

[0136] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0137] In this invention, the server includes: means for users to upload receipt images and payment data from their terminals to the server; means for the server to format the received data and send it to an AI engine; means for the AI ​​engine to analyze the data and allocate expenses using image recognition and natural language processing; means for the server to generate tax return documents based on the allocated data; means for the server to submit the generated tax return documents to an online system; means for store managers to photograph and upload daily expense receipts using a smartphone; and means for linking with payment terminals and accounting software to automatically import income and expenditure data. This automates the tax return process and significantly reduces the burden on users.

[0138] A "receipt" is a document issued to prove the amount paid for the purchase of goods or the use of services.

[0139] An "image" is a digital representation of visual information, typically obtained with a camera or scanner.

[0140] "Payment Data" refers to transaction information relating to the purchase of goods or the use of services, including details such as payment amount, date and time, and transaction partner.

[0141] "User" refers to a person or business entity that uses this system and is responsible for uploading receipts and payment data to the server.

[0142] "Terminal" refers to a digital device such as a smartphone, tablet, or PC, which a user uses to operate and upload data.

[0143] A "server" refers to a computer system that receives and processes data sent from users and provides various services.

[0144] An "AI engine" is software or a system that uses artificial intelligence technology to perform data analysis and classification tasks.

[0145] "Image recognition technology" is a general term for algorithms and technologies used to analyze digital images and extract specific patterns and information.

[0146] "Natural language processing technology" is a general term for algorithms and technologies for understanding text data and analyzing its meaning.

[0147] "Expenses" are expenditures necessary for business activities and are usually tax-deductible.

[0148] A "tax return" is a document that must be submitted to the tax office or other government agency, detailing income and expenses.

[0149] "Online system" refers to services and platforms that can be used via the Internet.

[0150] A "payment terminal" is a device for accepting electronic payments such as credit cards and debit cards.

[0151] "Accounting software" is a computer program used to record, manage, and analyze financial data.

[0152] "Income and expenditure data" refers to data that includes information on income and expenditure in business activities.

[0153] "Photography" refers to the act of capturing an image using a camera, smartphone, etc.

[0154] "Ingestion" refers to the process of receiving data from outside and integrating it into internal systems.

[0155] MODE FOR CARRYING OUT THE INVENTION

[0156] The present invention relates to a system that automates the process of filing tax returns in order to reduce the burden on store managers and sole proprietors. This system mainly consists of the following components:

[0157] 1. Upload your data

[0158] Users upload receipt images and payment data to the server using a device such as a smartphone. The device can be a smartphone, tablet, or PC with a camera function.

[0159] 2. Formatting and sending data

[0160] The server receives receipts and payment data sent by users, converts and formats the data into an appropriate format, and sends the formatted data to an artificial intelligence (AI) engine.

[0161] 3. Data analysis using an AI engine

[0162] The AI ​​engine uses image recognition and natural language processing technologies to analyze the received data. This analysis extracts information such as transaction date and time, amount, and transaction details from receipt images and payment data, and automatically classifies them into appropriate expense categories. The AI ​​model used can be, for example, the Transformers library.

[0163] 4. Generate tax return documents

[0164] The server automatically generates tax return documents based on the data categorized by the AI ​​engine, calculating the totals of income, expenditures, and expenses, and accurately filling in the necessary information.

[0165] 5. Online submission

[0166] The server submits the generated tax return documents to an online system (e.g., the e-Tax system), eliminating the need for the user to manually download and mail the documents.

[0167] Hardware and software used

[0168] This system uses the following hardware and software:

[0169] Hardware: smartphones, tablets, computers, payment terminals

[0170] Software: Server-side applications (e.g., Python), accounting software APIs (e.g., Moneybook, freee), natural language processing libraries (e.g., Transformers), PDF generation libraries (e.g., FPDF)

[0171] Specific examples

[0172] For example, if a user receives a receipt for "5,000 yen in transportation expenses on October 1, 2023," they can take a photo of the receipt with their smartphone and upload it to the server using a dedicated app. The server then formats the received image data and sends it to the AI ​​engine. The AI ​​engine then extracts text information such as "date," "amount," and "transportation expenses" from the image and classifies it as transportation expenses. The server then generates tax return documents based on this information and submits them to the e-Tax system.

[0173] Prompt Sentence Examples

[0174] When parsing the receipt text, you can use prompt statements such as:

[0175] The following text is the content of the receipt. Please indicate which expense category it falls under:

[0176] October 1, 2023 5,000 yen Transportation fee

[0177] As a result, this system will significantly reduce the burden on store managers and sole proprietors and enable efficient tax return filing.

[0178] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0179] Step 1:

[0180] The user uses the terminal to upload the receipt image and payment data to the server.

[0181] Input: Receipt images and transaction data taken and saved by the user.

[0182] Specific operations: Tap the "upload button" on your smartphone or computer, select images or data, and send them.

[0183] Output: Raw data uploaded to the server.

[0184] Step 2:

[0185] The server formats the data it receives.

[0186] Input: Uploaded raw data (images and transaction information).

[0187] Specific operation: The server removes noise from the image data, extracts text information, and formats the transaction information accordingly.

[0188] Output: Data formatted in a way that is suitable for an AI engine.

[0189] Step 3:

[0190] The server sends the formatted data to the AI ​​engine.

[0191] Input: Formatted data.

[0192] Specific operation: The server sends formatted text and image data to the AI ​​engine via API.

[0193] Output: Data sent to the AI ​​engine for analysis.

[0194] Step 4:

[0195] The AI ​​engine uses image recognition and natural language processing technologies to analyze the data and allocate expenses.

[0196] Input: The data sent from the server.

[0197] How it works: The AI ​​engine uses OCR technology to extract text, identifies dates, amounts, and categories using specific algorithms, and automatically classifies expenses using natural language processing.

[0198] Output: Categorised expense data.

[0199] Step 5:

[0200] The server generates tax return documents based on the allocated data.

[0201] Input: Data classified by the AI ​​engine (expense item data).

[0202] Specific operation: The server uses a PDF generation library (e.g. FPDF) to calculate the total income, expenses, and costs, and generate the tax return documents.

[0203] Output: Generated tax return PDF.

[0204] Step 6:

[0205] The server submits the generated tax return to the online system.

[0206] Input: Generated tax return PDF.

[0207] Specific operation: The server logs in to an online filing platform such as the e-Tax system and submits the generated PDF.

[0208] Output: Notification of submission completion and results.

[0209] Step 7:

[0210] The user receives a notification and confirms that the declaration has been completed.

[0211] Input: Submission completion notification from the server.

[0212] Specific operations: The user checks the notification on their smartphone or computer and operates the confirmation button for completing the declaration.

[0213] Output: A confirmation that the tax return is complete.

[0214] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0215] This invention is a system that significantly simplifies and streamlines the preparation of tax return documents, and further optimizes the user experience by combining it with an emotion engine that recognizes the user's emotions. This system significantly reduces the burden on sole proprietors by automating the complex process of allocating expenses, generating tax return documents, and even submitting them online.

[0216] Program processing overview

[0217] The program of this system has the following functions:

[0218] 1. The user uploads an image of the receipt and payment data.

[0219] Users use their smartphones or PCs to upload images of receipts and payment data to the server, using a dedicated application or web browser.

[0220] 2. The server receives the data, formats it, and sends it to the AI ​​engine

[0221] The server receives image data and payment data sent by the user, converts it into an appropriate format, and sends the formatted data to the AI ​​engine.

[0222] 3. The AI ​​engine analyzes the data and allocates expenses

[0223] The AI ​​engine uses image recognition and natural language processing technologies to extract necessary information (such as date, amount, and expense type) from receipt images and payment data. Based on the extracted information, it automatically allocates each expense to the appropriate expense category.

[0224] 4. The server generates tax return documents based on the sorted data.

[0225] The server stores the expense information sent from the AI ​​engine in a database and automatically generates tax return documents based on that information, calculating the totals of income, expenditures, and expenses and accurately filling out the information required for the return.

[0226] 5. Emotion engine recognizes user emotions

[0227] The server collects and analyzes the user's facial expressions and voice data through the emotion engine to recognize the user's emotions. The emotion engine determines whether the user is feeling stressed or relaxed, and uses this information to improve the user experience.

[0228] 6. The server submits the generated tax return to the online system.

[0229] The server then processes the generated tax return documents for submission to the e-Tax system, eliminating the need for users to download and mail the documents.

[0230] Specific examples

[0231] 1. The user takes a photo of the receipt with their smartphone and uploads it to the server using a dedicated app.

[0232] For example, a user takes a taxi and receives a receipt for the fare of 5,000 yen. The user takes a photo of the receipt with the smartphone camera and uploads it to the server via a dedicated app.

[0233] 2. The server formats the received image data and sends it to the AI ​​engine.

[0234] When the server receives the receipt image, it removes noise from the image and formats it so that text information can be easily extracted. The formatted image data is then sent to the AI ​​engine.

[0235] 3. The AI ​​engine extracts the necessary information from the receipt image and determines the expense item

[0236] The AI ​​engine uses image recognition technology to extract text information such as "October 1, 2023," "5,000 yen," and "transportation expenses." Natural language processing technology then categorizes this information as transportation expenses.

[0237] 4. The server automatically generates tax return documents

[0238] The server retrieves the information about the 5,000 yen allocated as transportation expenses from the AI ​​engine and automatically enters it into the appropriate field on the tax return. This calculates the total of income, expenditures, and expenses, and generates an accurate tax return.

[0239] 5. Emotion engine analyzes user emotions and optimizes the user experience

[0240] The emotion engine analyzes the user's facial expressions and tone of voice, and if the user is feeling stressed, it provides supportive messages to alleviate that stress. For example, it displays positive messages such as, "Your tax return preparation is going well. You're almost there!"

[0241] 6. The server submits the generated tax return to the online system.

[0242] The server logs into the e-Tax system and submits the generated tax return. Once submission is complete, the user is notified and can complete the procedure.

[0243] This system allows users to automate the entire tax return process by simply taking and uploading images of receipts. The introduction of an emotion engine also improves the user experience, making the tax return process less stressful.

[0244] The processing flow will be explained below.

[0245] Step 1:

[0246] The user opens the dedicated application using a smartphone or PC. After taking a picture of the receipt or importing the payment data, the user clicks the "Upload" button in the application. This operation causes the device to upload the receipt image and payment data to the server.

[0247] Step 2:

[0248] The server receives the image data and payment data sent by the user, and performs preprocessing (e.g., noise removal, resolution adjustment) on the received data to maintain the image resolution at an appropriate level.

[0249] Step 3:

[0250] The server sends the preprocessed image data to the AI ​​engine, where it is converted into an appropriate format and formatted for efficient analysis.

[0251] Step 4:

[0252] The AI ​​engine uses image recognition technology to extract text information from receipt images, such as date, amount, and category (transportation expenses, entertainment expenses, etc.) using a proprietary algorithm.

[0253] Step 5:

[0254] The AI ​​engine uses natural language processing technology to analyze the extracted text information and categorize it into the appropriate expense category. Specifically, it categorizes keywords such as "taxi" and "fare" as transportation expenses.

[0255] Step 6:

[0256] The server stores the classification information sent from the AI ​​engine in a database, which accumulates the allocated expense information and makes it available for subsequent processing.

[0257] Step 7:

[0258] The server automatically generates tax return documents based on the expense classification information retrieved from the database. Specifically, it inputs appropriate data into the income, expenditure, and expense information fields according to the tax return template, and calculates the total amount.

[0259] Step 8:

[0260] The emotion engine analyzes the user's facial expressions and voice data to recognize their emotions. It does this by analyzing data collected through the camera and microphone while the user is taking pictures or entering data.

[0261] Step 9:

[0262] The emotion engine provides feedback to the user based on the emotions it recognizes. For example, if the user is feeling stressed, the system will display a message saying, "We recommend you take a short break."

[0263] Step 10:

[0264] The server submits the generated tax return documents to the e-Tax system. At this time, the server uses the authentication information for logging in to the e-Tax system to carry out the appropriate upload procedure.

[0265] Step 11:

[0266] The server receives the results of the tax return submission and confirms that the submission was successful. It then sends a notification of the completion of the submission to the user.

[0267] This detailed processing step allows users to automate the entire tax return process by simply taking and uploading images of receipts. The introduction of an emotion engine also improves the user experience, making the tax return process less stressful.

[0268] Example 2

[0269] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0270] The traditional tax return preparation process is complex and time-consuming, placing a significant burden on many self-employed individuals. Additionally, sorting receipts and categorizing expenses is a tedious task that is prone to errors. Furthermore, the frustration users feel while completing the return is also a problem. To address this situation, there is a need for more efficient tax return preparation and an improved user experience.

[0271] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0272] In this invention, the server includes: means for a user to upload receipt images and payment data from a terminal to the server; means for the server to format the received data and send it to an AI engine; means for the AI ​​engine to analyze the data using image recognition technology and natural language processing technology and allocate expenses; means for the server to generate tax return documents based on the allocated data; means for the server to submit the generated tax return documents to an online system; means for the server to recognize the user's emotions through an emotion engine; and means for the emotion engine to analyze the user's facial expressions and voice data and determine the user's emotions. This realizes automation and efficiency in the preparation of tax return documents, reducing user stress and improving the experience.

[0273] A "receipt" is a document issued when a sale or service is provided that lists the details of the transaction.

[0274] "Payment Data" means data that includes transaction information related to the purchase of goods or payment for services.

[0275] "User" refers to individuals and businesses who use this system to file tax returns.

[0276] "Terminal" refers to the electronic device (smartphone, PC, etc.) that a user uses to operate the system.

[0277] The "server" is the central control unit that handles the entire processing of this system, and is a device that receives data from users, formats the data, connects with the AI ​​engine, and generates and submits tax return documents.

[0278] "Formatting" is the process of converting the format of received data, removing redundant parts of the data, and extracting necessary information to make it suitable for analysis.

[0279] An "AI engine" is a system that uses artificial intelligence technology to analyze data, and applies image recognition technology and natural language processing technology.

[0280] "Image recognition technology" is a technology that extracts specific information such as characters and objects from image data.

[0281] "Natural language processing technology" is a technology for analyzing the content of text data and understanding its meaning.

[0282] "Expense items" are categories for classifying types of expenses, and examples include "transportation expenses" and "entertainment expenses."

[0283] "Declaration documents" are various documents that must be submitted when filing a tax return, and contain details of income, expenditures, and expenses.

[0284] An "online system" is a system in which tax return procedures are carried out via the Internet, and examples include e-Tax.

[0285] An "emotion engine" is a system for analyzing a user's emotions, and determines the user's emotional state using facial expressions and voice data.

[0286] "Facial expression data" is data that includes information about the user's facial expression.

[0287] "Voice data" is data that includes information about the user's vocalizations.

[0288] This invention is a system that significantly simplifies and streamlines the preparation of tax return documents, and further optimizes the user experience by combining it with an emotion engine that recognizes the user's emotions. This system significantly reduces the burden on sole proprietors by automating the complex process of allocating expenses, generating tax return documents, and even submitting them online.

[0289] Specific program details

[0290] The system's program works as follows: First, the user uploads receipt images and payment data to the server using a smartphone or PC. Using a dedicated application or web browser, the user selects the data using the file selection button or drag-and-drop function, and then presses the send button.

[0291] The server receives image data and payment data sent by users. After receiving the data, it uses image processing algorithms to remove noise and clean it, and then formats it into a format that makes it easy to extract text. This formatted data is then sent to the AI ​​engine. Specific software used by the server includes OpenCV and Tesseract.

[0292] The AI ​​engine uses image recognition and natural language processing to extract necessary information from receipts, such as dates, amounts, and expense types. It then automatically categorizes each expense into the appropriate category based on the extracted information. Specific image recognition technologies include TensorFlow and PyTorch.

[0293] The server then stores the expense information sent from the AI ​​engine in a database and automatically generates tax return documents based on this information. The server calculates the totals of income, expenditures, and expenses, and generates documents that accurately list the information required for filing. An online platform using SaaS (Software as a Service) is used.

[0294] Through the emotion engine, the server collects and analyzes the user's facial expressions and voice data to recognize the user's emotions. The emotion engine determines whether the user is feeling stressed or relaxed, and uses this information to improve the user experience. Specific applications include the use of OpenPose and Microsoft Azure's Emotion API.

[0295] The server then finally processes the generated tax return documents for submission to the e-Tax system, eliminating the need for users to download and mail the documents.

[0296] Examples of concrete examples and prompts

[0297] 1. The user takes a photo of the receipt with their smartphone and uploads it to the server using a dedicated app:

[0298] For example, a user takes a taxi and receives a receipt for the fare of 5,000 yen. The user takes a photo of the receipt with the smartphone camera and uploads it to the server via a dedicated app.

[0299] 2. The server formats the received image data and sends it to the AI ​​engine:

[0300] When the server receives the receipt image, it removes noise from the image and formats it so that text information can be easily extracted. The formatted image data is then sent to the AI ​​engine.

[0301] 3. The AI ​​engine extracts the necessary information from the receipt image and determines the expense item:

[0302] The AI ​​engine uses image recognition technology to extract text information such as "October 1, 2023," "5,000 yen," and "transportation expenses." Natural language processing technology then categorizes this information as transportation expenses.

[0303] 4. The server automatically generates the tax return documents:

[0304] The server retrieves the information about the 5,000 yen allocated as transportation expenses from the AI ​​engine and automatically enters it into the appropriate field on the tax return. This calculates the total of income, expenditures, and expenses, and generates an accurate tax return.

[0305] 5. Emotion engine analyzes user emotions and optimizes user experience:

[0306] The emotion engine analyzes the user's facial expressions and tone of voice, and if the user is feeling stressed, it provides supportive messages to alleviate that stress. For example, it displays positive messages such as, "Your tax return preparation is going well. You're almost there!"

[0307] 6. The server submits the generated tax return to the online system:

[0308] The server logs into the e-Tax system and submits the generated tax return. Once submission is complete, the user is notified and can complete the procedure.

[0309] Prompt Sentence Examples

[0310] "An image of a taxi receipt taken with a smartphone is uploaded to a server. The image contains information such as 'October 1, 2023, 5,000 yen travel expenses'. Please explain how an AI engine can extract this information and allocate it to the appropriate expense category."

[0311] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0312] Step 1:

[0313] The user uploads an image of the receipt and payment data using the terminal.

[0314] Specific operation: The user takes a photo or scans a receipt or payment data using a smartphone or PC, then selects the captured image file or payment data through a dedicated application or web browser and clicks the upload button.

[0315] Input: Receipt image and payment data (e.g., taxi receipt image)

[0316] Output: Image data and payment data sent to the server

[0317] Step 2:

[0318] The server receives the data, formats it, and sends it to the AI ​​engine.

[0319] Specific operation: The server receives the receipt image sent by the user. It performs noise removal and image preprocessing on the received image data, formatting it so that text can be easily extracted. It then sends the formatted data (e.g., the noise-removed image and cleaned text data) to the AI ​​engine.

[0320] Input: Image data and payment data sent to the server

[0321] Output: Formatted data sent to the AI ​​engine

[0322] Step 3:

[0323] An AI engine analyzes the data and allocates expenses.

[0324] How it works: The AI ​​engine uses image recognition technology to extract dates, amounts, types of expenses, etc. from the received image data. It then uses natural language processing to understand the meaning of the extracted information and allocate it to the appropriate expense category (for example, "transportation expenses" or "entertainment expenses").

[0325] Input: Formatted data (e.g., a receipt image with noise removed)

[0326] Output: Allocated expense information (e.g., "October 1, 2023", "5,000 yen", "Transportation expenses")

[0327] Step 4:

[0328] The server generates tax return documents based on the allocated data.

[0329] Specific operation: The server saves the expense information sent from the AI ​​engine in a database and aggregates it. It then calculates the totals of income, expenses, and costs and automatically generates tax return documents. The generated tax return documents accurately contain the information necessary for filing (e.g., income, expenses, total amount).

[0330] Input: Allocated expense information

[0331] Output: Auto-generated tax return documents

[0332] Step 5:

[0333] The server recognizes the user's emotions through an emotion engine.

[0334] Specific operation: The server collects the user's facial expression and voice data and sends it to the emotion engine. The emotion engine analyzes the data and determines whether the user is feeling stressed or relaxed. If the user is feeling particularly stressed, it generates an appropriate support message and displays it to the user.

[0335] Input: User's facial expression data and voice data

[0336] Output: Emotion recognition results and supportive messages (e.g., "Your tax return preparation is going well. You're almost there!")

[0337] Step 6:

[0338] The server submits the generated tax return to the online system.

[0339] Specific operation: The server automatically sends the generated tax return documents to the e-Tax system. After sending, the server sends a success confirmation notice to the user, which saves the user the trouble of downloading and mailing the documents.

[0340] Input: Auto-generated tax return document

[0341] Output: Notification of completion of submission to the online system (e-Tax)

[0342] (Application example 2)

[0343] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0344] The present invention aims to reduce the burden on users and improve efficiency in expense management and tax return preparation. It also aims to improve the user experience throughout the entire tax return process by analyzing users' emotions and providing support to reduce stress. In particular, in environments where transactions are frequent, such as brick-and-mortar stores, it is necessary for store clerks to smoothly perform their daily tasks while managing expenses without stress.

[0345] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0346] In this invention, the server includes: a means for users to upload receipt images and payment data from their terminals to the server; a means for the server to format the received data and send it to an AI engine; a means for the AI ​​engine to analyze the data and allocate expenses using image recognition and natural language processing; a means for the server to generate tax return documents based on the allocated data; a means for the server to submit the generated tax return documents to an online system; and an emotion engine that analyzes the user's emotions and displays messages based on the emotions. This allows users to automate the entire tax return process simply by taking and uploading images of their receipts. The introduction of the emotion engine also improves the user experience, making the tax return process less stressful.

[0347] A "receipt image" is an image showing purchase details or a record of service usage that is taken by a user to prove expenses.

[0348] "Payment data" is digital data that includes a transaction record that occurs when a user makes a payment.

[0349] A "terminal" is an electronic device, such as a smartphone or computer, that a user operates to input and transmit data.

[0350] A "server" is a high-performance computer system for processing and storing data over a network.

[0351] "Formatting" refers to converting the data received by the server into an appropriate format for efficient processing.

[0352] An "AI engine" is a software system that uses artificial intelligence technology to analyze data and automatically perform specific tasks.

[0353] "Image recognition technology" is a technology that uses computer vision to extract specific information from images.

[0354] "Natural language processing technology" is a technology for analyzing meaning from text data and understanding and generating human language.

[0355] "Allocating expenses" means classifying expenses into specific categories.

[0356] A "tax return" is an official document that contains information required for filing a tax return.

[0357] An "online system" is a system for sending and receiving data and providing services using the Internet.

[0358] An "emotion engine" is a software system that analyzes a user's emotions and generates an appropriate response based on the results.

[0359] A "message" is text information that the emotion engine provides to the user based on the analysis results.

[0360] The present invention relates to a system that automatically processes receipt images and payment data to create tax return documents. Furthermore, by including an emotion engine that analyzes user emotions, the system aims to improve the user experience.

[0361] The system is configured as follows: First, the user uploads an image of the receipt and payment data to the server using a device such as a smartphone or PC. Specifically, a dedicated application or web browser is used. At this time, the user can take a photo of the receipt with a camera and send the image data.

[0362] The server receives the uploaded data, removes noise from the image data, and formats it in a way that makes it easier to extract text information. An image processing service like Google Cloud Vision API is a good choice for this purpose. The formatted data is then sent to the AI ​​engine.

[0363] The AI ​​engine analyzes data using image recognition and natural language processing technologies. Specifically, it uses deep learning libraries such as Keras to extract information such as date, amount, and expense item from receipt images and automatically classify them into appropriate categories.

[0364] The data is then sent back to the server, which then generates tax returns that accurately show income, expenditures, and total expenses. Again, a programming language such as Python is used to interface with a database management system to perform the necessary calculations.

[0365] Furthermore, the system is equipped with an emotion engine that analyzes the user's emotions. The emotion engine analyzes the facial expressions and voice of the store clerk to determine the level of stress. Facial recognition technology such as OpenCV and voice analysis technology are used for emotion analysis. If the user feels stressed, the emotion engine will support them by displaying encouraging messages such as, "Your declaration preparation is progressing smoothly. You're almost there!"

[0366] The completed tax return is then submitted to an online system (e.g., the e-Tax system) by the server, saving the user the trouble of manually submitting the return.

[0367] As a concrete example, consider the case of a convenience store clerk managing expenses while performing their daily duties. When making a transaction at the register, the clerk takes a photo of the receipt with their smartphone and uploads the data to a server using a dedicated app. The server then formats the image data, and an AI engine analyzes the data and allocates it to the appropriate expense category. This automatically generates tax return documents, which are ultimately submitted to the online system. The emotion engine also detects the clerk's stress and provides encouraging messages as needed, thereby reducing the clerk's stress.

[0368] Example prompt sentence:

[0369] "Just take a photo of your receipt with your smartphone and upload it using the app. We'll provide you with instant analysis and, if needed, a message of encouragement."

[0370] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0371] Step 1:

[0372] The user takes a photo of the receipt and payment data on the device and uploads it to the server using a dedicated application or a web browser. The image data of the receipt is used as input, and a data file is obtained as output, which is sent to the server.

[0373] Step 2:

[0374] The server formats the received image data. First, it uses the Google Cloud Vision API to remove noise from the image and convert it into a format that makes it easier to extract text information. It receives image data as input and obtains formatted image data as output.

[0375] Step 3:

[0376] The server sends the formatted image data to the AI ​​engine, which uses the formatted image data as input and obtains the data sent to the AI ​​engine as output.

[0377] Step 4:

[0378] The AI ​​engine analyzes the received image data. It uses image recognition technology to extract text information such as dates, amounts, and items, and then uses natural language processing technology to classify this information into appropriate expense categories. It receives formatted image data and extracted text information as input, and obtains classified expense information as output.

[0379] Step 5:

[0380] The server generates tax return documents based on the expense information returned by the AI ​​engine. The server calculates the totals of income, expenditures, and expenses through a database management system and compiles them into official tax return documents. It receives classified expense information as input and obtains the generated tax return documents as output.

[0381] Step 6:

[0382] The emotion engine analyzes the user's emotions. Facial expression and voice data collected on the device is sent to the server, which uses OpenCV and voice analysis technology to determine the level of stress. The engine receives the user's facial expression and voice data as input and obtains the emotion analysis results as output.

[0383] Step 7:

[0384] The server displays an appropriate message on the terminal based on the emotion analysis results. If the user is feeling stressed, an encouraging message is displayed to support the progress of the task. The emotion analysis results are received as input, and the message to be displayed is obtained as output.

[0385] Step 8:

[0386] The server submits the generated tax return to the online system, sends the tax return to an online platform such as the e-Tax system, and notifies the user of its completion, receiving the generated tax return as input and receiving a notification of completion of submission as output.

[0387] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0388] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0389] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0390] [Second embodiment]

[0391] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0392] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0393] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0394] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0395] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0396] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0397] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0398] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0399] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0400] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0401] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0402] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0403] This invention is a system that greatly simplifies and streamlines the preparation of tax return documents. This system significantly reduces the burden on sole proprietors by automating the complicated process of allocating expenses, generating tax return documents, and even submitting them online.

[0404] Program processing overview

[0405] The program of this system has the following functions:

[0406] 1. The user uploads an image of the receipt and payment data.

[0407] Users use their smartphones or PCs to upload images of receipts and payment data to the server, using a dedicated application or web browser.

[0408] 2. The server receives the data, formats it, and sends it to the AI ​​engine

[0409] The server receives image data and payment data sent by the user, converts it into an appropriate format, and sends the formatted data to the AI ​​engine.

[0410] 3. The AI ​​engine analyzes the data and allocates expenses

[0411] The AI ​​engine uses image recognition and natural language processing technologies to extract necessary information (such as date, amount, and expense type) from receipt images and payment data. Based on the extracted information, it automatically allocates each expense to the appropriate expense category.

[0412] 4. The server generates tax return documents based on the sorted data.

[0413] The server stores the expense information sent from the AI ​​engine in a database and automatically generates tax return documents based on that information, calculating the totals of income, expenditures, and expenses and accurately filling out the information required for the return.

[0414] 5. The server submits the generated tax return to the online system.

[0415] The server then processes the generated tax return documents for submission to an online system (such as e-Tax), eliminating the need for users to download and mail the documents.

[0416] Specific examples

[0417] 1. The user takes a photo of the receipt with their smartphone and uploads it to the server using a dedicated app.

[0418] For example, a user takes a taxi and receives a receipt for the fare of 5,000 yen. The user takes a photo of the receipt with the smartphone camera and uploads it to the server via a dedicated app.

[0419] 2. The server formats the received image data and sends it to the AI ​​engine.

[0420] When the server receives the receipt image, it removes noise from the image and formats it so that text information can be easily extracted. The formatted image data is then sent to the AI ​​engine.

[0421] 3. The AI ​​engine extracts the necessary information from the receipt image and determines the expense item

[0422] The AI ​​engine uses image recognition technology to extract text information such as "October 1, 2023," "5,000 yen," and "transportation expenses." Natural language processing technology then categorizes this information as transportation expenses.

[0423] 4. The server automatically generates tax return documents

[0424] The server retrieves the information about the 5,000 yen allocated as transportation expenses from the AI ​​engine and automatically enters it into the appropriate field on the tax return. This calculates the total of income, expenditures, and expenses, and generates an accurate tax return.

[0425] 5. The server submits the generated tax return to the online system.

[0426] The server logs into the e-Tax system and submits the generated tax return. The user is notified when the submission is complete.

[0427] This system allows users to carry out the complicated tax return procedures almost without thinking about them, enabling smooth data management and submission of tax return documents, allowing users to focus more on running their business.

[0428] The processing flow will be explained below.

[0429] Step 1:

[0430] The user opens the dedicated application using a smartphone or PC. After taking a picture of the receipt or importing the payment data, the user clicks the "Upload" button in the application. This operation causes the device to upload the receipt image and payment data to the server.

[0431] Step 2:

[0432] The server receives the image data and payment data sent by the user, and performs preprocessing (e.g., noise removal, resolution adjustment) on the received data to maintain the image resolution at an appropriate level.

[0433] Step 3:

[0434] The server sends the preprocessed image data to the AI ​​engine, where it is converted into an appropriate format and formatted for efficient analysis.

[0435] Step 4:

[0436] The AI ​​engine uses image recognition technology to extract text information from receipt images, such as date, amount, and category (transportation expenses, entertainment expenses, etc.) using a unique algorithm.

[0437] Step 5:

[0438] The AI ​​engine uses natural language processing technology to analyze the extracted text information and categorize it into the appropriate expense category. Specifically, it categorizes keywords such as "taxi" and "fare" as transportation expenses.

[0439] Step 6:

[0440] The server stores the classification information sent from the AI ​​engine in a database, which accumulates the allocated expense information and makes it available for subsequent processing.

[0441] Step 7:

[0442] The server automatically generates tax return documents based on the expense classification information retrieved from the database. Specifically, it inputs appropriate data into the income, expenditure, and expense information fields according to the tax return template, and calculates the total amount.

[0443] Step 8:

[0444] The server submits the generated tax return documents to the e-Tax system. At this time, the server uses the authentication information for logging in to the e-Tax system to carry out the appropriate upload procedure.

[0445] Step 9:

[0446] The server receives the results of the tax return submission and confirms that the submission was successful. It then sends a notification of the completion of the submission to the user.

[0447] This detailed processing step allows users to automate the entire tax return process by simply taking and uploading an image of their receipt.

[0448] Example 1

[0449] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0450] The complicated and time-consuming process of preparing tax returns by hand for sole proprietors has led to problems of human error and reduced work efficiency. Furthermore, it is a significant burden to individually categorize receipts, prepare tax return forms, and then submit them online.

[0451] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0452] In this invention, the server includes a means for a user to send business data from an information input device to a processing device, a means for converting and formatting the received data into an appropriate format by the processing device, and a means for sending the formatted data by the processing device to an analysis device. This automates the preparation of documents required for filing tax returns and the allocation of data to expense items, enabling online submission of tax returns.

[0453] An "information input device" is a device that allows a user to input business data and transmit it to a server, and includes smartphones, personal computers, and the like.

[0454] A "processor" is a server or computer system that receives data sent by a user and converts and formats it into an appropriate format.

[0455] "Business data" refers to various transaction information and receipt image data that are generated by users and sent to the server.

[0456] An "analysis device" is an AI engine or dedicated hardware that analyzes the formatted data sent from the processing device and classifies expenses.

[0457] "Image processing technology" refers to technology for extracting and processing necessary information from digital images, and includes noise removal and text recognition.

[0458] "Automatic analysis technology" is a technology that allows an AI engine to analyze data and classify information using natural language processing technology.

[0459] "Report" refers to tax returns and other business reports generated based on the allocated data.

[0460] "Online services" refers to various services provided via the Internet, including tax return submission systems such as e-Tax.

[0461] This invention is a system for streamlining the tax return process, where users submit business data and the data is analyzed, classified, and generated using a server and an AI engine. The system aims to automate many manual tasks and reduce the burden on users.

[0462] System configuration

[0463] The system consists of the following main components:

[0464] 1. Information input device

[0465] A device that allows users to input business data and send it to a server. Specifically, it includes smartphones (iOS or Android) and personal computers (Windows or Mac). Dedicated applications and web browsers (Chrome or Firefox) are used.

[0466] 2. Processing equipment

[0467] It is a server or computer system that receives data sent by users and converts and formats it into an appropriate format. It uses a Linux-based server and an image processing library (OpenCV). It also includes a database (MySQL or PostgreSQL).

[0468] 3. Analysis device

[0469] This is an AI engine and dedicated hardware that analyzes the formatted data sent from the processing device and classifies expenses. The AI ​​engine uses deep learning frameworks such as TensorFlow and PyTorch and an OCR library (Tesseract), and performs analysis using natural language processing libraries (spaCy and NLTK).

[0470] 4. Online Services

[0471] These are various services provided via the Internet, including tax return submission systems such as e-Tax.

[0472] Specific examples

[0473] Next, a concrete example of the operation of this system will be given.

[0474] 1. The user takes a photo of the receipt with their smartphone and uploads it to the server using a dedicated app.

[0475] Suppose a user takes a taxi and receives a receipt for the fare of 5,000 yen. The user takes a photo of this receipt with the smartphone camera and uploads the image to the server using a dedicated app. An example of a prompt sentence is, "Please take a photo of the restaurant receipt and upload it using the dedicated app. Please make sure that the amount and date are clearly visible."

[0476] 2. The server formats the received image data and sends it to the AI ​​engine.

[0477] The server uses OpenCV to remove noise from the images it receives and formats them so that text information can be easily extracted. The formatted image data is then sent to the AI ​​engine.

[0478] 3. The AI ​​engine extracts the necessary information from the receipt image and determines the expense item

[0479] The AI ​​engine uses Tesseract to extract information such as "October 1, 2023," "5,000 yen," and "transportation expenses," and then uses natural language processing technology (spaCy) to classify this as transportation expenses.

[0480] 4. The server automatically generates tax return documents

[0481] The server uses JasperReports to generate the necessary tax return documents based on the classification information from the AI ​​engine, and calculates the totals of income, expenses, and costs to accurately reflect them on the tax return documents.

[0482] 5. The server submits the generated tax return to the online system.

[0483] The server logs into the e-Tax system through the API and submits the generated tax return, which notifies the user and eliminates the need to download and mail the documents.

[0484] This system allows users to simplify the cumbersome manual process of filing tax returns, improving work efficiency.

[0485] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0486] Step 1:

[0487] The user uses an information input device (smartphone or PC) to send business data (receipt images and payment data) to the server. Specifically, the user uses a dedicated application or web browser to take a photo of the taxi receipt with the smartphone camera and upload the image. The input is image data taken by the user, which is then sent to the server.

[0488] Step 2:

[0489] The server converts and formats the received image data into an appropriate format. Specifically, the server uses an image processing library (OpenCV) to remove noise and format the data so that text information can be easily extracted. This processing improves the accuracy of reading the image data. The input is the raw image data sent by the user, and the output is formatted image data.

[0490] Step 3:

[0491] The server sends the formatted image data to the analysis device (AI engine). The server uses an API for data transmission to pass the formatted data to the AI ​​engine. The input is the formatted image data, and the output is the data sent to the analysis device.

[0492] Step 4:

[0493] The analysis device (AI engine) uses image processing technology (OCR) and automatic analysis technology to analyze the data and determine the expense items. Specifically, the AI ​​engine uses the OCR library (Tesseract) to extract text information such as "October 1, 2023" and "5,000 yen," and classifies it as "transportation expenses" using natural language processing technology (spaCy). The input is formatted image data, and the output is text information with classified expense items.

[0494] Step 5:

[0495] The server generates a report based on the expense classification data received from the analysis device. Specifically, the server uses JasperReports to calculate the totals of income, expenditures, and expenses, and automatically generates tax return documents. The input is text information with expense classifications, and the output is the automatically generated tax return documents.

[0496] Step 6:

[0497] The server submits the generated tax return to an online service (such as e-Tax). The server logs in to the e-Tax system via API and uploads the tax return. The input is the automatically generated tax return, and the output is a notification to the online service that the return has been submitted. This process notifies the user that the return has been submitted.

[0498] (Application example 1)

[0499] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0500] Currently, filing tax returns is a very complicated and time-consuming task for store owners and sole proprietors. In particular, processes such as organizing receipts, allocating expenses, and creating tax return documents and submitting them online require specialized knowledge and require a great deal of time and effort. This prevents them from focusing on running their business, resulting in a decline in work efficiency. The present invention aims to significantly reduce the burden of filing tax returns and provide a method for doing so more efficiently.

[0501] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0502] In this invention, the server includes: means for users to upload receipt images and payment data from their terminals to the server; means for the server to format the received data and send it to an AI engine; means for the AI ​​engine to analyze the data and allocate expenses using image recognition and natural language processing; means for the server to generate tax return documents based on the allocated data; means for the server to submit the generated tax return documents to an online system; means for store managers to photograph and upload daily expense receipts using a smartphone; and means for linking with payment terminals and accounting software to automatically import income and expenditure data. This automates the tax return process and significantly reduces the burden on users.

[0503] A "receipt" is a document issued to prove the amount paid for the purchase of goods or the use of services.

[0504] An "image" is a digital representation of visual information, typically obtained with a camera or scanner.

[0505] "Payment Data" refers to transaction information relating to the purchase of goods or the use of services, including details such as payment amount, date and time, and transaction partner.

[0506] "User" refers to a person or business entity that uses this system and is responsible for uploading receipts and payment data to the server.

[0507] "Terminal" refers to a digital device such as a smartphone, tablet, or PC, which a user uses to operate and upload data.

[0508] A "server" refers to a computer system that receives and processes data sent from users and provides various services.

[0509] An "AI engine" is software or a system that uses artificial intelligence technology to perform data analysis and classification tasks.

[0510] "Image recognition technology" is a general term for algorithms and technologies used to analyze digital images and extract specific patterns and information.

[0511] "Natural language processing technology" is a general term for algorithms and technologies for understanding text data and analyzing its meaning.

[0512] "Expenses" are expenditures necessary for business activities and are usually tax-deductible.

[0513] A "tax return" is a document that must be submitted to the tax office or other government agency, detailing income and expenses.

[0514] "Online system" refers to services and platforms that can be used via the Internet.

[0515] A "payment terminal" is a device for accepting electronic payments such as credit cards and debit cards.

[0516] "Accounting software" is a computer program used to record, manage, and analyze financial data.

[0517] "Income and expenditure data" refers to data that includes information on income and expenditure in business activities.

[0518] "Photography" refers to the act of capturing an image using a camera, smartphone, etc.

[0519] "Ingestion" refers to the process of receiving data from outside and integrating it into internal systems.

[0520] MODE FOR CARRYING OUT THE INVENTION

[0521] The present invention relates to a system that automates the process of filing tax returns in order to reduce the burden on store managers and sole proprietors. This system mainly consists of the following components:

[0522] 1. Upload your data

[0523] Users upload receipt images and payment data to the server using a device such as a smartphone. The device can be a smartphone, tablet, or PC with a camera function.

[0524] 2. Formatting and sending data

[0525] The server receives receipts and payment data sent by users, converts and formats the data into an appropriate format, and sends the formatted data to an artificial intelligence (AI) engine.

[0526] 3. Data analysis using an AI engine

[0527] The AI ​​engine uses image recognition and natural language processing technologies to analyze the received data. This analysis extracts information such as transaction date and time, amount, and transaction details from receipt images and payment data, and automatically classifies them into appropriate expense categories. The AI ​​model used can be, for example, the Transformers library.

[0528] 4. Generate tax return documents

[0529] The server automatically generates tax return documents based on the data categorized by the AI ​​engine, calculating the totals of income, expenditures, and expenses, and accurately filling in the necessary information.

[0530] 5. Online submission

[0531] The server submits the generated tax return documents to an online system (e.g., the e-Tax system), eliminating the need for the user to manually download and mail the documents.

[0532] Hardware and software used

[0533] This system uses the following hardware and software:

[0534] Hardware: smartphones, tablets, computers, payment terminals

[0535] Software: Server-side applications (e.g., Python), accounting software APIs (e.g., Moneybook, freee), natural language processing libraries (e.g., Transformers), PDF generation libraries (e.g., FPDF)

[0536] Specific examples

[0537] For example, if a user receives a receipt for "5,000 yen in transportation expenses on October 1, 2023," they can take a photo of the receipt with their smartphone and upload it to the server using a dedicated app. The server then formats the received image data and sends it to the AI ​​engine. The AI ​​engine then extracts text information such as "date," "amount," and "transportation expenses" from the image and classifies it as transportation expenses. The server then generates tax return documents based on this information and submits them to the e-Tax system.

[0538] Prompt Sentence Examples

[0539] When parsing the receipt text, you can use prompt statements such as:

[0540] The following text is the content of the receipt. Please indicate which expense category it falls under:

[0541] October 1, 2023 5,000 yen Transportation fee

[0542] As a result, this system will significantly reduce the burden on store managers and sole proprietors and enable efficient tax return filing.

[0543] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0544] Step 1:

[0545] The user uses the terminal to upload the receipt image and payment data to the server.

[0546] Input: Receipt images and transaction data taken and saved by the user.

[0547] Specific operations: Tap the "upload button" on your smartphone or computer, select images or data, and send them.

[0548] Output: Raw data uploaded to the server.

[0549] Step 2:

[0550] The server formats the data it receives.

[0551] Input: Uploaded raw data (images and transaction information).

[0552] Specific operation: The server removes noise from the image data, extracts text information, and formats the transaction information accordingly.

[0553] Output: Data formatted in a way that is suitable for an AI engine.

[0554] Step 3:

[0555] The server sends the formatted data to the AI ​​engine.

[0556] Input: Formatted data.

[0557] Specific operation: The server sends formatted text and image data to the AI ​​engine via API.

[0558] Output: Data sent to the AI ​​engine for analysis.

[0559] Step 4:

[0560] The AI ​​engine uses image recognition and natural language processing technologies to analyze the data and allocate expenses.

[0561] Input: The data sent from the server.

[0562] How it works: The AI ​​engine uses OCR technology to extract text, identifies dates, amounts, and categories using specific algorithms, and automatically classifies expenses using natural language processing.

[0563] Output: Categorised expense data.

[0564] Step 5:

[0565] The server generates tax return documents based on the allocated data.

[0566] Input: Data classified by the AI ​​engine (expense item data).

[0567] Specific operation: The server uses a PDF generation library (e.g. FPDF) to calculate the total income, expenses, and costs, and generate the tax return documents.

[0568] Output: Generated tax return PDF.

[0569] Step 6:

[0570] The server submits the generated tax return to the online system.

[0571] Input: Generated tax return PDF.

[0572] Specific operation: The server logs in to an online filing platform such as the e-Tax system and submits the generated PDF.

[0573] Output: Notification of submission completion and results.

[0574] Step 7:

[0575] The user receives a notification and confirms that the declaration has been completed.

[0576] Input: Submission completion notification from the server.

[0577] Specific operations: The user checks the notification on their smartphone or computer and operates the confirmation button for completing the declaration.

[0578] Output: A confirmation that the tax return is complete.

[0579] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0580] This invention is a system that significantly simplifies and streamlines the preparation of tax return documents, and further optimizes the user experience by combining it with an emotion engine that recognizes the user's emotions. This system significantly reduces the burden on sole proprietors by automating the complex process of allocating expenses, generating tax return documents, and even submitting them online.

[0581] Program processing overview

[0582] The program of this system has the following functions:

[0583] 1. The user uploads an image of the receipt and payment data.

[0584] Users use their smartphones or PCs to upload images of receipts and payment data to the server, using a dedicated application or web browser.

[0585] 2. The server receives the data, formats it, and sends it to the AI ​​engine

[0586] The server receives image data and payment data sent by the user, converts it into an appropriate format, and sends the formatted data to the AI ​​engine.

[0587] 3. The AI ​​engine analyzes the data and allocates expenses

[0588] The AI ​​engine uses image recognition and natural language processing technologies to extract necessary information (such as date, amount, and expense type) from receipt images and payment data. Based on the extracted information, it automatically allocates each expense to the appropriate expense category.

[0589] 4. The server generates tax return documents based on the sorted data.

[0590] The server stores the expense information sent from the AI ​​engine in a database and automatically generates tax return documents based on that information, calculating the totals of income, expenditures, and expenses and accurately filling out the information required for the return.

[0591] 5. Emotion engine recognizes user emotions

[0592] The server collects and analyzes the user's facial expressions and voice data through the emotion engine to recognize the user's emotions. The emotion engine determines whether the user is feeling stressed or relaxed, and uses this information to improve the user experience.

[0593] 6. The server submits the generated tax return to the online system.

[0594] The server then processes the generated tax return documents for submission to the e-Tax system, eliminating the need for users to download and mail the documents.

[0595] Specific examples

[0596] 1. The user takes a photo of the receipt with their smartphone and uploads it to the server using a dedicated app.

[0597] For example, a user takes a taxi and receives a receipt for the fare of 5,000 yen. The user takes a photo of the receipt with the smartphone camera and uploads it to the server via a dedicated app.

[0598] 2. The server formats the received image data and sends it to the AI ​​engine.

[0599] When the server receives the receipt image, it removes noise from the image and formats it so that text information can be easily extracted. The formatted image data is then sent to the AI ​​engine.

[0600] 3. The AI ​​engine extracts the necessary information from the receipt image and determines the expense item

[0601] The AI ​​engine uses image recognition technology to extract text information such as "October 1, 2023," "5,000 yen," and "transportation expenses." Natural language processing technology then categorizes this information as transportation expenses.

[0602] 4. The server automatically generates tax return documents

[0603] The server retrieves the information about the 5,000 yen allocated as transportation expenses from the AI ​​engine and automatically enters it into the appropriate field on the tax return. This calculates the total of income, expenditures, and expenses, and generates an accurate tax return.

[0604] 5. Emotion engine analyzes user emotions and optimizes the user experience

[0605] The emotion engine analyzes the user's facial expressions and tone of voice, and if the user is feeling stressed, it provides supportive messages to alleviate that stress. For example, it displays positive messages such as, "Your tax return preparation is going well. You're almost there!"

[0606] 6. The server submits the generated tax return to the online system.

[0607] The server logs into the e-Tax system and submits the generated tax return. Once submission is complete, the user is notified and can complete the procedure.

[0608] This system allows users to automate the entire tax return process by simply taking and uploading images of receipts. The introduction of an emotion engine also improves the user experience, making the tax return process less stressful.

[0609] The processing flow will be explained below.

[0610] Step 1:

[0611] The user opens the dedicated application using a smartphone or PC. After taking a picture of the receipt or importing the payment data, the user clicks the "Upload" button in the application. This operation causes the device to upload the receipt image and payment data to the server.

[0612] Step 2:

[0613] The server receives the image data and payment data sent by the user, and performs preprocessing (e.g., noise removal, resolution adjustment) on the received data to maintain the image resolution at an appropriate level.

[0614] Step 3:

[0615] The server sends the preprocessed image data to the AI ​​engine, where it is converted into an appropriate format and formatted for efficient analysis.

[0616] Step 4:

[0617] The AI ​​engine uses image recognition technology to extract text information from receipt images, such as date, amount, and category (transportation expenses, entertainment expenses, etc.) using a proprietary algorithm.

[0618] Step 5:

[0619] The AI ​​engine uses natural language processing technology to analyze the extracted text information and categorize it into the appropriate expense category. Specifically, it categorizes keywords such as "taxi" and "fare" as transportation expenses.

[0620] Step 6:

[0621] The server stores the classification information sent from the AI ​​engine in a database, which accumulates the allocated expense information and makes it available for subsequent processing.

[0622] Step 7:

[0623] The server automatically generates tax return documents based on the expense classification information retrieved from the database. Specifically, it inputs appropriate data into the income, expenditure, and expense information fields according to the tax return template, and calculates the total amount.

[0624] Step 8:

[0625] The emotion engine analyzes the user's facial expressions and voice data to recognize their emotions. It does this by analyzing data collected through the camera and microphone while the user is taking pictures or entering data.

[0626] Step 9:

[0627] The emotion engine provides feedback to the user based on the emotions it recognizes. For example, if the user is feeling stressed, the system will display a message saying, "We recommend you take a short break."

[0628] Step 10:

[0629] The server submits the generated tax return documents to the e-Tax system. At this time, the server uses the authentication information for logging in to the e-Tax system to carry out the appropriate upload procedure.

[0630] Step 11:

[0631] The server receives the results of the tax return submission and confirms that the submission was successful. It then sends a notification of the completion of the submission to the user.

[0632] This detailed processing step allows users to automate the entire tax return process by simply taking and uploading images of receipts. The introduction of an emotion engine also improves the user experience, making the tax return process less stressful.

[0633] Example 2

[0634] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0635] The traditional tax return preparation process is complex and time-consuming, placing a significant burden on many self-employed individuals. Additionally, sorting receipts and categorizing expenses is a tedious task that is prone to errors. Furthermore, the frustration users feel while completing the return is also a problem. To address this situation, there is a need for more efficient tax return preparation and an improved user experience.

[0636] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0637] In this invention, the server includes: means for a user to upload receipt images and payment data from a terminal to the server; means for the server to format the received data and send it to an AI engine; means for the AI ​​engine to analyze the data using image recognition technology and natural language processing technology and allocate expenses; means for the server to generate tax return documents based on the allocated data; means for the server to submit the generated tax return documents to an online system; means for the server to recognize the user's emotions through an emotion engine; and means for the emotion engine to analyze the user's facial expressions and voice data and determine the user's emotions. This realizes automation and efficiency in the preparation of tax return documents, reducing user stress and improving the experience.

[0638] A "receipt" is a document issued when a sale or service is provided that lists the details of the transaction.

[0639] "Payment Data" means data that includes transaction information related to the purchase of goods or payment for services.

[0640] "User" refers to individuals and businesses who use this system to file tax returns.

[0641] "Terminal" refers to the electronic device (smartphone, PC, etc.) that a user uses to operate the system.

[0642] The "server" is the central control unit that handles the entire processing of this system, and is a device that receives data from users, formats the data, connects with the AI ​​engine, and generates and submits tax return documents.

[0643] "Formatting" is the process of converting the format of received data, removing redundant parts of the data, and extracting necessary information to make it suitable for analysis.

[0644] An "AI engine" is a system that uses artificial intelligence technology to analyze data, and applies image recognition technology and natural language processing technology.

[0645] "Image recognition technology" is a technology that extracts specific information such as characters and objects from image data.

[0646] "Natural language processing technology" is a technology for analyzing the content of text data and understanding its meaning.

[0647] "Expense items" are categories for classifying types of expenses, and examples include "transportation expenses" and "entertainment expenses."

[0648] "Declaration documents" are various documents that must be submitted when filing a tax return, and contain details of income, expenditures, and expenses.

[0649] An "online system" is a system in which tax return procedures are carried out via the Internet, and examples include e-Tax.

[0650] An "emotion engine" is a system for analyzing a user's emotions, and determines the user's emotional state using facial expressions and voice data.

[0651] "Facial expression data" is data that includes information about the user's facial expression.

[0652] "Voice data" is data that includes information about the user's vocalizations.

[0653] This invention is a system that significantly simplifies and streamlines the preparation of tax return documents, and further optimizes the user experience by combining it with an emotion engine that recognizes the user's emotions. This system significantly reduces the burden on sole proprietors by automating the complex process of allocating expenses, generating tax return documents, and even submitting them online.

[0654] Specific program details

[0655] The system's program works as follows: First, the user uploads receipt images and payment data to the server using a smartphone or PC. Using a dedicated application or web browser, the user selects the data using the file selection button or drag-and-drop function, and then presses the send button.

[0656] The server receives image data and payment data sent by users. After receiving the data, it uses image processing algorithms to remove noise and clean it, and then formats it into a format that makes it easy to extract text. This formatted data is then sent to the AI ​​engine. Specific software used by the server includes OpenCV and Tesseract.

[0657] The AI ​​engine uses image recognition and natural language processing to extract necessary information from receipts, such as dates, amounts, and expense types. It then automatically categorizes each expense into the appropriate category based on the extracted information. Specific image recognition technologies include TensorFlow and PyTorch.

[0658] The server then stores the expense information sent from the AI ​​engine in a database and automatically generates tax return documents based on this information. The server calculates the totals of income, expenditures, and expenses, and generates documents that accurately list the information required for filing. An online platform using SaaS (Software as a Service) is used.

[0659] Through the emotion engine, the server collects and analyzes the user's facial expressions and voice data to recognize the user's emotions. The emotion engine determines whether the user is feeling stressed or relaxed, and uses this information to improve the user experience. Specific applications include the use of OpenPose and Microsoft Azure's Emotion API.

[0660] The server then finally processes the generated tax return documents for submission to the e-Tax system, eliminating the need for users to download and mail the documents.

[0661] Examples of concrete examples and prompts

[0662] 1. The user takes a photo of the receipt with their smartphone and uploads it to the server using a dedicated app:

[0663] For example, a user takes a taxi and receives a receipt for the fare of 5,000 yen. The user takes a photo of the receipt with the smartphone camera and uploads it to the server via a dedicated app.

[0664] 2. The server formats the received image data and sends it to the AI ​​engine:

[0665] When the server receives the receipt image, it removes noise from the image and formats it so that text information can be easily extracted. The formatted image data is then sent to the AI ​​engine.

[0666] 3. The AI ​​engine extracts the necessary information from the receipt image and determines the expense item:

[0667] The AI ​​engine uses image recognition technology to extract text information such as "October 1, 2023," "5,000 yen," and "transportation expenses." Natural language processing technology then categorizes this information as transportation expenses.

[0668] 4. The server automatically generates the tax return documents:

[0669] The server retrieves the information about the 5,000 yen allocated as transportation expenses from the AI ​​engine and automatically enters it into the appropriate field on the tax return. This calculates the total of income, expenditures, and expenses, and generates an accurate tax return.

[0670] 5. Emotion engine analyzes user emotions and optimizes user experience:

[0671] The emotion engine analyzes the user's facial expressions and tone of voice, and if the user is feeling stressed, it provides supportive messages to alleviate that stress. For example, it displays positive messages such as, "Your tax return preparation is going well. You're almost there!"

[0672] 6. The server submits the generated tax return to the online system:

[0673] The server logs into the e-Tax system and submits the generated tax return. Once submission is complete, the user is notified and can complete the procedure.

[0674] Prompt Sentence Examples

[0675] "An image of a taxi receipt taken with a smartphone is uploaded to a server. The image contains information such as 'October 1, 2023, 5,000 yen travel expenses'. Please explain how an AI engine can extract this information and allocate it to the appropriate expense category."

[0676] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0677] Step 1:

[0678] The user uploads an image of the receipt and payment data using the terminal.

[0679] Specific operation: The user takes a photo or scans a receipt or payment data using a smartphone or PC, then selects the captured image file or payment data through a dedicated application or web browser and clicks the upload button.

[0680] Input: Receipt image and payment data (e.g., taxi receipt image)

[0681] Output: Image data and payment data sent to the server

[0682] Step 2:

[0683] The server receives the data, formats it, and sends it to the AI ​​engine.

[0684] Specific operation: The server receives the receipt image sent by the user. It performs noise removal and image preprocessing on the received image data, formatting it so that text can be easily extracted. It then sends the formatted data (e.g., the noise-removed image and cleaned text data) to the AI ​​engine.

[0685] Input: Image data and payment data sent to the server

[0686] Output: Formatted data sent to the AI ​​engine

[0687] Step 3:

[0688] An AI engine analyzes the data and allocates expenses.

[0689] How it works: The AI ​​engine uses image recognition technology to extract dates, amounts, types of expenses, etc. from the received image data. It then uses natural language processing to understand the meaning of the extracted information and allocate it to the appropriate expense category (for example, "transportation expenses" or "entertainment expenses").

[0690] Input: Formatted data (e.g., a receipt image with noise removed)

[0691] Output: Allocated expense information (e.g., "October 1, 2023", "5,000 yen", "Transportation expenses")

[0692] Step 4:

[0693] The server generates tax return documents based on the allocated data.

[0694] Specific operation: The server saves the expense information sent from the AI ​​engine in a database and aggregates it. It then calculates the totals of income, expenses, and costs and automatically generates tax return documents. The generated tax return documents accurately contain the information necessary for filing (e.g., income, expenses, total amount).

[0695] Input: Allocated expense information

[0696] Output: Auto-generated tax return documents

[0697] Step 5:

[0698] The server recognizes the user's emotions through an emotion engine.

[0699] Specific operation: The server collects the user's facial expression and voice data and sends it to the emotion engine. The emotion engine analyzes the data and determines whether the user is feeling stressed or relaxed. If the user is feeling particularly stressed, it generates an appropriate support message and displays it to the user.

[0700] Input: User's facial expression data and voice data

[0701] Output: Emotion recognition results and supportive messages (e.g., "Your tax return preparation is going well. You're almost there!")

[0702] Step 6:

[0703] The server submits the generated tax return to the online system.

[0704] Specific operation: The server automatically sends the generated tax return documents to the e-Tax system. After sending, the server sends a success confirmation notice to the user, which saves the user the trouble of downloading and mailing the documents.

[0705] Input: Auto-generated tax return document

[0706] Output: Notification of completion of submission to the online system (e-Tax)

[0707] (Application example 2)

[0708] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0709] The present invention aims to reduce the burden on users and improve efficiency in expense management and tax return preparation. It also aims to improve the user experience throughout the entire tax return process by analyzing users' emotions and providing support to reduce stress. In particular, in environments where transactions are frequent, such as brick-and-mortar stores, it is necessary for store clerks to smoothly perform their daily tasks while managing expenses without stress.

[0710] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0711] In this invention, the server includes: a means for users to upload receipt images and payment data from their terminals to the server; a means for the server to format the received data and send it to an AI engine; a means for the AI ​​engine to analyze the data and allocate expenses using image recognition and natural language processing; a means for the server to generate tax return documents based on the allocated data; a means for the server to submit the generated tax return documents to an online system; and an emotion engine that analyzes the user's emotions and displays messages based on the emotions. This allows users to automate the entire tax return process simply by taking and uploading images of their receipts. The introduction of the emotion engine also improves the user experience, making the tax return process less stressful.

[0712] A "receipt image" is an image showing purchase details or a record of service usage that is taken by a user to prove expenses.

[0713] "Payment data" is digital data that includes a transaction record that occurs when a user makes a payment.

[0714] A "terminal" is an electronic device, such as a smartphone or computer, that a user operates to input and transmit data.

[0715] A "server" is a high-performance computer system for processing and storing data over a network.

[0716] "Formatting" refers to converting the data received by the server into an appropriate format for efficient processing.

[0717] An "AI engine" is a software system that uses artificial intelligence technology to analyze data and automatically perform specific tasks.

[0718] "Image recognition technology" is a technology that uses computer vision to extract specific information from images.

[0719] "Natural language processing technology" is a technology for analyzing meaning from text data and understanding and generating human language.

[0720] "Allocating expenses" means classifying expenses into specific categories.

[0721] A "tax return" is an official document that contains information required for filing a tax return.

[0722] An "online system" is a system for sending and receiving data and providing services using the Internet.

[0723] An "emotion engine" is a software system that analyzes a user's emotions and generates an appropriate response based on the results.

[0724] A "message" is text information that the emotion engine provides to the user based on the analysis results.

[0725] The present invention relates to a system that automatically processes receipt images and payment data to create tax return documents. Furthermore, by including an emotion engine that analyzes user emotions, the system aims to improve the user experience.

[0726] The system is configured as follows: First, the user uploads an image of the receipt and payment data to the server using a device such as a smartphone or PC. Specifically, a dedicated application or web browser is used. At this time, the user can take a photo of the receipt with a camera and send the image data.

[0727] The server receives the uploaded data, removes noise from the image data, and formats it in a way that makes it easier to extract text information. An image processing service like Google Cloud Vision API is a good choice for this purpose. The formatted data is then sent to the AI ​​engine.

[0728] The AI ​​engine analyzes data using image recognition and natural language processing technologies. Specifically, it uses deep learning libraries such as Keras to extract information such as date, amount, and expense item from receipt images and automatically classify them into appropriate categories.

[0729] The data is then sent back to the server, which then generates tax returns that accurately show income, expenditures, and total expenses. Again, a programming language such as Python is used to interface with a database management system to perform the necessary calculations.

[0730] Furthermore, the system is equipped with an emotion engine that analyzes the user's emotions. The emotion engine analyzes the facial expressions and voice of the store clerk to determine the level of stress. Facial recognition technology such as OpenCV and voice analysis technology are used for emotion analysis. If the user feels stressed, the emotion engine will support them by displaying encouraging messages such as, "Your declaration preparation is progressing smoothly. You're almost there!"

[0731] The completed tax return is then submitted to an online system (e.g., the e-Tax system) by the server, saving the user the trouble of manually submitting the return.

[0732] As a concrete example, consider the case of a convenience store clerk managing expenses while performing their daily duties. When making a transaction at the register, the clerk takes a photo of the receipt with their smartphone and uploads the data to a server using a dedicated app. The server then formats the image data, and an AI engine analyzes the data and allocates it to the appropriate expense category. This automatically generates tax return documents, which are ultimately submitted to the online system. The emotion engine also detects the clerk's stress and provides encouraging messages as needed, thereby reducing the clerk's stress.

[0733] Example prompt sentence:

[0734] "Just take a photo of your receipt with your smartphone and upload it using the app. We'll provide you with instant analysis and, if needed, a message of encouragement."

[0735] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0736] Step 1:

[0737] The user takes a photo of the receipt and payment data on the device and uploads it to the server using a dedicated application or a web browser. The image data of the receipt is used as input, and a data file is obtained as output, which is sent to the server.

[0738] Step 2:

[0739] The server formats the received image data. First, it uses the Google Cloud Vision API to remove noise from the image and convert it into a format that makes it easier to extract text information. It receives image data as input and obtains formatted image data as output.

[0740] Step 3:

[0741] The server sends the formatted image data to the AI ​​engine, which uses the formatted image data as input and obtains the data sent to the AI ​​engine as output.

[0742] Step 4:

[0743] The AI ​​engine analyzes the received image data. It uses image recognition technology to extract text information such as dates, amounts, and items, and then uses natural language processing technology to classify this information into appropriate expense categories. It receives formatted image data and extracted text information as input, and obtains classified expense information as output.

[0744] Step 5:

[0745] The server generates tax return documents based on the expense information returned by the AI ​​engine. The server calculates the totals of income, expenditures, and expenses through a database management system and compiles them into official tax return documents. It receives classified expense information as input and obtains the generated tax return documents as output.

[0746] Step 6:

[0747] The emotion engine analyzes the user's emotions. Facial expression and voice data collected on the device is sent to the server, which uses OpenCV and voice analysis technology to determine the level of stress. The engine receives the user's facial expression and voice data as input and obtains the emotion analysis results as output.

[0748] Step 7:

[0749] The server displays an appropriate message on the terminal based on the emotion analysis results. If the user is feeling stressed, an encouraging message is displayed to support the progress of the task. The emotion analysis results are received as input, and the message to be displayed is obtained as output.

[0750] Step 8:

[0751] The server submits the generated tax return to the online system, sends the tax return to an online platform such as the e-Tax system, and notifies the user of its completion, receiving the generated tax return as input and receiving a notification of completion of submission as output.

[0752] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0753] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0754] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0755] [Third embodiment]

[0756] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0757] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0758] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0759] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0760] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0761] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0762] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0763] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0764] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0765] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0766] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0767] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0768] This invention is a system that greatly simplifies and streamlines the preparation of tax return documents. This system significantly reduces the burden on sole proprietors by automating the complicated process of allocating expenses, generating tax return documents, and even submitting them online.

[0769] Program processing overview

[0770] The program of this system has the following functions:

[0771] 1. The user uploads an image of the receipt and payment data.

[0772] Users use their smartphones or PCs to upload images of receipts and payment data to the server, using a dedicated application or web browser.

[0773] 2. The server receives the data, formats it, and sends it to the AI ​​engine

[0774] The server receives image data and payment data sent by the user, converts it into an appropriate format, and sends the formatted data to the AI ​​engine.

[0775] 3. The AI ​​engine analyzes the data and allocates expenses

[0776] The AI ​​engine uses image recognition and natural language processing technologies to extract necessary information (such as date, amount, and expense type) from receipt images and payment data. Based on the extracted information, it automatically allocates each expense to the appropriate expense category.

[0777] 4. The server generates tax return documents based on the sorted data.

[0778] The server stores the expense information sent from the AI ​​engine in a database and automatically generates tax return documents based on that information, calculating the totals of income, expenditures, and expenses and accurately filling out the information required for the return.

[0779] 5. The server submits the generated tax return to the online system.

[0780] The server then processes the generated tax return documents for submission to an online system (such as e-Tax), eliminating the need for users to download and mail the documents.

[0781] Specific examples

[0782] 1. The user takes a photo of the receipt with their smartphone and uploads it to the server using a dedicated app.

[0783] For example, a user takes a taxi and receives a receipt for the fare of 5,000 yen. The user takes a photo of the receipt with the smartphone camera and uploads it to the server via a dedicated app.

[0784] 2. The server formats the received image data and sends it to the AI ​​engine.

[0785] When the server receives the receipt image, it removes noise from the image and formats it so that text information can be easily extracted. The formatted image data is then sent to the AI ​​engine.

[0786] 3. The AI ​​engine extracts the necessary information from the receipt image and determines the expense item

[0787] The AI ​​engine uses image recognition technology to extract text information such as "October 1, 2023," "5,000 yen," and "transportation expenses." Natural language processing technology then categorizes this information as transportation expenses.

[0788] 4. The server automatically generates tax return documents

[0789] The server retrieves the information about the 5,000 yen allocated as transportation expenses from the AI ​​engine and automatically enters it into the appropriate field on the tax return. This calculates the total of income, expenditures, and expenses, and generates an accurate tax return.

[0790] 5. The server submits the generated tax return to the online system.

[0791] The server logs into the e-Tax system and submits the generated tax return. The user is notified when the submission is complete.

[0792] This system allows users to carry out the complicated tax return procedures almost without thinking about them, enabling smooth data management and submission of tax return documents, allowing users to focus more on running their business.

[0793] The processing flow will be explained below.

[0794] Step 1:

[0795] The user opens the dedicated application using a smartphone or PC. After taking a picture of the receipt or importing the payment data, the user clicks the "Upload" button in the application. This operation causes the device to upload the receipt image and payment data to the server.

[0796] Step 2:

[0797] The server receives the image data and payment data sent by the user, and performs preprocessing (e.g., noise removal, resolution adjustment) on the received data to maintain the image resolution at an appropriate level.

[0798] Step 3:

[0799] The server sends the preprocessed image data to the AI ​​engine, where it is converted into an appropriate format and formatted for efficient analysis.

[0800] Step 4:

[0801] The AI ​​engine uses image recognition technology to extract text information from receipt images, such as date, amount, and category (transportation expenses, entertainment expenses, etc.) using a unique algorithm.

[0802] Step 5:

[0803] The AI ​​engine uses natural language processing technology to analyze the extracted text information and categorize it into the appropriate expense category. Specifically, it categorizes keywords such as "taxi" and "fare" as transportation expenses.

[0804] Step 6:

[0805] The server stores the classification information sent from the AI ​​engine in a database, which accumulates the allocated expense information and makes it available for subsequent processing.

[0806] Step 7:

[0807] The server automatically generates tax return documents based on the expense classification information retrieved from the database. Specifically, it inputs appropriate data into the income, expenditure, and expense information fields according to the tax return template, and calculates the total amount.

[0808] Step 8:

[0809] The server submits the generated tax return documents to the e-Tax system. At this time, the server uses the authentication information for logging in to the e-Tax system to carry out the appropriate upload procedure.

[0810] Step 9:

[0811] The server receives the results of the tax return submission and confirms that the submission was successful. It then sends a notification of the completion of the submission to the user.

[0812] This detailed processing step allows users to automate the entire tax return process by simply taking and uploading an image of their receipt.

[0813] Example 1

[0814] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0815] The complicated and time-consuming process of preparing tax returns by hand for sole proprietors has led to problems of human error and reduced work efficiency. Furthermore, it is a significant burden to individually categorize receipts, prepare tax return forms, and then submit them online.

[0816] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0817] In this invention, the server includes a means for a user to send business data from an information input device to a processing device, a means for converting and formatting the received data into an appropriate format by the processing device, and a means for sending the formatted data by the processing device to an analysis device. This automates the preparation of documents required for filing tax returns and the allocation of data to expense items, enabling online submission of tax returns.

[0818] An "information input device" is a device that allows a user to input business data and transmit it to a server, and includes smartphones, personal computers, and the like.

[0819] A "processor" is a server or computer system that receives data sent by a user and converts and formats it into an appropriate format.

[0820] "Business data" refers to various transaction information and receipt image data that are generated by users and sent to the server.

[0821] An "analysis device" is an AI engine or dedicated hardware that analyzes the formatted data sent from the processing device and classifies expenses.

[0822] "Image processing technology" refers to technology for extracting and processing necessary information from digital images, and includes noise removal and text recognition.

[0823] "Automatic analysis technology" is a technology that allows an AI engine to analyze data and classify information using natural language processing technology.

[0824] "Report" refers to tax returns and other business reports generated based on the allocated data.

[0825] "Online services" refers to various services provided via the Internet, including tax return submission systems such as e-Tax.

[0826] This invention is a system for streamlining the tax return process, where users submit business data and the data is analyzed, classified, and generated using a server and an AI engine. The system aims to automate many manual tasks and reduce the burden on users.

[0827] System configuration

[0828] The system consists of the following main components:

[0829] 1. Information input device

[0830] A device that allows users to input business data and send it to a server. Specifically, it includes smartphones (iOS or Android) and personal computers (Windows or Mac). Dedicated applications and web browsers (Chrome or Firefox) are used.

[0831] 2. Processing equipment

[0832] It is a server or computer system that receives data sent by users and converts and formats it into an appropriate format. It uses a Linux-based server and an image processing library (OpenCV). It also includes a database (MySQL or PostgreSQL).

[0833] 3. Analysis device

[0834] This is an AI engine and dedicated hardware that analyzes the formatted data sent from the processing device and classifies expenses. The AI ​​engine uses deep learning frameworks such as TensorFlow and PyTorch and an OCR library (Tesseract), and performs analysis using natural language processing libraries (spaCy and NLTK).

[0835] 4. Online Services

[0836] These are various services provided via the Internet, including tax return submission systems such as e-Tax.

[0837] Specific examples

[0838] Next, a concrete example of the operation of this system will be given.

[0839] 1. The user takes a photo of the receipt with their smartphone and uploads it to the server using a dedicated app.

[0840] Suppose a user takes a taxi and receives a receipt for the fare of 5,000 yen. The user takes a photo of this receipt with the smartphone camera and uploads the image to the server using a dedicated app. An example of a prompt sentence is, "Please take a photo of the restaurant receipt and upload it using the dedicated app. Please make sure that the amount and date are clearly visible."

[0841] 2. The server formats the received image data and sends it to the AI ​​engine.

[0842] The server uses OpenCV to remove noise from the images it receives and formats them so that text information can be easily extracted. The formatted image data is then sent to the AI ​​engine.

[0843] 3. The AI ​​engine extracts the necessary information from the receipt image and determines the expense item

[0844] The AI ​​engine uses Tesseract to extract information such as "October 1, 2023," "5,000 yen," and "transportation expenses," and then uses natural language processing technology (spaCy) to classify this as transportation expenses.

[0845] 4. The server automatically generates tax return documents

[0846] The server uses JasperReports to generate the necessary tax return documents based on the classification information from the AI ​​engine, and calculates the totals of income, expenses, and costs to accurately reflect them on the tax return documents.

[0847] 5. The server submits the generated tax return to the online system.

[0848] The server logs into the e-Tax system through the API and submits the generated tax return, which notifies the user and eliminates the need to download and mail the documents.

[0849] This system allows users to simplify the cumbersome manual process of filing tax returns, improving work efficiency.

[0850] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0851] Step 1:

[0852] The user uses an information input device (smartphone or PC) to send business data (receipt images and payment data) to the server. Specifically, the user uses a dedicated application or web browser to take a photo of the taxi receipt with the smartphone camera and upload the image. The input is image data taken by the user, which is then sent to the server.

[0853] Step 2:

[0854] The server converts and formats the received image data into an appropriate format. Specifically, the server uses an image processing library (OpenCV) to remove noise and format the data so that text information can be easily extracted. This processing improves the accuracy of reading the image data. The input is the raw image data sent by the user, and the output is formatted image data.

[0855] Step 3:

[0856] The server sends the formatted image data to the analysis device (AI engine). The server uses an API for data transmission to pass the formatted data to the AI ​​engine. The input is the formatted image data, and the output is the data sent to the analysis device.

[0857] Step 4:

[0858] The analysis device (AI engine) uses image processing technology (OCR) and automatic analysis technology to analyze the data and determine the expense items. Specifically, the AI ​​engine uses the OCR library (Tesseract) to extract text information such as "October 1, 2023" and "5,000 yen," and classifies it as "transportation expenses" using natural language processing technology (spaCy). The input is formatted image data, and the output is text information with classified expense items.

[0859] Step 5:

[0860] The server generates a report based on the expense classification data received from the analysis device. Specifically, the server uses JasperReports to calculate the totals of income, expenditures, and expenses, and automatically generates tax return documents. The input is text information with expense classifications, and the output is the automatically generated tax return documents.

[0861] Step 6:

[0862] The server submits the generated tax return to an online service (such as e-Tax). The server logs in to the e-Tax system via API and uploads the tax return. The input is the automatically generated tax return, and the output is a notification to the online service that the return has been submitted. This process notifies the user that the return has been submitted.

[0863] (Application example 1)

[0864] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0865] Currently, filing tax returns is a very complicated and time-consuming task for store owners and sole proprietors. In particular, processes such as organizing receipts, allocating expenses, and creating tax return documents and submitting them online require specialized knowledge and require a great deal of time and effort. This prevents them from focusing on running their business, resulting in a decline in work efficiency. The present invention aims to significantly reduce the burden of filing tax returns and provide a method for doing so more efficiently.

[0866] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0867] In this invention, the server includes: means for users to upload receipt images and payment data from their terminals to the server; means for the server to format the received data and send it to an AI engine; means for the AI ​​engine to analyze the data and allocate expenses using image recognition and natural language processing; means for the server to generate tax return documents based on the allocated data; means for the server to submit the generated tax return documents to an online system; means for store managers to photograph and upload daily expense receipts using a smartphone; and means for linking with payment terminals and accounting software to automatically import income and expenditure data. This automates the tax return process and significantly reduces the burden on users.

[0868] A "receipt" is a document issued to prove the amount paid for the purchase of goods or the use of services.

[0869] An "image" is a digital representation of visual information, typically obtained with a camera or scanner.

[0870] "Payment Data" refers to transaction information relating to the purchase of goods or the use of services, including details such as payment amount, date and time, and transaction partner.

[0871] "User" refers to a person or business entity that uses this system and is responsible for uploading receipts and payment data to the server.

[0872] "Terminal" refers to a digital device such as a smartphone, tablet, or PC, which a user uses to operate and upload data.

[0873] A "server" refers to a computer system that receives and processes data sent from users and provides various services.

[0874] An "AI engine" is software or a system that uses artificial intelligence technology to perform data analysis and classification tasks.

[0875] "Image recognition technology" is a general term for algorithms and technologies used to analyze digital images and extract specific patterns and information.

[0876] "Natural language processing technology" is a general term for algorithms and technologies for understanding text data and analyzing its meaning.

[0877] "Expenses" are expenditures necessary for business activities and are usually tax-deductible.

[0878] A "tax return" is a document that must be submitted to the tax office or other government agency, detailing income and expenses.

[0879] "Online system" refers to services and platforms that can be used via the Internet.

[0880] A "payment terminal" is a device for accepting electronic payments such as credit cards and debit cards.

[0881] "Accounting software" is a computer program used to record, manage, and analyze financial data.

[0882] "Income and expenditure data" refers to data that includes information on income and expenditure in business activities.

[0883] "Photography" refers to the act of capturing an image using a camera, smartphone, etc.

[0884] "Ingestion" refers to the process of receiving data from outside and integrating it into internal systems.

[0885] MODE FOR CARRYING OUT THE INVENTION

[0886] The present invention relates to a system that automates the process of filing tax returns in order to reduce the burden on store managers and sole proprietors. This system mainly consists of the following components:

[0887] 1. Upload your data

[0888] Users upload receipt images and payment data to the server using a device such as a smartphone. The device can be a smartphone, tablet, or PC with a camera function.

[0889] 2. Formatting and sending data

[0890] The server receives receipts and payment data sent by users, converts and formats the data into an appropriate format, and sends the formatted data to an artificial intelligence (AI) engine.

[0891] 3. Data analysis using an AI engine

[0892] The AI ​​engine uses image recognition and natural language processing technologies to analyze the received data. This analysis extracts information such as transaction date and time, amount, and transaction details from receipt images and payment data, and automatically classifies them into appropriate expense categories. The AI ​​model used can be, for example, the Transformers library.

[0893] 4. Generate tax return documents

[0894] The server automatically generates tax return documents based on the data categorized by the AI ​​engine, calculating the totals of income, expenditures, and expenses, and accurately filling in the necessary information.

[0895] 5. Online submission

[0896] The server submits the generated tax return documents to an online system (e.g., the e-Tax system), eliminating the need for the user to manually download and mail the documents.

[0897] Hardware and software used

[0898] This system uses the following hardware and software:

[0899] Hardware: smartphones, tablets, computers, payment terminals

[0900] Software: Server-side applications (e.g., Python), accounting software APIs (e.g., Moneybook, freee), natural language processing libraries (e.g., Transformers), PDF generation libraries (e.g., FPDF)

[0901] Specific examples

[0902] For example, if a user receives a receipt for "5,000 yen in transportation expenses on October 1, 2023," they can take a photo of the receipt with their smartphone and upload it to the server using a dedicated app. The server then formats the received image data and sends it to the AI ​​engine. The AI ​​engine then extracts text information such as "date," "amount," and "transportation expenses" from the image and classifies it as transportation expenses. The server then generates tax return documents based on this information and submits them to the e-Tax system.

[0903] Prompt Sentence Examples

[0904] When parsing the receipt text, you can use prompt statements such as:

[0905] The following text is the content of the receipt. Please indicate which expense category it falls under:

[0906] October 1, 2023 5,000 yen Transportation fee

[0907] As a result, this system will significantly reduce the burden on store managers and sole proprietors and enable efficient tax return filing.

[0908] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0909] Step 1:

[0910] The user uses the terminal to upload the receipt image and payment data to the server.

[0911] Input: Receipt images and transaction data taken and saved by the user.

[0912] Specific operations: Tap the "upload button" on your smartphone or computer, select images or data, and send them.

[0913] Output: Raw data uploaded to the server.

[0914] Step 2:

[0915] The server formats the data it receives.

[0916] Input: Uploaded raw data (images and transaction information).

[0917] Specific operation: The server removes noise from the image data, extracts text information, and formats the transaction information accordingly.

[0918] Output: Data formatted in a way that is suitable for an AI engine.

[0919] Step 3:

[0920] The server sends the formatted data to the AI ​​engine.

[0921] Input: Formatted data.

[0922] Specific operation: The server sends formatted text and image data to the AI ​​engine via API.

[0923] Output: Data sent to the AI ​​engine for analysis.

[0924] Step 4:

[0925] The AI ​​engine uses image recognition and natural language processing technologies to analyze the data and allocate expenses.

[0926] Input: The data sent from the server.

[0927] How it works: The AI ​​engine uses OCR technology to extract text, identifies dates, amounts, and categories using specific algorithms, and automatically classifies expenses using natural language processing.

[0928] Output: Categorised expense data.

[0929] Step 5:

[0930] The server generates tax return documents based on the allocated data.

[0931] Input: Data classified by the AI ​​engine (expense item data).

[0932] Specific operation: The server uses a PDF generation library (e.g. FPDF) to calculate the total income, expenses, and costs, and generate the tax return documents.

[0933] Output: Generated tax return PDF.

[0934] Step 6:

[0935] The server submits the generated tax return to the online system.

[0936] Input: Generated tax return PDF.

[0937] Specific operation: The server logs in to an online filing platform such as the e-Tax system and submits the generated PDF.

[0938] Output: Notification of submission completion and results.

[0939] Step 7:

[0940] The user receives a notification and confirms that the declaration has been completed.

[0941] Input: Submission completion notification from the server.

[0942] Specific operations: The user checks the notification on their smartphone or computer and operates the confirmation button for completing the declaration.

[0943] Output: A confirmation that the tax return is complete.

[0944] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0945] This invention is a system that significantly simplifies and streamlines the preparation of tax return documents, and further optimizes the user experience by combining it with an emotion engine that recognizes the user's emotions. This system significantly reduces the burden on sole proprietors by automating the complex process of allocating expenses, generating tax return documents, and even submitting them online.

[0946] Program processing overview

[0947] The program of this system has the following functions:

[0948] 1. The user uploads an image of the receipt and payment data.

[0949] Users use their smartphones or PCs to upload images of receipts and payment data to the server, using a dedicated application or web browser.

[0950] 2. The server receives the data, formats it, and sends it to the AI ​​engine

[0951] The server receives image data and payment data sent by the user, converts it into an appropriate format, and sends the formatted data to the AI ​​engine.

[0952] 3. The AI ​​engine analyzes the data and allocates expenses

[0953] The AI ​​engine uses image recognition and natural language processing technologies to extract necessary information (such as date, amount, and expense type) from receipt images and payment data. Based on the extracted information, it automatically allocates each expense to the appropriate expense category.

[0954] 4. The server generates tax return documents based on the sorted data.

[0955] The server stores the expense information sent from the AI ​​engine in a database and automatically generates tax return documents based on that information, calculating the totals of income, expenditures, and expenses and accurately filling out the information required for the return.

[0956] 5. Emotion engine recognizes user emotions

[0957] The server collects and analyzes the user's facial expressions and voice data through the emotion engine to recognize the user's emotions. The emotion engine determines whether the user is feeling stressed or relaxed, and uses this information to improve the user experience.

[0958] 6. The server submits the generated tax return to the online system.

[0959] The server then processes the generated tax return documents for submission to the e-Tax system, eliminating the need for users to download and mail the documents.

[0960] Specific examples

[0961] 1. The user takes a photo of the receipt with their smartphone and uploads it to the server using a dedicated app.

[0962] For example, a user takes a taxi and receives a receipt for the fare of 5,000 yen. The user takes a photo of the receipt with the smartphone camera and uploads it to the server via a dedicated app.

[0963] 2. The server formats the received image data and sends it to the AI ​​engine.

[0964] When the server receives the receipt image, it removes noise from the image and formats it so that text information can be easily extracted. The formatted image data is then sent to the AI ​​engine.

[0965] 3. The AI ​​engine extracts the necessary information from the receipt image and determines the expense item

[0966] The AI ​​engine uses image recognition technology to extract text information such as "October 1, 2023," "5,000 yen," and "transportation expenses." Natural language processing technology then categorizes this information as transportation expenses.

[0967] 4. The server automatically generates tax return documents

[0968] The server retrieves the information about the 5,000 yen allocated as transportation expenses from the AI ​​engine and automatically enters it into the appropriate field on the tax return. This calculates the total of income, expenditures, and expenses, and generates an accurate tax return.

[0969] 5. Emotion engine analyzes user emotions and optimizes the user experience

[0970] The emotion engine analyzes the user's facial expressions and tone of voice, and if the user is feeling stressed, it provides supportive messages to alleviate that stress. For example, it displays positive messages such as, "Your tax return preparation is going well. You're almost there!"

[0971] 6. The server submits the generated tax return to the online system.

[0972] The server logs into the e-Tax system and submits the generated tax return. Once submission is complete, the user is notified and can complete the procedure.

[0973] This system allows users to automate the entire tax return process by simply taking and uploading images of receipts. The introduction of an emotion engine also improves the user experience, making the tax return process less stressful.

[0974] The processing flow will be explained below.

[0975] Step 1:

[0976] The user opens the dedicated application using a smartphone or PC. After taking a picture of the receipt or importing the payment data, the user clicks the "Upload" button in the application. This operation causes the device to upload the receipt image and payment data to the server.

[0977] Step 2:

[0978] The server receives the image data and payment data sent by the user, and performs preprocessing (e.g., noise removal, resolution adjustment) on the received data to maintain the image resolution at an appropriate level.

[0979] Step 3:

[0980] The server sends the preprocessed image data to the AI ​​engine, where it is converted into an appropriate format and formatted for efficient analysis.

[0981] Step 4:

[0982] The AI ​​engine uses image recognition technology to extract text information from receipt images, such as date, amount, and category (transportation expenses, entertainment expenses, etc.) using a proprietary algorithm.

[0983] Step 5:

[0984] The AI ​​engine uses natural language processing technology to analyze the extracted text information and categorize it into the appropriate expense category. Specifically, it categorizes keywords such as "taxi" and "fare" as transportation expenses.

[0985] Step 6:

[0986] The server stores the classification information sent from the AI ​​engine in a database, which accumulates the allocated expense information and makes it available for subsequent processing.

[0987] Step 7:

[0988] The server automatically generates tax return documents based on the expense classification information retrieved from the database. Specifically, it inputs appropriate data into the income, expenditure, and expense information fields according to the tax return template, and calculates the total amount.

[0989] Step 8:

[0990] The emotion engine analyzes the user's facial expressions and voice data to recognize their emotions. It does this by analyzing data collected through the camera and microphone while the user is taking pictures or entering data.

[0991] Step 9:

[0992] The emotion engine provides feedback to the user based on the emotions it recognizes. For example, if the user is feeling stressed, the system will display a message saying, "We recommend you take a short break."

[0993] Step 10:

[0994] The server submits the generated tax return documents to the e-Tax system. At this time, the server uses the authentication information for logging in to the e-Tax system to carry out the appropriate upload procedure.

[0995] Step 11:

[0996] The server receives the results of the tax return submission and confirms that the submission was successful. It then sends a notification of the completion of the submission to the user.

[0997] This detailed processing step allows users to automate the entire tax return process by simply taking and uploading images of receipts. The introduction of an emotion engine also improves the user experience, making the tax return process less stressful.

[0998] Example 2

[0999] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1000] The traditional tax return preparation process is complex and time-consuming, placing a significant burden on many self-employed individuals. Additionally, sorting receipts and categorizing expenses is a tedious task that is prone to errors. Furthermore, the frustration users feel while completing the return is also a problem. To address this situation, there is a need for more efficient tax return preparation and an improved user experience.

[1001] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1002] In this invention, the server includes: means for a user to upload receipt images and payment data from a terminal to the server; means for the server to format the received data and send it to an AI engine; means for the AI ​​engine to analyze the data using image recognition technology and natural language processing technology and allocate expenses; means for the server to generate tax return documents based on the allocated data; means for the server to submit the generated tax return documents to an online system; means for the server to recognize the user's emotions through an emotion engine; and means for the emotion engine to analyze the user's facial expressions and voice data and determine the user's emotions. This realizes automation and efficiency in the preparation of tax return documents, reducing user stress and improving the experience.

[1003] A "receipt" is a document issued when a sale or service is provided that lists the details of the transaction.

[1004] "Payment Data" means data that includes transaction information related to the purchase of goods or payment for services.

[1005] "User" refers to individuals and businesses who use this system to file tax returns.

[1006] "Terminal" refers to the electronic device (smartphone, PC, etc.) that a user uses to operate the system.

[1007] The "server" is the central control unit that handles the entire processing of this system, and is a device that receives data from users, formats the data, connects with the AI ​​engine, and generates and submits tax return documents.

[1008] "Formatting" is the process of converting the format of received data, removing redundant parts of the data, and extracting necessary information to make it suitable for analysis.

[1009] An "AI engine" is a system that uses artificial intelligence technology to analyze data, and applies image recognition technology and natural language processing technology.

[1010] "Image recognition technology" is a technology that extracts specific information such as characters and objects from image data.

[1011] "Natural language processing technology" is a technology for analyzing the content of text data and understanding its meaning.

[1012] "Expense items" are categories for classifying types of expenses, and examples include "transportation expenses" and "entertainment expenses."

[1013] "Declaration documents" are various documents that must be submitted when filing a tax return, and contain details of income, expenditures, and expenses.

[1014] An "online system" is a system in which tax return procedures are carried out via the Internet, and examples include e-Tax.

[1015] An "emotion engine" is a system for analyzing a user's emotions, and determines the user's emotional state using facial expressions and voice data.

[1016] "Facial expression data" is data that includes information about the user's facial expression.

[1017] "Voice data" is data that includes information about the user's vocalizations.

[1018] This invention is a system that significantly simplifies and streamlines the preparation of tax return documents, and further optimizes the user experience by combining it with an emotion engine that recognizes the user's emotions. This system significantly reduces the burden on sole proprietors by automating the complex process of allocating expenses, generating tax return documents, and even submitting them online.

[1019] Specific program details

[1020] The system's program works as follows: First, the user uploads receipt images and payment data to the server using a smartphone or PC. Using a dedicated application or web browser, the user selects the data using the file selection button or drag-and-drop function, and then presses the send button.

[1021] The server receives image data and payment data sent by users. After receiving the data, it uses image processing algorithms to remove noise and clean it, and then formats it into a format that makes it easy to extract text. This formatted data is then sent to the AI ​​engine. Specific software used by the server includes OpenCV and Tesseract.

[1022] The AI ​​engine uses image recognition and natural language processing to extract necessary information from receipts, such as dates, amounts, and expense types. It then automatically categorizes each expense into the appropriate category based on the extracted information. Specific image recognition technologies include TensorFlow and PyTorch.

[1023] The server then stores the expense information sent from the AI ​​engine in a database and automatically generates tax return documents based on this information. The server calculates the totals of income, expenditures, and expenses, and generates documents that accurately list the information required for filing. An online platform using SaaS (Software as a Service) is used.

[1024] Through the emotion engine, the server collects and analyzes the user's facial expressions and voice data to recognize the user's emotions. The emotion engine determines whether the user is feeling stressed or relaxed, and uses this information to improve the user experience. Specific applications include the use of OpenPose and Microsoft Azure's Emotion API.

[1025] The server then finally processes the generated tax return documents for submission to the e-Tax system, eliminating the need for users to download and mail the documents.

[1026] Examples of concrete examples and prompts

[1027] 1. The user takes a photo of the receipt with their smartphone and uploads it to the server using a dedicated app:

[1028] For example, a user takes a taxi and receives a receipt for the fare of 5,000 yen. The user takes a photo of the receipt with the smartphone camera and uploads it to the server via a dedicated app.

[1029] 2. The server formats the received image data and sends it to the AI ​​engine:

[1030] When the server receives the receipt image, it removes noise from the image and formats it so that text information can be easily extracted. The formatted image data is then sent to the AI ​​engine.

[1031] 3. The AI ​​engine extracts the necessary information from the receipt image and determines the expense item:

[1032] The AI ​​engine uses image recognition technology to extract text information such as "October 1, 2023," "5,000 yen," and "transportation expenses." Natural language processing technology then categorizes this information as transportation expenses.

[1033] 4. The server automatically generates the tax return documents:

[1034] The server retrieves the information about the 5,000 yen allocated as transportation expenses from the AI ​​engine and automatically enters it into the appropriate field on the tax return. This calculates the total of income, expenditures, and expenses, and generates an accurate tax return.

[1035] 5. Emotion engine analyzes user emotions and optimizes user experience:

[1036] The emotion engine analyzes the user's facial expressions and tone of voice, and if the user is feeling stressed, it provides supportive messages to alleviate that stress. For example, it displays positive messages such as, "Your tax return preparation is going well. You're almost there!"

[1037] 6. The server submits the generated tax return to the online system:

[1038] The server logs into the e-Tax system and submits the generated tax return. Once submission is complete, the user is notified and can complete the procedure.

[1039] Prompt Sentence Examples

[1040] "An image of a taxi receipt taken with a smartphone is uploaded to a server. The image contains information such as 'October 1, 2023, 5,000 yen travel expenses'. Please explain how an AI engine can extract this information and allocate it to the appropriate expense category."

[1041] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1042] Step 1:

[1043] The user uploads an image of the receipt and payment data using the terminal.

[1044] Specific operation: The user takes a photo or scans a receipt or payment data using a smartphone or PC, then selects the captured image file or payment data through a dedicated application or web browser and clicks the upload button.

[1045] Input: Receipt image and payment data (e.g., taxi receipt image)

[1046] Output: Image data and payment data sent to the server

[1047] Step 2:

[1048] The server receives the data, formats it, and sends it to the AI ​​engine.

[1049] Specific operation: The server receives the receipt image sent by the user. It performs noise removal and image preprocessing on the received image data, formatting it so that text can be easily extracted. It then sends the formatted data (e.g., the noise-removed image and cleaned text data) to the AI ​​engine.

[1050] Input: Image data and payment data sent to the server

[1051] Output: Formatted data sent to the AI ​​engine

[1052] Step 3:

[1053] An AI engine analyzes the data and allocates expenses.

[1054] How it works: The AI ​​engine uses image recognition technology to extract dates, amounts, types of expenses, etc. from the received image data. It then uses natural language processing to understand the meaning of the extracted information and allocate it to the appropriate expense category (for example, "transportation expenses" or "entertainment expenses").

[1055] Input: Formatted data (e.g., a receipt image with noise removed)

[1056] Output: Allocated expense information (e.g., "October 1, 2023", "5,000 yen", "Transportation expenses")

[1057] Step 4:

[1058] The server generates tax return documents based on the allocated data.

[1059] Specific operation: The server saves the expense information sent from the AI ​​engine in a database and aggregates it. It then calculates the totals of income, expenses, and costs and automatically generates tax return documents. The generated tax return documents accurately contain the information necessary for filing (e.g., income, expenses, total amount).

[1060] Input: Allocated expense information

[1061] Output: Auto-generated tax return documents

[1062] Step 5:

[1063] The server recognizes the user's emotions through an emotion engine.

[1064] Specific operation: The server collects the user's facial expression and voice data and sends it to the emotion engine. The emotion engine analyzes the data and determines whether the user is feeling stressed or relaxed. If the user is feeling particularly stressed, it generates an appropriate support message and displays it to the user.

[1065] Input: User's facial expression data and voice data

[1066] Output: Emotion recognition results and supportive messages (e.g., "Your tax return preparation is going well. You're almost there!")

[1067] Step 6:

[1068] The server submits the generated tax return to the online system.

[1069] Specific operation: The server automatically sends the generated tax return documents to the e-Tax system. After sending, the server sends a success confirmation notice to the user, which saves the user the trouble of downloading and mailing the documents.

[1070] Input: Auto-generated tax return document

[1071] Output: Notification of completion of submission to the online system (e-Tax)

[1072] (Application example 2)

[1073] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1074] The present invention aims to reduce the burden on users and improve efficiency in expense management and tax return preparation. It also aims to improve the user experience throughout the entire tax return process by analyzing users' emotions and providing support to reduce stress. In particular, in environments where transactions are frequent, such as brick-and-mortar stores, it is necessary for store clerks to smoothly perform their daily tasks while managing expenses without stress.

[1075] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1076] In this invention, the server includes: a means for users to upload receipt images and payment data from their terminals to the server; a means for the server to format the received data and send it to an AI engine; a means for the AI ​​engine to analyze the data and allocate expenses using image recognition and natural language processing; a means for the server to generate tax return documents based on the allocated data; a means for the server to submit the generated tax return documents to an online system; and an emotion engine that analyzes the user's emotions and displays messages based on the emotions. This allows users to automate the entire tax return process simply by taking and uploading images of their receipts. The introduction of the emotion engine also improves the user experience, making the tax return process less stressful.

[1077] A "receipt image" is an image showing purchase details or a record of service usage that is taken by a user to prove expenses.

[1078] "Payment data" is digital data that includes a transaction record that occurs when a user makes a payment.

[1079] A "terminal" is an electronic device, such as a smartphone or computer, that a user operates to input and transmit data.

[1080] A "server" is a high-performance computer system for processing and storing data over a network.

[1081] "Formatting" refers to converting the data received by the server into an appropriate format for efficient processing.

[1082] An "AI engine" is a software system that uses artificial intelligence technology to analyze data and automatically perform specific tasks.

[1083] "Image recognition technology" is a technology that uses computer vision to extract specific information from images.

[1084] "Natural language processing technology" is a technology for analyzing meaning from text data and understanding and generating human language.

[1085] "Allocating expenses" means classifying expenses into specific categories.

[1086] A "tax return" is an official document that contains information required for filing a tax return.

[1087] An "online system" is a system for sending and receiving data and providing services using the Internet.

[1088] An "emotion engine" is a software system that analyzes a user's emotions and generates an appropriate response based on the results.

[1089] A "message" is text information that the emotion engine provides to the user based on the analysis results.

[1090] The present invention relates to a system that automatically processes receipt images and payment data to create tax return documents. Furthermore, by including an emotion engine that analyzes user emotions, the system aims to improve the user experience.

[1091] The system is configured as follows: First, the user uploads an image of the receipt and payment data to the server using a device such as a smartphone or PC. Specifically, a dedicated application or web browser is used. At this time, the user can take a photo of the receipt with a camera and send the image data.

[1092] The server receives the uploaded data, removes noise from the image data, and formats it in a way that makes it easier to extract text information. An image processing service like Google Cloud Vision API is a good choice for this purpose. The formatted data is then sent to the AI ​​engine.

[1093] The AI ​​engine analyzes data using image recognition and natural language processing technologies. Specifically, it uses deep learning libraries such as Keras to extract information such as date, amount, and expense item from receipt images and automatically classify them into appropriate categories.

[1094] The data is then sent back to the server, which then generates tax returns that accurately show income, expenditures, and total expenses. Again, a programming language such as Python is used to interface with a database management system to perform the necessary calculations.

[1095] Furthermore, the system is equipped with an emotion engine that analyzes the user's emotions. The emotion engine analyzes the facial expressions and voice of the store clerk to determine the level of stress. Facial recognition technology such as OpenCV and voice analysis technology are used for emotion analysis. If the user feels stressed, the emotion engine will support them by displaying encouraging messages such as, "Your declaration preparation is progressing smoothly. You're almost there!"

[1096] The completed tax return is then submitted to an online system (e.g., the e-Tax system) by the server, saving the user the trouble of manually submitting the return.

[1097] As a concrete example, consider the case of a convenience store clerk managing expenses while performing their daily duties. When making a transaction at the register, the clerk takes a photo of the receipt with their smartphone and uploads the data to a server using a dedicated app. The server then formats the image data, and an AI engine analyzes the data and allocates it to the appropriate expense category. This automatically generates tax return documents, which are ultimately submitted to the online system. The emotion engine also detects the clerk's stress and provides encouraging messages as needed, thereby reducing the clerk's stress.

[1098] Example prompt sentence:

[1099] "Just take a photo of your receipt with your smartphone and upload it using the app. We'll provide you with instant analysis and, if needed, a message of encouragement."

[1100] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1101] Step 1:

[1102] The user takes a photo of the receipt and payment data on the device and uploads it to the server using a dedicated application or a web browser. The image data of the receipt is used as input, and a data file is obtained as output, which is sent to the server.

[1103] Step 2:

[1104] The server formats the received image data. First, it uses the Google Cloud Vision API to remove noise from the image and convert it into a format that makes it easier to extract text information. It receives image data as input and obtains formatted image data as output.

[1105] Step 3:

[1106] The server sends the formatted image data to the AI ​​engine, which uses the formatted image data as input and obtains the data sent to the AI ​​engine as output.

[1107] Step 4:

[1108] The AI ​​engine analyzes the received image data. It uses image recognition technology to extract text information such as dates, amounts, and items, and then uses natural language processing technology to classify this information into appropriate expense categories. It receives formatted image data and extracted text information as input, and obtains classified expense information as output.

[1109] Step 5:

[1110] The server generates tax return documents based on the expense information returned by the AI ​​engine. The server calculates the totals of income, expenditures, and expenses through a database management system and compiles them into official tax return documents. It receives classified expense information as input and obtains the generated tax return documents as output.

[1111] Step 6:

[1112] The emotion engine analyzes the user's emotions. Facial expression and voice data collected on the device is sent to the server, which uses OpenCV and voice analysis technology to determine the level of stress. The engine receives the user's facial expression and voice data as input and obtains the emotion analysis results as output.

[1113] Step 7:

[1114] The server displays an appropriate message on the terminal based on the emotion analysis results. If the user is feeling stressed, an encouraging message is displayed to support the progress of the task. The emotion analysis results are received as input, and the message to be displayed is obtained as output.

[1115] Step 8:

[1116] The server submits the generated tax return to the online system, sends the tax return to an online platform such as the e-Tax system, and notifies the user of its completion, receiving the generated tax return as input and receiving a notification of completion of submission as output.

[1117] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1118] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1119] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1120] [Fourth embodiment]

[1121] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1122] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1123] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1124] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1125] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1126] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1127] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1128] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1129] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1130] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1131] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1132] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1133] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1134] This invention is a system that greatly simplifies and streamlines the preparation of tax return documents. This system significantly reduces the burden on sole proprietors by automating the complicated process of allocating expenses, generating tax return documents, and even submitting them online.

[1135] Program processing overview

[1136] The program of this system has the following functions:

[1137] 1. The user uploads an image of the receipt and payment data.

[1138] Users use their smartphones or PCs to upload images of receipts and payment data to the server, using a dedicated application or web browser.

[1139] 2. The server receives the data, formats it, and sends it to the AI ​​engine

[1140] The server receives image data and payment data sent by the user, converts it into an appropriate format, and sends the formatted data to the AI ​​engine.

[1141] 3. The AI ​​engine analyzes the data and allocates expenses

[1142] The AI ​​engine uses image recognition and natural language processing technologies to extract necessary information (such as date, amount, and expense type) from receipt images and payment data. Based on the extracted information, it automatically allocates each expense to the appropriate expense category.

[1143] 4. The server generates tax return documents based on the sorted data.

[1144] The server stores the expense information sent from the AI ​​engine in a database and automatically generates tax return documents based on that information, calculating the totals of income, expenditures, and expenses and accurately filling out the information required for the return.

[1145] 5. The server submits the generated tax return to the online system.

[1146] The server then processes the generated tax return documents for submission to an online system (such as e-Tax), eliminating the need for users to download and mail the documents.

[1147] Specific examples

[1148] 1. The user takes a photo of the receipt with their smartphone and uploads it to the server using a dedicated app.

[1149] For example, a user takes a taxi and receives a receipt for the fare of 5,000 yen. The user takes a photo of the receipt with the smartphone camera and uploads it to the server via a dedicated app.

[1150] 2. The server formats the received image data and sends it to the AI ​​engine.

[1151] When the server receives the receipt image, it removes noise from the image and formats it so that text information can be easily extracted. The formatted image data is then sent to the AI ​​engine.

[1152] 3. The AI ​​engine extracts the necessary information from the receipt image and determines the expense item

[1153] The AI ​​engine uses image recognition technology to extract text information such as "October 1, 2023," "5,000 yen," and "transportation expenses." Natural language processing technology then categorizes this information as transportation expenses.

[1154] 4. The server automatically generates tax return documents

[1155] The server retrieves the information about the 5,000 yen allocated as transportation expenses from the AI ​​engine and automatically enters it into the appropriate field on the tax return. This calculates the total of income, expenditures, and expenses, and generates an accurate tax return.

[1156] 5. The server submits the generated tax return to the online system.

[1157] The server logs into the e-Tax system and submits the generated tax return. The user is notified when the submission is complete.

[1158] This system allows users to carry out the complicated tax return procedures almost without thinking about them, enabling smooth data management and submission of tax return documents, allowing users to focus more on running their business.

[1159] The processing flow will be explained below.

[1160] Step 1:

[1161] The user opens the dedicated application using a smartphone or PC. After taking a picture of the receipt or importing the payment data, the user clicks the "Upload" button in the application. This operation causes the device to upload the receipt image and payment data to the server.

[1162] Step 2:

[1163] The server receives the image data and payment data sent by the user, and performs preprocessing (e.g., noise removal, resolution adjustment) on the received data to maintain the image resolution at an appropriate level.

[1164] Step 3:

[1165] The server sends the preprocessed image data to the AI ​​engine, where it is converted into an appropriate format and formatted for efficient analysis.

[1166] Step 4:

[1167] The AI ​​engine uses image recognition technology to extract text information from receipt images, such as date, amount, and category (transportation expenses, entertainment expenses, etc.) using a unique algorithm.

[1168] Step 5:

[1169] The AI ​​engine uses natural language processing technology to analyze the extracted text information and categorize it into the appropriate expense category. Specifically, it categorizes keywords such as "taxi" and "fare" as transportation expenses.

[1170] Step 6:

[1171] The server stores the classification information sent from the AI ​​engine in a database, which accumulates the allocated expense information and makes it available for subsequent processing.

[1172] Step 7:

[1173] The server automatically generates tax return documents based on the expense classification information retrieved from the database. Specifically, it inputs appropriate data into the income, expenditure, and expense information fields according to the tax return template, and calculates the total amount.

[1174] Step 8:

[1175] The server submits the generated tax return documents to the e-Tax system. At this time, the server uses the authentication information for logging in to the e-Tax system to carry out the appropriate upload procedure.

[1176] Step 9:

[1177] The server receives the results of the tax return submission and confirms that the submission was successful. It then sends a notification of the completion of the submission to the user.

[1178] This detailed processing step allows users to automate the entire tax return process by simply taking and uploading an image of their receipt.

[1179] Example 1

[1180] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1181] The complicated and time-consuming process of preparing tax returns by hand for sole proprietors has led to problems of human error and reduced work efficiency. Furthermore, it is a significant burden to individually categorize receipts, prepare tax return forms, and then submit them online.

[1182] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1183] In this invention, the server includes a means for a user to send business data from an information input device to a processing device, a means for converting and formatting the received data into an appropriate format by the processing device, and a means for sending the formatted data by the processing device to an analysis device. This automates the preparation of documents required for filing tax returns and the allocation of data to expense items, enabling online submission of tax returns.

[1184] An "information input device" is a device that allows a user to input business data and transmit it to a server, and includes smartphones, personal computers, and the like.

[1185] A "processor" is a server or computer system that receives data sent by a user and converts and formats it into an appropriate format.

[1186] "Business data" refers to various transaction information and receipt image data that are generated by users and sent to the server.

[1187] An "analysis device" is an AI engine or dedicated hardware that analyzes the formatted data sent from the processing device and classifies expenses.

[1188] "Image processing technology" refers to technology for extracting and processing necessary information from digital images, and includes noise removal and text recognition.

[1189] "Automatic analysis technology" is a technology that allows an AI engine to analyze data and classify information using natural language processing technology.

[1190] "Report" refers to tax returns and other business reports generated based on the allocated data.

[1191] "Online services" refers to various services provided via the Internet, including tax return submission systems such as e-Tax.

[1192] This invention is a system for streamlining the tax return process, where users submit business data and the data is analyzed, classified, and generated using a server and an AI engine. The system aims to automate many manual tasks and reduce the burden on users.

[1193] System configuration

[1194] The system consists of the following main components:

[1195] 1. Information input device

[1196] A device that allows users to input business data and send it to a server. Specifically, it includes smartphones (iOS or Android) and personal computers (Windows or Mac). Dedicated applications and web browsers (Chrome or Firefox) are used.

[1197] 2. Processing equipment

[1198] It is a server or computer system that receives data sent by users and converts and formats it into an appropriate format. It uses a Linux-based server and an image processing library (OpenCV). It also includes a database (MySQL or PostgreSQL).

[1199] 3. Analysis device

[1200] This is an AI engine and dedicated hardware that analyzes the formatted data sent from the processing device and classifies expenses. The AI ​​engine uses deep learning frameworks such as TensorFlow and PyTorch and an OCR library (Tesseract), and performs analysis using natural language processing libraries (spaCy and NLTK).

[1201] 4. Online Services

[1202] These are various services provided via the Internet, including tax return submission systems such as e-Tax.

[1203] Specific examples

[1204] Next, a concrete example of the operation of this system will be given.

[1205] 1. The user takes a photo of the receipt with their smartphone and uploads it to the server using a dedicated app.

[1206] Suppose a user takes a taxi and receives a receipt for the fare of 5,000 yen. The user takes a photo of this receipt with the smartphone camera and uploads the image to the server using a dedicated app. An example of a prompt sentence is, "Please take a photo of the restaurant receipt and upload it using the dedicated app. Please make sure that the amount and date are clearly visible."

[1207] 2. The server formats the received image data and sends it to the AI ​​engine.

[1208] The server uses OpenCV to remove noise from the images it receives and formats them so that text information can be easily extracted. The formatted image data is then sent to the AI ​​engine.

[1209] 3. The AI ​​engine extracts the necessary information from the receipt image and determines the expense item

[1210] The AI ​​engine uses Tesseract to extract information such as "October 1, 2023," "5,000 yen," and "transportation expenses," and then uses natural language processing technology (spaCy) to classify this as transportation expenses.

[1211] 4. The server automatically generates tax return documents

[1212] The server uses JasperReports to generate the necessary tax return documents based on the classification information from the AI ​​engine, and calculates the totals of income, expenses, and costs to accurately reflect them on the tax return documents.

[1213] 5. The server submits the generated tax return to the online system.

[1214] The server logs into the e-Tax system through the API and submits the generated tax return, which notifies the user and eliminates the need to download and mail the documents.

[1215] This system allows users to simplify the cumbersome manual process of filing tax returns, improving work efficiency.

[1216] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1217] Step 1:

[1218] The user uses an information input device (smartphone or PC) to send business data (receipt images and payment data) to the server. Specifically, the user uses a dedicated application or web browser to take a photo of the taxi receipt with the smartphone camera and upload the image. The input is image data taken by the user, which is then sent to the server.

[1219] Step 2:

[1220] The server converts and formats the received image data into an appropriate format. Specifically, the server uses an image processing library (OpenCV) to remove noise and format the data so that text information can be easily extracted. This processing improves the accuracy of reading the image data. The input is the raw image data sent by the user, and the output is formatted image data.

[1221] Step 3:

[1222] The server sends the formatted image data to the analysis device (AI engine). The server uses an API for data transmission to pass the formatted data to the AI ​​engine. The input is the formatted image data, and the output is the data sent to the analysis device.

[1223] Step 4:

[1224] The analysis device (AI engine) uses image processing technology (OCR) and automatic analysis technology to analyze the data and determine the expense items. Specifically, the AI ​​engine uses the OCR library (Tesseract) to extract text information such as "October 1, 2023" and "5,000 yen," and classifies it as "transportation expenses" using natural language processing technology (spaCy). The input is formatted image data, and the output is text information with classified expense items.

[1225] Step 5:

[1226] The server generates a report based on the expense classification data received from the analysis device. Specifically, the server uses JasperReports to calculate the totals of income, expenditures, and expenses, and automatically generates tax return documents. The input is text information with expense classifications, and the output is the automatically generated tax return documents.

[1227] Step 6:

[1228] The server submits the generated tax return to an online service (such as e-Tax). The server logs in to the e-Tax system via API and uploads the tax return. The input is the automatically generated tax return, and the output is a notification to the online service that the return has been submitted. This process notifies the user that the return has been submitted.

[1229] (Application example 1)

[1230] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1231] Currently, filing tax returns is a very complicated and time-consuming task for store owners and sole proprietors. In particular, processes such as organizing receipts, allocating expenses, and creating tax return documents and submitting them online require specialized knowledge and require a great deal of time and effort. This prevents them from focusing on running their business, resulting in a decline in work efficiency. The present invention aims to significantly reduce the burden of filing tax returns and provide a method for doing so more efficiently.

[1232] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1233] In this invention, the server includes: means for users to upload receipt images and payment data from their terminals to the server; means for the server to format the received data and send it to an AI engine; means for the AI ​​engine to analyze the data and allocate expenses using image recognition and natural language processing; means for the server to generate tax return documents based on the allocated data; means for the server to submit the generated tax return documents to an online system; means for store managers to photograph and upload daily expense receipts using a smartphone; and means for linking with payment terminals and accounting software to automatically import income and expenditure data. This automates the tax return process and significantly reduces the burden on users.

[1234] A "receipt" is a document issued to prove the amount paid for the purchase of goods or the use of services.

[1235] An "image" is a digital representation of visual information, typically obtained with a camera or scanner.

[1236] "Payment Data" refers to transaction information relating to the purchase of goods or the use of services, including details such as payment amount, date and time, and transaction partner.

[1237] "User" refers to a person or business entity that uses this system and is responsible for uploading receipts and payment data to the server.

[1238] "Terminal" refers to a digital device such as a smartphone, tablet, or PC, which a user uses to operate and upload data.

[1239] A "server" refers to a computer system that receives and processes data sent from users and provides various services.

[1240] An "AI engine" is software or a system that uses artificial intelligence technology to perform data analysis and classification tasks.

[1241] "Image recognition technology" is a general term for algorithms and technologies used to analyze digital images and extract specific patterns and information.

[1242] "Natural language processing technology" is a general term for algorithms and technologies for understanding text data and analyzing its meaning.

[1243] "Expenses" are expenditures necessary for business activities and are usually tax-deductible.

[1244] A "tax return" is a document that must be submitted to the tax office or other government agency, detailing income and expenses.

[1245] "Online system" refers to services and platforms that can be used via the Internet.

[1246] A "payment terminal" is a device for accepting electronic payments such as credit cards and debit cards.

[1247] "Accounting software" is a computer program used to record, manage, and analyze financial data.

[1248] "Income and expenditure data" refers to data that includes information on income and expenditure in business activities.

[1249] "Photography" refers to the act of capturing an image using a camera, smartphone, etc.

[1250] "Ingestion" refers to the process of receiving data from outside and integrating it into internal systems.

[1251] MODE FOR CARRYING OUT THE INVENTION

[1252] The present invention relates to a system that automates the process of filing tax returns in order to reduce the burden on store managers and sole proprietors. This system mainly consists of the following components:

[1253] 1. Upload your data

[1254] Users upload receipt images and payment data to the server using a device such as a smartphone. The device can be a smartphone, tablet, or PC with a camera function.

[1255] 2. Formatting and sending data

[1256] The server receives receipts and payment data sent by users, converts and formats the data into an appropriate format, and sends the formatted data to an artificial intelligence (AI) engine.

[1257] 3. Data analysis using an AI engine

[1258] The AI ​​engine uses image recognition and natural language processing technologies to analyze the received data. This analysis extracts information such as transaction date and time, amount, and transaction details from receipt images and payment data, and automatically classifies them into appropriate expense categories. The AI ​​model used can be, for example, the Transformers library.

[1259] 4. Generate tax return documents

[1260] The server automatically generates tax return documents based on the data categorized by the AI ​​engine, calculating the totals of income, expenditures, and expenses, and accurately filling in the necessary information.

[1261] 5. Online submission

[1262] The server submits the generated tax return documents to an online system (e.g., the e-Tax system), eliminating the need for the user to manually download and mail the documents.

[1263] Hardware and software used

[1264] This system uses the following hardware and software:

[1265] Hardware: smartphones, tablets, computers, payment terminals

[1266] Software: Server-side applications (e.g., Python), accounting software APIs (e.g., Moneybook, freee), natural language processing libraries (e.g., Transformers), PDF generation libraries (e.g., FPDF)

[1267] Specific examples

[1268] For example, if a user receives a receipt for "5,000 yen in transportation expenses on October 1, 2023," they can take a photo of the receipt with their smartphone and upload it to the server using a dedicated app. The server then formats the received image data and sends it to the AI ​​engine. The AI ​​engine then extracts text information such as "date," "amount," and "transportation expenses" from the image and classifies it as transportation expenses. The server then generates tax return documents based on this information and submits them to the e-Tax system.

[1269] Prompt Sentence Examples

[1270] When parsing the receipt text, you can use prompt statements such as:

[1271] The following text is the content of the receipt. Please indicate which expense category it falls under:

[1272] October 1, 2023 5,000 yen Transportation fee

[1273] As a result, this system will significantly reduce the burden on store managers and sole proprietors and enable efficient tax return filing.

[1274] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1275] Step 1:

[1276] The user uses the terminal to upload the receipt image and payment data to the server.

[1277] Input: Receipt images and transaction data taken and saved by the user.

[1278] Specific operations: Tap the "upload button" on your smartphone or computer, select images or data, and send them.

[1279] Output: Raw data uploaded to the server.

[1280] Step 2:

[1281] The server formats the data it receives.

[1282] Input: Uploaded raw data (images and transaction information).

[1283] Specific operation: The server removes noise from the image data, extracts text information, and formats the transaction information accordingly.

[1284] Output: Data formatted in a way that is suitable for an AI engine.

[1285] Step 3:

[1286] The server sends the formatted data to the AI ​​engine.

[1287] Input: Formatted data.

[1288] Specific operation: The server sends formatted text and image data to the AI ​​engine via API.

[1289] Output: Data sent to the AI ​​engine for analysis.

[1290] Step 4:

[1291] The AI ​​engine uses image recognition and natural language processing technologies to analyze the data and allocate expenses.

[1292] Input: The data sent from the server.

[1293] How it works: The AI ​​engine uses OCR technology to extract text, identifies dates, amounts, and categories using specific algorithms, and automatically classifies expenses using natural language processing.

[1294] Output: Categorised expense data.

[1295] Step 5:

[1296] The server generates tax return documents based on the allocated data.

[1297] Input: Data classified by the AI ​​engine (expense item data).

[1298] Specific operation: The server uses a PDF generation library (e.g. FPDF) to calculate the total income, expenses, and costs, and generate the tax return documents.

[1299] Output: Generated tax return PDF.

[1300] Step 6:

[1301] The server submits the generated tax return to the online system.

[1302] Input: Generated tax return PDF.

[1303] Specific operation: The server logs in to an online filing platform such as the e-Tax system and submits the generated PDF.

[1304] Output: Notification of submission completion and results.

[1305] Step 7:

[1306] The user receives a notification and confirms that the declaration has been completed.

[1307] Input: Submission completion notification from the server.

[1308] Specific operations: The user checks the notification on their smartphone or computer and operates the confirmation button for completing the declaration.

[1309] Output: A confirmation that the tax return is complete.

[1310] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1311] This invention is a system that significantly simplifies and streamlines the preparation of tax return documents, and further optimizes the user experience by combining it with an emotion engine that recognizes the user's emotions. This system significantly reduces the burden on sole proprietors by automating the complex process of allocating expenses, generating tax return documents, and even submitting them online.

[1312] Program processing overview

[1313] The program of this system has the following functions:

[1314] 1. The user uploads an image of the receipt and payment data.

[1315] Users use their smartphones or PCs to upload images of receipts and payment data to the server, using a dedicated application or web browser.

[1316] 2. The server receives the data, formats it, and sends it to the AI ​​engine

[1317] The server receives image data and payment data sent by the user, converts it into an appropriate format, and sends the formatted data to the AI ​​engine.

[1318] 3. The AI ​​engine analyzes the data and allocates expenses

[1319] The AI ​​engine uses image recognition and natural language processing technologies to extract necessary information (such as date, amount, and expense type) from receipt images and payment data. Based on the extracted information, it automatically allocates each expense to the appropriate expense category.

[1320] 4. The server generates tax return documents based on the sorted data.

[1321] The server stores the expense information sent from the AI ​​engine in a database and automatically generates tax return documents based on that information, calculating the totals of income, expenditures, and expenses and accurately filling out the information required for the return.

[1322] 5. Emotion engine recognizes user emotions

[1323] The server collects and analyzes the user's facial expressions and voice data through the emotion engine to recognize the user's emotions. The emotion engine determines whether the user is feeling stressed or relaxed, and uses this information to improve the user experience.

[1324] 6. The server submits the generated tax return to the online system.

[1325] The server then processes the generated tax return documents for submission to the e-Tax system, eliminating the need for users to download and mail the documents.

[1326] Specific examples

[1327] 1. The user takes a photo of the receipt with their smartphone and uploads it to the server using a dedicated app.

[1328] For example, a user takes a taxi and receives a receipt for the fare of 5,000 yen. The user takes a photo of the receipt with the smartphone camera and uploads it to the server via a dedicated app.

[1329] 2. The server formats the received image data and sends it to the AI ​​engine.

[1330] When the server receives the receipt image, it removes noise from the image and formats it so that text information can be easily extracted. The formatted image data is then sent to the AI ​​engine.

[1331] 3. The AI ​​engine extracts the necessary information from the receipt image and determines the expense item

[1332] The AI ​​engine uses image recognition technology to extract text information such as "October 1, 2023," "5,000 yen," and "transportation expenses." Natural language processing technology then categorizes this information as transportation expenses.

[1333] 4. The server automatically generates tax return documents

[1334] The server retrieves the information about the 5,000 yen allocated as transportation expenses from the AI ​​engine and automatically enters it into the appropriate field on the tax return. This calculates the total of income, expenditures, and expenses, and generates an accurate tax return.

[1335] 5. Emotion engine analyzes user emotions and optimizes the user experience

[1336] The emotion engine analyzes the user's facial expressions and tone of voice, and if the user is feeling stressed, it provides supportive messages to alleviate that stress. For example, it displays positive messages such as, "Your tax return preparation is going well. You're almost there!"

[1337] 6. The server submits the generated tax return to the online system.

[1338] The server logs into the e-Tax system and submits the generated tax return. Once submission is complete, the user is notified and can complete the procedure.

[1339] This system allows users to automate the entire tax return process by simply taking and uploading images of receipts. The introduction of an emotion engine also improves the user experience, making the tax return process less stressful.

[1340] The processing flow will be explained below.

[1341] Step 1:

[1342] The user opens the dedicated application using a smartphone or PC. After taking a picture of the receipt or importing the payment data, the user clicks the "Upload" button in the application. This operation causes the device to upload the receipt image and payment data to the server.

[1343] Step 2:

[1344] The server receives the image data and payment data sent by the user, and performs preprocessing (e.g., noise removal, resolution adjustment) on the received data to maintain the image resolution at an appropriate level.

[1345] Step 3:

[1346] The server sends the preprocessed image data to the AI ​​engine, where it is converted into an appropriate format and formatted for efficient analysis.

[1347] Step 4:

[1348] The AI ​​engine uses image recognition technology to extract text information from receipt images, such as date, amount, and category (transportation expenses, entertainment expenses, etc.) using a proprietary algorithm.

[1349] Step 5:

[1350] The AI ​​engine uses natural language processing technology to analyze the extracted text information and categorize it into the appropriate expense category. Specifically, it categorizes keywords such as "taxi" and "fare" as transportation expenses.

[1351] Step 6:

[1352] The server stores the classification information sent from the AI ​​engine in a database, which accumulates the allocated expense information and makes it available for subsequent processing.

[1353] Step 7:

[1354] The server automatically generates tax return documents based on the expense classification information retrieved from the database. Specifically, it inputs appropriate data into the income, expenditure, and expense information fields according to the tax return template, and calculates the total amount.

[1355] Step 8:

[1356] The emotion engine analyzes the user's facial expressions and voice data to recognize their emotions. It does this by analyzing data collected through the camera and microphone while the user is taking pictures or entering data.

[1357] Step 9:

[1358] The emotion engine provides feedback to the user based on the emotions it recognizes. For example, if the user is feeling stressed, the system will display a message saying, "We recommend you take a short break."

[1359] Step 10:

[1360] The server submits the generated tax return documents to the e-Tax system. At this time, the server uses the authentication information for logging in to the e-Tax system to carry out the appropriate upload procedure.

[1361] Step 11:

[1362] The server receives the results of the tax return submission and confirms that the submission was successful. It then sends a notification of the completion of the submission to the user.

[1363] This detailed processing step allows users to automate the entire tax return process by simply taking and uploading images of receipts. The introduction of an emotion engine also improves the user experience, making the tax return process less stressful.

[1364] Example 2

[1365] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1366] The traditional tax return preparation process is complex and time-consuming, placing a significant burden on many self-employed individuals. Additionally, sorting receipts and categorizing expenses is a tedious task that is prone to errors. Furthermore, the frustration users feel while completing the return is also a problem. To address this situation, there is a need for more efficient tax return preparation and an improved user experience.

[1367] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1368] In this invention, the server includes: means for a user to upload receipt images and payment data from a terminal to the server; means for the server to format the received data and send it to an AI engine; means for the AI ​​engine to analyze the data using image recognition technology and natural language processing technology and allocate expenses; means for the server to generate tax return documents based on the allocated data; means for the server to submit the generated tax return documents to an online system; means for the server to recognize the user's emotions through an emotion engine; and means for the emotion engine to analyze the user's facial expressions and voice data and determine the user's emotions. This realizes automation and efficiency in the preparation of tax return documents, reducing user stress and improving the experience.

[1369] A "receipt" is a document issued when a sale or service is provided that lists the details of the transaction.

[1370] "Payment Data" means data that includes transaction information related to the purchase of goods or payment for services.

[1371] "User" refers to individuals and businesses who use this system to file tax returns.

[1372] "Terminal" refers to the electronic device (smartphone, PC, etc.) that a user uses to operate the system.

[1373] The "server" is the central control unit that handles the entire processing of this system, and is a device that receives data from users, formats the data, connects with the AI ​​engine, and generates and submits tax return documents.

[1374] "Formatting" is the process of converting the format of received data, removing redundant parts of the data, and extracting necessary information to make it suitable for analysis.

[1375] An "AI engine" is a system that uses artificial intelligence technology to analyze data, and applies image recognition technology and natural language processing technology.

[1376] "Image recognition technology" is a technology that extracts specific information such as characters and objects from image data.

[1377] "Natural language processing technology" is a technology for analyzing the content of text data and understanding its meaning.

[1378] "Expense items" are categories for classifying types of expenses, and examples include "transportation expenses" and "entertainment expenses."

[1379] "Declaration documents" are various documents that must be submitted when filing a tax return, and contain details of income, expenditures, and expenses.

[1380] An "online system" is a system in which tax return procedures are carried out via the Internet, and examples include e-Tax.

[1381] An "emotion engine" is a system for analyzing a user's emotions, and determines the user's emotional state using facial expressions and voice data.

[1382] "Facial expression data" is data that includes information about the user's facial expression.

[1383] "Voice data" is data that includes information about the user's vocalizations.

[1384] This invention is a system that significantly simplifies and streamlines the preparation of tax return documents, and further optimizes the user experience by combining it with an emotion engine that recognizes the user's emotions. This system significantly reduces the burden on sole proprietors by automating the complex process of allocating expenses, generating tax return documents, and even submitting them online.

[1385] Specific program details

[1386] The system's program works as follows: First, the user uploads receipt images and payment data to the server using a smartphone or PC. Using a dedicated application or web browser, the user selects the data using the file selection button or drag-and-drop function, and then presses the send button.

[1387] The server receives image data and payment data sent by users. After receiving the data, it uses image processing algorithms to remove noise and clean it, and then formats it into a format that makes it easy to extract text. This formatted data is then sent to the AI ​​engine. Specific software used by the server includes OpenCV and Tesseract.

[1388] The AI ​​engine uses image recognition and natural language processing to extract necessary information from receipts, such as dates, amounts, and expense types. It then automatically categorizes each expense into the appropriate category based on the extracted information. Specific image recognition technologies include TensorFlow and PyTorch.

[1389] The server then stores the expense information sent from the AI ​​engine in a database and automatically generates tax return documents based on this information. The server calculates the totals of income, expenditures, and expenses, and generates documents that accurately list the information required for filing. An online platform using SaaS (Software as a Service) is used.

[1390] Through the emotion engine, the server collects and analyzes the user's facial expressions and voice data to recognize the user's emotions. The emotion engine determines whether the user is feeling stressed or relaxed, and uses this information to improve the user experience. Specific applications include the use of OpenPose and Microsoft Azure's Emotion API.

[1391] The server then finally processes the generated tax return documents for submission to the e-Tax system, eliminating the need for users to download and mail the documents.

[1392] Examples of concrete examples and prompts

[1393] 1. The user takes a photo of the receipt with their smartphone and uploads it to the server using a dedicated app:

[1394] For example, a user takes a taxi and receives a receipt for the fare of 5,000 yen. The user takes a photo of the receipt with the smartphone camera and uploads it to the server via a dedicated app.

[1395] 2. The server formats the received image data and sends it to the AI ​​engine:

[1396] When the server receives the receipt image, it removes noise from the image and formats it so that text information can be easily extracted. The formatted image data is then sent to the AI ​​engine.

[1397] 3. The AI ​​engine extracts the necessary information from the receipt image and determines the expense item:

[1398] The AI ​​engine uses image recognition technology to extract text information such as "October 1, 2023," "5,000 yen," and "transportation expenses." Natural language processing technology then categorizes this information as transportation expenses.

[1399] 4. The server automatically generates the tax return documents:

[1400] The server retrieves the information about the 5,000 yen allocated as transportation expenses from the AI ​​engine and automatically enters it into the appropriate field on the tax return. This calculates the total of income, expenditures, and expenses, and generates an accurate tax return.

[1401] 5. Emotion engine analyzes user emotions and optimizes user experience:

[1402] The emotion engine analyzes the user's facial expressions and tone of voice, and if the user is feeling stressed, it provides supportive messages to alleviate that stress. For example, it displays positive messages such as, "Your tax return preparation is going well. You're almost there!"

[1403] 6. The server submits the generated tax return to the online system:

[1404] The server logs into the e-Tax system and submits the generated tax return. Once submission is complete, the user is notified and can complete the procedure.

[1405] Prompt Sentence Examples

[1406] "An image of a taxi receipt taken with a smartphone is uploaded to a server. The image contains information such as 'October 1, 2023, 5,000 yen travel expenses'. Please explain how an AI engine can extract this information and allocate it to the appropriate expense category."

[1407] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1408] Step 1:

[1409] The user uploads an image of the receipt and payment data using the terminal.

[1410] Specific operation: The user takes a photo or scans a receipt or payment data using a smartphone or PC, then selects the captured image file or payment data through a dedicated application or web browser and clicks the upload button.

[1411] Input: Receipt image and payment data (e.g., taxi receipt image)

[1412] Output: Image data and payment data sent to the server

[1413] Step 2:

[1414] The server receives the data, formats it, and sends it to the AI ​​engine.

[1415] Specific operation: The server receives the receipt image sent by the user. It performs noise removal and image preprocessing on the received image data, formatting it so that text can be easily extracted. It then sends the formatted data (e.g., the noise-removed image and cleaned text data) to the AI ​​engine.

[1416] Input: Image data and payment data sent to the server

[1417] Output: Formatted data sent to the AI ​​engine

[1418] Step 3:

[1419] An AI engine analyzes the data and allocates expenses.

[1420] How it works: The AI ​​engine uses image recognition technology to extract dates, amounts, types of expenses, etc. from the received image data. It then uses natural language processing to understand the meaning of the extracted information and allocate it to the appropriate expense category (for example, "transportation expenses" or "entertainment expenses").

[1421] Input: Formatted data (e.g., a receipt image with noise removed)

[1422] Output: Allocated expense information (e.g., "October 1, 2023", "5,000 yen", "Transportation expenses")

[1423] Step 4:

[1424] The server generates tax return documents based on the allocated data.

[1425] Specific operation: The server saves the expense information sent from the AI ​​engine in a database and aggregates it. It then calculates the totals of income, expenses, and costs and automatically generates tax return documents. The generated tax return documents accurately contain the information necessary for filing (e.g., income, expenses, total amount).

[1426] Input: Allocated expense information

[1427] Output: Auto-generated tax return documents

[1428] Step 5:

[1429] The server recognizes the user's emotions through an emotion engine.

[1430] Specific operation: The server collects the user's facial expression and voice data and sends it to the emotion engine. The emotion engine analyzes the data and determines whether the user is feeling stressed or relaxed. If the user is feeling particularly stressed, it generates an appropriate support message and displays it to the user.

[1431] Input: User's facial expression data and voice data

[1432] Output: Emotion recognition results and supportive messages (e.g., "Your tax return preparation is going well. You're almost there!")

[1433] Step 6:

[1434] The server submits the generated tax return to the online system.

[1435] Specific operation: The server automatically sends the generated tax return documents to the e-Tax system. After sending, the server sends a success confirmation notice to the user, which saves the user the trouble of downloading and mailing the documents.

[1436] Input: Auto-generated tax return document

[1437] Output: Notification of completion of submission to the online system (e-Tax)

[1438] (Application example 2)

[1439] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1440] The present invention aims to reduce the burden on users and improve efficiency in expense management and tax return preparation. It also aims to improve the user experience throughout the entire tax return process by analyzing users' emotions and providing support to reduce stress. In particular, in environments where transactions are frequent, such as brick-and-mortar stores, it is necessary for store clerks to smoothly perform their daily tasks while managing expenses without stress.

[1441] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1442] In this invention, the server includes: a means for users to upload receipt images and payment data from their terminals to the server; a means for the server to format the received data and send it to an AI engine; a means for the AI ​​engine to analyze the data and allocate expenses using image recognition and natural language processing; a means for the server to generate tax return documents based on the allocated data; a means for the server to submit the generated tax return documents to an online system; and an emotion engine that analyzes the user's emotions and displays messages based on the emotions. This allows users to automate the entire tax return process simply by taking and uploading images of their receipts. The introduction of the emotion engine also improves the user experience, making the tax return process less stressful.

[1443] A "receipt image" is an image showing purchase details or a record of service usage that is taken by a user to prove expenses.

[1444] "Payment data" is digital data that includes a transaction record that occurs when a user makes a payment.

[1445] A "terminal" is an electronic device, such as a smartphone or computer, that a user operates to input and transmit data.

[1446] A "server" is a high-performance computer system for processing and storing data over a network.

[1447] "Formatting" refers to converting the data received by the server into an appropriate format for efficient processing.

[1448] An "AI engine" is a software system that uses artificial intelligence technology to analyze data and automatically perform specific tasks.

[1449] "Image recognition technology" is a technology that uses computer vision to extract specific information from images.

[1450] "Natural language processing technology" is a technology for analyzing meaning from text data and understanding and generating human language.

[1451] "Allocating expenses" means classifying expenses into specific categories.

[1452] A "tax return" is an official document that contains information required for filing a tax return.

[1453] An "online system" is a system for sending and receiving data and providing services using the Internet.

[1454] An "emotion engine" is a software system that analyzes a user's emotions and generates an appropriate response based on the results.

[1455] A "message" is text information that the emotion engine provides to the user based on the analysis results.

[1456] The present invention relates to a system that automatically processes receipt images and payment data to create tax return documents. Furthermore, by including an emotion engine that analyzes user emotions, the system aims to improve the user experience.

[1457] The system is configured as follows: First, the user uploads an image of the receipt and payment data to the server using a device such as a smartphone or PC. Specifically, a dedicated application or web browser is used. At this time, the user can take a photo of the receipt with a camera and send the image data.

[1458] The server receives the uploaded data, removes noise from the image data, and formats it in a way that makes it easier to extract text information. An image processing service like Google Cloud Vision API is a good choice for this purpose. The formatted data is then sent to the AI ​​engine.

[1459] The AI ​​engine analyzes data using image recognition and natural language processing technologies. Specifically, it uses deep learning libraries such as Keras to extract information such as date, amount, and expense item from receipt images and automatically classify them into appropriate categories.

[1460] The data is then sent back to the server, which then generates tax returns that accurately show income, expenditures, and total expenses. Again, a programming language such as Python is used to interface with a database management system to perform the necessary calculations.

[1461] Furthermore, the system is equipped with an emotion engine that analyzes the user's emotions. The emotion engine analyzes the facial expressions and voice of the store clerk to determine the level of stress. Facial recognition technology such as OpenCV and voice analysis technology are used for emotion analysis. If the user feels stressed, the emotion engine will support them by displaying encouraging messages such as, "Your declaration preparation is progressing smoothly. You're almost there!"

[1462] The completed tax return is then submitted to an online system (e.g., the e-Tax system) by the server, saving the user the trouble of manually submitting the return.

[1463] As a concrete example, consider the case of a convenience store clerk managing expenses while performing their daily duties. When making a transaction at the register, the clerk takes a photo of the receipt with their smartphone and uploads the data to a server using a dedicated app. The server then formats the image data, and an AI engine analyzes the data and allocates it to the appropriate expense category. This automatically generates tax return documents, which are ultimately submitted to the online system. The emotion engine also detects the clerk's stress and provides encouraging messages as needed, thereby reducing the clerk's stress.

[1464] Example prompt sentence:

[1465] "Just take a photo of your receipt with your smartphone and upload it using the app. We'll provide you with instant analysis and, if needed, a message of encouragement."

[1466] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1467] Step 1:

[1468] The user takes a photo of the receipt and payment data on the device and uploads it to the server using a dedicated application or a web browser. The image data of the receipt is used as input, and a data file is obtained as output, which is sent to the server.

[1469] Step 2:

[1470] The server formats the received image data. First, it uses the Google Cloud Vision API to remove noise from the image and convert it into a format that makes it easier to extract text information. It receives image data as input and obtains formatted image data as output.

[1471] Step 3:

[1472] The server sends the formatted image data to the AI ​​engine, which uses the formatted image data as input and obtains the data sent to the AI ​​engine as output.

[1473] Step 4:

[1474] The AI ​​engine analyzes the received image data. It uses image recognition technology to extract text information such as dates, amounts, and items, and then uses natural language processing technology to classify this information into appropriate expense categories. It receives formatted image data and extracted text information as input, and obtains classified expense information as output.

[1475] Step 5:

[1476] The server generates tax return documents based on the expense information returned by the AI ​​engine. The server calculates the totals of income, expenditures, and expenses through a database management system and compiles them into official tax return documents. It receives classified expense information as input and obtains the generated tax return documents as output.

[1477] Step 6:

[1478] The emotion engine analyzes the user's emotions. Facial expression and voice data collected on the device is sent to the server, which uses OpenCV and voice analysis technology to determine the level of stress. The engine receives the user's facial expression and voice data as input and obtains the emotion analysis results as output.

[1479] Step 7:

[1480] The server displays an appropriate message on the terminal based on the emotion analysis results. If the user is feeling stressed, an encouraging message is displayed to support the progress of the task. The emotion analysis results are received as input, and the message to be displayed is obtained as output.

[1481] Step 8:

[1482] The server submits the generated tax return to the online system, sends the tax return to an online platform such as the e-Tax system, and notifies the user of its completion, receiving the generated tax return as input and receiving a notification of completion of submission as output.

[1483] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1484] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1485] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1486] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1487] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1488] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1489] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1490] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1491] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1492] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1493] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1494] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1495] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1496] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1497] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1498] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1499] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1500] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1501] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1502] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1503] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1504] The following is further disclosed regarding the above embodiment.

[1505] (Claim 1)

[1506] A means for users to upload receipt images and payment data from their devices to the server;

[1507] A means for the server to format the data received and send it to the AI ​​engine;

[1508] The AI ​​engine uses image recognition and natural language processing technologies to analyze data and allocate expenses.

[1509] A means for the server to generate tax return documents based on the allocated data;

[1510] a means for the server to submit the generated tax return to an online system;

[1511] A system including:

[1512] (Claim 2)

[1513] The system according to claim 1, which converts and formats image data of receipts and payment data uploaded by a user from a terminal to a server into an appropriate format.

[1514] (Claim 3)

[1515] The system of claim 1, wherein the AI ​​engine uses natural language processing technology to analyze the meaning of the extracted text information and automatically classify it into appropriate expense categories.

[1516] "Example 1"

[1517] (Claim 1)

[1518] A means for a user to transmit business data from an information input device to a processing device;

[1519] means for converting and shaping data received by said processing device into an appropriate format;

[1520] means for transmitting the data shaped by the processing device to an analysis device;

[1521] means for the analysis device to analyze and classify data using image processing techniques and automatic analysis techniques;

[1522] means for generating a report based on the data classified by the processing device;

[1523] means for the processing device to submit the generated report to an online service;

[1524] A system including:

[1525] (Claim 2)

[1526] 2. The system according to claim 1, wherein business data transmitted by a user from an information input device to a processing device is converted and formatted in an appropriate format.

[1527] (Claim 3)

[1528] 2. The system of claim 1, wherein the analysis device uses automatic analysis technology to analyze meaning from the extracted information and classify it appropriately.

[1529] "Application Example 1"

[1530] (Claim 1)

[1531] A means for users to upload receipt images and payment data from their devices to the server;

[1532] A means for the server to format the data received and send it to the AI ​​engine;

[1533] The AI ​​engine uses image recognition and natural language processing technologies to analyze data and allocate expenses.

[1534] A means for the server to generate tax return documents based on the allocated data;

[1535] a means for the server to submit the generated tax return to an online system;

[1536] Store managers can use their smartphones to take photos and upload daily expense receipts,

[1537] It connects to payment terminals and accounting software to automatically import income and expenditure data,

[1538] A system including:

[1539] (Claim 2)

[1540] The system according to claim 1, which converts and formats image data of receipts and payment data uploaded by a user from a terminal to a server into an appropriate format.

[1541] (Claim 3)

[1542] The system of claim 1, wherein the AI ​​engine uses natural language processing technology to analyze the meaning of the extracted text information and automatically classify it into appropriate expense categories.

[1543] "Example 2: Combining Emotion Engines"

[1544] (Claim 1)

[1545] A means for users to upload receipt images and payment data from their devices to the server;

[1546] A means for the server to format the data received and send it to the AI ​​engine;

[1547] The AI ​​engine uses image recognition and natural language processing technologies to analyze data and allocate expenses.

[1548] A means for the server to generate tax return documents based on the allocated data;

[1549] a means for the server to submit the generated tax return to an online system;

[1550] A means for the server to recognize user emotions through an emotion engine;

[1551] A means for the emotion engine to analyze the user's facial expression and voice data and determine the user's emotion;

[1552] A system including:

[1553] (Claim 2)

[1554] The system according to claim 1, which converts and formats image data of receipts and payment data uploaded by a user from a terminal to a server into an appropriate format.

[1555] (Claim 3)

[1556] The system of claim 1, wherein the AI ​​engine uses natural language processing technology to analyze the meaning of the extracted text information and automatically classify it into appropriate expense categories.

[1557] "Application example 2 when combining emotion engines"

[1558] (Claim 1)

[1559] A means for users to upload receipt images and payment data from their devices to the server;

[1560] A means for the server to format the data received and send it to the AI ​​engine;

[1561] The AI ​​engine uses image recognition and natural language processing technologies to analyze data and allocate expenses.

[1562] A means for the server to generate tax return documents based on the allocated data;

[1563] a means for the server to submit the generated tax return to an online system;

[1564] means for analyzing a user's emotions and displaying messages based on the emotions;

[1565] A system including:

[1566] (Claim 2)

[1567] The system according to claim 1, which converts and formats image data of receipts and payment data uploaded by a user from a terminal to a server into an appropriate format.

[1568] (Claim 3)

[1569] The system of claim 1, wherein the AI ​​engine uses natural language processing technology to analyze the meaning of the extracted text information and automatically classify it into appropriate expense categories. [Explanation of symbols]

[1570] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for users to upload receipt images and payment data from their devices to the server; A means for the server to format the data received and send it to the AI ​​engine; The AI ​​engine uses image recognition and natural language processing technologies to analyze data and allocate expenses. A means for the server to generate tax return documents based on the allocated data; a means for the server to submit the generated tax return to an online system; A system including:

2. 2. The system according to claim 1, wherein image data of receipts and payment data uploaded by a user from a terminal to a server are converted and formatted in an appropriate format.

3. The system according to claim 1, wherein the AI ​​engine uses natural language processing technology to analyze the meaning of the extracted text information and automatically classify it into appropriate expense items.

Images